POST, GET, and the iTMS

by Scot Hacker

Related link:

After a (very) long period of hibernation, I recently resurrected an old site of mine, The Archive of Misheard Lyrics. But after populating the site with thousands of deep links into the iTunes music store, I've found them largely incompatible with the architecture of my site.

For years, I've wanted to stop serving so many random ads on the site and leverage its music-centric content to sell music. The iTunes affiliate program seemed like the perfect opportunity, so I signed up as an affiliate and began the process of populating my databases with special affiliate links to songs and artists.

The other big goal of the redesign was to leverage the collaborative power of the internet and let readers vote on the funniness (or lack thereof) of published lyrics and unpublished submissions. That system came together nicely, but that's where the iTMS connection started to turn sour.

The voting mechanism works through a simple form, submitted through a standard http POST action. PHP session tracking ensures that users can't vote on a given lyric more than once per session. But here's the rub: When you click a link (any link) into iTMS, it makes an http request. That link is processed by Apple, which sends back an http header telling the browser to launch an external application (iTunes). Because Apple is essentially resubmitting the URL of the linking page to itself, the browser thinks the page you just voted on is being re-POSTed. The browser then throws the standard "Are you sure you want to resubmit this form?" dialog, which is very confusing for the user.

To see this in action, visit the site and try some of the "Hear more from this artist" links near the tops of lyric pages. They should work fine. Then try voting on a lyric and then click the "Hear more" link at the top of the resulting page. Oops. Miserable user experience.

The alternative, of course, would be to rewrite the code around GET rather than POST, but that would be a misuse of GET, which is supposed to be used only for retrieving, not submitting information. In addition, using GET would result in all the form variables being sent in the URL string, which is messy. And hidden form variables would also become visible in the URL.

You can sort of see why Apple went this route -- it saves them from having to invent a new internet protocol (which would be a more elegant means of handing the hand-off problem) or from resorting to little downloadable "launch" files, which would litter users' desktops. But I have to wonder whether they considered the ramifications of their technique for sites that use POST for navigation.

Trying to get help -- or even feedback -- from the affiliates program has been a lesson in frustration. My emails to them have been totally ignored.

Unfortunately, the POST problem wasn't the end of my frustrations. It turns out that a lot of the links provided through the LinkShare back-end (Apple partners with LinkShare for the affiliates program; you have to use their back-end to generate your custom links) simply don't work. See the six (currently static) album covers in the left column of the site? Only two of them actually take you to that album in iTMS. The other four take you to the iTMS homepage. All six links were pasted directly out of the LinkShare link maker, and should work as-is. This problem is totally unrelated to the POST problem -- they're just dishing up broken links to affiliates, period.

My original plan for the album covers was to parse the LinkShare-provided iTunes RSS feeds, which are separated into loose genres, and associate them with the genre of the lyric currently being viewed. I envisioned this system where people would always see context-relevant, freshly delivered columns of iTMS-linked album covers. But as I started working with Apple's feeds, I found that they don't deliver RSS in such a way that you can simply use a parser such as Magpie to just grab album covers and links. The only way to get album covers from their RSS is as a series of table blobs, where you end up also displaying the prices, publisher, etc. To get what I want out of their feeds, I'm going to have to do a bunch of scraping and (fragile) post-processing. It would have been so easy for them to deliver nice, atomic XML elements that sites could display as they saw fit.

Finally, I was expecting to find some kind of ad rotation mechanism for affiliates. See those static Stevie Wonder banners at the top of all the lyrics pages? I should be able to drop in a block of code and have those rotated out automatically from iTMS. Instead, the only option is for me to return to LinkShare every few days and grab some new static code. Maybe I'll store a few dozen in a database and do my own randomizer for them. But why should this be so difficult?

An ideal way to run things would be for them to say, "Here's your partner ID. Now you can build this into a link that looks like this for a simple song link, or like this for an album cover, etc." That way I could simply grab URLs from iTunes itself and write code to integrate my partner ID into the links. But because of the way they format links, you're basically forced to scoop every link you want to use -- which comes complete with a hard-coded image link -- out of LinkShare.

I'm trying to sell music for Apple here. You'd think they'd welcome all the help they can get. This whole process has been incredibly frustrating. Maybe I've drunk too much of the Apple Kool-Aid, but I really expect better from them. At this point, I'm starting to consider alternatives to iTMS, such as Rhapsody. But I'm really not eager to face another database population job. If I don't hear from something useful from someone at Apple soon, it's back to the drawing board.

Is there something I'm missing? Or shouldn't this all be a whole lot easier?


2005-12-13 11:20:37
In the HTTP spec...

It uses the phrase "SHOULD NOT" which in W3C-language (to the best of my translation abilities) means that they'd much rather you didn't, but understand you may need to, do something a certain way.

From the spec (

9.1.1 Safe Methods

Implementors should be aware that the software represents the user in
their interactions over the Internet, and should be careful to allow
the user to be aware of any actions they might take which may have an
unexpected significance to themselves or others.

In particular, the convention has been established that the GET and
HEAD methods SHOULD NOT have the significance of taking an action
other than retrieval. These methods ought to be considered "safe".
This allows user agents to represent other methods, such as POST, PUT
and DELETE, in a special way, so that the user is made aware of the
fact that a possibly unsafe action is being requested.

Naturally, it is not possible to ensure that the server does not
generate side-effects as a result of performing a GET request; in
fact, some dynamic resources consider that a feature. The important
distinction here is that the user did not request the side-effects,
so therefore cannot be held accountable for them.

2005-12-13 15:35:55
Should not
Right - nothing technically stops me from re-doing the site with GET rather than POST, but I would hate to because :

A) The URLs would become messy, and since they would contain variables, would not bookmark nicely.

B) I don't like the idea of changing the fundamental architecture of the site just to satisfy the needs of iTMS.

I'm wondering now if there's a way to accept the vote on a dynamic page behind the scenes, then redirect the user immediately to a non-POSTed version of the page. Maybe I'll look into that.

2005-12-14 01:31:26
In the HTTP spec…

In particular, on the web, using GET because POST is “inconvenient” means that anyone can put <img src=""> on their site, and any visitors to that site who have an account with will find their data is mysteriously gone.

SHOULD does not mean “oh pretty please please do it this way okay,” it means THERE ARE DIRE CONSEQUENCES IF YOU DON’T AND YOU MUST BE AWARE OF THEM BEFORE YOU CONSIDER IT.

2005-12-16 11:47:33
Problem fixed
FWIW, I went ahead and implemented a hack to work around the POST problem. Now, when a vote has occurred it processes the vote on the POST, then uses a PHP header Location: directive to pass the user to the next page. The result is that the user receives the next page as if they had clicked there from a normal link, so links to iTMS on the resulting page don't throw a "Re-POST?" dialog. Not quite as clean, but it works.
2005-12-23 07:00:08
itunes datafeed
why dodn't you just get the xml datafeed from itunes? it's about 2gb, and has everything you need so you don't have to use the took me about 2 months to get approved