[General Topics] Menneisyys:
3.2.	Double requests upon file download - the RapidShare problem

In the above-linked AximSite thread, I've been told of a strange problem affecting Opera Mobile and NetFront, making it impossible to download anything off the well-known file distributor service RapidShare.

I've found the protocol-level cause for the problem after some investigation: NetFront and Opera Mobile, when downloading any resource, resends the requests (in both GET and POST mode) sent out right after the download starts. This is what causes the RapidShare service to immediately close the transfer and just return an error page. (NetFront reissues exactly the same request (for example, two times the same POST), while Opera Mobile, if it sent out a POST request first, sends out a GET second - to the same URL. Both issue additional no-cache headers in the second request when needed.)

Note that the desktop Opera doesn't suffer from this problem - only Opera Mobile.

This means if you have problems with these browsers and RapidShare (or, for that matter, any Web page that only accepts one download request and just returns an error page upon the second), use an alternate Web browser like the built-in PIE or Minimo. Note that, specifically with RapidShare, Thunderhawk won't work either because it won't be able to count down - its JavaScript compliance is limited.

(Some debugging screenshots of what's happening, using the excellent, albeit expensive and easily crashing debugger proxy Charles (this time, I haven't used my debugger proxy). With PIE, the POST request sending the captcha code and the direct URL of the file to be downloaded is here; here's the response (the file itself), after which PIE doesn't send out another request(s). The initial request is exactly the same with the (buggy) NetFront; so is RapidShare's response. However, right after starting the download by the user, NetFront issues a second request for the same resource, which results in an error page to be returned by the server.)

3.3.	Referrers passed to download? The Caiman problem

I've (re-)tested the "Referer" problem of PIE explained here (please read it for further info) with all the new browser versions. The best news is that, except for, it seems, Thunderhawk, none of the major browsers have related problems - including, fortunately, WM5. Kudos to Microsoft - PIE / IEM is really getting more and more stable and standards-compliant.

(This, of course, still means you can't use PIE under previous operating systems on sites requiring explicit Referer passing. There, use alternate browsers or some download managers which also allow for Referer setting.)

3.4.	Comparison chart

In the following chart, I've listed all the above cases, with some additional ones. These explain whether a given Web browser parses the contents of a given Web resource returned with the application main MIME type and either the well-known widely used octet-stream subtype or with a deliberately unknown subtype. I've also thoroughly tested where the Web browsers look for type information and where they get their filenames from. I don't explain these columns as they are "only" of interest to Microsoft, Opera, Bitstream and Access developers and anyone knowing the HTTP protocol well enough to understand the separate cases. These columns are of no interest to ordinary, non-geek people - not even with explanation.

Note that, as usual with my thorough protocol compliance tests, I’ve used a custom-developed HTTP server emulator to test all this. It (the source – as usual, I release it as sources so that you can also have a look at it) is available HERE and should be used the following way:

the first parameter should contain the name of the file to return. This makes it possible to test the browsers with a lot of different types of files – image files, PDF files etc. The point in this the ability to find out that, say, does the browser try to find out the real type of files that are sent back as application/octet-stream or, horribile dictu, application/HERE-IS-SOME-UNKNOWN-SUBTYPE. (Also see the second parameter on setting this type)

if the second parameter is a single Y, application/octet-stream will be returned as the file type; otherwise, the custom and by Web browsers unknown application/x-download. The point in this parameter is that, testing the Web browsers with both combinations, you can see whether they differ in how they handle different subtypes of the application MIME type. Implementing and testing this was really useful and instructive: as you can see in the “Save (+) or render (-)., depending on the type” group, PIE treats files of type application/octet-stream differently than files with an unknown subtype.

Finally, the third parameter is either a “Y” (then, it returns the real name of the file) or a custom-given name in the Content-Disposition header. For example, if you supply an “anything.dic” name as this parameter, it’ll return this name and not the real name of the file. Implementing this conditional, I was able to test whether the Web browsers do parse the contents of the file to find out whether it’s of a different type than its name suggests in the Content-Disposition header. The results gained by using the two possible cases can be seen in the first two columns (“Peeks into file to decide its type to override the type given by C-D filename / URL?” and “Takes into account C-D filename?”) of the “It decides for the content type & gets the filename by...” group.