[General Topics] Menneisyys:
2.2.1.2 How Web administrators should treat Thunderhawk clients?

Incidentally, speaking of server-side User-Agent checking, if you operate a, say, Web site offering content in a Central-European language (that is, a language sufficiently close to Western languages, alphabet-wise; that is, in where using Western characters instead of some special, local characters - for example, using instead of the "official" ), then, upon sensing the client's using Thunderhawk, you may want to force the content encoding back to 8859-1. That is, return the content as plain 8859-1 (Western) document. This will make sure non-Western characters will be converted back to their closest Western equivalent, resulting in a far better user experience.

2.2.1.3 Generic advice for Webadmins of web hosters: NEVER set the charset attribute in the Content-Type HTTP response header!

Also, the results clearly show if you're providing Web hosting service to customers, you should never set the character set the ‘charset' attribute in the Content-Type HTTP response because your customers won't be able to override it with their own encodings. For example, if you're a Central-European Webspace provider and use the local, non-Western language as default charset returned straight in the HTTP header, you'll make all your customers unable to provide content in the Western-European charset.

The situation is the same in the opposite direction: if you set the Western-European charset and you still have for example Russian customers that would like to publish their Russian pages on your server, most Web browsers won't be able to render these, not even if these folks explicitly try to override your encoding settings.

A real-world example: earlier, this all has been a huge problem with my current webspace provider. Due to cost considerations (I didn't want to pay big bucks to host a webpage I don't use for commercial stuff, just as a database back-end for my articles, images and other downloads), I've chosen a (compared to, for example, the Finnish Web hosting fees) very cheap Central-European webspace provider. It, however, before July 2006, also set the above-mentioned ‘charset' attribute to Central-European encoding, which made it impossible to put Word-generated/-exported English language pages on my page without first changing all the non-standard (extended) punctuation marks to their non-extended (and, therefore, less spectacular) counterparts. That is, when I, for example, posted a comparison chart HTML file there (I can't include comparison charts in Pocket PC discussion forums because of the forum restrictions and wide charts in my Smartphone and Pocket PC Magazine Blog - in these cases, I must link them from my back-end), I always had to change (with a generic search-replace in, say, Notepad) these characters back to the non-extended version.

Note that you can avoid all this hassle in Microsoft Word before starting to write your article / post by disabling the two checkboxes ("Straight quotes" with "Smart quotes" and Hyphens (--) with dash (-)) in Tools / Autocorrect Options / Autoformat As You Type. If you find it necessary and want to avoid problems with, for example, your NetFront or non-Western readers, make sure you do this (that is, disable the two checkboxes).

Now that, during a recent Web server update, this annoyance has been removed, I can post anything without mass-replacing 'special' characters (or disabling autocorrection in Word).

All in all: the inability to override (in everything but NetFront) the HTTP response header means a decent web hoster should never return the language / character encoding parameter in his / her Web pages if there is chance users would want to return pages in different encoding. This is, fortunately, the case with the majority of current web hosters.

3. Form Posting

The second main problem area is not displaying non-Western characters but posting such contents to Web servers via Web forms. In these tests, I've tested the same; with exactly the same input (punctuation, non-Western characters) and I've also added Western accented characters like ä and ö: I've posted these characters to the Web server and checked what they have become.

I've used two sites for this purpose: my PPCMag blog, which is 8859-1 (Western charset) by default and a Central-European server using the Windows-1250 (Central-European) encoding on all its pages. (The page encoding setting has direct effect on what is uploaded back to the server from a form.)

Click here for the chart

As can clearly be seen, the situation isn't at all bad with the three desktop browsers (I've encountered no compatibility issues at all - all special and even Central-European characters were visible in their original - unconverted! - form after posting, even when posted to an 8859-1 server).

With Pocket PC-based browsers, on the other hand, posting special / non-Western characters turned out to be much more problematic, particularly - as opposed to the desktop posting case - when posting to a 8859-1 (Western) server. Then, it was only Thunderhawk that was able to upload these characters; all the other clients either don't upload these characters at all or converted them.

An explicitly windows-1250 server was a bit better as far as NetFront is concerned: now, it was able to upload non-standard Western punctuation and non-Western accents.

Unfortunately, Opera Mobile has never been able to upload any kind of extended Western punctuation. This is a real bug that really should be fixed.

4. Non-8-bit file formats

In addition to 8-bit file formats (ISO-8859-1, Windows-1250 etc), there are some other, non-8-bit file formats. One of them is Unicode, of which a test page is here. Another is UTF-8 (test page here; OK, I know this is, technically, a 8-bit file format, using 2 or 3 bytes for extended 8859-1 or Unicode characters. I didn't want to create a different category for it.)

The former is almost never used on the Web (albeit it's possible some, say, Chinese or Japanese site will use it); the latter pretty extensively in the non-Western language areas. Its penetration in Central-Europe (excluding languages using cyrillic characters) may be 10-20% (because the special characters of these languages are easy to map into a 8-bit chartable); in China/Japan or other languages using alphabets containing Kanjis and other special (and numerous) characters, around 100%.

In this test, I've tested whether the Pocket PC web browsers are able to read these pages (see the above two links if you want to give them a try). As can be seen, the situation is pretty good: the common UTF-8 is read by all browsers. IEM and Minimo fail to render Unicode files, though.

This is, again, not a big problem at all - I've yet to see a Web page that uses Unicode instead of UTF-8. Note that this is one of the very slight differences between Minimo and its desktop big brother, Mozilla / Firefox. The latter, as with the desktop Opera and Internet Explorer, is able to render Unicode files too.

Click here for the chart