While I reference the issue of a mixed-codec directory in order to make clear the flaw in the operating assumption, the actual problem I am interesting in here, and which I conclude my statements with a reference to, is handling things like sys.path (hence this being a reply to the article).

To respond to your comments, however: actually, the behavior on, let's say a Linux box, if you mount one of those aforementioned filesystems, will not be "exactly correct". Python 3 will attempt to encode the Unicode string using the current locale, pass it to the underlying Unix open() function, which will then have no clue what to do as it hits the filesystem.

In fact, rather than just idly claiming this, I went ahead and set up exactly this test setup on one of my servers. I created a python3 script in a known specific source encoding (UTF-8) and asked it, in each of two different locales, to make a file that included an accented character, while mounted on an HFS+ disk image.

hfs+# mount | grep hfs
/.../hfs+.img on /.../hfs+ type hfsplus (rw,force)
hfs+# cat test.py
#!/usr/bin/python3
# -*- coding: utf-8 -*-
open("helloä", "w")
hfs+# LANG=en_US.UTF-8 ./test.py
hfs+# LANG=fr_FR.ISO-8859-1 ./test.py
hfs+# LANG=en_US.UTF-8 ls -la
-rw-r--r-- 1 root root 0 2012-07-11 00:57 hello?
-rw-r--r-- 1 root root 0 2012-07-11 00:57 helloä
-rwxr-xr-x 1 root root 64 2012-07-11 00:56 test.py*
hfs+#

As you can see, the behavior here is really poor for something that claims to support Unicode. What we would expect to have happen is that, as I opened the file with a Unicode name on a Unicode filesystem that I would actually get the specific Unicode string that I had wanted.

Instead, because Python 3 is only pretending to understand the semantics of filesystems with regards to character sets, and in fact has no way of taking advantage of the Unicode support in HFS+, we ended up with the encoding of the user's locale environment breaking our filenames.

This would be akin to me doing JSON.encode() in a browser, and having a Unicode JavaScript string get converted into different JSON (which is also represented as a Unicode JavaScript string) depending on what language the user's browser is configured to use: that is a miserable Unicode failure, and not a success story.

FWIW, the exact same code does work on Mac OS X (using the correct locale name of fr_FR.ISO8859-1): both versions of the script get the exact same filename. To be honest, I was somewhat surprised that they got that right ;P. I sadly do not have a Windows computer handy, as I'd love to see how it handles NTFS (which is technically UCS-2 and doesn't have the weird canonicalization behavior that HFS+ sometimes does: I can easily imagine broken corner cases with invalid UTF-16 surrogate pairs).

Regardless, I really do not believe that you need to break the behavior on Linux in order to make Mac OS X work correctly. Even if such a tradeoff were required, I question whether it should be resolved with Linux on the losing end. I am not going to say that the correct solution doesn't even use Unicode strings: it just needs to more intelligently handle who is in charge of the encodings and what they semantically mean than Python 3 is prepared to do.

As far as possible solutions go, I certainly do not believe that the private Unicode codepoint solution is either sufficient (as it doesn't solve the de novo filename creation problem) nor even remotely reasonable (as if I attempt to then communicate these filenames with other systems, or heaven-forbid wanted to store a filename with a private Unicode codepoint in it, I'm now screwed).

(edit: Wow, BTW. They added this "solution" after I had already given up on Python 3k, so I hadn't followed up to see what they did. It seems like they ended up not using "private codepoints" as they were considering in 2008, but are instead using surrogate-halves: they are encoding "malformed UTF-8 sequences as malformed UTF-16 sequences"[1]. Given that you are allowed to store malformed UTF-16 on NTFS, I wonder how they expect that to work. Still doesn't solve either of my complaints, though.)

1: http://hyperreal.org/~est/utf-8b/releases/utf-8b-20060413043...