> Locale suffixes ".utf8" and ".UTF-8" are hacks by specific Linux distributions, and people want their Python code to work even if their system has not implemented this hack.

I do want my Python code to work, yes. But I assure you that if my locale says ru_RU.KOI8-R, when I say "work", I don't mean "dump garbled stuff on my screen, because you output UTF-8 when I specifically asked you to use KOI8-R".

I also don't see why the UTF-8 encoding specifier in locale is a "hack". UTF-8 is just that, an encoding. Locales are a generic mechanism for dealing with encodings. What makes UTF-8 special in that regard, and why is an UTF-8 locale a hack?

> The locale "C" does not really mean you want your program to explode when it sees a non-ASCII byte. What Python does here does not promote compatibility in any way.

That's true for any locale, if a character comes up in the output that cannot be encoded in it - it should just use some reasonable substitution. And if you're actually dealing with binary data, then you should be reading and writing bytes objects, not printing strings, and then the whole question of encoding is moot.

> BSD has no UTF-8 locales and has never pretended that locales work with Unicode.

That would be a surprise to my FreeBSD installation, which is running with en_US.UTF-8. Do you mean that the locales don't come generated by default? I'm not sure why it's a big deal ... I mean, Arch doesn't come with any locales generated by default, but I don't think anyone would claim that it doesn't support UTF-8.

One other thing BSD does, is that it doesn't guarantee that wchar_t is Unicode in all locales (Linux/glibc does). But that's a different and orthogonal issue.

> Windows' equivalent of locales is extremely deprecated and never actually supported UTF-8. The modern Windows APIs that deal with standard in and out correspond to Python's Unicode string type, not its bytes type; encoding the Unicode I/O into bytes is not Python's responsibility.

Win32 doesn't have any string-based I/O functions, except for console. So if you're printing directly to the screen (and ignoring / not supporting redirection!), then yes, you can just invoke WriteConsole, give it a UTF-16 string, and be done with it. But if you're printing to a file, or to stdin/out as a handle, then you only have ReadFile/WriteFile, which require a byte array; you're expected to do your own encoding.

By the way, and speaking of proper Unicode handling APIs - because of this disparity, and the desire to avoid a conversion unnecessarily (due to the lack of UTF-8 locale making it lossy) so as to allow printing out Unicode, Python actually has to resort to hacks to do this on Win32. Specifically, it has a special class - _io._Win32ConsoleIO - that is implemented using ReadConsole/WriteConsole. If output is a Win32 console, then that class is used, and Unicode is fully supported in the output. If output is redirected, then the standard implementation (that does encoding + Read/WriteFile) is used instead.

> Really every operating system but Linux has this figured out, and in practice, Linux users want UTF-8 regardless of what their locale says.

I strongly disagree. The users who want it but don't know how to configure it, are already using a distro that says that it should be UTF-8. The users who are using something that doesn't, either do it for a reason, or are knowledgeable enough to know what's going on and why, and deal with it accordingly. Either way, no harm is done by just using the locale setting. On Ubuntu etc, it will just be UTF-8, and everyone will be happy.