use Sys::Binmode;
Tags: perl
Published: 2021-03-27 10:34:01

Character encoding is an often-misunderstood aspect of Perl. Perl itself has significant bugs in the area. I recently published a CPAN module called [Sys::Binmode](https://metacpan.org/pod/Sys::Binmode) which fixes most of those. If you write Perl you _probably_ should use it in all new code.

I know that’s a “tall” claim, but …

# Check this out:
```
my $foo = "\xff\x{100}";
chop $foo;
print $foo, $/;
exec "echo", $foo;
```
This _looks_ like it ought to print two identical lines, right? But in fact, it prints:
```
�
ÿ
```
Why? To answer that we have to learn a bit of Perl’s internals. Read on!

# Background: What’s in a string?

In theory, Perl strings store _code points_, nothing more. They don’t store “bytes” or “characters”, but just code points—i.e., unsigned integers. (In that sense, Perl is more like JavaScript than C!)

That, of course, is just an abstraction: all programming languages use bytes _internally_ to store strings. How does Perl decide which bytes to use for which code points? As it happens, Perl can do that in either of two formats: a “narrow” format that can store code points 0-255 only, and a “wide” format that can store any arbitrary code point. Which of those formats Perl uses for a given string _is up to Perl_; things that aren’t Perl _generally_ shouldn’t care about it.

For this abstraction to work, whether Perl stores a given string as “narrow” or “wide” must make no difference to a Perl program. And indeed, if you `print $foo` you’ll get the same result regardless of which internal format Perl uses to store `$foo`. Same for `syswrite` and `send`.

_Most_ of Perl’s built-ins, though—e.g., `exec`, `open`, `mkdir`, etc.—don’t work this way.

Look again at our 4-line program above. In line 1 we create a string with 2 code points: 255 and 256. In line 2 we chop off the latter code point, so now `$foo` just has 255. In line 3 we print that string and a newline; in line 4 we run `echo` to do the same thing. Ideally lines 3 and 4 should achieve the same output. But for you they probably didn’t. Why?

Let’s rerun that program but this time pipe it to `xxd` to see exactly what’s being output:
```
> pbpaste | perl | xxd
00000000: ff0a c3bf 0a
```
`0a` is just the newline character. So line 3 printed a single byte, 0xff (plus newline), while `echo` on line 4 printed 2 bytes—0xc3 0xbf (and a newline). Line 3 is correct: a string that contains code point 255 should output byte 255 (i.e., 0xff). What’s going on with line 4?

Recall that, of Perl’s internal string-storage formats, only the “wide” one can handle code points above 255. Since `$foo` on line 1 contains code point 256, Perl stores that string in “wide” format. Then in line 2 we get rid of code point 256. Now we have just 255. Perl _could_ thus switch our string to its “narrow” (i.e., 0-255) format, but it happens—as of Perl 5.32, anyway—not to.

This _should_ make no difference since those internal storage details are behind Perl’s string abstraction. That’s the case with `print`, but `exec` misbehaves: it outputs Perl’s raw internal buffer rather than the proper code-point-to-byte conversion that `print` uses. Perl, though, doesn’t publicly define the _contents_ of that internal buffer. Thus we have [undefined behaviour](https://en.wikipedia.org/wiki/Undefined_behavior), aka “nasal demons”, built _directly_ into Perl!

This is a leak in Perl’s string-storage abstraction, and it’s what Sys::Binmode fixes.

(Extra credit: remove the `\x{100}` and line 2 from our program above, and rerun it. The two lines should now be the same. Why?)

# Enter Sys::Binmode

Sys::Binmode fixes `exec` and many other Perl built-ins by force-converting those built-ins’ arguments to Perl’s internal “narrow” string storage format. This fixes the abstraction leak: now, no matter how these strings are stored, Perl gives them to the operating system the same way.

Try it: do `cpan Sys::Binmode`, then rerun our program with `perl -MSys::Binmode`. It’ll now print two identical lines.

## Special Case: Non-POSIX OSes (e.g., Windows)

Windows programmers may see a problem here: Perl’s “narrow” string storage format can only store bytes, so any time we want to give arbitrary Unicode characters to the operating system—which doesn’t exist for POSIX OSes like Linux—we’re stuck.

As it happens, though, Perl doesn’t actually _use_ the Windows APIs that would allow sending arbitrary Unicode characters anyway. If Perl ever changed that Sys::Binmode would need an update, but for now it can work the same way as on POSIX OSes without compromising any functionality.

# Use in Existing Code?

Note that I say to use Sys::Binmode in **new** code, not _all_ code. This is because existing code may actually _depend_ on Perl’s abstraction leak.

Look again at `exec`’s broken behaviour above. For code point 255 it printed the bytes of Perl’s “wide” storage format, which for that string was 2 bytes: 0xc3 and 0xbf. Notice that that broken behaviour actually made our terminal print something useful: `ÿ`. As it happens, those 2 bytes from Perl’s internals are UTF-8 for 255. That’s because Perl’s “wide” internal format is actually just ([a “lax” variant of](https://metacpan.org/pod/Encode#UTF-8-vs.-utf8-vs.-UTF8)) UTF-8, so anything that outputs Perl’s internals will output UTF-8 if Perl stores the string in “wide” format.

Ordinarily to output a string in UTF-8 you _encode_ it thus explicitly, e.g., `encode('UTF-8', $str)`. `exec` _appears_ to be automatically `encode()`ing for us, but it’s actually just outputting whatever Perl happens to store internally. So if Perl decides to store a string “wide”, it’ll give UTF-8 to `exec` … but if Perl decides to store that string “narrow”, then you’ll get something else! We could try to second-guess Perl’s internal decision-making, but that’s dangerous: how Perl decides to store its strings is undocumented and always subject to change.

Sys::Binmode will suppress that unreliable “auto-encode” behaviour, which forces us to encode our strings properly before giving them to `exec` and friends. Of course, that’s what we should have done all along!

# Conclusion

I can think of no situation where Sys::Binmode effects any undesirable change to Perl in new code. It surely fixes bugs like in our `exec` demo program. Assuming that I’m correct that, for new code, this module _only_ avoids problems without introducing any, it should be used in all new code.

Convinced? :-)