Perl’s SvPV Menace
Tags: perl
Published: 2021-05-27 02:42:30

If you’ve read my [Perl, Unicode, and Bytes](https://dev.to/fgasper/perl-unicode-and-bytes-5cg7) or [Sys::Binmode](https://dev.to/fgasper/use-sys-binmode-4e6o) posts, you know about the complexities of character encoding in Perl. A bit after I wrote that first post I had a little epiphany I thought worth sharing.

One day I noticed that [URI::XSEscape](https://metacpan.org/pod/URI::XSEscape) was mangling its output: I’d pass in `épée` and get out `%C3%83%C2%A9p%C3%83%C2%A9e`. I recognized this as an extra UTF-8 encode: rather than URI-encoding my 6 bytes of `épée`, it was UTF-8 encoding—so now 10 bytes—then URI-encoding that.

I pulled out [Devel::Peek](https://metacpan.org/pod/Devel::Peek) and saw that something prior to the URI-encoding step had “upgraded” my string’s internal storage: Perl itself stored my string as 10 bytes, even though the Perl scalar still consisted of 6 _characters_. Ordinarily this is nothing of importance since Perl code doesn’t need to care how Perl itself stores its strings.

… until it _does_ need to care, that is.

# What is SvPV?

Perl’s C API—the set of macros and functions available to work with Perl from C—is a classic C API: lots of different ways to do _almost_ the same thing. To translate a Perl scalar to a C signed integer, for example, you can use `SvIV`, `SvIV_nomg`, `SvIVX`, or `SvIVx`. (`IV` here signifying an “integer value”) A similar set of macros exists for unsigned integers (`UV`s).

Converting a Perl scalar to a C string is similar. There are many tools available, but the 3 “fundamental” ones are:

- `SvPVbyte`: Takes the code points of your Perl string and gives back a C buffer whose bytes match those code points. Thus, any code point that exceeds 255 doesn’t work, and an exception is thrown.

- `SvPVutf8`: Like `SvPVbyte` but gives the _UTF-8-encoded_ bytes for your Perl string’s code points. This works for any code point that Perl can store, but for code points 128-255 it’ll give different results from `SvPVbyte`. (cf. [perldoc perlunicode](https://perldoc.perl.org/perlunicode#The-%22Unicode-Bug%22))

- `SvPV`: Gives you the Perl string’s internal buffer, aka its “PV” (“pointer value”). It could be bytes, or it could be UTF-8. It’s like a C analogue to Perl’s `use bytes`.

`SvPV`, of course, is what URI::XSEscape was using.

For `SvPV` to be meaningful it has to be used in tandem with `SvUTF8`, a macro that tells you which form the PV is: bytes, or UTF-8. So if `SvUTF8` is true, then `SvPV`’s output is UTF-8; otherwise `SvPV`’s output is bytes. But URI::XSEscape wasn’t checking `SvUTF8`; it was just URI-encoding `SvPV` directly.

The big problem with `SvPV` is that the number of contexts _other_ than Perl where it’s sensible to have a C string that could be bytes _or_ UTF-8 is … small. Nevertheless, uses of this macro (and its variants) to interact with contexts outside Perl are all over CPAN.

URI::XSEscape, like [its pure-Perl counterpart](https://metacpan.org/pod/URI::Escape), presents interfaces appropriate for both “byte-oriented” and “character-oriented” Perl code (cf. [Perl, Unicode, and Bytes](https://dev.to/fgasper/perl-unicode-and-bytes-5cg7)). Since the byte-oriented interface is what I was using, switching URI::XSEscape from `SvPV` to `SvPVbyte` was the simple fix to this problem.

In essence, C code like URI::XSEscape should approach Perl strings the same way that pure-Perl code does, without caring about Perl’s internal string storage. _Most_ C code should thus avoid `SvPV` for the same reason that most Perl should not `use bytes`.

# The plot thickens …

A quick scan through some popular XS modules showed more occurrences of this problem:

- [DBD::SQLite] (https://metacpan.org/pod/DBD::SQLite)
- [Net::Curl](https://metacpan.org/pod/Net::Curl)
- [DNS::Unbound](https://metacpan.org/pod/DNS::Unbound) (mea culpa!)
- [DNS::LDNS](https://metacpan.org/pod/DNS::LDNS)
- [YAML::Syck](https://metacpan.org/pod/YAML::Syck)
- [HTTP::Parser::XS](https://metacpan.org/pod/HTTP::Parser::XS)
- [Socket](https://metacpan.org/pod/Socket) (a core module!)

These offer a non-default mode that auto-encodes to UTF-8, but their default setup has the same bug:

- [CDB_File](https://metacpan.org/pod/CDB_File)
- [LMDB_File](https://metacpan.org/pod/LMDB_File)

There are likely many more; those are just ones I’ve found.

# How did this come to be?

I suspect it’s that:

- `SvPV` is the shortest of the above-named methods for converting a Perl scalar to a C string. Thus, it’s easier to type and looks less “intimidating”.

- Historically, Perl’s documentation favoured `SvPV` in its examples of scalar-to-string conversion; the other two were seldom discussed. [I fixed this recently](https://github.com/Perl/perl5/commit/3c3f883d1ac1fc6048277d2d60015c66c211ac9b), but it’ll be years before everyone’s local `perldoc` reflects that change.

- Perl’s default [XS typemap](https://perldoc.perl.org/perlxstypemap) uses `SvPV` (without consulting `SvUTF8`) to convert a scalar to a string. Thus, the following XSUB, called as `printstr($mystr)`:
```
void
printstr (const char *str)
  CODE:
    fprintf(stdout, str);
```
… prints Perl’s internals, which a Perl caller isn’t supposed to care about. Ideally language defaults like this would be the “safe” ways to do things, but this particular one is nonsensical.

# Does this problem affect your code?

A simple way to test for this problem is to `utf8::upgrade` your strings before you give them to the tested code—ensuring, of course, that you’re testing with some code points in the 128-255 range. Your test should verify that your program’s behaviour is the same with `utf8::upgrade`d strings as with non-upgraded strings.

You wouldn’t normally upgrade strings manually in production (since it makes your Perl code think about Perl’s internals, which it shouldn’t do), but for testing it’s fine and useful.

For example, I found the URI::XSEscape problem by doing:
```
my $foo = "épée";
utf8::upgrade($foo);
print URI::XSEscape::uri_escape($foo);
```

# Not just any old bug …

The worst part of all this is that modules like CDB_File _can’t_ replace `SvPV` without breaking existing applications that may depend on that `use bytes`-ish behaviour. So there’s not much to do except build new, corrected interfaces, deprecating the old ones … which of course will _eventually_ necessitate changes to existing code. For Perl “gurus” that may be simple, but for everyone else changing existing code could be expensive, painful, and even harmful to Perl’s reputation as a language that prizes backward compatibility.

# But that’s not all …

XS code isn’t the only place where this bug appears; _Perl itself_ has it, too! Read all about it at “[use Sys::Binmode;](https://dev.to/fgasper/use-sys-binmode-4e6o)”.

# How can we fix this?

I think _most_ code that uses `SvPV` to convert a Perl string to a C string intends for Perl code points to correspond to bytes in the C string; thus, such code should actually use `SvPVbyte` or one of its variants. (UTF-8-aware C code, of course, would use `SvPVutf8`.) Toward that end, we **MUST** discourage further use of `SvPV`. I propose to the Perl community, then, a few  changes: some that don’t break anything, and others that will probably break some things:

## Fixing this: The easy parts!

1) Rename `SvPV` and friends. We can’t _remove_ them, but we can create longer, “scarier-looking” aliases for them and use those names in the documentation. I propose `SvPVinternal`, `SvPVinternal_const`, etc.

2) Make `xsubpp` warn when it sees SvPV or variants in a typemap.

3) Use [Sys::Binmode](https://dev.to/fgasper/use-sys-binmode-4e6o) in all new code to fix Perl’s own buggy behaviour.

4) Submit bug reports! Audit the XS modules that you use, and if you find different behaviour between upgraded and downgraded strings, let the maintainers know—ideally by sending them patches!

## Fixing this: The hard part …

You can’t make an omelet without breaking some eggs, and you often can’t fix things like this without breaking _some_ current applications. Nevertheless …

5) Make `char *` and `const char *` in Perl’s default typemap use `SvPVbyte`. (Actually `SvPVbyte_nolen`, but hey.) For the vast majority of XS modules this probably would be just a bug fix, though for apps that depend on a `use bytes`-ish status quo there would be breakage. Thankfully, though: a) the most widely-used XS modules (e.g., [MIME::Base64](https://metacpan.org/pod/MIME::Base64), [JSON::XS](https://metacpan.org/pod/JSON::XS)) where this _could_ be a problem don’t appear to be vulnerable, and b) any breakage would be easy to fix: module authors merely have to adopt `SvPVutf8` if that’s what they want, optionally creating separate functions if support for both is desired.

6) Make [Sys::Binmode](https://metacpan.org/pod/Sys::Binmode)’s behaviour Perl’s own behaviour. This is more contentious because it sidesteps the much larger problem of Perl’s lacklustre support for Windows filesystems; still, Sys::Binmode-type behaviour is no _worse_ than Perl’s status quo, and it fixes a significant leak in Perl’s string abstraction.

## Fixing this: The moon-shot …

7) Perl needs to differentiate byte sequences from text strings. This would fix a plethora of “shin-bumpers” that afflict users of the language. This is a fairly difficult problem to solve, but I don’t think it’s insurmountable.

# In the meantime …

Absent fixes like the above, we just have to avoid this issue. You’ll always have consistent behaviour if you send *encoded* strings to the operating system and *downgrade* them prior to output; this way Perl doesn’t store any strings as UTF-8, so `SvPV` and `SvPVbyte` give the same result.

IMPORTANT: If you don’t _decode_ your strings, then by definition they’re already encoded, so in this case _don’t_ encode them manually, or you’ll mangle your output.