> I'm referring to #2 there: "In ASCII compatible mode, Unicode character classes are not allowed."

But that doesn't throw out Unicode support. ASCII compatible mode is literally defined as "Unicode support is disabled and the atom of matching is the individual byte instead of the codepoint." But that only applies to the region of the pattern in which Unicode mode is disabled. Searching on `&[u8]` doesn't automatically enable ASCII compatible mode. It merely makes it possible to enable it in contexts that would otherwise match invalid UTF-8.

For example, you cannot search for `(?-u:[^a])` on `&str` because `(?-u:[^a])` matches any individual byte except for the ASCII `a`. That includes things like `\xFF` or even a prefix of a valid UTF-8 encoded Unicode scalar value. But if you search on `&[u8]`, then constructs like `(?-u:[^a])` are allowed. But disabling Unicode mode when searching a `&str` is still allowed, but only if it's still guaranteed to match valid UTF-8. For example:

$ echo 'ſ' | rg '(?i-u:s)'
$ echo 'ſ' | rg '(?i:s)'
ſ

But when searching `&[u8]`, Unicode mode is still enabled by default. `rg` searches `&[u8]`, and you can see right above that it's still doing a Unicode aware case insensitive search.

> Yeah I read it; it's very good! (like everything you write) I think you're right that you can't handwave away the validation overhead, and that in some cases it matters. I just think ripgrep is pretty niche: there's gazillions of websites out there and like, 10 greps. If you're writing a grep and you want to optimize the shit out of it, well welcome to working around not validating. Otherwise, the safety tradeoffs and weirdo lack of specification aren't worth it.

It's not just grep. It's pretty much any tool that wants to deal with arbitrary file content where it's useful.

To be clear, I am not staking a position like the OP where I'm saying "conventionally UTF-8" instead of "required UTF-8" is always better or should be the preferred design in a primitive string data type. My own personal bias is certainly towards that, but as someone who is also a steward of Rust's standard library, I do not limit myself to the world of grep or Unix command line tools.

My point here is not that you're wrong about requiring UTF-8 validation as a better design, but rather, that you aren't quite getting the details right when talking about the downsides of conventional UTF-8.

I'm not clear on what you mean by "safety trade-offs" or "weirdo lack of specification." The Unicode consortium has defined rules for dealing with invalid sequences of bytes. bstr's API docs have an entire section about this[1], and its behavior even comes from a W3C standard.

The difference between "conventional UTF-8" and "required UTF-8" is not really one of safety or lack of specification. In either design, you can expose the same logical semantics. The real differences tend to come in two flavors. First is that when you have a UTF-8 guarantee and your representation is UTF-8, you can do UTF-8 decoding potentially faster than you could otherwise (safely) do because you can eliminate certain conditional branches and error checking. (But this advantage is rendered moot if validation forces you to do two passes over a string that is large.) The second benefit is that it forces a stronger discipline of discarding junk data at the boundaries. It makes failure modes as a result of bad data more explicit. It is this second benefit that I think is the most compelling advantage personally, and why I think, on balance, Rust's design is better. But it's not obviously The Correct Choice.

[1]: https://docs.rs/bstr/latest/bstr/#handling-of-invalid-utf-8