rost-tok-ro-raw

Romanian BPE tokenizer trained on raw (mixed-encoding) text. Published for comparison; prefer the normalized one.

Measured fertility

tokenizer	tokens/word on Romanian
English-only (control)	2.8421
Romanian, raw (mixed encodings)	1.5660
Romanian, normalized	1.5571
Romanian, diacritics stripped	1.5190 — but 2.0083 on correct Romanian

A Romanian tokenizer is worth 1.8× over an English one. Stripping diacritics looks 2.5% better on de-diacriticised text and is 29% worse on correct Romanian, because it byte-falls-back on every ș ț ă â î.

See https://github.com/StefanIancu/rost for the full experiment.