rost-tok-ro-normalized

Romanian BPE tokenizer, 32,768 vocab, trained on cedilla-normalized text.

Measured fertility

tokenizer	tokens/word on Romanian
English-only (control)	2.8421
Romanian, raw (mixed encodings)	1.5660
Romanian, normalized	1.5571
Romanian, diacritics stripped	1.5190 — but 2.0083 on correct Romanian

A Romanian tokenizer is worth 1.8× over an English one. Stripping diacritics looks 2.5% better on de-diacriticised text and is 29% worse on correct Romanian, because it byte-falls-back on every ș ț ă â î.

See https://github.com/StefanIancu/rost for the full experiment.