intx82/byt5-textnorm-ru

Russian ByT5 Text Normalization

A ByT5-based text normalization model for converting written Russian text into a spoken-form representation suitable for text-to-speech preprocessing.

The model is fine-tuned from google/byt5-small.

Intended use

The model converts written forms such as numbers, dates, times, measurements, percentages, fractions, mathematical expressions, references, and other normalization targets into Russian spoken text while preserving the surrounding sentence.

Example:

Input:
Встреча назначена на 18:45.

Output:
Встреча назначена на восемнадцать сорок пять.

The model is intended primarily as a preprocessing stage for Russian TTS systems.

Model

• Architecture: ByT5 / T5 encoder-decoder
• Base model: google/byt5-small
• Language: Russian
• Maximum source length used during training: 200 tokens
• Maximum target length: 384 tokens
• Decoding used during evaluation: greedy decoding

Training data

The model was trained through several successive dataset-generation, hard-example, corrective, and fine-tuning stages.

Across the datasets used during the training process, there are:

238,023 unique input examples

The final fine-tuning dataset contained:

train: 37046
validation: 1446
test: 1520
total: 40012

Final dataset signature:

a26a3b376f8f5796

The datasets were produced using deterministic normalization rules, model-assisted generation and review, semantic validation, linguistic validation, and deterministic verification.

Later training stages increasingly focused on difficult and corrective examples rather than simply increasing the amount of general training data.

Validation results

Final controlled validation:

Metric	Score
Exact match	78.01%
CER	2.32%
Semantic valid	89.56%
Verifier valid	82.23%

Exact match is intentionally strict. Multiple spoken realizations of the same written form may be semantically valid.

Independent OMath benchmark

The model was additionally evaluated on a fixed set of 1,000 previously unseen, naturally occurring numeric-heavy Russian sentences extracted from the OMath mathematics textbook source corpus.

Source revision:

a337a64825f6dcbbb7f96aded2622593689b1d4e

Results:

Metric	Score
Samples	1000
Exact match	10.1%
Semantic valid	92.1%
Verifier valid	91.6%

Selected category-level semantic-valid rates:

Category	Samples	Semantic valid
Math	555	97.66%
Cardinal	246	77.24%
Mixed	104	95.19%
Fraction	33	90.91%
Measure	11	100%
Ordinal	10	100%
Range	6	100%
Duration	7	100%

The independent benchmark is intentionally numeric-heavy and significantly different from the training distribution.

The benchmark itself is not distributed with this model. Only aggregate evaluation metrics and source provenance are included.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "YOUR_HF_NAME/byt5-textnorm-ru"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    use_fast=False,
)

model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "Встреча назначена на 18:45."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=200,
)

output = model.generate(
    **inputs,
    max_length=384,
    num_beams=1,
    do_sample=False,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Known limitations

The main remaining weakness observed during independent evaluation is normalization of cardinal numbers in difficult grammatical contexts.

Long or structurally complex inputs can occasionally cause truncation or repetition.

The model is optimized for modern Russian TTS normalization rather than historical or pre-reform Russian orthography.

Some mathematical expressions have more than one acceptable spoken representation, so exact string matching substantially understates semantic accuracy.

License

The model is derived from google/byt5-small, which is distributed under the Apache License 2.0.