Model Card for maximxls/text-normalization-ru-terrible

Normalization for Russian text. Couldn't find any existing solutions (besides algorithms, don't like those) so made this.

Model Details

Model Description

Tiny T5 trained from scratch for normalizing Russian texts:

translating numbers into words
expanding abbreviations into phonetic letter combinations
transliterating english into russian letters
whatever else was in the dataset (see below)

Model Sources

Training code repository: https://github.com/maximxlss/text_normalization
Main dataset: https://www.kaggle.com/c/text-normalization-challenge-russian-language

Uses

Useful in TTS, for example with Silero to make it read numbers and English words (even if not perfectly, it's at least not ignoring)

Quick Start

from transformers import (
    T5ForConditionalGeneration,
    PreTrainedTokenizerFast,
)


model_path = "maximxls/text-normalization-ru-terrible"

tokenizer = PreTrainedTokenizerFast.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path)


example_text = "Я ходил в McDonald's 10 июля 2022 года."

inp_ids = tokenizer(
    example_text,
    return_tensors="pt",
).input_ids
out_ids = model.generate(inp_ids, max_new_tokens=128)[0]
out = tokenizer.decode(out_ids, skip_special_tokens=True)

print(out)

я ходил в макдоналд'эс десятого июля две тысячи двадцать второго года.

Bias, Risks, and Limitations

Very much unreliable:

For some reason, sometimes skips over first couple of tokens. Might be benificial to add some extra padding or whatever so it would be more stable. Wasn't able to solve it in training.
Sometimes is pretty unstable with repeating or missing words (especially with transliteration)

Training Details

Training Data

Data from this Kaggle challenge (761435 sentences) aswell as a bit of extra data written by me.

Training Procedure

Preprocessing

See preprocessing.py

Training Hyperparameters

See train.py

I have reset lr manually several times during training, see metrics.

Details

See README on github for a step-by-step overview of the training procedure.

Technical Specifications

Hardware

Couple tens of hours of RTX 3090 Ti compute on my personal PC (21.65 epochs)