---
license: mit
language:
- ru
library_name: transformers
tags:
- text-generation-inference
---
# Model Card for maximxls/text-normalization-ru-terrible

Normalization for Russian text. Couldn't find any existing solutions (besides algorithms, don't like those) so made this.

## Model Details

### Model Description

Tiny T5 trained from scratch for normalizing Russian texts:
- translating numbers into words
- expanding abbreviations into phonetic letter combinations
- transliterating english into russian letters
- whatever else was in the dataset (see below)

### Model Sources

- **Training code repository:** https://github.com/maximxlss/text_normalization
- **Main dataset:** https://www.kaggle.com/c/text-normalization-challenge-russian-language

## Uses

Useful in TTS, for example with Silero to make it read numbers and English words (even if not perfectly, it's at least not ignoring)

### Quick Start

```Python
from transformers import (
    T5ForConditionalGeneration,
    PreTrainedTokenizerFast,
)


model_path = "maximxls/text-normalization-ru-terrible"

tokenizer = PreTrainedTokenizerFast.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path)


example_text = "Я ходил в McDonald's 10 июля 2022 года."

inp_ids = tokenizer(
    example_text,
    return_tensors="pt",
).input_ids
out_ids = model.generate(inp_ids, max_new_tokens=128)[0]
out = tokenizer.decode(out_ids, skip_special_tokens=True)

print(out)
```

`я ходил в макдоналд'эс десятого июля две тысячи двадцать второго года.`

## Bias, Risks, and Limitations

**Very much unreliable:**
- For some reason, sometimes skips over first couple of tokens. Might be benificial to add some extra padding or whatever so it would be more stable. Wasn't able to solve it in training.
- Sometimes is pretty unstable with repeating or missing words (especially with transliteration)

## Training Details

### Training Data

Data from [this Kaggle challenge](https://www.kaggle.com/c/text-normalization-challenge-russian-language) (761435 sentences) aswell as a bit of extra data written by me.

### Training Procedure 

#### Preprocessing

See [`preprocessing.py`](https://github.com/maximxlss/text_normalization/blob/master/preprocess.py)

#### Training Hyperparameters

See [`train.py`](https://github.com/maximxlss/text_normalization/blob/master/train.py)

I have reset lr manually several times during training, see metrics.

#### Details

See [`README` on github](https://github.com/maximxlss/text_normalization) for a step-by-step overview of the training procedure.

## Technical Specifications

#### Hardware

Couple tens of hours of RTX 3090 Ti compute on my personal PC (21.65 epochs)