perevod-en-ru: a small English→Russian Transformer

Model details

An encoder–decoder Transformer with 9,602,304 parameters, built from scratch in the course Transformers: from scratch to production: lesson 5's design (pre-norm with RMSNorm, SwiGLU, RoPE) with lesson 6's tuned hyperparameters. Run it with the package perevod: perevod "I think, therefore I am."

Intended use

Short, everyday English sentences, and learning how translation models work. It is a course model: small enough to train on a free GPU and read in full.

Out of scope

Anything that matters. News, documents, legal or medical text, long sentences: see the evaluation.

Training data

About half a million English–Russian sentence pairs from Tatoeba (dataset xmarva/tfs-tatoeba-en-ru): short sentences written by volunteers for language learners.

Evaluation

chrF and BLEU with sacreBLEU (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.6.0), beam search with 4 hypotheses.

test set	chrF
Tatoeba, 2000 sentences	61.1
WMT20 news, 200 sentences	25.8

By the length of the English sentence, on the Tatoeba sample:

length	sentences	chrF
1–5 words	792	63.8
6–8	881	65.3
9–12	246	61.5
13 and more	81	40.7

Limitations and biases

The model's quality falls quickly outside short everyday sentences: on news it loses more than half of its chrF and often writes fluent Russian unrelated to the source.

Gender: in "My doctor said she was tired" and eleven variants with other professions, the Russian verb is feminine in 8% of translations; the model follows the profession's stereotypical gender, not the pronoun.

An empty input makes the model write a sentence; the package returns an empty translation instead. Inputs longer than 256 tokens are cut, with a warning.