Russian → English Transformer (from scratch)
A compact encoder–decoder Transformer trained from scratch (no pretrained weights) for Russian→English translation. Built as a learning project — the tokenizer, model, training loop, and beam-search decoding are all hand-written.
- Parameters: ~11.5M
- Architecture: 4 encoder + 4 decoder layers,
d_model=256, 8 heads,d_ff=1024, sinusoidal positional encoding, tied input/output embeddings - Tokenizer: byte-level BPE, vocab 16,000 (shared RU/EN), included as
tokenizer.json - Data: 200,000 opus-100 RU–EN pairs
- Training: 60 epochs max, early-stopped ~epoch 40 (patience 5), Adam + Noam LR schedule, label smoothing 0.1, batch size 64
Results (held-out test split, 1,951 sentences)
| Decoding | BLEU | chrF |
|---|---|---|
| Greedy | 25.04 | 47.07 |
| Beam-5 | 25.91 | 47.85 |
Validation BLEU was 26.96. Note that opus-100 (subtitle-derived) contains some misaligned reference pairs, so these BLEU numbers slightly underestimate true quality.
Usage
# pip install torch tokenizers huggingface_hub
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("prplguyy/ru-en-transformer")
sys.path.insert(0, path)
from translator import translate
print(translate("Привет, как у тебя дела сегодня?", method="beam"))
# -> "Hey, how are you doing today?"
The repo bundles everything needed to run inference on CPU: model.pt (weights),
tokenizer.json, and the model/decoding code (config.py, model.py, decoding.py,
translator.py).
Limitations
Small from-scratch model: strong on everyday conversational sentences, but expect rough edges on rare proper names, idioms, and long or technical text. English→Russian is not supported (trained one direction only).
Links
- 🕹️ Live demo: https://transformertranslaterussian2english.streamlit.app/
- 💻 Source / training code: https://github.com/prplguyy/transformerTranslateRussianEnglish