mdpead/en-cy-translation

English → Welsh Neural Machine Translation

A Transformer-based sequence-to-sequence model for English to Welsh translation, implemented from scratch in PyTorch. Full training code available at github.com/mdpead/en-cy-translation.

Usage

from transformers import pipeline
pipe = pipeline("translation", model="mdpead/en-cy-translation")
pipe("Hello, how are you?")

Or with the model and tokenizer directly:

from src.hf_wrapper import EnCyForTranslation
from transformers import PreTrainedTokenizerFast

model = EnCyForTranslation.from_pretrained("mdpead/en-cy-translation")
tokenizer = PreTrainedTokenizerFast.from_pretrained("mdpead/en-cy-translation")

inputs = tokenizer("Hello, how are you?", return_tensors="pt")
output_ids = model.generate(**inputs, max_length=256)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Architecture

Component	Detail
Model	Encoder-Decoder Transformer
Embedding dim (d_model)	512
Attention heads	8
Encoder / Decoder layers	6 / 6
Feed-forward dim (d_ff)	2048
Vocabulary size	16,000
Max sequence length	256 tokens
Tokenizer	WordPiece (shared bilingual vocabulary)

Training uses mixed-precision (AMP), gradient accumulation, and a warmup inverse square-root learning rate schedule.

Training

• Dataset: techiaith/cardiff-university-tm-en-cy (~1.3M sentence pairs)
• Steps: 50,000
• Effective batch size: 25,000 tokens
• Optimiser: AdamW (β₁=0.9, β₂=0.98, ε=1e-9)
• Learning rate: 1e-3 with 2,000 warmup steps, inverse square root decay

Benchmark

Evaluated with beam search (beam size 4) against three publicly available EN→CY models across three datasets:

• FLORES+: openlanguagedata/flores_plus devtest split (1012 sentences, Wikipedia text)
• Cardiff: techiaith/cardiff-university-tm-en-cy 10% held-out test split (1000 sentences, institutional text)
• Tatoeba: agentlans/tatoeba-english-translations Welsh subset (1613 sentences, casual/short)

FLORES+ (out-of-distribution, Wikipedia)

Model	Params	BLEU	spBLEU	chrF	chrF++
en-cy-translation	~50M	46.29	51.89	68.39	66.38
NLLB-200 (distilled)	600M	36.14	37.57	58.61	56.55
Opus-MT	74M	13.94	13.74	33.82	32.25
Small-100	330M	1.96	2.86	18.83	16.14

Cardiff University TM (in-distribution, institutional)

Model	BLEU	spBLEU	chrF	chrF++
en-cy-translation	57.49	63.87	76.77	74.64
NLLB-200 (distilled)	37.10	39.51	61.36	58.69
Opus-MT	11.05	10.69	31.19	29.17
Small-100	1.54	2.02	17.25	14.52

Tatoeba (casual, short sentences)

Model	BLEU	spBLEU	chrF	chrF++
en-cy-translation	47.37	50.21	66.57	64.22
NLLB-200 (distilled)	39.12	39.15	58.21	56.17
Opus-MT	29.86	30.11	48.25	46.72
Small-100	0.52	0.75	11.63	10.84

Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

python scripts/train.py --config base

Checkpoints are saved to runs/<run-name>/checkpoints/ every checkpoint_steps steps. Training resumes automatically from the latest checkpoint.

Project Structure

├── configs/          # YAML training configs
├── scripts/
│   ├── train.py      # Training entry point
│   └── push_to_hub.py
└── src/
    ├── model.py      # Transformer implementation
    ├── tokenizer.py  # WordPiece tokenizer
    ├── datasets.py   # Dataset loading
    ├── dataloader.py # Token-bucketed DataLoader
    ├── train.py      # Training loop and checkpointing
    ├── generation.py # Autoregressive decoding
    └── hf_wrapper.py # HuggingFace PreTrainedModel wrapper

License

CC BY 4.0 — derived from the Cardiff University Translation Memory dataset, also licensed CC BY 4.0. Attribution to Cardiff University Language Technologies Unit.