mateza/mateza-opus-ft-v3

Mateza OPUS-FT v3: English ↔ Kinyarwanda translation

Nova Opus by Mateza — training and serving code for English ↔ Kinyarwanda OPUS-MT fine-tunes.

Two compact translation models, English → Kinyarwanda and Kinyarwanda → English, behind one Python interface. They're fine-tuned from Helsinki-NLP's OPUS-MT models on 160,912 screened sentence pairs, in three rounds (v1 → v2 → v3). Later rounds continue from Ice Block lake slices (curriculum, exams, data-driven training_runs).

Organization	Mateza (withmateza on GitHub)
GitHub	https://github.com/withmateza/nova-opus
Hugging Face (weights)	mateza/mateza-opus-ft-v3
Hugging Face (curated packs)	mateza/ice-block-curated-packs
Sibling (data lake)	Ice-Block
Nova Chat	https://nova.mateza.rw/
System One	https://nova.mateza.rw/system-one
Training UI	https://nova.mateza.rw/training
Developer guide	CONTRIBUTING.md
Quality (v3 card)	41.65 mean chrF++ on FLORES+ devtest (beam 4); 40.81 greedy. That's +8.5 over the untouched OPUS-MT models (33.18).
Size	75M parameters per direction; 290 MB (fp32) or 75 MB (CTranslate2 int8)
Speed	~11 sentences/s on a laptop CPU and ~310 ms for one sentence with the included CTranslate2 int8 models. On a T4 GPU, ~71–80 sentences/s.
Licence	Apache-2.0 (weights); see Data and attribution for the training data

Authors: Mateza Kayondo and Promesse Rukundo. Organization: Mateza.

---

Current state (Nov 2026) — v5.5 is the working version

Topic	Status
Working version	v5.5 — curated basics + kin-ling continue-from-v5.2 (docs/v5.5-training-spec.md). Helpers: scripts/start_v55.sh, register_run.py. STEPS=1500 (never blind 10k). Train/promote only after exams clear.
Champion (Nova default)	v5.2-curriculum — capability ladder en-rw 21/22 (tied with v5.1). Live current_id on nova.mateza.rw.
Lake (verified)	1,279,699 pairs — general 1,098,868 · legal 165,785 · dictionary 9,774 · primer 4,582 · medical 528 · greetings 162. Curated v5.5 fill ~+3,071 rows. Prefer packs over more NLLB.
System One	Router only at nova.mateza.rw/system-one. Not a Marian replacement.
GPU	Host behind nova.mateza.rw. Catalog via train.py + register_run.py → Ice Block training_runs (19 runs verified).

Version history (v5 → v5.4.2)

Version	Status
v5 / v5.1	Ladder work; v5.1 tied 21/22
v5.2	Champion — 21/22; Nova current
v5.3 / v5.4.x / v5.4.2	Exam-fail / dictionary / NLLB / System One-in-loop — regressed vs v5.2; do not promote
v5.5	Current working version — lake curated fill done; Marian continue pending promotion

Before / after (v5.2 vs v5.5)

	v5.2 champion	v5.5 working
Capability en-rw	21/22	Target ≥ 21/22 + kin-ling (not promoted yet)
Continue-from	cleaned lake / prior FT	v5.2 best/ only
STEPS	various (10k hurt later)	1500
Basics in lake	primer+greetings ~0.16%	primer 4,582 + greetings 162

HF card body still describes published v3 baseline weights. Curriculum champions live as GPU runs/…/best until a promoted upload.

---

What this repo does

This is the training and serving code for Mateza OPUS-FT: Marian checkpoints (en-rw/, rw-en/), CTranslate2 int8 (ct2-int8/), translator.py, train.py, FLORES+/capability eval, Ice Block exam runners, and register_run.py for the Nova training catalog.

Ice Block holds the Postgres lake (pairs, quality labels, curriculum fixtures, exams, training-run IDs). This repo reads that lake, trains, grades exams. nova-lab loads the checkpoints on nova.mateza.rw.

Clone:

git clone https://github.com/withmateza/nova-opus.git
cd nova-opus
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# optional fast CPU/GPU engine
pip install ctranslate2

Do not commit .env. Point ICE_BLOCK_DATABASE_URL or DATABASE_URL at a local Ice Block database when training or running exams. Weights are gitignored; pull from Hugging Face or train locally.

---

Quick start

pip install torch transformers sentencepiece sacremoses
pip install ctranslate2            # optional: the fast CPU/GPU engine

import os, sys
from huggingface_hub import hf_hub_download
sys.path.insert(0, os.path.dirname(hf_hub_download("mateza/mateza-opus-ft-v3", "translator.py")))
from translator import Translator

t = Translator("mateza/mateza-opus-ft-v3")               # PyTorch; GPU if available
t.translate("Thank you very much.", "en->rw")                 # 'Urakoze cyane.'
t.translate(["Murakoze cyane.", "Abana bari ku ishuri."], "rw->en")
# ['Thank you very much.', 'Children are at school.']

fast = Translator("mateza/mateza-opus-ft-v3", backend="ct2")   # CTranslate2 int8: fastest
fast.translate("Students must register before the end of the month.", "en->rw")
# 'Abanyeshuri bagomba kwiyandikisha mbere y'uko ukwezi kurangira.'

Command line (clone or download the repo first). Quote the direction, because a shell reads a bare > as a redirect:

python translator.py --repo . --direction "en->rw" "Good afternoon." "The children are at school."
python translator.py --repo . --direction "rw->en" --backend ct2 "Abana bari ku ishuri."
python translator.py --repo . --direction "en->rw" --beams 4 "Wash your hands with soap."   # best quality

Plain transformers, without translator.py

Each direction is a standard Marian model in its own subfolder:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo = "mateza/mateza-opus-ft-v3"
tok = AutoTokenizer.from_pretrained(repo, subfolder="en-rw")
model = AutoModelForSeq2SeqLM.from_pretrained(repo, subfolder="en-rw").eval()

batch = tok(["Thank you very much.", "The children are at school."], return_tensors="pt", padding=True)
out = model.generate(**batch, num_beams=1, max_new_tokens=128)   # num_beams=1: greedy, fast
print(tok.batch_decode(out, skip_special_tokens=True))

Use subfolder="rw-en" for Kinyarwanda → English. The saved generation config defaults to beam 4, so pass num_beams=1 for speed.

CTranslate2 directly

import ctranslate2
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

repo = snapshot_download("mateza/mateza-opus-ft-v3", allow_patterns=["ct2-int8/en-rw/*", "en-rw/*"])
tok = AutoTokenizer.from_pretrained(f"{repo}/en-rw")
engine = ctranslate2.Translator(f"{repo}/ct2-int8/en-rw", device="cpu", compute_type="int8")

src = [tok.convert_ids_to_tokens(tok(s).input_ids) for s in ["Thank you very much."]]
res = engine.translate_batch(src, beam_size=1)
print([tok.decode(tok.convert_tokens_to_ids(r.hypotheses[0]), skip_special_tokens=True) for r in res])

On a GPU, use device="cuda", compute_type="int8_float16".

---

How the translation is made fast

The speed comes from five choices. Each one is measured below, on the same 100 FLORES+ devtest sentences per direction, on an ordinary laptop CPU (Intel 12th-gen, 10 threads, no GPU). Reproduce it with python benchmark.py.

1. A compact model instead of a large one

Each direction is a 75M-parameter Marian Transformer: 6 encoder and 6 decoder layers, width 512. Mateza Nova Pocket, the model we started from, has about 250M parameters and scored 14.41 chrF++. These models score 41.65 at less than a third of the size. Instead of making a big model do everything, we fine-tuned a small model that was already a specialist (OPUS-MT) on better data.

2. Greedy decoding by default

Beam search keeps 4 candidate translations alive at every step. Greedy keeps one. On this data, greedy costs about 1 chrF++ (41.65 → 40.81 on the full devtest) and is ~1.6× faster one sentence at a time. On a T4 GPU, greedy batched throughput is ~2.1× that of beam 4 (71–80 vs 33–39 sentences/s). Translator uses greedy unless you pass beams=4.

3. Length-sorted batching

Translating sentences one at a time leaves the CPU or GPU mostly idle. Translator.translate(list) sorts the input by length and sends 32 sentences per call. Similar lengths mean little padding, and results come back in the original order. This alone gives ~5.4–5.9× more throughput at the same quality.

4. CTranslate2 int8: an optimised inference engine

ct2-int8/ holds the same two models converted with CTranslate2:

• a C++ Transformer engine with 8-bit integer weights;
• 75 MB instead of 290 MB per direction;
• ~2.3–2.5× the throughput of batched PyTorch;
• ~4.3× faster for a single sentence;
• a quality change of −0.2 to −0.4 chrF++ greedy. Beam 4 even scored slightly higher on this sample.

We also tried PyTorch's built-in int8 dynamic quantisation (quantize="int8"). It gave no reliable speed-up on this CPU and cost 0.4–0.8 chrF++, so we don't recommend it. It's kept in translator.py only for comparison.

5. Small things that add up

• Lazy loading: a direction's model loads on first use, so a caller that only needs en→rw loads one model.
• Inference mode: torch.inference_mode() turns off gradient tracking.
• Length cap: the output is capped at 2 × input length + 16 tokens, so a bad input can't run on for 512 tokens.
• fp16 on GPU: half precision by default on a GPU.

Measured results (CPU, 100 FLORES+ devtest sentences per direction)

Setting	en→rw sent/s	rw→en sent/s	One sentence, p50	chrF++ en→rw / rw→en
One sentence at a time, beam 4 (naive)	0.53	0.48	2.24–2.29 s	42.66 / 40.26
One sentence at a time, greedy	0.83	0.79	1.34–1.41 s	41.49 / 39.58
Batched (32, length-sorted), greedy	4.52	4.66	1.32–1.36 s	41.49 / 39.58
Batched, greedy, PyTorch int8	4.14	5.69	1.46–1.53 s	40.67 / 39.18
Batched, greedy, CTranslate2 int8	10.59	11.61	0.31–0.32 s	41.32 / 39.17
Batched, beam 4, CTranslate2 int8	3.19	3.58	0.49 s	43.11 / 40.15

Overall, going from the naive setup to CTranslate2 int8 greedy gives 20–24× the throughput and makes each sentence ~7× faster, for about 1 chrF++. Raw numbers are in eval/benchmark-cpu.json.

Notes:

• The chrF++ values here are on a 100-sentence sample. The full 1,012-sentence results are in the next section.
• The "one sentence, p50" column times single-sentence calls, so batching doesn't change it. Batching helps throughput.
• On your hardware, run python benchmark.py (add --n 200 for more sentences). FLORES+ is gated: accept its terms on Hugging Face and log in to get the quality column.

On a GPU (Tesla T4, PyTorch, full devtest, from our evaluation): greedy batched reaches 71.1 (en→rw) and 79.5 (rw→en) sentences/s, with single-sentence p50 latency of 216 / 185 ms. Beam 4 batched reaches 33.1 / 38.8 sentences/s.

---

Quality

FLORES+ devtest: 1,012 sentences, never used for training or model selection. Scored with sacrebleu chrF++ (nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.6.0).

System	Decoding	en→rw chrF++	rw→en chrF++	Mean	BLEU en→rw / rw→en
Mateza Nova Pocket (fine-tuned)	greedy	13.66	15.15	14.41	0.25 / 0.29
OPUS-MT, untouched	beam 4	32.77	33.58	33.18	6.93 / 10.88
v1 (DU data)	beam 4	36.83	36.41	36.62	8.61 / 13.00
v2 (+ education + NLLB)	beam 4	40.59	39.13	39.86	12.20 / 14.90
v3 (this release)	greedy	41.49	40.12	40.81	12.94 / 15.89
v3 (this release)	beam 4	42.60	40.69	41.65	13.63 / 16.62

• v3 on the Digital Umuganda dev set (1,490 sentences, greedy): en→rw 65.06, rw→en 62.15 chrF++.
• No empty outputs. Degenerate (repetitive) outputs ≤ 0.3%.
• Full reports are in eval/: devtest-summary.json, devtest-beam4-summary.json and du-dev-summary.json. Each also includes v2 for comparison.

Sample translations (v3, greedy)

Input	Output
Thank you very much.	Urakoze cyane.	✅
The children are at school.	Abana bari ku ishuri.	✅
Students must register before the end of the month.	Abanyeshuri bagomba kwiyandikisha mbere y'uko ukwezi kurangira.	✅
The meeting starts at two in the afternoon.	Inama itangira saa munani z'umugoroba.	✅ time correct; "z'amanywa" fits "afternoon" better
Murakoze cyane.	Thank you very much.	✅
Good morning, how are you?	Mu gitondo, uri mu buhe buryo?	❌ word for word; "Mwaramutse, amakuru?" expected
Mwaramutse, amakuru yanyu?	If so, are you?	❌ greeting not understood

Limitations

• Greetings and everyday conversation are still a known gap. Lake audit: primer+greetings were ~0.16% of ~1.28M pairs. v5.5 prioritises curated packs + kin-ling exams over more NLLB.
• Clock times use Rwandan hours, which count from 6 a.m. ("saa mbiri" = 8 a.m.). They're often right but not reliable. Check any time, date or number.
• No native-speaker review yet. The scores are automatic metrics. Don't use the output unreviewed for medical, legal or other high-stakes text.
• Two models, not one. Each direction is a separate model, loaded only when used. A single multilingual model is ongoing work.
• Input length: up to 256 tokens per sentence (about 150–200 words); longer input is cut off. Translate long documents sentence by sentence.

Work with Ice Block (migrate → ingest → train → exam → serve)

Set ICE_BLOCK_ROOT to your clone of Ice-Block (default in the helper script: ~/ice-block). The helper sources Ice Block .env so DATABASE_URL never appears on the command line.

1. Migrate and ingest (Ice Block)

cd "$ICE_BLOCK_ROOT"
cp .env.example .env   # set DATABASE_URL; never commit .env
cargo run -p ice-block-migration -- up
# download HF datasets into data/raw/ then:
cargo run -p ice-block-sample-ingest --bin full-ingest --release
cargo run -p ice-block-sample-ingest --bin backfill_quality --release
cargo run -p ice-block-sample-ingest --release --bin seed-capability-curriculum
cargo run -p ice-block-sample-ingest --release --bin seed-exams

2. Train from the lake

Dry run (no Ice Block, synthetic rows):

python train.py --direction en-rw --synthetic 64 --steps 100 --opal-backend keep

Data-driven (preferred):

export ICE_BLOCK_ROOT=/path/to/Ice-Block
./scripts/train_from_iceblock.sh status
./scripts/train_from_iceblock.sh prepare --name opus-v4-clean-en-rw --direction en-rw
./scripts/train_from_iceblock.sh start --training-run-id UUID --steps 10000
# or one shot:
./scripts/train_from_iceblock.sh prepare-and-start --name opus-v4-clean-en-rw --direction en-rw --steps 10000

Equivalent Python (after prepare-training-run printed a UUID):

python train.py --direction en-rw --training-run-id UUID --steps 10000 --batch 8 --grad-accum 4 --fp16 --no-opal

3. Exam and curriculum

# Known-answer exam (writes exam_sittings / exam_item_results in Ice Block)
python scripts/run_exam.py --model v5.2
python scripts/run_exam.py --model v3          # HF baseline

# Full capability ladder (primer → conversation → imperative → …)
python scripts/eval_capability_curriculum.py \
  --model runs/v5.2-curriculum-en-rw/en-rw/best --label v5.2 --subfolder en-rw

# FLORES+ held-out (gated HF dataset; accept terms and `huggingface-cli login`)
python scripts/eval_flores_quick.py --model runs/v5.2-curriculum-en-rw/en-rw/best
python benchmark.py

Exam snapshot: v5.2 and v5.1 both 21/22 capability en-rw; later 5.4.x worse — keep champion = v5.2.

Promotion should require: required curriculum pass (≥ v5.2) + no serious mbaza-val / FLORES regression + gain on the target domain. See Ice Block docs/capability-curriculum.md and docs/v5.5-training-spec.md.

4. v5.5 continue (from v5.2)

bash scripts/start_v55.sh status
bash scripts/start_v55.sh queue
python register_run.py --help    # catalog into Ice Block / Nova training UI

5. Serve

Local translator (this repo):

python translator.py --repo . --direction "en->rw" "Good afternoon."
python translator.py --repo . --direction "rw->en" --backend ct2 "Abana bari ku ishuri."

Lake API (pairs/exams, Ice Block): cargo run -p ice-block-api → http://0.0.0.0:8080.

Public lab (nova-lab):

Surface	URL
Chat	https://nova.mateza.rw/
System One	https://nova.mateza.rw/system-one
Training	https://nova.mateza.rw/training

Verified API smoke (direction is en-rw / rw-en, not en->rw):

curl -s https://nova.mateza.rw/api/models | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['current_id'], d['champion_id'])"
curl -s https://nova.mateza.rw/api/lake-stats
curl -s -X POST https://nova.mateza.rw/api/translate \
  -H 'content-type: application/json' \
  -d '{"text":"Thank you very much.","direction":"en-rw"}'
# → hypothesis "Urakoze cyane."

There is no /health on the public lab today — use /api/models or /api/lake-stats.

End-to-end setup: CONTRIBUTING.md.

---

Fetch data (copy-paste)

1. Postgres — pairs + training_runs (GPU / Ice Block host)

cd /home/ubuntu/ice-block   # or your ICE_BLOCK_ROOT
set -a; source .env; set +a   # never commit .env

psql "$DATABASE_URL" -c "SELECT COUNT(*) AS pairs FROM translation_pairs;"
psql "$DATABASE_URL" -c "SELECT data_type, COUNT(*) FROM translation_pairs GROUP BY 1 ORDER BY 2 DESC;"
psql "$DATABASE_URL" -c "SELECT id::text, name, direction, status, created_at::date FROM training_runs ORDER BY created_at DESC LIMIT 8;"

2. Hugging Face (weights + curated packs)

# public v3 baseline (smoke: one file)
hf download mateza/mateza-opus-ft-v3 translator.py --local-dir ./hf-opus-v3

# curated JSONL packs (not the full 1.28M lake)
hf download mateza/ice-block-curated-packs --repo-type dataset \
  --include 'curated/greetings/pack.jsonl' --local-dir ./hf-curated

3. Local curated packs (Ice Block tree)

ls data/curated
# conjugation  du-everyday  greetings  math  primer  questions
find data/curated -name '*.jsonl' | sort
# e.g. data/curated/greetings/pack.jsonl
#      data/curated/primer/everyday.jsonl

Sample pairs (from curated packs):

en	rw
Hello	Muraho
Come here	Ngwino hano
Sit down	Icara hasi
I write	nandika

Sample exam items (nova-capability-ladder):

prompt	expected	langs
Yes	Yego	en→rw
Good morning	Mwaramutse	en→rw
Muraho	Hello	rw→en
Come here	Ngwino hano	en→rw

---

Train / exam commands (verified CLI — no blind 10k)

Activate the repo venv on the GPU box first (source .venv/bin/activate). Plain system Python lacks torch / opal.

export ICE_BLOCK_ROOT=/home/ubuntu/ice-block
cd /home/ubuntu/mateza-opus && source .venv/bin/activate

# status only — shows STEPS=1500 and init checkpoint paths
bash scripts/start_v55.sh status

# dry-run train (synthetic, 2 steps) — safe smoke; does not use the lake
python train.py --direction en-rw --synthetic 32 --steps 2 \
  --opal-backend keep --no-opal --output-dir /tmp/opus-dry-smoke

# real v5.5 continue (when you intend to train — STEPS=1500, mbaza-val eval)
# start_v55 currently inits from v5.4.2-system-one best/ (ops override);
# Nova chat current stays v5.2 until a run clears the 21/22 gate.
# bash scripts/start_v55.sh en-rw
# bash scripts/start_v55.sh queue   # en-rw then rw-en when GPU free

# exams (writes exam_sittings into Ice Block)
python scripts/run_exam.py --help
python scripts/run_exam.py --model-id v5.2

Pipeline: ingest (Ice Block) → prepare-training-run → train.py / start_v55 → run_exam / capability ladder → register_run → nova-lab serve.

---

Training (v1–v3 history)

Base models	Helsinki-NLP/opus-mt-en-rw, Helsinki-NLP/opus-mt-rw-en (Apache-2.0)
v1	Fine-tuned on Digital Umuganda data: LR 3e-5, batch 32, early stopping on FLORES+ dev
v2	Continued from v1 on DU + education + NLLB (60k): 2 epochs, LR 2e-5
v3	Continued from v2 on the full mix below: 4 epochs, LR 2e-5, batch 32, label smoothing 0.1, fp16, max length 256
Model selection	Best epoch by FLORES+ dev chrF++. FLORES+ devtest was used only for the final scores.
Hardware	One Tesla T4 (Google Colab)

v3 training data: 160,912 sentence pairs

Source	Pairs	Screening
Digital Umuganda (general, conversational)	26,984	Mateza's admission screens: language ID, LaBSE alignment, PII, scripture, deduplication
mbazaNLP education (Coursera, Atingi, Wikipedia)	36,433	Same Mateza screening
NLLB web-mined eng–kin	97,495	Out of 10.08M rows: religious-content screen, language ID ≥ 0.9, 3–60 words, length ratio, markup, deduplication, FLORES+ overlap removed, LaBSE ≥ 0.75

No FLORES+ sentence is in the training data (checked).

Data and attribution

The model weights are released under Apache-2.0, like the base models. The training data comes from:

• Digital Umuganda, Kinyarwanda–English machine translation dataset, CC-BY-4.0;
• mbazaNLP, NMT Education parallel data en–kin, CC-BY-2.0;
• NLLB mined bitext (allenai/nllb), ODC-BY 1.0;
• OPUS-MT base models: Jörg Tiedemann and Santhosh Thottingal, OPUS-MT: Building open translation services for the World (EAMT 2020).

Evaluation uses FLORES+ (CC-BY-SA-4.0); it isn't redistributed here.

Files

Path	Content
en-rw/, rw-en/	The two models: Marian, safetensors, fp32, with tokenizers
ct2-int8/en-rw/, ct2-int8/rw-en/	The same models in CTranslate2 int8 format (fast inference)
translator.py	Translator: one translate(text, direction) for both directions; batching, greedy/beam, fp16, CTranslate2 backend
benchmark.py	Speed and quality benchmark of the decoding settings, as reported above
eval/	FLORES+ devtest (greedy, beam 4) and DU dev reports, training and data reports, CPU benchmark
requirements.txt	Dependencies
train.py	Continue fine-tune; Ice Block --training-run-id or --train-jsonl
register_run.py	Register finished runs into Ice Block / Nova training catalog
scripts/train_from_iceblock.sh	Prepare lake slice + start train (reads Ice Block .env)
scripts/start_v55.sh	v5.5 examfix continue from v5.2 best/
scripts/start_v542.sh	v5.4.2 System One–in-loop variant
scripts/run_exam.py	Grade Ice Block known-answer exam
scripts/eval_capability_curriculum.py	Score capability_rules / fixtures
scripts/eval_flores_quick.py	FLORES+ chrF++ / BLEU
docs/v5.5-training-spec.md	Current training / promotion spec
CONTRIBUTING.md	Developer guide (setup → train → exam → deploy)

Citation

Nova Opus is a Mateza project. Authors: Mateza Kayondo and Promesse Rukundo. Cite the published v3 weights as:

@misc{mateza-opus-ft-v3,
  title        = {Mateza OPUS-FT v3: English-Kinyarwanda translation},
  author       = {Kayondo, Mateza and Rukundo, Promesse},
  year         = {2026},
  note         = {Fine-tuned from Helsinki-NLP OPUS-MT en-rw and rw-en. Organization: Mateza.},
  url          = {https://huggingface.co/mateza/mateza-opus-ft-v3},
  howpublished = {Hugging Face}
}