opus-mt-en-ur-vocational

Fine-tuned version of Helsinki-NLP/opus-mt-en-ur on the VocabUR corpus for domain-adaptive English→Urdu translation of vocational trade instruction.

Model Description

The base opus-mt-en-ur model was trained on OPUS corpora (primarily religious texts, legal documents, and subtitles) and performs poorly on trade-specific language out of the box (BLEU 3.08 on vocational held-out set). Fine-tuning on VocabUR raises it to 20.83 BLEU, a ×6.8 relative improvement.

A companion NLLB-based model (mujtabaumarr/nllb-600M-vocational-urdu) achieves higher absolute performance (49.80 BLEU) on the same held-out set; that model is recommended for production use. This model is provided for research comparison.

Training Data

VocabUR — 3,442 training segments from ~32 hours of English vocational training video across three trade domains: electrical work, carpentry/woodworking, and plumbing. Human-validated translations by native Urdu speakers with trade knowledge. Transliteration of technical terms (e.g. stringers → سٹرنگرز, PEX → پی ای ایکس) is treated as the correct domain norm.

Evaluation

Evaluated on a video-independent held-out set (56 segments from 4 videos excluded entirely from the training pipeline). BLEU computed with the flores200 tokenizer (sacrebleu nrefs:1|case:mixed|eff:no|tok:flores200|smooth:exp|version:2.6.0).

Model	BLEU (flores200)	chrF
opus-mt-en-ur zero-shot	3.08	19.30
opus-mt-en-ur fine-tuned (this model)	20.83	42.91

Per-Domain BLEU (internal test split, n=354)

Domain	Zero-shot	Fine-tuned	Gain
Carpentry	2.9	19.9	+17.0
Plumbing	7.0	26.5	+19.5
Electrical	4.4	22.5	+18.1

Training Configuration

Hyperparameter	Value
Base model	Helsinki-NLP/opus-mt-en-ur
Epochs	5
Learning rate	5e-5
Effective batch size	16
Max sequence length	256 tokens
Precision	fp16 mixed
Hardware	NVIDIA T4 16GB
Training time	~50 minutes
Best checkpoint	Epoch 5 (val loss 0.842)

Usage

from transformers import MarianMTModel, MarianTokenizer

model_name = "mujtabaumarr/opus-mt-en-ur-vocational"
tokenizer  = MarianTokenizer.from_pretrained(model_name)
model      = MarianMTModel.from_pretrained(model_name)

text = ["Install the shutoff valve before connecting the PEX line."]
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
translated = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.batch_decode(translated, skip_special_tokens=True))

Citation

@inproceedings{umar2025vocabur,
  title     = {{VocabUR}: Domain-Adaptive {English--Urdu} Translation for Vocational Education},
  author    = {Umar, Mujtaba},
  booktitle = {Proceedings of [Conference Name]},
  year      = {2025}
}

Limitations

• Evaluated on a small held-out set (56 segments); headline metrics carry wide confidence intervals.
• Single-reference BLEU/chrF may penalize valid transliteration variants.
• Source videos are North American English; South Asian trade vocabulary may differ.
• Not a substitute for certified trade training or expert supervision.