No Language Left Behind: Scaling Human-Centered Machine Translation
Paper • 2207.04672 • Published • 5
Request access to m2mTani-MT
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
m2mTani-MT is released for non-commercial research and educational use under CC BY-NC 4.0. The Tani languages are community languages of Arunachal Pradesh, and machine translations of them can be wrong in ways that are hard to spot without a fluent speaker. Requests are reviewed manually. Please tell us who you are and how you plan to use the model.
Log in or Sign Up to review the conditions and access this model content.
m2mTani-MT: many-to-many translation for five Tani languages, Hindi and English
m2mTani-MT is NLLB-200 distilled 600M fine-tuned for many-to-many translation among five Tani languages of Arunachal Pradesh (Adi, Apatani, Galo, Nyishi and Tagin), Hindi and English: 7 languages, 42 translation directions, in one model.
NLLB-200 has never seen any of the five Tani languages. This model adds a language token for each of them and trains all directions together, so Tani ↔ Tani, Tani ↔ Hindi and Tani ↔ English translation share the same parameters.
| Developed by | Tungon Dugi, Department of Computer Science and Engineering, National Institute of Technology |
| Model type | Encoder-decoder Transformer (M2M100ForConditionalGeneration), 12 + 12 layers, d_model 1024, ≈615M parameters |
| Base model | facebook/nllb-200-distilled-600M |
| Checkpoint | m2m-tani-nllb-32k-steps: 32,000 optimizer steps of fine-tuning |
| Languages | Adi, Apatani, Galo, Nyishi, Tagin, Hindi, English |
| License | CC BY-NC 4.0, the same as the base model |
| Paper | m2m-Tani-MT: Finetuning Multilingual Machine Translation Model for 5 Tani languages of Arunachal Pradesh (under peer review) |
| Demo | m2mTani-MT Space (gated: sign in with an account that has been granted access to this model) |
| Training code | GitHub: m2mTani-MT |
Language codes
The model uses NLLB-200 / FLORES-200 style codes (ISO 639-3 + ISO 15924 script). The five Tani codes are new tokens added to the NLLB tokenizer.
| Language | Code | Family | In NLLB-200? |
|---|---|---|---|
| Adi | adi_Latn |
Tani (Sino-Tibetan) | No (new) |
| Apatani | apt_Latn |
Tani (Sino-Tibetan) | No (new) |
| Galo | adl_Latn |
Tani (Sino-Tibetan) | No (new) |
| Nyishi | njz_Latn |
Tani (Sino-Tibetan) | No (new) |
| Tagin | tgj_Latn |
Tani (Sino-Tibetan) | No (new) |
| Hindi | hin_Deva |
Indo-Aryan | Yes |
| English | eng_Latn |
Germanic | Yes |
The other NLLB-200 language tokens are still in the tokenizer, but the model was fine-tuned only on the seven languages above. Translation into or out of any other language is not supported or evaluated.
How to use
Access is gated. Request access on this page, then log in with a token from the approved account
(hf auth login, or set HF_TOKEN).
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = "repleeka/m2mTani-MT"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(
model_id, dtype=torch.float16 if device == "cuda" else torch.float32
).to(device).eval()
def translate(texts, src_lang, tgt_lang, num_beams=4, max_new_tokens=256):
tokenizer.src_lang = src_lang # input = [src_lang] tokens </s>
batch = tokenizer(texts, return_tensors="pt", padding=True, truncation=True, max_length=256).to(device)
with torch.inference_mode():
out = model.generate(
**batch,
forced_bos_token_id=tokenizer.convert_tokens_to_ids(tgt_lang), # output starts with the target code
num_beams=num_beams,
max_new_tokens=max_new_tokens,
)
return tokenizer.batch_decode(out, skip_special_tokens=True)
print(translate(["Where are you going?"], "eng_Latn", "tgj_Latn")) # English -> Tagin
print(translate(["Where are you going?"], "eng_Latn", "njz_Latn")) # English -> Nyishi
print(translate(["ngo metikslo kénna amiko."], "adi_Latn", "eng_Latn")) # Adi -> English
Notes:
- Set
src_langto the language of the input andforced_bos_token_idto the target language code. These two tokens are the only signal of the translation direction. - The model was trained on segments of at most 128 tokens (Bible verses, sentences, words and phrases). Translate long documents sentence by sentence.
- Beam search (
num_beams=4) was used for test-set evaluation. Greedy decoding (num_beams=1) is faster and slightly worse. - Tani text in the training data is lowercase, and its spelling follows the source documents of each language
(for example Adi
íandéare letters of the Adi alphabet, and a colon inside a word, as inde:mo, marks a long vowel). Input that follows these conventions works best.
Training data
The model was trained on the mTani corpora, a parallel corpus in which every text in a Tani language or Hindi is paired with one or more English translations. Pairs between two non-English languages are built from the same Bible verse in both languages.
| Language | Parallel pairs with English | Main domains |
|---|---|---|
| Adi | 565,218 | Bible (Old and New Testament) |
| Apatani | 176,388 | Bible (New Testament), dictionary, everyday phrases and conversation |
| Galo | 171,894 | Bible (New Testament), dictionary entries and example sentences |
| Nyishi | 585,364 | Bible (Old and New Testament), dictionary, everyday phrases, news |
| Tagin | 363,606 | Bible (New Testament), the GinLish corpus, dictionary, everyday conversation, news, synthetic (KIFS) |
| Hindi | 1,089,496 | Bible (Old and New Testament) |
| Total | ≈2.95M |
The pair counts include the multiple English versions of each Bible verse. A verse is typically paired with about 20 openly licensed English Bible versions from eBible.org.
Bible text is not redistributed with this model or its GitHub repository. Bible translations are the property of their publishers. Links to the sources are given in the GitHub repository. The GitHub repository contains a 2,000-pair non-Bible sample of the training data for illustration.
Training procedure
Training is reproducible with the notebook in the GitHub repository.
- New language tokens.
adi_Latn,apt_Latn,adl_Latn,njz_Latnandtgj_Latnwere added as special tokens. Each new embedding was initialised to the mean of thelus_Latn(Mizo) andkac_Latn(Jingpho) embeddings. Both are Latin-script Sino-Tibetan languages of the region that NLLB-200 already knows. The output layer shares these weights. - Split. 90 / 5 / 5 train / dev / test. Bible text is split by whole chapters, with the same chapters held out in every language, so a verse never appears in two splits in any language or pair. Other text is split by groups of source texts linked through a shared English translation. A leakage audit drops a held-out item if its English occurs in training or if its source nearly matches a training text (3-gram consonant-skeleton Jaccard ≥ 0.8).
- Examples per epoch. Every training source text with up to 3 of its English versions, in both directions. Languages are sampled with temperature α = 0.3. Synthetic (KIFS) rows get weight 0.5. Every training verse found in 2 or more non-English languages adds up to 3 random language pairs, in both directions.
- Optimisation. Adafactor, learning rate 1e-4, 2,000 warm-up steps then inverse-square-root decay, label smoothing 0.1, bf16 autocast, ≈16K tokens per optimizer step (2,048-token batches × 8 accumulation steps), length buckets up to 128 tokens.
- Model selection. Mean dev chrF++ over all 42 directions (greedy, 100 items per direction), computed every 2,000 steps. The released checkpoint was taken at 32,000 steps.
- Hardware. One NVIDIA RTX 4000 Ada (20 GB).
Evaluation
The test set is held out by Bible chapter and audited for leakage, as described above. The metrics are chrF++ (primary) and BLEU (sacreBLEU), with 95% bootstrap confidence intervals, for all 42 directions. Results are reported in the accompanying paper, which is currently under peer review. This card will link to them once the paper is published.
Intended use
- Research on machine translation for low-resource and Tani languages.
- Education, and drafting or assisting translation work by people who know the languages.
- Language documentation and community language technology, with review by fluent speakers.
Out-of-scope use
- Commercial use (prohibited by CC BY-NC 4.0).
- Medical, legal, safety-critical or official communication without review by a fluent speaker.
- Treating output as authoritative text in a Tani language, for example publishing it without review, or using it to create new "reference" data.
- Languages other than the seven listed above.
Bias, risks and limitations
- Domain skew. Most of the training text is Bible translation. The model is strongest on Biblical and formal register, and can produce Biblical vocabulary or phrasing for everyday input. Hindi was learned through Bible text only.
- Uneven data. Adi and Nyishi have the full Old and New Testament. Galo, Apatani and Tagin have the New Testament plus smaller everyday and dictionary data. Quality differs a lot between languages and directions.
- Tani ↔ Tani and Tani ↔ Hindi directions were learned from verse-aligned Bible text only.
- Spelling variation. Tani orthographies are not fully standardised, and different sources spell the same word differently. The model reproduces the conventions of its training sources and may not match a reader's preferred orthography.
- Hallucination. Like every NMT model, it can produce fluent output that is wrong, especially for names, numbers, dates and long or out-of-domain inputs.
- Cultural content. Kinship terms, rituals and traditional knowledge are under-represented and may be mistranslated.
Citation
If you use this model, please cite:
@unpublished{dugi2026m2mtanimt,
title = {m2m-Tani-MT: Finetuning Multilingual Machine Translation Model for 5 Tani languages of Arunachal Pradesh},
author = {Dugi, Tungon},
note = {Department of Computer Science and Engineering, National Institute of Technology Arunachal Pradesh. Manuscript under peer review},
year = {2026}
}
Title: m2m-Tani-MT: Finetuning Multilingual Machine Translation Model for 5 Tani languages of Arunachal Pradesh Author: Tungon Dugi Department: Computer Science and Engineering Institute: National Institute of Technology Arunachal Pradesh Manuscript status: Under peer review
Please also cite the base model:
@article{nllb2022,
title = {No Language Left Behind: Scaling Human-Centered Machine Translation},
author = {{NLLB Team} and Costa-juss{\`a}, Marta R. and Cross, James and others},
journal = {arXiv preprint arXiv:2207.04672},
year = {2022}
}
Contact
For questions, access issues or collaboration, open a discussion in the Community tab of this repository.
- Downloads last month
- -
Model tree for repleeka/m2mTani-MT
Base model
facebook/nllb-200-distilled-600M