script-langid/nllb-lid-220-sinhala-pali-sanskrit

NLLB LID-220: Sinhala-Script Pali & Sanskrit

A fastText language identification model that extends Meta's NLLB LID-218 with two new Sinhala-script classes — Pali (pli_Sinh) and Sanskrit (san_Sinh) — without catastrophic forgetting of the original 218 languages.

Pali and Sanskrit are frequently written in Sinhala script in Sri Lankan Buddhist and scholarly corpora. Off-the-shelf LID models classify all such text as Sinhala, since sin_Sinh is the only Sinhala-script label they know. This model separates the three.

Labels

220 total: the full NLLB-218 label set, plus pli_Sinh and san_Sinh.

The three Sinhala-script classes are:

Label	Language	Script
sin_Sinh	Sinhala	Sinhala
pli_Sinh	Pali	Sinhala
san_Sinh	Sanskrit	Sinhala

Usage

import fasttext
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="script-langid/nllb-lid-220-sinhala-pali-sanskrit",
    filename="model.bin",
)
model = fasttext.load_model(path)

labels, probs = model.predict("බුද්ධං සරණං ගච්ඡාමි", k=3)
print(labels, probs)
# ('__label__pli_Sinh', '__label__san_Sinh', '__label__sin_Sinh')
# [0.6205, 0.3157, 0.0604]

Strip the __label__ prefix to get the language code. Pass k=1 for a single prediction.

Training

Base model	facebook/fasttext-language-identification (NLLB LID-218)
Method	Label-space extension + supervised fine-tuning
Epochs	3
Learning rate	0.01
Dimension	256
New classes	pli_Sinh, san_Sinh

The label space was extended from 218 to 220 and the model fine-tuned on a Sinhala/Pali/Sanskrit corpus. A low learning rate and small epoch count were used deliberately to acquire the new classes while preserving the pretrained representations for the original 218.

Evaluation

Evaluated on three benchmarks — CommonLID, FLORES+, and WiLI-2018 — across both the new Sinhala-script classes and a representative sample of original NLLB languages.

Per-language F1

Language	CommonLID	FLORES+	WiLI-2018
Sinhala (sin_Sinh)	0.9625	0.9760	0.9760
Pali (pli_Sinh)	0.9500	0.9526	0.9526
Sanskrit (san_Sinh)	0.9053	0.9053	0.9053
Sanskrit (Devanagari)	0.9310	0.9915	0.9909
English	0.9072	1.0000	0.9455
Tamil	0.9875	1.0000	0.9940
Hindi	0.9588	0.9921	0.9879
Bengali	0.9874	1.0000	0.9429
Arabic	0.9919	0.6667*	n/a
French	0.9192	0.9995	0.9900
German	0.9407	0.9995	0.9899

Aggregate

Benchmark	Samples	Accuracy	Macro F1
CommonLID	77,974	0.9165	0.9492
FLORES+	16,155	0.9151	0.9530
WiLI-2018	14,047	0.9618	0.8795

* The FLORES+ Arabic score reflects a benchmark label-mapping artifact (precision 1.0000, recall 0.5000 — two Arabic variants collapsed into one evaluation class), not a model regression. Arabic scores 0.9919 F1 on CommonLID.

No catastrophic forgetting

The original NLLB languages retain near-ceiling performance after the extension — Tamil, Bengali, and English all reach 1.0000 F1 on FLORES+, with Hindi, French, and German above 0.99. The two new classes are learned without degrading the pretrained 218.

Limitations

• Sanskrit in Sinhala script has the lowest precision (0.8582) of the three target classes; its most common confusion is with Pali, which is expected given their lexical and orthographic overlap.
• Pali shows high precision (0.9932) but lower recall (0.9105) — the model is conservative about assigning pli_Sinh.
• Short inputs are less reliable, as with all fastText LID models. Prefer at least a full sentence.
• The model inherits the domain characteristics of its fine-tuning corpus; performance on out-of-domain Sinhala-script text may differ.

License

cc-by-nc-4.0, inherited from the NLLB base model (facebook/fasttext-language-identification). Non-commercial use only.

Citation

If you use this model, please cite the NLLB base model alongside this work:

@article{nllb2022,
  title={No Language Left Behind: Scaling Human-Centered Machine Translation},
  author={{NLLB Team}},
  year={2022}
}