script-langid/glotlid-2104-sinhala-pali-sanskrit

GlotLID-2104: Sinhala-Script Pali & Sanskrit

A fastText language identification model that extends GlotLID v3 with two new Sinhala-script classes — Pali (pli_Sinh) and Sanskrit (san_Sinh) — without catastrophic forgetting of the original 2102 languages.

Pali and Sanskrit are frequently written in Sinhala script in Sri Lankan Buddhist and scholarly corpora. Off-the-shelf LID models classify all such text as Sinhala, since sin_Sinh is the only Sinhala-script label they know. This model separates the three.

Labels

2104 total: GlotLID v3's full 2102-label set, plus pli_Sinh and san_Sinh.

The Sinhala-script classes are:

Label	Language	Script
sin_Sinh	Sinhala	Sinhala
pli_Sinh	Pali	Sinhala
san_Sinh	Sanskrit	Sinhala
und_Sinh	Undetermined (inherited from base)	Sinhala

Usage

import fasttext
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="script-langid/glotlid-2104-sinhala-pali-sanskrit",
    filename="model.bin",
)
model = fasttext.load_model(path)

labels, probs = model.predict("බුද්ධං සරණං ගච්ඡාමි", k=3)
print(labels, probs)

Strip the __label__ prefix to get the language code. Pass k=1 for a single prediction.

Training

Base model	cis-lmu/glotlid (GlotLID v3, 2102 labels)
Method	Label-space extension + supervised fine-tuning
Epochs	5
Learning rate	0.0005
Dimension	256
New classes	pli_Sinh, san_Sinh

The label space was extended from 2102 to 2104 and the model fine-tuned on a Sinhala/Pali/Sanskrit corpus. A low learning rate over few epochs was used deliberately to acquire the new classes while preserving the pretrained representations for the original 2102.

An ablation over checkpoints shows why these hyperparameters matter — the new classes need enough training to be learned at all, but the base languages stay intact throughout:

Checkpoint	Pali F1	Sanskrit F1
Extended only (no fine-tuning)	0.0000	0.0000
1 epoch, lr 0.01	0.0066	0.1505
3 epochs, lr 0.01	0.7836	0.6388
5 epochs, lr 0.0005 (this model)	0.9566	0.9019

Evaluation

Evaluated on three benchmarks — CommonLID, FLORES+, and WiLI-2018 — across both the new Sinhala-script classes and a representative sample of original GlotLID languages.

Per-language F1

Language	CommonLID	FLORES+	WiLI-2018
Sinhala (sin_Sinh)	0.9589	0.9688	0.9688
Pali (pli_Sinh)	0.9566	0.9566	0.9566
Sanskrit (san_Sinh)	0.9019	0.9019	0.9019
Sanskrit (Devanagari)	0.9612	0.9955	0.9914
English	0.9270	1.0000	0.9367
Tamil	0.9818	1.0000	0.9950
Hindi	0.9646	0.9926	0.9126
Bengali	0.9792	0.9995	0.9451
Arabic	0.9603	0.5962*	n/a
French	0.9256	1.0000	0.9875
German	0.9211	1.0000	0.9744

Aggregate

Benchmark	Samples	Accuracy	Macro F1
CommonLID	77,974	0.9016	0.9489
FLORES+	16,155	0.9049	0.9465
WiLI-2018	14,047	0.9490	0.8700

* The FLORES+ Arabic score reflects a benchmark label-mapping artifact (two Arabic variants collapsed into one evaluation class), not a model regression. Arabic scores 0.9603 F1 on CommonLID. The base GlotLID model scores 0.6054 on the same FLORES+ split, so this is inherited from the benchmark setup rather than introduced by fine-tuning.

No catastrophic forgetting

Comparing this model against the unmodified GlotLID v3 baseline on the original languages:

Language	Base GlotLID v3	This model	Δ
Sanskrit (Devanagari)	0.9634	0.9612	−0.0022
English	0.9255	0.9270	+0.0015
Tamil	0.9878	0.9818	−0.0060
Hindi	0.9713	0.9646	−0.0067
Bengali	0.9792	0.9792	0.0000
Arabic	0.9631	0.9603	−0.0028
French	0.9255	0.9256	+0.0001
German	0.9215	0.9211	−0.0004

(CommonLID, the largest benchmark at 77,974 samples.)

The original languages move by at most 0.0067 F1, while Pali and Sanskrit go from 0.0000 to 0.9566 and 0.9019. The base model scores 0.5451 on Sinhala because it cannot distinguish Pali and Sanskrit from it; this model reaches 0.9589 by separating all three.

Limitations

• Sanskrit in Sinhala script is the weakest of the three target classes (0.9019 F1); its most common confusion is with Pali, which is expected given their lexical and orthographic overlap.
• Short inputs are less reliable, as with all fastText LID models. Prefer at least a full sentence.
• The model inherits the domain characteristics of its fine-tuning corpus; performance on out-of-domain Sinhala-script text may differ.
• The base und_Sinh (undetermined Sinhala-script) label is retained and can still fire.

License

Apache-2.0, inherited from the GlotLID base model (cis-lmu/glotlid).

Citation

If you use this model, please cite GlotLID alongside this work:

@inproceedings{kargaran2023glotlid,
  title     = {{GlotLID}: Language Identification for Low-Resource Languages},
  author    = {Kargaran, Amir Hossein and Imani, Ayyoob and Yvon, Fran\c{c}ois and Sch{\"u}tze, Hinrich},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year      = {2023}
}