germanic-lid

A 6.3 MB fastText language identifier for English and the Germanic languages, with a single other class for every non-Germanic language. It is built for filtering large corpora on CPU, at about 12,500 lines/s on one core.

Labels (16): en de nl af fy lb gsw yi is fo da sv nb nn gem_other other

• gsw: Alemannic / Swiss German.
• gem_other: other Germanic varieties (Low German, Limburgish, Bavarian, North and Saterland Frisian, West Flemish, Zeelandic, Pennsylvania German, Kölsch, Palatine German). In a filter, treat gem_other as other, unless you want those varieties.
• other: everything non-Germanic, plus non-language noise.

Files

File	What
germanic-lid.ftz	quantised fastText model (the default)
labels.txt	the 16 labels
germanic_lid.py	wrapper: GermanicLID(path).predict(lines, threshold) and a CLI to predict, filter or annotate JSONL and parquet
results/	metrics JSON and full tables (per-label P/R/F1 with bootstrap CIs, confusion matrices, hard pairs, threshold curves)
pareto/germanic-lid-d32.ftz	2.6 MB, same accuracy as the default within CI, about 30% faster
pareto/germanic-lid-d32.bin	162 MB, unquantised, 26k lines/s/core: the best choice for bulk filtering
pareto/germanic-lid-d128.bin	1.1 GB, the most accurate (macro-F1 0.989 on the independent sets)
pareto/PARETO.md, pareto/*.png	accuracy, speed and size of 22 systems (ours and off-the-shelf)
pareto/archive/	every other trained variant (char NB/LR, CLD3-style tiny network, the superseded v1)

Usage

from germanic_lid import GermanicLID
lid = GermanicLID("germanic-lid.ftz")
lid.predict(["Hvar er næsta strætóstoppistöð?", "Dette er ein test."], threshold=0.7)

python germanic_lid.py filter --model germanic-lid.ftz --input in.jsonl --field text --keep is,fo --threshold 0.7 --output out.jsonl
python germanic_lid.py filter --model germanic-lid.ftz --input in.parquet --field text --keep nb,nn --threshold 0.7 --output out.parquet

The wrapper normalises the text before prediction: NFKC, lowercasing, URLs and e-mail addresses removed, digits mapped to 0. It then applies a script gate: a Germanic label needs mostly Latin letters (Hebrew for yi), and any other line becomes other. If you call fastText directly, apply the same normalisation.

Recipe

• fastText supervised, softmax; dim 64; char n-grams 2-5 plus words (minCount 50); 3 epochs; lr 0.8.
• Quantised with qnorm, cutoff 150k, dsub 2 and 1 retrain epoch.
• Trained on 7.95M lines: 400k-600k per large class; small classes upsampled to at most 3×; 2.6M lines for other.
• Dedup by normalised-line hash; lines that occur under two labels are dropped; splits by document.

Training data

Source	Used for	Licence
OpenLID-v3 dataset (HPLT)	most labels; other (180 languages + noise)	per-source; all allow non-commercial use
Wikipedia 20231101	every Germanic label and gem_other; close confusables in other (Sami, Greenlandic, Celtic, Romance minority languages, Sorbian, Finnic minority languages)	CC BY-SA 4.0
FineWeb-2 train	fo, fy, lb, gsw, yi; a line is kept only if GlotLID agrees on that single line	ODC-By 1.0
Our cleaned Icelandic corpus (private)	is	mixed source licences

Not used: glotlid-corpus (gated, no access); FLORES+ (gated, no access); Scots Wikipedia (quality problem).

Results

Our model (6.3 MB) against OpenLID-v3 (1.23 GB) and GlotLID v3 (1.69 GB), with the baselines' labels mapped to ours. Macro-F1 is over the 12 labels all three support. Accuracy covers all 16 labels.

Set	Ours	OpenLID-v3	GlotLID
FLORES-200 devtest, macro-F1	0.995	0.993	0.997*
UDHR, macro-F1	0.976	0.887	0.990*
Icelandic held-out documents, accuracy	0.939	0.909	0.898
In-domain held-out, accuracy	0.946	0.784	0.916
Under 20 characters, accuracy	0.764	0.606	0.706
20-50 characters, accuracy	0.913	0.742	0.879
Synthetic web noise, accuracy	0.909	0.616	0.801
FineWeb-2 web lines (agreement with GlotLID-derived labels)	0.907	0.782	0.899
Lines/s, one core	12,500	4,700	1,500

* GlotLID's training data includes FLORES dev and probably UDHR-like text.

Hard pairs on FLORES (share of lines of A labelled B):

Pair	Ours	OpenLID-v3	GlotLID
nb→nn / nn→nb	3.3% / 0.5%	2.4% / 1.8%	0.5% / 2.4%
nb→da	0.8%	1.5%	0.3%
is↔fo	0.2% / 0.2%	0.1% / 0.3%	0.1% / 0.0%
nl↔af, de↔lb	0.1% or less	0.1% or less	0.0%

For gsw, fy and gem_other, see the per-label tables in results/; OpenLID-v3 has none of them. 95% bootstrap CIs for every number are in results/tables_v2.md.

Threshold:

• t = 0.7 raises in-domain Germanic precision from 0.943 to 0.977 and costs 2.7 points of recall;
• on FLORES, precision is 0.999 at 98.7% recall.

Choosing a model

Every fastText file works with the same wrapper (--model <file>).

• Bulk filtering: pareto/germanic-lid-d32.bin.
• Small and portable: germanic-lid.ftz or pareto/germanic-lid-d32.ftz.
• Short text or maximum accuracy: pareto/germanic-lid-d128.bin.
• See pareto/PARETO.md.

Caveats

• Closed label set: unseen Latin-script languages can receive a Germanic label, most often gem_other or en. Use a threshold.
• nb/nn is the weakest pair. Lines under 20 characters reach only about 76% accuracy (all systems struggle there).
• gsw means Alemannic broadly (Wikipedia als includes Alsatian and Swabian).
• The FineWeb-2 portion inherits GlotLID's line-level decisions for fo, fy, lb, gsw and yi.
• Research use only because of the mixed source licences.

Citation

@misc{einarsson2026germaniclid,
  author       = {Einarsson, Hafsteinn},
  title        = {germanic-lid: a fastText language identifier for English and the Germanic languages},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/hafsteinn/germanic-lid}}
}