kyr0/defuss-lid-1.0

defuss-lid 1.0 — compact pretrained language-identification packs

Four ready-to-load language-identification packs repacked from the released fastText lid.176 and Google CLD3 (cld3-asm 4.0.0) weights. No model was trained from scratch. The packs are product-quantized (PQ) with int8 classifier heads and run in Node, Bun and modern browsers through the dependency-free defuss-lid runtime — no inference service, no native addons.

Packs

Pack	File	Runtime	Languages	Bytes	SHA-256 (first 16)
der_3	der_3.dlid	fastText	de, en, ru	78,048	a5f7731d030a0142
deferi_6	deferi_6.dlid	fastText	de, en, fr, es, ru, it	127,488	b72e2ba0d0994958
major_20	major_20.cld3q	CLD3	en, de, fr, it, ru, id, es, pt, nl, ur, vi, th, zh, ja, pl, tr, ar, ko, bn, hi	110,428	95854544f8bc194e
full	full.cld3q	CLD3	all 109 CLD3 labels	130,404	24af762041e06b07

Full hashes and the per-set build manifest are in config.json and match the source repository's reports/v0.3/build-lock.json.

Usage

npm install defuss-lid

Each pack is a package export; Node and Bun read the bundled asset from disk, browsers fetch it (the node/default export conditions select the loader):

import {load} from 'defuss-lid/packs/der_3';
const lid = await load();
console.log(lid.predict('Guten Morgen!'));  // [{language: 'de', score: …}]

import {load} from 'defuss-lid/packs/full';
const lid = await load();
console.log(lid.predict('Bonjour tout le monde.'));  // [{language: 'fr', …, reliable: true}]
lid.dispose();

To use raw downloads from this repo instead of the npm assets: .dlid files load with LidModel.fromBuffer / loadModel from defuss-lid (or defuss-lid/fasttext). .cld3q files load with createCld3Pack / loadCld3Pack from defuss-lid/cld3-pack and additionally require the shared WASM core at runtime/cld3-core.bin.gz (SHA-256 in config.json; the loader verifies it). The node-condition pack loaders default to the npm-bundled core; pass {coreURL: '…/cld3-core.bin.gz'} when using a HuggingFace download.

Measured accuracy

Held-out top-1 accuracy (%) on FLORES-200 devtest sentences (1,012 per language, aligned) and MASSIVE 1.1 test utterances (2,974 per locale), measured 2026-11-02 and re-verified byte-identical on an independent machine. Scores are not calibrated probabilities.

Pack	FLORES full	FLORES first 20 code points	MASSIVE
der_3	100.00	98.42	98.54
deferi_6	99.97	93.74	96.92
major_20	99.65	91.39	95.03
full	97.77	78.80	88.60

Cohorts: 3,036 / 6,072 / 20,240 / 78,936 FLORES records and 8,922 / 17,844 / 62,454 / 154,648 MASSIVE records for the 3/6/20/full sets. Full-inventory evaluations cover the 78 (FLORES) and 51 (MASSIVE) labels shared by both models, not every label. Closed-set packs compete only inside their own allowlist; cross-set scores are not comparable.

Limitations

• Smaller packs trade away short-text and off-domain accuracy; full loses mostly on the artificial 20-code-point slice. There is no domain-independent best pack — evaluate on your own distribution.
• The fastText packs (der_3, deferi_6) prune input features observed in a 100-sentences-per-language FLORES dev calibration corpus; valid features outside it are absent by construction. Hashed n-gram buckets can collide across languages.
• No unsupported-language rejection and no mixed-language/code-switching accuracy is claimed. Restricting candidates does not make unsupported-language detection reliable.
• CLD3 packs reconstruct the native WASM layout at load: they reduce download/storage size, not inference RAM.

Licenses and attribution

Multi-licensed; see NOTICE.md (bundled here) for full attribution:

• Runtime code: MIT
• der_3, deferi_6 (derived from fastText lid.176): CC-BY-SA-3.0
• major_20, full (derived from CLD3): Apache-2.0

Provenance

Source repository, benchmark evidence (40 reports with raw predictions, paired bootstrap intervals, environment records) and the one-command reproduction pipeline: defuss-lid v0.3.0. Pack bytes are SHA-256-locked; the four files in this repo are byte-identical to the measured artifacts.