kyr0/fastText-pq-dno

defuss-lid fastText DNO-PQ packs

fastText lid.176 for a few languages, 82% to 89% smaller, with 20% to 21% fewer errors than lid.176 across the measured test slices.

Two fastText language-identification packs, der_3 (English, German, Russian) and deferi_6 (plus French, Italian and Spanish), made from Meta's lid.176.ftz by the DNO-PQ (Direct Neural Optimization - Product Quantization) method. lid.176's row vectors are re-encoded into 4 product-quantization codes per row instead of 8, and bucket ids are stored as 2-byte gaps, so a bucket row takes 7 bytes instead of 13 and 74% to 78% more rows fit the same file size. Four steps then tailor the packs to their languages, all on held-out text:

• the rows were chosen from lid.176's 50,000 rows by what each one is worth, then exchanged once by exact gain;
• the shared PQ codebooks and norms were refitted;
• the hierarchical-softmax node rows were refitted and stored as int8.

Compared with lid.176 restricted to the same languages, the packs are 82% to 89% smaller. They make 33% fewer errors on MASSIVE utterances, 22% to 29% fewer on the two-word starts of the benchmark sentences, 21% to 31% fewer on the first 20 code points of FLORES sentences and 15% to 22% fewer on those of NTREX-128 news. deferi_6 beats lid.176 on 9 of the 10 test slices; der_3 trails it by a few texts on Tatoeba sentences and NTREX two-word starts (see below).

Runtime

The packs run on the defuss-lid fastText runtime, plain TypeScript with no dependencies, in browsers, Node.js and Bun: defuss-lid. They need a runtime that reads bucket ids stored as 2-byte gaps (bucket_gaps), which the source repository added after release 0.4.0; 0.4.0 and older reject the files.

Packs

Pack	File	Languages	Rows	Bytes	gzip -9	SHA-256 (first 16)
der_3	der_3.dlid	en, de, ru	7,733	77,952	70,450	6119405a387d8568
deferi_6	deferi_6.dlid	en, de, ru, fr, it, es	13,915	127,424	111,006	edd01aadbaeb4daf

Size compared with lid.176

Measured 2026-11-03; gzip at level 9 with fflate, as in the source repository's size table. "lid.176" is lid.176.ftz with its output restricted to the pack's labels and every input row kept (3/fasttext-native-exact.dlid and 6/fasttext-native-exact.dlid of kyr0/defuss-lid-1.0). "All-in" adds the fastText runtime an application downloads besides the pack (33,270 bytes, 11,367 gzipped).

	lid.176	DNO-PQ	Reduction
der_3 weights	702,886 B	77,952 B	−88.9%
der_3 weights, gzipped	570,719 B	70,450 B	−87.7%
deferi_6 weights	703,315 B	127,424 B	−81.9%
deferi_6 weights, gzipped	571,066 B	111,006 B	−80.6%
der_3 all-in, gzipped	582,086 B	81,817 B	−85.9%
deferi_6 all-in, gzipped	582,433 B	122,373 B	−79.0%

The released lid.176.ftz itself, with all 176 labels, is 938,013 bytes.

Usage

import {readFile} from 'node:fs/promises';
import {LidModel} from 'defuss-lid';

const lid = LidModel.fromBuffer(await readFile('deferi_6.dlid'));
console.log(lid.predict('Buongiorno a tutti!'));  // [{language: 'it', score: 1.96…}]
console.log(lid.predict('Buongiorno a tutti!', {boost: false}));  // [{language: 'it', score: 0.96…}], the model alone
console.log(lid.predict('Dzień dobry', {boost: false, threshold: 0.5}));  // [], Polish is not in the pack

In a browser, loadModel(url) from defuss-lid fetches a pack.

Scores are lid.176's unconditional probabilities: text in a language outside the pack scores low for every label, so a threshold on {boost: false} scores can reject it.

Letter boost: the defuss-lid runtime adds to each language's score the share of the text's letters its alphabet explains (up to 1). {boost: false} returns the model's own probability.

Measured accuracy

Top-1 accuracy (%), measured with the defuss-lid runtime (dnp-tools/fastText/eval.mjs in the source repository). "lid.176" is the output-restricted lid.176.ftz above, "DNO-PQ" the pack as defuss-lid runs it by default, with the letter boost. "Errors vs lid.176" is the relative change in the error rate, (DNO-PQ errors − lid.176 errors) / lid.176 errors; −50% means half the errors.

• Benchmark (the defuss-lid benchmark cohorts): FLORES-200 devtest sentences (1,012 per language), their first 20 code points, MASSIVE 1.1 test utterances, and the distinct first two words of every FLORES and MASSIVE benchmark sentence per language. No benchmark text was used to choose rows, fit or select the edit.
• Tatoeba test split: 1,000 sentences per language, held out by sentence id from the Tatoeba text the edit used.
• NTREX-128: 1,000 news sentences per language, from a source no fit or selection text came from. The edit's news text is Wikinews; a Wikinews sentence equal to a test sentence was left out.

der_3 (English, German, Russian)

Slice	Rows	lid.176	DNO-PQ	Errors vs lid.176	DNO-PQ, boost off	Errors vs lid.176, boost off
FLORES-200 devtest, sentences	3,036	100.00	100.00	0.0%	100.00	0.0%
FLORES-200 devtest, first 20 code points	3,036	98.62	99.04	−31.0%	98.98	−31.1%
MASSIVE 1.1 test, utterances	8,922	99.16	99.44	−33.3%	99.39	−34.9%
Two-word starts (FLORES + MASSIVE)	7,278	96.52	97.51	−28.5%	97.44	−35.2%
Tatoeba test split, sentences	2,999	99.77	99.73	+14.3%	99.73	+14.3%
Tatoeba test split, first 20 code points	2,999	99.50	99.60	−20.0%	99.60	−20.0%
Tatoeba test split, two-word starts	2,279	98.11	98.55	−23.3%	98.55	−26.7%
NTREX-128, sentences	3,000	99.93	99.93	0.0%	99.93	0.0%
NTREX-128, first 20 code points	3,000	97.83	98.17	−15.4%	98.13	−15.2%
NTREX-128, two-word starts	2,637	95.37	95.22	+3.3%	95.15	+0.8%

deferi_6 (English, German, Russian, French, Italian, Spanish)

Slice	Rows	lid.176	DNO-PQ	Errors vs lid.176	DNO-PQ, boost off	Errors vs lid.176, boost off
FLORES-200 devtest, sentences	6,072	100.00	99.97	2 errors, lid.176 none	99.97	2 errors, lid.176 none
FLORES-200 devtest, first 20 code points	6,072	94.60	95.75	−21.3%	95.73	−23.4%
MASSIVE 1.1 test, utterances	17,844	97.88	98.58	−32.8%	98.56	−33.4%
Two-word starts (FLORES + MASSIVE)	14,297	89.10	91.51	−22.1%	91.47	−23.5%
Tatoeba test split, sentences	5,999	99.42	99.43	−2.9%	99.42	−2.8%
Tatoeba test split, first 20 code points	5,999	97.88	98.37	−22.8%	98.33	−21.9%
Tatoeba test split, two-word starts	4,585	91.67	92.78	−13.4%	92.72	−13.5%
NTREX-128, sentences	6,000	99.80	99.85	−25.0%	99.85	−25.0%
NTREX-128, first 20 code points	6,000	92.72	94.35	−22.4%	94.28	−22.0%
NTREX-128, two-word starts	5,147	85.25	86.54	−8.7%	86.48	−9.8%

The losses are few texts: der_3 gets 8 of 2,999 Tatoeba sentences wrong against lid.176's 7 and 126 of 2,637 NTREX two-word starts against 122; deferi_6 gets 2 of 6,072 FLORES sentences wrong against none.

Per-language losses against lid.176

Measured on the answers without letter boost. Losses of more than 0.5 points:

• der_3: German −1.05 points on NTREX two-word starts.
• deferi_6: English −1.48 on FLORES first 20 code points and −1.01 on NTREX two-word starts; German −0.58 on NTREX two-word starts.

The losses concentrate on very short text, where a table with 3.6 (deferi_6, 13,915 rows) to 6.5 (der_3, 7,733 rows) times fewer rows than lid.176's 50,000 has fewer n-grams to go on.

How it was made

1. Re-encoding. lid.176 stores each row as 8 codes into 2-value codebooks plus a norm code. Weighted k-means re-encodes every row's direction into 4 codes into 4-value codebooks (256 centroids each), keeping lid.176's norm code; a row's squared error weighs by how much held-out text reads it. A first refit of the codebooks, norms and node rows on every row that text reads follows. Bucket ids are stored as gaps between sorted ids, 2 bytes each.
2. Rows. From the rows that held-out text reads, rows are removed in rounds, those whose exact removal costs the least loss per byte first, until the pack is no larger than the defuss-lid v0.3 pack of the same languages.
3. Refit. The hierarchical-softmax node rows, the 4 × 256 × 4 PQ codebook values and the 256 norms (4,352 numbers shared by all rows) are refitted with L-BFGS on weighted cross-entropy. The outcomes are the pack's labels plus "none of these", the probability of the tree branches that lead to labels outside the pack. Text in other languages keeps that mass, so a threshold still rejects it. Long texts lid.176 gets right and text in other languages are fitted 70% towards lid.176's own distribution, and an L2 penalty pulls towards lid.176's values. The node rows are rounded to int8 per row, and the codebooks and norms are refitted on the rounded rows.
4. Exchange. With the refitted numbers, the 10% of rows the pack lacks that would gain the most per byte join, the least valuable rows leave until the size is the same again, and step 3 runs again. The exchange was kept because it lowered the loss over all fit text.
5. Guard. On a selection split, no language may lose more than 0.25 points on long text or 1 point on short text against the v0.3 pack when a paired exact test says the loss is not noise.

Edit data: FLORES-200 dev, MASSIVE 1.1 train and dev, Tatoeba (up to 20,000 fit sentences per language, split by sentence id) and Wikinews (10,000 fit and 1,000 selection sentences per language, split by article). Each sentence also enters as its first 1, 2, 3 and 5 words and its first 10, 20 and 40 code points. Every language has the same weight, shared equally by the four sources. A Wikinews article holding a FLORES sentence was left out, because FLORES was translated from English Wikinews articles.

Limitations

• A pack answers with one of its own languages; use a threshold on {boost: false} scores to reject others.
• Text without any of the pack's rows gets no answer ([]).
• Accuracy is measured on the six languages' benchmark, Tatoeba and NTREX text only.
• defuss-lid 0.4.0 and older cannot read the packs (see Runtime).

Licenses

• Weights: CC BY-SA 3.0, adapted from fastText lid.176.ftz (Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov and the fastText team). ShareAlike: adaptations keep this license.
• Edit data, not redistributed: FLORES-200 (CC BY-SA 4.0), MASSIVE (CC BY 4.0), Tatoeba (CC BY 2.0 FR), Wikinews (CC BY 2.5, CC BY 4.0, French pages also CC BY-SA 4.0).
• Test data only: NTREX-128 (CC BY-SA 4.0).

See NOTICE.md.