kyr0/cld3-pq-dno

defuss-lid CLD3 DNO-PQ packs

Google CLD3 strongly compressed and massively improved.

Two CLD3 language-identification packs, major_20 and full, made from Google's CLD3 weights by my new DNO-PQ (Direct Neural Optimization - Product Quantization) method: the embeddings are product-quantized, and the hidden and softmax layers were refitted on held-out text, starting from Google's weights, converted to int8. The packs run on the unmodified CLD3 WebAssembly build of cld3-asm 4.0.0.

Unlike typical compressed models, that suffer from quality loss after quantization and product quantization, this model improves massively over the base model FP32. Compared with Google's original CLD3:

• both packs are 62% to 66% smaller
• and make 16% to 91% fewer labeling errors, depending on the language measured

For two-word starts, which is a significantly harder task for language identification than longer sentence LID, this model makes 51% fewer errors than Google's original with major_20 (80.88% to 90.57% accuracy) and 21% fewer with full (69.38% to 75.74%) on the benchmark.

On two-word starts of other held-out text, the errors fall by 62% and 26% on the Tatoeba test split and by 35% and 17% on NTREX-128 news.

These figures are with the letter boost at its default weight of 20, the default of defuss-lid releases after 0.4.0; 0.4.0 used weight 1, and its figures are in the history of this card.

Runtime

The model packs are compatible with the defuss-lid CLD3 runtime:

Github: defuss-lid

The runtime has been tested in-browser, in Node.js and with bun.

Packs

Pack	File	Languages	Bytes	gzip -9	SHA-256 (first 16)
major_20	major_20.cld3q	en, de, fr, it, ru, id, es, pt, nl, ur, vi, th, zh, ja, pl, tr, ar, ko, bn, hi	~110 kB	~99 kB	03503bf3e896d64a
full	full.cld3q	all 109 CLD3 labels	~130 kB	~117 kB	1c468715f981034d

Size compared with Google's CLD3

Measured 2026-11-03; gzip at level 9 with fflate, as in the source repository's size table. "Google original" is Google's released CLD3 weights in the same container, unchanged; "all-in" adds what an application downloads besides the pack: the CLD3 pack runtime JavaScript and the shared WebAssembly core (188,549 bytes, 108,233 gzipped).

	Google original	DNO-PQ	Reduction
full weights	366,028 B	130,548 B	−64.3%
full weights, gzipped	343,197 B	116,637 B	−66.0%
major_20 weights (same 20 labels)	290,876 B	110,576 B	−62.0%
major_20 weights, gzipped	273,188 B	99,901 B	−63.4%
full all-in, same runtime	554,577 B	319,097 B	−42.5%
full all-in, same runtime, gzipped	451,430 B	224,870 B	−50.2%
full all-in against npm cld3-asm 4.0.0 (its WebAssembly bundle lib/browser/cld3.js alone)	1,058,372 B	319,097 B	−69.8%
the same, gzipped	516,234 B	224,870 B	−56.4%

Usage

The packs load with createCld3Pack / loadCld3Pack from defuss-lid/cld3-pack and need the shared WASM core cld3-core.bin.gz, which the npm package ships as runtime/cld3-core.bin.gz (the loader verifies its SHA-256):

import {readFile} from 'node:fs/promises';
import {createCld3Pack} from 'defuss-lid/cld3-pack';

const lid = await createCld3Pack(await readFile('full.cld3q'), {core: await readFile('cld3-core.bin.gz')});
console.log(lid.predict('Bonjour tout le monde.'));  // [{language: 'fr', score: 20.92…, reliable: true}]
console.log(lid.predict('Dzień dobry')[0].score);  // 20.99…
console.log(lid.predict('Dzień dobry', {boost: false})[0].score);  // 0.99…, the model alone
lid.dispose();

Letter boost: Letter boosting is a defuss-lid algorithm enhancement that further improves the results: the score is the model's probability plus 20 times the share of the text's letters the language's alphabet explains (so up to 21), and a language whose weighted share alone beats the answer's boosted score replaces it. The weight 20 (CLD3_BOOST_WEIGHT) was tuned on held-out selection text, never on the test text below: accuracy rose with the weight up to 20 and then stayed flat. {boost: N} sets another weight, and {boost: false} returns the model's own probability.

In a browser, loadCld3Pack(packURL, {coreURL}) fetches both files.

Measured accuracy

Top-1 accuracy (%), measured with the real WebAssembly build (dnp-tools/cld3/eval.mjs). "Google original" is Google's released CLD3 weights restricted to the same labels, run without letter boost as Google's CLD3 has none. "DNO-PQ" is the pack as defuss-lid runs it by default, with the letter boost: 20 times each language's share of the text's letters is added to the model scores. "Errors vs Google" is the relative change in the error rate, (DNO-PQ errors − Google errors) / Google errors; −50% means half the errors. Scores are not calibrated probabilities.

• Benchmark (the defuss-lid benchmark cohorts): FLORES-200 devtest sentences (1,012 per language), their first 20 code points, MASSIVE 1.1 test utterances, and two-word starts: the distinct first two words of every FLORES and MASSIVE benchmark sentence per language (Unicode word boundaries, so Chinese, Japanese and Thai get words too). No benchmark text was used to fit or select the edit.
• Tatoeba test split: 20% of each language's Tatoeba sentences by sentence id, held out from the edit's Tatoeba fit and selection splits.
• NTREX-128: up to 1,000 news sentences per language from a source no fit or selection text came from.
• full only: FLORES-200 devtest of the 19 CLD3 languages outside the benchmark cohort, and synthetic romanized text for the six -Latn labels.

major_20 (20 languages)

Slice	Rows	Google original	DNO-PQ	Errors vs Google	DNO-PQ, boost off	Errors vs Google, boost off
FLORES-200 devtest, sentences	20,240	99.64	99.94	-83.3%	99.91	-75.0%
FLORES-200 devtest, first 20 code points	20,240	91.77	94.37	-31.6%	94.10	-28.3%
MASSIVE 1.1 test, utterances	62,454	95.20	97.66	-51.3%	97.54	-48.7%
Two-word starts (FLORES + MASSIVE)	54,043	80.88	90.57	-50.7%	90.21	-48.8%
Tatoeba test split, sentences	19,559	96.38	98.49	-58.4%	98.38	-55.3%
Tatoeba test split, first 20 code points	19,559	92.39	96.73	-57.0%	96.58	-55.1%
Tatoeba test split, two-word starts	15,698	81.72	93.04	-61.9%	92.78	-60.5%
NTREX-128, sentences	20,000	98.64	99.66	-75.0%	99.60	-70.6%
NTREX-128, first 20 code points	20,000	88.93	92.41	-31.4%	91.99	-27.6%
NTREX-128, two-word starts	17,441	78.30	85.83	-34.7%	85.29	-32.2%

full (all 109 CLD3 labels)

Slice	Rows	Google original	DNO-PQ	Errors vs Google	DNO-PQ, boost off	Errors vs Google, boost off
FLORES-200 devtest, sentences	78,936	97.81	98.16	-16.0%	98.13	-14.5%
FLORES-200 devtest, first 20 code points	78,936	79.62	83.30	-18.0%	82.63	-14.8%
MASSIVE 1.1 test, utterances	154,648	89.09	92.21	-28.6%	92.02	-26.8%
Two-word starts (FLORES + MASSIVE)	162,919	69.38	75.74	-20.8%	75.20	-19.0%
FLORES-200 devtest, 19 extra languages, sentences	19,226	93.94	98.60	-76.8%	98.60	-76.8%
FLORES-200 devtest, 19 extra languages, first 20 code points	19,226	72.77	82.80	-36.9%	82.49	-35.7%
FLORES-200 devtest, 19 extra languages, two-word starts	15,790	58.24	70.39	-29.1%	69.92	-28.0%
Tatoeba test split, sentences	64,334	86.32	89.96	-26.6%	89.71	-24.7%
Tatoeba test split, first 20 code points	64,334	77.38	84.02	-29.4%	83.59	-27.5%
Tatoeba test split, two-word starts	52,438	61.56	71.47	-25.8%	70.87	-24.2%
NTREX-128, sentences	90,999	95.23	96.98	-36.7%	96.82	-33.4%
NTREX-128, first 20 code points	90,999	74.61	80.10	-21.6%	79.20	-18.1%
NTREX-128, two-word starts	79,653	62.71	68.90	-16.6%	68.00	-14.2%
Romanized test text (synthetic), sentences	6,045	78.66	98.00	-90.6%	97.88	-90.1%
Romanized test text (synthetic), first 20 code points	6,045	50.22	74.97	-49.7%	74.81	-49.4%
Romanized test text (synthetic), two-word starts	5,205	31.80	56.77	-36.6%	56.31	-35.9%

Two-word start errors by cause

The remaining wrong two-word starts of the DNO-PQ packs without letter boost, each filed under its first matching cause by scripts/sentence_starts.mjs (which picks its own example sentence for a few duplicate starts, so its totals differ from the tables above by at most 2 starts).

Cause	major_20	full
shared: the two words also start text of the predicted language	811	2,999
words: each word occurs in the predicted language's text	604	3,088
alphabet: the predicted language's letters fit worse than the true one's	408	2,681
compression: Google's original weights answer correctly	793	5,260
unseen: the predicted language has no benchmark text	0	8,316
model: none of the above	2,678	18,064
Errors	5,294	40,408
Wrong answers flagged reliable	2,116	7,267

Per-language losses against Google's original

Measured on the answers without letter boost, the only per-language figures recorded.

major_20: no language loses more than 0.9 points on any slice (Japanese on NTREX two-word starts; Italian 0.7 on FLORES first 20 code points).

full: most languages gain, some lose. The losses of more than 1 point:

• Serbian: −9.6 points on NTREX sentences, mostly from the 353 of 1,000 NTREX Serbian sentences that put Latin look-alike letters inside Cyrillic words; on ordinary text −2.8 on Tatoeba sentences, −3.8 on FLORES first 20 code points and −4.7 to −6.9 on two-word starts.
• Croatian: −3.8 on FLORES sentences, −1.8 on Tatoeba, −4.0 on NTREX sentences, −3.8 and −3.7 on FLORES and NTREX first 20 code points, −1.8 to −7.6 on two-word starts (CLD3 confuses it with Bosnian and Serbian; Google's original gets 57.5% of its FLORES sentences).
• Malay: −5.7 on FLORES sentences, −2.4 on MASSIVE, −7.3 on Tatoeba, −4.7 on NTREX, −1.1 to −6.2 on first 20 code points and two-word starts (CLD3 confuses it with Indonesian).
• Norwegian: −2.7 on MASSIVE, −3.3 on FLORES devtest, −6.6 on Tatoeba, −6.1 on NTREX (to Danish, which gains).
• Slovenian: −2.8 on FLORES first 20 code points, −1.2 on NTREX sentences, −7.0 and −6.0 on NTREX first 20 code points and two-word starts.
• Marathi: −3.2 on benchmark and NTREX two-word starts, −2.5 on NTREX first 20 code points.
• Zulu: −1.6 on FLORES devtest, −2.6 on NTREX (CLD3 confuses it with Xhosa).
• Hmong: −4.2 on NTREX first 20 code points.
• 1 to 3 points: Ukrainian (−2.3 on Tatoeba, −1.1 on its first 20 code points, −2.2 on NTREX two-word starts), Dutch (−1.3 and −3.0 on benchmark and NTREX two-word starts), Javanese and Spanish (first 20 code points), Tagalog (NTREX), Bulgarian (NTREX two-word starts), Hawaiian and Azerbaijani (Tatoeba first 20 code points).
• Small Tatoeba test sets: Nyanja (10 sentences), Shona (11), Corsican (13) and Zulu (16) lose one to three sentences or starts each, 6 to 30 points.

How the packs were made

• Changed: all 39,629 hidden- and softmax-layer parameters of full (21,028 of major_20), 18% and 11% of the network's float parameters. The 178,520 embedding values are unchanged. The hidden layer moved far from Google's weights (272% relative L2 change in full, measured against the int8 copy the edit started from), so this is a refit of the two dense layers on frozen embeddings, not a sparse patch.
• Fit data: FLORES-200 dev, MASSIVE 1.1 train and dev, Tatoeba fit and selection splits for every CLD3 label Tatoeba has, and synthetic romanized text. Each sentence also enters as its first 1, 2, 3 and 5 words and its first 10, 20 and 40 code points.
• Constraints: L2 pull towards the starting weights (λ = 1e-6), long texts the starting pack gets right anchored half to its own answer, confusable pairs (hr/bs, da/no, id/ms, xh/zu) keeping the starting pack's split, and a per-language guard with a paired significance test on the selection split; a pack is written only when no language loses there.
• Quantization: both layers stored as int8 with one scale per column, the softmax refitted on the rounded hidden layer's activations.

Limitations

• The losses listed above are real trade-offs; evaluate on your own distribution.
• The romanized test text used to align the direct neural edits was synthetically generated; Google's float32 CLD3 gives it the intended label for 57.6% (zh-Latn) to 98.2% (el-Latn) of sentences.
• Several Tatoeba test sets are small (24 labels under 100 sentences).
• Tatoeba publishes unversioned weekly exports; a rebuild reproduces these packs only from the exports downloaded.
• No unsupported-language rejection and no code-switching accuracy is claimed.
• CLD3 packs rebuild the native WASM layout at load: they reduce download size, not inference RAM.

Licenses and attribution

Apache-2.0, like Google CLD3 (Copyright 2016 Google Inc.); see NOTICE.md. Alignment data, not redistributed: FLORES-200 dev (CC BY-SA 4.0), MASSIVE 1.1 train and dev (CC BY 4.0), Tatoeba (CC BY 2.0 FR). Test data only: NTREX-128 (CC BY-SA 4.0), FLORES-200 devtest, MASSIVE test.