Fidelity measurement: KL(reference || this quant) on frozen tokens, receipt-backed
What this is. A third-party fidelity measurement of drowzeys/keys-GLM-5.3-EXL3 @ ebf3c8bb0ed869b8f96a6ade9c8d365a49bdbad5 (exl3-trellis, 3 bits per weight as declared) against its unquantized reference, made with quant-fidelity-suite. Measured by malaiwah, not by the model's author. Every number below is read from a sealed receipt named at the bottom.
| KL(reference ‖ candidate), mean tokenwise, nats | 0.10233258694757999 |
| top-1 agreement | 0.9112652662432829 |
| KL median / p95 / p99 / max | 0.007512645535868924 / 0.47124999949160773 / 1.6232298221454846 / 10.579389872756519 |
| reference root dataset | malaiwah/glm53-fidelity-root-v1 @ 9c4a29ee10f393ed2fdbdb9262c1192ddb1507b4 (capture 9eba97dddb4ff2e2…) |
| panel | panel--glm53.malaiwah.corpus5x5-v1, 25 contexts, 51175 scored positions |
| direction / vocabulary / accumulation | KL(reference |
| method | decode-and-run, weights only: exl3-trellis-decode-to-bf16 -- each exl3 payload group (trellis/suh/svh/codebook marker) decoded to bf16 per module on the capture device via exllamav3's transcribed codebooks (codebook mul1 as declared -- the capture reads each module's own marker -- at 3 declared bits), non-routed tensors carried as shipped; same engine, schedule and device as the reference capture |
| determinism | two fresh processes captured the candidate; both sealed captures carry content digest 650ae59e576d146f… (self-comparison 0.0) |
| comparability class | strict |
Scope (scope_digest): attn.o=native:fp16@16|attn.other=quantized:mixed|attn.qkv=native:fp16@16|embed_tokens=native:bf16@16|lm_head=native:fp16@16|mlp.down=native:fp16@16|mlp.gate=native:fp16@16|mlp.up=native:fp16@16|moe.experts=quantized:exl3-mcg@3|moe.experts=quantized:exl3-mul1@3|moe.router=quantized:mixed|moe.shared_expert=native:fp16@16|mtp=quantized:mixed|norm=native:bf16@16|head=native|kv=bf16
Disclosures on the comparison receipt:
native_head_replay(info): HEAD-1d: each side replayed through its own sealed head (reference 864f488a0074, candidate 850d8eb118ae); head error is inside the measurement, as under HEAD-2, and nothing is substituted. The heads differ in content.
What this number does not do. It is a same-lane distance from one reference capture on one panel. It does not rank this artifact against numbers measured on another panel, lane or reference, and a same-lane root does not retroactively upgrade rows measured against another teacher. Per-window scatter exceeds the gap between adjacent bit-widths; compare only within a group whose comparability keys match.
Receipts.
- comparison receipt
receipt_sha256ee284e4a09c173b721ca728d5474fa989b1648d1728ef122eb27fbf5339c025e - root-qualification receipt
receipt_sha256c4c6dfa089269fd261ec74831eafeb4d2fc5973d80703ba12d8d74031b5b7be4(canonical dataset_sha2568c1f186a670d68526128958eecf33905da494aa40f0e9e0d1e84ec8648ee178b, repeatfe50a0937208e5d9b7f73d3e3c5f6e2dbd258c2154c536d36b61d29af2daaaae) - job
job_id_full7df6722ceca0322232fa75f1690cfe0fa2ddda6b496054d0751bafd6ee545490 - candidate capture dataset published:
malaiwah/glm53-fidelity-exl3-drowzeys-v1@6d9256e5b0798a0115e0f1e164f0cd3deaf90a15
Reproduce: fetch the two datasets named above and run fidelity-dataset compare --reference <root> --candidate <this> --own-heads; the receipt's estimator block is the exact recipe. Questions and corrections are welcome here; the registry files this as a third-party row (measured_by enumerated, never conflated with author-reported numbers).
-Mean KL 0.10 nats, top-1 agreement 91% is the Q3 class. very nice since this is 3.0bpws (Q3). retaining 91% of the full model. could go 3.5 bpws or 4 bit.. to get it to 95-97% for kLD of 0.02-0.05. but i want the speed and larger context on dgx spark and this is the lowest quant anyone should do as more aggressive quant will drop the agreement to below 91%.
thank you for doing independent testing for this.
Thank you!