bge-m3 query-side clients at 3–4 bits, calibrated on Czech text — summary and links
Two GGUF query encoders for bge-m3 dense retrieval (355 and 321 MiB instead of 1 104 MiB fp16), compatible with document vectors from the unchanged model (CLS pooling, 1024-d). On the model's own fp32 index of 55 071 Czech supreme-court segments the Q4_K_M file keeps 99.2 % of nDCG@10 (cosine 0.990, top-10 overlap 0.864) and the Q3_K file 98.3 % (cosine 0.970); the same Czech-calibrated files keep 99.6 / 100.0 % on SciFact, so the calibration language does not cost English.
Along the way: of the four families we measured on this index, the CLS-pooled XLM-R encoder tolerates quantization best (bge-m3 ≥ jina-v5-small > Qwen3-Embedding-0.6B, whose 3-bit files fail); the 250 002 × 1024 token table is half the file and its 4-bit version costs ≤ 0.2 points here, against 1.0 on Qwen3-Embedding. llama-imatrix refuses encoders that append EOS; the recipe on the card shows the one-line workaround.
- verify against your own index: https://github.com/rosecky/embedding-quantization-public
- report (§3.2, Czech case study): https://github.com/rosecky/embedding-quantization-public/blob/main/docs/release/technical_report.md
- the tables: https://github.com/rosecky/embedding-quantization-public/blob/main/results/tables/release2.md
Limits: synthetic Czech queries (doc2query, relevant = source segment), one English corpus; nothing here is a claim about the base model's quality. Questions and attacks welcome here.