harrier-0.6b query clients: 192–235 MiB, 94–99 % of fp32 on the unchanged index — summary and links

#1
by honza-rosecky - opened
thinletter org

Three GGUF query encoders for microsoft/harrier-oss-v1-0.6b (1 143 MiB fp16) that keep the document index unchanged. GPTQ with activation order rounded directly onto llama.cpp's K-quant grids: the Q3_K file (235 MiB, generic English calibration) keeps 98.9–101 % of nDCG@10 on SciFact / NFCorpus / ArguAna / SciDocs; the Q2_K files (192 MiB) keep 98 % on SciFact and 93–94 % on SciDocs, where calibration on synthetic queries over the corpus adds +0.01–0.02 over generic text. They run unchanged in llama.cpp and in the browser (wllama, WebGPU or WASM).

Along the way: at 3.4 bits the calibration text hardly matters and llama-quantize --imatrix reaches the same quality with the same budget, so the defensible value is the verification recipe, not the quantizer; at 2.6 bits the language of the calibration text matters before its domain (Czech vs English generic text: +0.09 nDCG on a Czech index with another model). Negative results are in the report with the same care as the positive ones.

Limits: 0.6B models only, four English corpora plus one Czech case study; read differences under 0.01 nDCG@10 as ties. Questions and attacks welcome here.

Sign up or log in to comment