F2LLM-v2-1.7B ONNX, 4/8-bit quantized

Quantized ONNX versions of codefuse-ai/F2LLM-v2-1.7B for ONNX Runtime. This is a quantized derivative: the model, its training and its licence (Apache-2.0) belong to its original authors. All 17 variants we built are here, including those below our quality gate, with their measurements.

Recommended: mix50-gptq-sq50-a4, 1,629 MB instead of 6,885 MB, 1,788 MB RAM instead of 5,603 MB; nDCG@10 vs FP32 +0.19 [-0.44, +0.85] points on FreshRet Small, -0.38 [-1.22, +0.28] on MIRACL.

Usage

pip install "onnxruntime>=1.31" tokenizers numpy huggingface_hub
hf download nynefold/F2LLM-v2-1.7B-ONNX inference.py --local-dir .

from inference import load

model = load("nynefold/F2LLM-v2-1.7B-ONNX")  # recommended variant; load("nynefold/F2LLM-v2-1.7B-ONNX", variant="...") for another
q = model.embed_queries(["How do I reset the router?"])
d = model.embed_documents(["Hold the reset button for ten seconds.", "The router has four LAN ports."])
print(q @ d.T)  # cosine similarities (vectors are L2-normalized)

inference.py (in this repo) needs only ONNX Runtime, tokenizers and NumPy. Given a repo id, it downloads the variant as plain files into ~/.cache/nynefold/<repo id> (set NYNEFOLD_CACHE to change the folder); ONNX Runtime does not load model.onnx.data through the Hub cache's symlinks. load("path/to/folder") uses a local copy. To run the graphs another way:

Inputs input_ids and attention_mask, int64 [batch, seq], right-padded.
Query prefix "Instruct: Given a question, retrieve passages that can help answer the question.\nQuery: ", no document prefix. At most 512 tokens (the final end token is kept).
Last-token pooling is inside the graph; output embedding [batch, dim].
The output is L2-normalized; compare by dot product.
config.json holds all of this in machine-readable form.

Variants

Each folder holds model.onnx + model.onnx.data and the recipe.json it was built from. Quality deltas are paired against FP32 in nDCG@10 points with 95% bootstrap intervals.

variant	size MB	RAM MB	stage A cosine mean (gate 0.99)	FreshRet Small nDCG@10 vs FP32	MIRACL nDCG@10 vs FP32	status
FP32 (not included)	6,885	5,603	1	reference	reference
`e8-w4-gptq-a0`	1,266	1,111	0.9540			below quality gate
`e8-w4-gptq-a4`	1,266	1,442	0.9370			below quality gate
`e8-w4-gptq-gu0`	1,266	1,276	0.9508			below quality gate
`e8-w4-gptq-sq50-a4`	1,266	1,442	0.9490			below quality gate
`mix02-gptq-a4`	1,280	1,455	0.9471			below quality gate
`mix10-gptq-gu0`	1,338	1,392	0.9808			below quality gate
`mix30-gptq-gu0`	1,483	1,530	0.9872			below quality gate
`mix30-gptq-sq50-a4`	1,483	1,649	0.9895			below quality gate
`mix50-gptq-gu0`	1,629	1,670	0.9943			passes stage A only
`mix50-gptq-sq50-a4`	1,629	1,788	0.9922	+0.19 [-0.44, +0.85]	-0.38 [-1.22, +0.28]	recommended
`e8-w8-rtn-a0`	1,993	1,863	0.9953	+0.33 [-0.01, +0.74]		passes
`e8-w8-rtn-a4`	1,993	2,135	0.9772			below quality gate
`w8-qkv0`	1,993	2,097	0.9780			below quality gate
`w8-qkvgu0`	1,993	1,974	0.9929			passes stage A only
`w8-sq50-a4`	1,993	2,135	0.9919			passes stage A only
`w8-sq75-a4`	1,993	2,135	0.9924			passes stage A only
`e32-w8-rtn-a4`	2,877	2,116	0.9772			below quality gate

Recommended variant chosen on FreshRet Small quality; MIRACL is the held-out check (120 dev queries in en, es, fr, de, zh, ar from mteb/MIRACLReranking, MTEB's reranking version of MIRACL, with BM25 top-20 pools). Calibration and stage A use other, disjoint MIRACL dev queries.
"Below quality gate": stage A fidelity under the gate, or more than 1.0 nDCG@10 point lost on FreshRet Small. "Passes stage A only": not measured on FreshRet Small (finalists only), so its retrieval quality is unconfirmed.
RAM: peak process memory while embedding one 256-token passage, in a process with only ONNX Runtime, tokenizers and NumPy loaded.
a4 variants use INT8 compute (accuracy_level 4) and are meant for CPUs with fast INT8 instructions; a0 variants compute in FP32 from the quantized weights.
Tested with ONNX Runtime 1.31. Variant names are explained in nynefold/retrieval-quant.

How they were made

Linear layers use ONNX Runtime's MatMulNBits and the token table GatherBlockQuantized, with bits, algorithm and compute mode set per tensor (GPTQ, RTN, clip search; mixed 4/8-bit from a sensitivity scan; SmoothQuant where noted). The method, gates, all results and the pipeline comparison with the other models are in nynefold/retrieval-quant. The evaluation set is nynefold/FreshRet-Small, with mteb/MIRACLReranking for calibration, fidelity and the held-out check; all six repos are in the collection retrieval-quant: quantized ONNX retrievers.