clef-flash-NVFP4-Hybrid

Unofficial mixed-precision quantization of Cloudflare/clef-flash, a 9B decision model that scores every option of every typed question (choice / score / true-false) in one forward pass. Not affiliated with or endorsed by Cloudflare.

Layers 0-16 are FP8, layers 17-31 are NVFP4. A layer-by-layer scan (below) showed that all of the damage from 4-bit quantization sits in the first half of the network, so this layout keeps the sensitive half in 8 bits and takes the 4-bit speed-up where it is free. It agrees with the BF16 original on about 98% of answers.

The two big lookup tables (token embeddings and lm_head) are stored as 8-bit with one scale per row, which takes the download from 10.7 GB to 8.7 GB and frees about 2.4 GiB of GPU memory, at no measurable cost in accuracy or speed.

It ships with the same vLLM plugin and Jev/SystemOne-compatible HTTP server (POST /v1/systemone) as kurcontko/clef-flash-NVFP4.

Updated 2026-10-07: the weight file and the plugin changed together. The token-embedding and lm_head tables moved out of model.safetensors into clef_head/tables_int8.safetensors (8-bit), which made the download 2 GB smaller with the same accuracy and speed. If you downloaded this repo before that date:

• Re-download the whole repo and reinstall the plugin (pip install ./<repo>/vllm_plugin, version 0.2.0 or later). The new weights only work with the new plugin.
• Never pair these weights with an older plugin. The checkpoint now carries a version marker, so an older plugin stops at startup with TypeError: ... unexpected keyword argument 'requires_clef_vllm'. For a few hours on 2026-10-07 the marker was missing: in that state an older plugin starts normally and returns wrong answers, because the two tables stay uninitialized. If you pulled only the weights that day, pull the repo again.
• Your old download keeps working as it is, as long as you keep its weights and its plugin together.

Read before using

• Text and JSON state only. The vision tower is kept in BF16 but has not been tested; the server rejects images / videos with HTTP 400. If you need images, alpha-x-ai/clef-flash-NVFP4 supports them.
• Use the bundled vLLM plugin, pinned to vLLM 0.28.x. The checkpoint does not run efficiently in plain transformers.
• Hardware: NVIDIA Blackwell (the NVFP4 layers need FP4 tensor cores). Tested on one RTX 5070 Ti 16 GB (SM 12.0) only.
• Needs the bundled plugin to load at all: the lookup tables are in this repo's own 8-bit format (clef_head/tables_int8.safetensors), so model.safetensors alone is not a complete checkpoint.
• Evaluated on 1,400 records from 7 public tasks with our own harness, not on Cloudflare's Decision Index.

Which build to pick

All rows measured the same way: this repo's server with --gdn-backend flashinfer on one RTX 5070 Ti, the same 1,400 records, 64 concurrent requests. Agree is the share of questions with the same top answer as BF16; KL is the mean KL divergence from BF16 per question (lower is closer).

Build	Layout	Agree	KL	Throughput	Size
kurcontko/clef-flash-NVFP4	NVFP4 everywhere	94.4%	0.024	22.4k tok/s	9.1 GB
alpha-x-ai/clef-flash-NVFP4	their mixed NVFP4/FP8 layout	97.8%	0.0048	15.4k tok/s	10.3 GB
this repo	layers 0-16 FP8, 17-31 NVFP4	98.1-98.6%	0.0023-0.0027	14.1k tok/s	8.7 GB
kurcontko/clef-flash-FP8-Dynamic	FP8 everywhere	98.8%	0.002	~10k tok/s	12.2 GB

Single-request latency is the same for every build (about 32-46 ms for 240-300 token requests, depending on the card), because at batch size 1 it is set by per-layer overhead, not by arithmetic. Pick by throughput and fidelity: the all-NVFP4 build is the fastest, this one is the closest to BF16 among the NVFP4 builds, and alpha-x-ai's sits between them and also handles images. Their checkpoint ran unmodified through this repo's server; the difference in agreement between it and this build is within run-to-run noise, the difference in KL is not. The FP8 throughput was measured earlier without --gdn-backend flashinfer. This repo's throughput was measured before its tables were made 8-bit; a back-to-back test on the machine's second, slower 5070 Ti gave the same throughput for both versions (12.5k tok/s each), so the row stands. Sizes are the weight files in each repo.

Quickstart: SystemOne API server

hf download kurcontko/clef-flash-NVFP4-Hybrid --local-dir clef-flash-NVFP4-Hybrid

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD/clef-flash-NVFP4-Hybrid:/model:ro \
  --entrypoint bash vllm/vllm-openai:v0.28.0 -c \
  "cp -r /model/vllm_plugin /tmp/p && pip install -q /tmp/p && clef-systemone --model /model --dp 1"

On RTX 50-series add --gdn-backend flashinfer (about 10% faster; experimental, see the clef-flash-NVFP4 card). The server warms up before /health reports ready.

curl localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "clef-flash",
  "state": "Our checkout started returning errors and orders are blocked.",
  "questions": {
    "department": {"type": "choice", "instructions": "Which team should handle the message?",
                   "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}},
    "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},
    "outage": {"type": "noul", "instructions": "Is a service down?"}}}'

Request and response bodies match the systemone() function of the original release. The offline Python API is the same as in the clef-flash-NVFP4 card.

A clean install from this repo with the command above started in about 4 minutes, left 3.05 GiB for KV cache on a 16 GB card, and measured 98.3% agreement with BF16 (KL 0.0026) at 11.8k tok/s with default settings, or 98.6% (KL 0.0023) at 12.5k tok/s with --gdn-backend flashinfer (both on the slower of the two test cards).

Results

Accuracy and agreement with BF16, per task

1,400 records, 200 per task, from held-out splits. Acc is against gold labels. BF16 is the original release in transformers 5.10.2; this model was measured in vLLM with real FP8/FP4 kernels (offline run).

Task	BF16 acc	This model acc	Agree	KL
BANKING77 (77-way intent)	96.5	97.0	99.5	0.0008
ARC-Challenge	99.5	99.5	100.0	0.0006
MMLU	94.5	94.0	99.5	0.0026
HellaSwag	99.0	99.0	100.0	0.0006
ANLI R3	50.0	49.0	95.0	0.0043
BoolQ	87.5	87.5	99.5	0.0029
Multi-question ANLI R1	73.0	74.0	97.3	0.0038
All	84.1	84.2	98.4	0.0026

Across four runs (offline and through the HTTP server, before and after the tables were made 8-bit) this model measured 98.1-98.7% agreement and KL 0.0023-0.0028; BF16 itself varies by about 0.2 points between runs. Accuracy differences under 1 point are within noise at 200 records per task.

Where 4-bit quantization hurts: a layer-by-layer scan

For each layer, one group of projections at a time was switched to NVFP4 (weights and activations) while everything else stayed BF16, and the mean KL divergence from pure BF16 was measured on 190 eval records. The number is the damage that group alone does in 4 bits.

Layer	Attention type	Attention projections in NVFP4: KL	MLP in NVFP4: KL
0	linear attention	0.0015	0.0004
1	linear attention	0.0008	0.0005
2	linear attention	0.0020	0.0010
3	full attention	0.0010	0.0004
4	linear attention	0.0008	0.0008
5	linear attention	0.0004	0.0013
6	linear attention	0.0018	0.0010
7	full attention	0.0015	0.0009
8	linear attention	0.0009	0.0010
9	linear attention	0.0008	0.0009
10	linear attention	0.0007	0.0011
11	full attention	0.0021	0.0007
12	linear attention	0.0006	0.0007
13	linear attention	0.0010	0.0006
14	linear attention	0.0004	0.0004
15	full attention	0.0010	0.0003
16	linear attention	0.0002	0.0002
17-31	both	at most 0.0001 each	at most 0.0001 each

Layers 0-16 account for 97% of the summed damage (0.0300 of 0.0309); layers 17-31 together add 0.0009. The single-group numbers add up to about what the all-NVFP4 build measures in practice (KL 0.024 on the full set), so the effects are close to additive. scripts/sensitivity_scan.py reproduces the scan and scripts/sensitivity_scan.json holds every number.

Quantization details

• Tool: llm-compressor 0.14.0 (compressed-tensors 0.19.0), one QuantizationModifier with two config groups; run on a DGX Spark. scripts/quantize_mixed.py --split 17 reproduces it.
• Layers 0-16: FP8 (E4M3) W8A8, per-channel weight scales, per-token dynamic activation scales.
• Layers 17-31: NVFP4 W4A4, 16-element groups with FP8 group scales and one FP32 global scale per tensor; activation global scales calibrated on 672 records (96 from the train split of each eval task).
• Covered projections in every layer: MLP gate/up/down_proj, full-attention q/k/v/o_proj, linear-attention in_proj_qkv / in_proj_z / out_proj.
• Lookup tables in 8-bit: Clef never multiplies by its token-embedding or lm_head matrices: embeddings are looked up by token id, and the joint head only reads the lm_head rows of each request's option tokens. Both are stored as int8 with one scale per row (scripts/shrink_tables.py): relative error 0.9% and 1.1%, mean row cosine 0.99996 and 0.99994. A back-to-back test against the BF16-table version showed the same agreement, latency and throughput, with KV cache room on a 16 GB card rising from 0.62 to 3.07 GiB.
• Kept in BF16: the vision tower, the Gated DeltaNet gate projections in_proj_a / in_proj_b, norms, and the joint schema head.
• Fused scale: in_proj_qkv and in_proj_z share one NVFP4 global scale, as vLLM's fused GEMM requires.
• A pitfall with mixed layouts: vLLM matches the config's layer patterns against its own module names (...model.layers.N...), not the Hugging Face ones, so patterns must not anchor on language_model.layers. A pattern that fails to match leaves the layer unquantized in BF16 and the model no longer fits a 16 GB card.

Files

Path	Purpose
model.safetensors, config.json, recipe.yaml	Quantized backbone without the two lookup tables (6.4 GB, compressed-tensors mixed-precision)
clef_head/tables_int8.safetensors	Token embeddings and lm_head as int8 + per-row scale (2.0 GB)
clef_head/joint_head*	Joint schema head, BF16; weights unchanged from the original release, config has one added key (requires_clef_vllm)
joint_schema_model.py	Original release code (unchanged)
tokenizer*.json, chat_template.jinja, processor_config.json, generation_config.json	Tokenizer and processor (unchanged)
vllm_plugin/	vLLM plugin and clef-systemone server
scripts/	build_data.py, quantize_mixed.py, shrink_tables.py, sensitivity_scan.py + its results, clef_load.py

License and attribution

Apache-2.0, following Cloudflare/clef-flash (post-trained from Qwen/Qwen3.5-9B). All credit for the model goes to its original authors; this repository only adds the quantization, the vLLM plugin and the server. The idea of mixing FP8 and NVFP4 by layer for this model was published first by alpha-x-ai; the layout here comes from our own scan. Not an official Cloudflare release.