isaranto/clef-flash-FP8-block

clef-flash FP8 (block)

This is an FP8 version of Cloudflare's clef-flash decision model. The Wikimedia Foundation Machine Learning team made it to serve clef-flash on a smaller GPU, a single 24GB partition. It is the same model: only the large layers of the language model are in FP8, and everything else is bit-identical to Cloudflare's release.

For what clef-flash does, its API, benchmarks and limitations, see the original model card.

What changed

• FP8 layers. The MLP, full-attention and large linear-attention projections of the language model (200 weights) are FP8 (e4m3), with one scale for each 128×128 block. Activations are quantized to FP8 at run time, in groups of 128.
• Unchanged layers. Everything else is bit-identical to Cloudflare's files: the vision encoder, the embeddings, lm_head, the norms, the small in_proj_a/in_proj_b layers, the Clef joint schema head (joint_head.safetensors) and Cloudflare's joint_schema_model.py. These are the same layers that Qwen keeps in BF16 in its own FP8 releases.
• config.json. It is Cloudflare's file with a quantization_config added.
• Size. The weights go from 17.5 GiB to 11.1 GiB on disk. In vLLM, the backbone takes 9.68 GiB instead of 15.77 GiB (measured on an AMD MI210).

How it was made

• Source. Cloudflare/clef-flash at revision 41318bf2106d135c7a92da745fec4b255279b4de. Before quantizing, every file was checked against the SHA-256 that Hugging Face publishes for that revision.
• Tool. llm-compressor 0.13.0, scheme FP8_BLOCK (compressed-tensors 0.18.0, transformers 5.12.1, torch 2.10.0).
• No calibration data. The result depends only on Cloudflare's weights. The same files and versions give the same bytes, so anyone can reproduce it and compare. MANIFEST.json lists the versions and the SHA-256 of every file.
• Checked after quantizing. Every tensor was read back and compared with Cloudflare's. The 200 FP8 weights have a mean relative error of 2.66%, and all other tensors are bit-identical.
• One deviation from a default llm-compressor run. For calibration, llm-compressor converts Qwen's norm weights to 1 + w and back in BF16, which rounds them. FP8_BLOCK without calibration data does not use the norms, so we skipped that step. The norms here are exactly Cloudflare's.

The scripts that made this copy, and the comparison with BF16, are in scripts/. You can use them to reproduce the weights and compare the result with ours.

Results compared to BF16

We sent the same 59 requests (54 text in 5 languages and 5 Wikimedia Commons images, 123 questions in total) to the BF16 and the FP8 model, one at a time. Both ran in our model server with vLLM 0.22.1 on an AMD MI210. The full report is in scripts/results/parity-mi210.md.

	Text (108 questions)	Images (15 questions)
Same answer (same choice, same yes/no at 0.5, same most likely score)	108/108	15/15
Mean change in an option's probability	0.009	0.006
99th percentile of the change	0.062	0.022
Largest change	0.087	0.029

No request failed, and the token counts were the same. The largest changes are on answers that the model is unsure about, mostly choices between three options. There, FP8 moves some probability from one option to another without changing the most likely option. For example, for one choice question BF16 gave 0.49 / 0.27 / 0.24, and FP8 gave 0.41 / 0.26 / 0.33. If you use the probabilities with a threshold, check the effect on your own data. We have not yet compared the accuracy of the two models on labelled data.

Memory in vLLM (MI210, gpu_memory_utilization=0.5)	BF16	FP8
Weights	15.77 GiB	9.68 GiB
Free for the KV cache	13.28 GiB	19.19 GiB

The MI210 has no FP8 hardware, so vLLM emulates FP8 there. Results on an MI300X, which has FP8 hardware, can be slightly different.

How to use it

As with the original, this is a Qwen3.5 backbone plus Cloudflare's joint schema head. vLLM can serve the backbone but not the head. We serve it like this:

• Backbone in vLLM 0.22.1. It runs as a pooling model (runner="pooling", convert="embed", task token_embed) and returns the last hidden state of every token. vLLM reads the FP8 format from config.json, so the settings are the same as for the BF16 model.
• Head in PyTorch. The joint schema head in joint_schema_model.py runs on those hidden states and returns a probability for every option of every question.
• Same API. Requests and responses follow Cloudflare's systemone format.

Our model server is in the Lift Wing inference-services repository, under src/models/decision_models/clef/.

Tested on: vLLM 0.22.1 with ROCm on an AMD MI210. The MI210 has no FP8 hardware, so vLLM emulates FP8 there. We have not yet tested it on an MI300X or on a 24GB partition. We have not tested loading this model with transformers or SGLang.

License

Apache-2.0, the same as the original model. QUANTIZATION.md lists the files that were changed. Please credit and cite Cloudflare's clef-flash.

Questions and issues: T440223 on Wikimedia Phabricator.