clef-flash-4bit

A pre-quantized 4-bit copy of Cloudflare/clef-flash, a 9B System One decision model (Qwen3.5-9B backbone + joint schema head) with a Jev/SystemOne-compatible interface. It exists so that GPUs without bf16 and with little memory (for example a Colab T4) can download about a third of the original size and load the model without quantizing it on the fly.

• Backbone: quantized with bitsandbytes NF4, compute dtype float16 (the quantization settings are stored in config.json).
• Joint schema head, loader code (joint_schema_model.py), processor and tokenizer: copied unchanged from the original repository.
• License: Apache-2.0, like the original. All credit for the model goes to Cloudflare; see the original model card for its description, evaluations and limitations.

Usage

Exactly as the original, without passing a quantization config:

import sys, torch
from huggingface_hub import snapshot_download

path = snapshot_download("irodkin/clef-flash-4bit")
sys.path.insert(0, path)
from joint_schema_model import load_release_model, systemone

model, processor = load_release_model(path, device="cuda", dtype=torch.float16)
response = systemone(model, processor, {
    "model": "clef-flash",
    "state": "Customer says the parcel arrived damaged and wants their money back.",
    "questions": {"refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"}},
})
print(response["answers"])

Needs transformers ≥ 5.10, bitsandbytes, torchvision and pillow.

Check

Against quantizing the original on the fly with the same settings, the pre-quantized checkpoint gives identical decisions (maximum probability difference 0.0000 on 16 noul/choice questions) and loads with 7.9 GiB of GPU memory. Download size: about 8.5 GB instead of 19 GB (the embeddings and the vision encoder stay in 16-bit). Used as the retrieval controller in a course lab, it matched on-the-fly 4-bit quantization exactly (accuracy 1.00 / 1.00 / 0.87 on single-hop / 2-hop / 3-hop questions).