Qwen3.5-4B-nvfp4

NVFP4 (W4A4) quantization of Qwen/Qwen3.5-4B, produced with llm-compressor. lm_head is kept in the original precision.

Quality

KL divergence of the quantized model's next-token distribution vs the bf16 baseline (128 samples/bucket, seq len 2048). prose is WikiText-2; agentic is tool-calling turns (schema + query + call) rendered with the chat template.

bucket	tokens	mean KL	top-1 agreement
prose	24,130	0.1049	83.59%
agentic	77,522	1.3375	77.06%
overall	101,652	1.0449	78.61%

4-bit weights and activations trade noticeable accuracy for size. For a near-lossless option use Qwen3.5-4B-fp8.

Usage

vLLM:

vllm serve mahadev9/Qwen3.5-4B-nvfp4

transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mahadev9/Qwen3.5-4B-nvfp4", torch_dtype="auto", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("mahadev9/Qwen3.5-4B-nvfp4")