DeepSeek-R1-Distill-Qwen-7B-NVFP4A16

NVFP4A16 quantization of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, produced with llm-compressor and published only after passing an automated accuracy gate against the bf16 baseline (see Certification).

Model overview

• Scheme: NVFP4A16 - 4-bit NVFP4 weights, 16-bit (bf16) activations (weight-only).
• Why weight-only: activations are transient (never stored), so keeping them at 16-bit costs ~nothing on disk - weights dominate size - while avoiding the token-level quality loss that 4-bit activation quantization (W4A4) causes. You get almost all the compression with accuracy much closer to the original.
• Format: compressed-tensors; loads natively in vLLM (no --quantization flag).

Use with vLLM

from vllm import LLM, SamplingParams

llm = LLM(model="uist-labs/DeepSeek-R1-Distill-Qwen-7B-NVFP4A16")
prompts = ["Give me a short introduction to large language models."]
params = SamplingParams(temperature=0.7, top_p=0.8, max_tokens=256)
for out in llm.generate(prompts, params):
    print(out.outputs[0].text)

Hardware requirements

This checkpoint uses weight-only NVFP4: it runs via vLLM's FP4 Marlin kernel and does not require Blackwell GPUs or native FP4 tensor cores. On non-Blackwell hardware the 4-bit weights are dequantized to 16-bit for the GEMM, trading some compute throughput for the smaller memory footprint (the KV cache and activations are 16-bit either way, so the saving is on the linear weights).

Validated (measured through assay's gate): Ada (sm_89: e.g. L4, RTX 4090). Other >= sm_80 (Ampere / Hopper) GPUs are expected to work by the same weight-only Marlin path but are not yet independently validated here. Turing (sm_75) is excluded: a known vLLM issue makes the NVFP4 Marlin GEMM emit incorrect output on Turing, so we do not claim it until that is fixed and measured.

Creation

• Tool: llm-compressor, scheme NVFP4A16, targets Linear, ignore lm_head.
• Calibration: 512 samples from HuggingFaceH4/ultrachat_200k at 2048-token sequences.
• Pipeline: UIST Labs' assay benchmark-gated quantization pipeline - quantize -> benchmark -> gate -> publish.

Evaluation

Measured with lm-evaluation-harness (vLLM backend) on the bf16 baseline and this checkpoint. retention is quantized / baseline (higher is better; for perplexity, lower raw value is better).

task	metric	baseline	quantized	delta	+/-stderr	retention
aime24_avg	exact_match,avg	0.5250	0.5229	-0.0021	0.0913	0.9960
aime25_avg	exact_match,avg	0.4083	0.3979	-0.0104	0.0992	0.9745
minerva_math500	math_verify,none	0.8360	0.8240	-0.0120	0.0238	0.9856
gpqa_diamond_cot_zeroshot	exact_match,flexible-extract	0.3182	0.3384	+0.0202	0.0473	1.0635
wikitext	word_perplexity	31.2854	31.4295	+0.1442	-	1.0046

Gate: PASS - no task regressed beyond k=2 combined stderr (* = statistically significant regression at the recipe's k) No task showed a statistically significant regression.

Methodology and limitations

• Apples-to-apples deltas. The bf16 baseline and this checkpoint were evaluated with the identical harness and settings, so the delta/retention columns are a fair like-for-like comparison - which is what a quantization gate should measure: change from the original, honestly.
• Chat-mode evaluation. Tasks are evaluated with the model's chat template applied. The template shifts the absolute scores on both the baseline and the quantized model - most visibly on the multiple-choice tasks - so read the deltas, not the absolute values. The comparison stays valid because both sides are evaluated with identical settings.
• Per-task statistical power varies. aime25_avg has the largest combined standard error (0.099) and is power-limited - read its delta as directional only. minerva_math500 (0.024) and perplexity carry the certification's binding, low-variance signal. The gate tests each task for a statistically significant regression, so an underpowered task cannot fail a sound quant on noise alone.
• Retention near or above 100% means "no measurable loss," not "better." Where a task ticks up, that is within benchmark noise (small benchmark sets vary run to run) plus a touch of quantization acting as mild regularization - read the whole table as "indistinguishable from the original," not as an improvement.
• Weight-only tradeoff. Weights are 4-bit; activations stay 16-bit. That keeps quality close to the original at nearly the full disk-size saving, at a modest inference-speed cost versus a fully 4-bit (W4A4) variant. If you need maximum throughput and can accept more degradation, a W4A4 build is a different point on that curve.
• Bias, risks, and inherited behavior. This is a quantization of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B and inherits its capabilities, biases, and limitations unchanged - quantization faithfully reproduces the base model's behavior (the gate above certifies exactly that), it does not add or remove bias. For intended use, safety, and ethical considerations, refer to the base model's card. Absolute-quality claims (multilingual, coding, safety) are the base model's; we certify only that quantization preserves them within the stated bar.

Certification

UIST Labs publishes a quantized checkpoint only if it clears a hard, stated accuracy bar against its own bf16 baseline - we would rather withhold a release than ship an unverified one. This checkpoint passed all of:

• No statistically significant per-task accuracy regression: one-sided, a task fails only if its drop exceeds k=2 times the combined standard error of the baseline and quantized scores
• Perplexity increase <= 3%

The deltas above are the actual measured numbers, not vendor estimates. This gate runs on every UIST Labs quantization release.

Citation

If you use this checkpoint, please cite both this quantized release and the base model.

@misc{uistlabs_deepseek_r1_distill_qwen_7b_nvfp4a16,
  title        = {DeepSeek-R1-Distill-Qwen-7B-NVFP4A16: benchmark-gated NVFP4 quantization of DeepSeek-R1-Distill-Qwen-7B},
  author       = {{UIST Labs}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/uist-labs/DeepSeek-R1-Distill-Qwen-7B-NVFP4A16}},
  note         = {Quantized and certified with the assay pipeline; published only after passing an automated accuracy gate against the bf16 baseline.}
}

Please also cite the base model, deepseek-ai/DeepSeek-R1-Distill-Qwen-7B.

Produced by assay v0.5.0. - UIST Labs