You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Checkpoint released to reproduce the QATFactory paper (arXiv 2609.39223). Base model deepseek-ai/DeepSeek-R1-Distill-Llama-8B (MIT; DeepSeek-R1-Distill-Llama-8B is derived from Llama-3.1-8B, under the Llama 3.1 Community License).
Log in or Sign Up to review the conditions and access this model content.
DeepSeek-R1-Distill-Llama-8B NVFP4 QAD (QATFactory)
DeepSeek-R1-Distill-Llama-8B quantized to NVFP4 with weights fake-quantized during training (W4A16), recovered with quantization-aware distillation (QAD) from the BF16 model on the model's own reasoning traces. Reproduces the DeepSeek-R1-Distill-Llama-8B NVFP4 "QAD (trained with W4A16)" row of the QATFactory paper's Table 2 with the released code, recipe and regenerated training corpus. Deployed at W4A4.
Serve with vLLM (compressed-tensors nvfp4-pack-quantized; native NVFP4 needs Blackwell GPUs):
vllm serve qatfactory/DeepSeek-R1-Distill-Llama-8B-NVFP4-QAD --max-model-len 65536
Training
- Recipe:
configs/llm/deepseek_r1_distill_llama_8b_nvfp4_qad.yamlat QATFactory commit4d43e52869716b3445dad106d8466f245eb32a66(main), run on 4 GPUs with--train.gradient_accumulation_steps 4(same global batch 16 as the recipe's 8 GPUs x 2),--model.attn_implementation sdpa, checkpoints every 512 steps; the last job resumed with the resume fixes now in QATFactory (2c6a70c). - Data:
qatfactory/openperfectblend-100k-deepseek-r1-distill-llama-8b-thinkat revision62a84d9dd1dbf6aca0c9db7ce4583d553fb7bc08(file sha256bb7b7e5be768511c94daf14d3b8739c4d1dd6457aa200aa663a9167131f32e2a). - Hardware and software: 1 node of 4x NVIDIA GB300, image vllm/vllm-openai:v0.30.0-aarch64 (sha256:4864d46625cbc3307623e29ac742030655e27249feba7b97ec925ce4cc4dfb56): torch 2.13.0+cu130, transformers 5.17.0, accelerate 1.15.0, plus flash-linear-attention 0.6.0 and datasets 5.0.0.
- Run: one epoch, 6,030 optimizer steps at global batch 16, lr 1e-6 cosine, forward KL to the frozen BF16 teacher, 8K context, weights fake-quantized to NVFP4; trained over several 4-hour jobs, each resuming from the last checkpoint. Final held-out (128 conversations) forward KL 0.0285 and teacher-student top-1 agreement 0.938.
Evaluation
Served with vLLM 0.30.0 (data-parallel 4 on GB300, native NVFP4 W4A4, max-model-len 65536; tokenizer_class set to PreTrainedTokenizerFast, which this repo ships, since the original snapshot decodes byte tokens under transformers 5.x) and evaluated with the released harness (scripts/eval/quality/, evalscope 1.9.1, temperature 1.0, top-p 0.95, top-k 20, 32K new tokens): GPQA-Diamond, Winogrande and HellaSwag (300), MMLU-Pro and MMMLU (21 per subset), AIME25 (5 samples per problem), LiveCodeBench v6 (300) and BigCodeBench (300 of 1,140). KL: forward KL to the BF16 model over top-20 prompt logprobs, 100 samples x 8K tokens each of UltraChat, the Pile and WikiText (scripts/eval/quality/run_kld.sh). The BF16 column was measured with the same harness.
| Benchmark | BF16 | This QAD | Paper QAD (W4A16-trained) |
|---|---|---|---|
| GPQA-Diamond | 44.4 | 48.0 | 46.1 |
| MMLU-Pro | 53.4 | 47.6 | 50.4 |
| MMMLU | 54.4 | 52.0 | 54.0 |
| Winogrande | 69.3 | 61.7 | 68.9 |
| HellaSwag | 52.3 | 53.3 | 55.3 |
| AIME25 | 38.7 | 30.0 | 32.7 |
| LiveCodeBench v6 | 61.3 | 60.7 | 58.5 |
| BigCodeBench | 17.3 | 19.7 | 15.3 |
| Average (8) | 48.9 | 46.6 | 47.7 |
| KL UltraChat / Pile / WikiText | - | 0.0905 / 0.1214 / 0.1365 | 0.0855 / 0.1174 / 0.1311 |
| KL mean | - | 0.1161 | 0.1113 |
LiveCodeBench is graded with evalscope's local executor (no Docker on the evaluation cluster). BigCodeBench check programs are built by the harness unchanged and executed in an aarch64 rebuild of the official bigcodebench-evaluate image (all pinned requirements installed; no 4 GB memory cap; tensorflow does not import, which affects 1 of the 300 sampled tasks). Single-sample benchmarks of 198-300 questions carry 2-3 points of sampling noise; our BF16 GPQA-Diamond is 6 points below the paper's with the same protocol. The BF16 column is the same BF16 run for every checkpoint in this series.
Provenance
Files under provenance/: trainer_state.json (loss, held-out KL and top-1 history), kld-ds8b-nvfp4-qad.json (per-dataset KL), versions.txt (training environment).
- Downloads last month
- -
Model tree for qatfactory/DeepSeek-R1-Distill-Llama-8B-NVFP4-QAD
Base model
deepseek-ai/DeepSeek-R1-Distill-Llama-8B