happynood/DeepSeek-R1-Distill-Qwen-1.5B-GGUF

DeepSeek-R1-Distill-Qwen-1.5B — GGUF, benchmarked by QuantThink

GGUF quantizations of deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, originally converted by bartowski (re-hosted here unmodified — same files, same quantization recipe, all credit for the GGUF conversion itself belongs to bartowski). What's new here is the benchmark data: real, measured accuracy/thinking-length/cost-to-solve for each quant level from QuantThink, a reproducible benchmark measuring how quantization affects small reasoning (long chain-of-thought) models on a 4GB consumer GPU.

Measured results (RTX 3050 Laptop, 4GB VRAM)

First-pass numbers (N=6 problems x 2 seeds = 12 samples per cell — disclosed small-N first pass, not yet the full statistically-rigorous sweep; see docs/RUN_REAL.md for the complete write-up and caveats):

Quant	File	GSM8K Acc	GSM8K TL	MATH-500 Acc	MATH-500 TL	Peak VRAM
fp16	DeepSeek-R1-Distill-Qwen-1.5B-f16.gguf	0.667	469.8	0.417	1465.3	3.50 GB
Q8_0	DeepSeek-R1-Distill-Qwen-1.5B-Q8_0.gguf	0.750	385.5	0.500	1335.9	2.16 GB
Q5_K_M	DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M.gguf	0.667	244.3	0.417	1215.9	1.67 GB
Q4_K_M	DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf	0.583	420.8	0.333	1280.7	1.55 GB

Q4_K_M is the only quant showing a measurable accuracy drop vs. fp16 (-8.3 points on both benchmarks) at this sample size; Q8_0 and Q5_K_M show no measurable degradation. See the GitHub repo's docs/RUN_REAL.md for KV-cache-quantization results (including a genuine finding: Q4 KV cache causes total generation collapse for this model, not smooth degradation), thinking-cap results, and the Memory-Budget Frontier this data feeds.

Sampling used for these numbers

Temperature 0.6, top-p 0.95 (per the model card's own recommendation), fixed seed set [0, 1] — not greedy decoding, since reasoning models are known to degrade under greedy sampling.

Reproduce

git clone https://github.com/Happynood/quant-reasoning-bench
cd quant-reasoning-bench
uv sync --extra llama-cpp
uv run quantthink run --config configs/phase1/Q4_K_M_E1.yaml --output result.json --manifest manifest.json

Related

• GitHub repository
• quantthink-results — all real result.json files
• quantthink-leaderboard — interactive leaderboard
• Original GGUF conversion: bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF
• Original model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B