vexp-ai/horizon-draft-0.5b

Horizon Draft 0.5B

A tiny standalone draft model for speculative decoding with DeepSeek-R1-Distill-Qwen-7B as the target, built for consumer CPUs and llama.cpp -md. Part of the Horizon project: a verification-first layer for local LLMs, by the team behind vexp.

What it is

• Base: Qwen2.5-0.5B, full-finetuned for one epoch (~2.5 h on a single 24 GB GPU, ~$3 of compute) on 30k reasoning traces distilled from DeepSeek-R1 (13k math, 13k code, 4k science), formatted with the target's chat template.
• Vocabulary-aligned with the target: trained with the target's tokenizer and the embedding padded to n_vocab 152064, so it passes llama.cpp's strict speculative vocabulary check and drops straight into -md. A stock Qwen2.5-0.5B does not pair (different special tokens, 151936 vocab), and through permissive paths it reaches only τ≈1.

Measured results (mainstream DDR4 desktop, Ryzen 9 3900X, 12 threads)

Config	Code generation	Reasoning segment	RSS
7B Q4_K_M autoregressive	8.1 t/s	8.1 t/s	7.7 GB
+ this drafter (Q8_0, γ=8)	10.7-15.1 t/s (mean 13.3, 1.65×)	7.1-9.4 (neutral)	8.3 GB
+ same-family 1.5B draft	7.1-10.9 (no gain)	5.3 (hurts)	9.4 GB

Acceptance on three coding tasks: 36-56% on code, 20-30% on chain-of-thought text. Two practical lessons we ship with the numbers: on CPU the draft must be nearly free (a 1.5B draft with similar acceptance gains nothing), and speculation pays on code, not on reasoning: enable it per segment.

Usage (llama.cpp)

llama-speculative -m DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf \
  -md horizon-draft-0.5b-q8_0.gguf \
  --spec-draft-n-max 8 -t 12 -n 512 --temp 0 -p "..."

Caveat: some recent llama-server builds silently skip speculative decoding (unified KV cache default) while still loading the draft. Verify acceptance stats are non-zero; the llama-speculative binary is the reliable path today.

Limitations

• Target-specific by design: it predicts DeepSeek-R1-Distill-Qwen-7B's output distribution. Pair it with other targets at your own risk.
• Trained for one epoch on 30k traces: acceptance has headroom (more epochs, more diverse code styles, DSpark-style parallel drafting are the declared next steps).
• Not an instruct model: do not use it standalone for generation.

Reproduce

Training script (train/train_draft.py), corpus recipe and the full measurement logs are in the Horizon repository.