Llama-3.2-3B-Instruct-DSpark

A DSpark draft model for speculative decoding with unsloth/Llama-3.2-3B-Instruct as the verifier, trained with speculators. The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. Mean acceptance length is 3.41 tokens committed per verification step, up to 5.53 on math_reasoning.

Training code: rasyosef/train-dspark-draft-models.

Trained on 100,000 samples.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Llama-3.2-3B-Instruct-DSpark --port 8000 --gpu-memory-utilization 0.8

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.

Details

5 Qwen3 layers (hidden size 3072, intermediate size 8192, 24 attention heads over 8 KV heads, sliding-window attention with a 2048-token window), ~0.9B params, bfloat16. Block size 8, draft vocabulary reduced to 50,000, aux hidden-state layers 2/8/14/20/26, confidence head with Markov (rank 256).

Trained for 3 epochs at lr 3e-4 (AdamW, cosine schedule with 4% warmup) on 100,000 Open PerfectBlend prompts regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Prompts prepared at 1024 tokens; training sequence length 8192, up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front. speculators 0.8.0.dev207, vLLM 0.28.0, transformers 5.15.1, torch 2.13.0.

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 9.0 at block size 8.

subset	acceptance_length	pos_0	pos_1	pos_2	pos_3	pos_4	pos_5	pos_6	pos_7
math_reasoning	5.529	89.6%	78.4%	67.9%	58.8%	50.5%	43.3%	36.0%	28.2%
HumanEval	5.058	85.6%	72.6%	61.6%	52.3%	43.7%	36.3%	29.9%	23.8%
tool_call	3.492	73.6%	54.2%	39.5%	28.6%	20.8%	14.8%	10.5%	7.2%
question	2.952	67.7%	43.8%	28.8%	19.7%	13.8%	10.0%	6.8%	4.6%
rag	2.917	67.6%	45.8%	30.8%	20.4%	12.6%	7.6%	4.4%	2.4%
writing	2.908	67.5%	43.3%	28.0%	18.7%	13.0%	9.3%	6.4%	4.5%
translation	2.653	67.0%	42.7%	25.2%	14.2%	7.9%	4.4%	2.6%	1.3%
summarization	2.597	65.6%	40.5%	23.9%	13.9%	8.0%	4.3%	2.2%	1.2%
qa	2.241	55.0%	31.6%	17.9%	9.4%	5.0%	2.7%	1.6%	0.8%

Weighted across all subsets: 3.412 over 69,061 verification steps.

Acceptance is still highest where the verifier's next token is most predictable — math, code, structured tool calls. math_reasoning leads HumanEval by about half a token and holds a margin at every position in the block; math_reasoning's pos_5 (43.3%) is above summarization's pos_1 (40.5%), and both subsets still commit better than one token in four at pos_7. tool_call sits on its own at 3.49, between the two groups.

Five of the six prose-like subsets start in a tight band at pos_0 (65.6–67.7%) and separate only deeper into the block; qa is the exception, starting well below the rest at 55.0% and staying last throughout. question and writing decay gently and are still above 4% at pos_7. rag has the strongest pos_1 of the group (45.8%) but falls away after pos_3, and summarization, translation, and qa are all under 1.5% at pos_7 — so the tail of the block is worth something on open-ended generation but essentially nothing on summarization, translation, and qa.

Speed

Up to 3.88x on HumanEval (34.8 → 135.1 tok/s) and 2.56x overall (34.6 → 88.4 tok/s). The gains follow acceptance length: math and code clear 3.5x, while qa, summarization, and translation stay under 2x.

subset	n	base tok/s	spec tok/s	ratio	median
HumanEval	20	34.8	135.1	3.88x	3.88x
math_reasoning	20	35.1	124.1	3.53x	3.58x
writing	19	35.0	95.3	2.72x	2.52x
question	19	35.1	95.2	2.71x	2.50x
tool_call	20	34.3	85.8	2.50x	2.33x
rag	14	33.1	73.4	2.22x	2.36x
translation	20	35.3	64.6	1.83x	1.77x
summarization	20	32.8	58.3	1.77x	1.75x
qa	16	35.3	53.1	1.50x	1.58x
OVERALL	168	34.6	88.4	2.56x	2.21x

Throughput with the drafter attached (spec) against the verifier decoding alone (base), DSpark block 8, bs=1, vLLM on 2xT4 with the verifier and drafter on separate cards, fp16. ratio compares aggregate tok/s; median is the median of the per-request ratios.

Limitations

Works only with Llama-3.2-3B-Instruct and is not usable as a standalone model. Acceptance falls off steeply past the first few positions on prose-like traffic (summarization, translation, qa), so a block size of 8 is mostly wasted there — the gains concentrate in math, code, and tool calls. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

License

Llama 3.2 Community License, inherited from the verifier. The speculators training code is Apache-2.0.