Speculative Decoding Lab
Systematic, reproducible empirical evaluation of modern speculative decoding (EAGLE-3, DFlash, DSpark vs. vanilla and model-free) on an RTX 4060 Ti 16 GB with Gemma 4 12B and Qwen3-8B GGUF-quantized models: acceptance by cognitive domain, quantization effects, and hardware break-even maps.
Paper: PDF · arXiv-ready source bundle · source (markdown) · paper page
Hardware used
The study was developed and benchmarked on this machine, so all reported numbers (speedups, acceptance, break-even) are tied to these specs. The code itself is portable: it only needs llama.cpp (CUDA build) and enough VRAM for the target model, and can be re-run on any other hardware.
| Component | Detail |
|---|---|
| GPU | NVIDIA RTX 4060 Ti 16 GB (Ada Lovelace, compute 8.9) |
| CPU | Intel i5-12400 (6C/12T, 4.4 GHz) |
| RAM / Disk | 31 GB / ~185 GB free |
| OS | Ubuntu 24.04 · driver 595.84 |
Environment principle: everything user-space, no sudo (uv + CUDA runfile in $HOME).
Stack
- Python 3.12 + uv —
pyproject.toml+uv.lock(numpy, ruff, pyright, cmake, ninja). - CUDA toolkit 12.8.1 in
~/cuda(runfile--toolkitpath). Note: thenvccbinary is not on PyPI (the wheels only ship ptxas). - llama.cpp master 0713275 (v22) (CUDA build) in
~/llama.cpp(built from source; no CUDA binaries are published for Linux). - GGUF models in
models/(not versioned; downloaded withcurl -C -).
Setup (once)
uv sync # create .venv with all dependencies
source scripts/spec-env.sh # PATH + CUDA_HOME + env vars (idempotent)
Quality (required before committing)
uv run ruff check . # lint
uv run ruff format --check . # formatting
uv run pyright # types
Execution regime
Overnight sessions with pause/resume: scripts/bench_accept.py --resume
(incremental JSONL per prompt; server-based runner with α/τ), downloads with
scripts/dl.py, runs in tmux/nohup. Measurement campaigns are launched with
scripts/run_final_chain.sh, scripts/run_curves_chain.sh and
scripts/run_ksweep_chain.sh; final analysis with scripts/analyze_final.py.
Structure
scripts/ environment, downloads, benchmark runner and analysis (bench_accept.py, analyze_final.py)
experiments/ runs/ (raw, not versioned) and prompts/ (JSONL subsamples)
models/ GGUFs (not versioned)
For agents
Read AGENTS.md before working in this repo (conventions, git
rules, sessions and handoff).
License
MIT — see LICENSE.