gitarist/Qwen3-8B-BPDQ-coded-2.5bit

Qwen3-8B — BPDQ coded planes, 2.5 bits

Qwen3-8B with the linear weights stored as three bit-planes per weight and the two low planes coded per block of 8 weights. The first plane is stored as is, the second as an even-weight codeword (7 bits), and the third as one of the 8 codewords of the Reed–Muller code RM(1,3) (3 bits). That is 18 bits per 8 weights. With fp16 coefficients per 256 columns, the total is 2.5 bits/weight. The embedding is 4-bit and the head is 8-bit. 3.1 GB on disk.

Recipe: block-Babai encoding, then 3056 steps of KD QAT, then 250 steps of GRPO.

Benchmarks

	fp16	this model
bits/weight (linears)	16	2.5
KL to fp16, WikiText-2	0	0.254
GSM8K nothink	91.7	84.2
GSM8K think @1024	83.0	87.6
HumanEval	82.9	71.3
MBPP	66.6	57.6
IFEval (prompt strict)	81.7	76.3
TriviaQA 5-shot	56.7	37.3
BFCL simple	94.5	93.8
BFCL multiple	96.5	93.0
BFCL parallel	94.0	93.5
BFCL parallel-multiple	91.0	87.0
BFCL irrelevance	83.8	81.7
BFCL multi-turn	45.0	31.0
12-task average	80.6	74.5

Decode throughput, vLLM 0.28, one RTX 4090: 260 tok/s at batch 1, 970 at 4, 3280 at 16, 5040 at 32, 6560 at 64, 7700 at 256.

Run it

CUDA only (sm80+).

import sys; sys.path.insert(0, MODEL_DIR)
import bpdq   # registers the "bpdq" quantization method with vLLM
from vllm import LLM
llm = LLM(model=MODEL_DIR, quantization="bpdq", dtype="float16")

or vllm serve MODEL_DIR --quantization bpdq --dtype float16 after import bpdq. bpdq.load(MODEL_DIR) gives a transformers model.