mbehr90/Kolibri-1-nvfp4

Kolibri-1 - NVFP4 (weight-only, routed experts)

The 57,600 routed-expert projections (50 layers x 384 experts x gate/up/down): FP4 E2M1 values with an FP8 E4M3 scale per 16 weights and one fp32 global scale shared by the gate/up pair of each expert. Attention, shared experts, routers, embeddings and lm_head stay BF16. No calibration data: activations stay BF16 (NVFP4A16, no input_global_scale). vLLM therefore runs it through Marlin as W4A16 on every GPU, including Blackwell: the FP4 tensor-core kernels (FlashInfer TRT-LLM / CUTLASS) require NVFP4-quantized activations and are not selected for this checkpoint. The benefit is memory (4-bit weights in HBM) and the bandwidth that comes with it, not FP4 arithmetic. For FP4 tensor cores you need a calibrated W4A4 build.

Quantized at tensor level from Aleph-Alpha/Kolibri-1-BF16 (145.5 GiB) with a ~300-line script and no model instantiation; no calibration data was used. Checkpoint size: 44.4 GiB in the compressed-tensors format. vLLM kernel on H100: CompressedTensorsW4A4Nvfp4MoEMethod (use_a16) -> Marlin on Hopper.

Kolibri 1 is Aleph Alpha's 78B-parameter / 3.46B-active mixture-of-experts reasoning model (50 layers, 384 experts per layer, 6 routed + 1 shared, 4:1 sliding-window:global attention, German + English, Apache 2.0). See the original model card for capabilities, prompting, reasoning effort and tool calling.

Serving

Kolibri 1 needs the aleph-alpha-inference vLLM plugin (vLLM 0.29.x). The container ghcr.io/aleph-alpha/aleph-alpha-inference:1.0.0-vllm0.29.0 has it installed.

vllm serve mbehr90/Kolibri-1-nvfp4 --tensor-parallel-size 4 --enable-expert-parallel \
  --kv-cache-dtype fp8 --reasoning-parser kolibri1 --tool-call-parser kolibri1 --enable-auto-tool-choice

Benchmark (4x H100 80 GB, vLLM 0.29, TP=4, FP8 KV cache)

Throughput: vllm bench serve, random prompts, --ignore-eos, output tokens/s per load point; rows without a suffix use expert parallelism, plain TP rows do not. Accuracy via the chat endpoint at temperature 0: GSM8K (first 200 questions and the full 1319-question test set, with thinking reasoning_effort=low and without), MMLU (1000 English questions) and MMMLU DE_DE (1000 German questions), each with thinking off (letter-only answer, 16 tokens) and with thinking (reasoning_effort=low, 4096-token budget; answers that run out of budget count as wrong, 3-5 % for every variant). The no-think multiple-choice numbers are a relative comparison only, not comparable to published MMLU scores. Every variant stays within about one point of the BF16 source on all of them.

config	weights/GPU	1 stream (1k in / 512 out) tok/s	16 streams tok/s	64 streams tok/s	16k prompt, 2 streams tok/s	TPOT 1 stream	TTFT 16k	GSM8K 200 think	GSM8K 200 no-think	GSM8K full think	GSM8K full no-think	MMLU 1000 (en) no-think	MMMLU 1000 (de) no-think	MMLU 1000 think	MMMLU 1000 think
BF16 (source)	36.5 GiB	170	1867	4586	251	5.8 ms	372 ms	95.5 %	95.0 %	95.1 %	94.4 %	74.4 %	70.3 %	84.7 %	80.0 %
FP8 block 128x128 (official)	18.5 GiB	146	1805	5151	222	6.7 ms	387 ms	95.5 %	94.0 %	95.1 %	94.2 %	73.9 %	71.0 %	84.5 %	80.6 %
FP8 per-channel	18.6 GiB	128	1694	4983	210	7.6 ms	377 ms	95.0 %	95.5 %	95.5 %	93.9 %	74.1 %	71.5 %	85.3 %	81.0 %
INT8 W8A8	18.6 GiB	173	2055	5260	252	5.6 ms	403 ms	95.5 %	94.0 %	94.7 %	94.8 %	73.8 %	71.3 %	84.2 %	80.3 %
NVFP4 (Marlin W4A16)	11.3 GiB	163	2192	6087	236	6.0 ms	394 ms	96.0 %	93.0 %	95.0 %	94.8 %	74.3 %	70.6 %	84.4 %	80.1 %
INT4 W4A16 g128 (Marlin)	10.4 GiB	170	2248	6259	243	5.8 ms	386 ms	96.5 %	92.5 %	95.1 %	94.0 %	73.8 %	70.5 %	84.7 %	80.0 %
INT4 x FP8 (W4A8, Hopper)	11.3 GiB	125	1752	5147	209	7.9 ms	353 ms	95.0 %	95.5 %	94.9 %	94.1 %	73.8 %	70.0 %	83.9 %	79.2 %
BF16, plain TP	36.5 GiB	191	1847	4929	-	5.1 ms	-	95.5 %	95.0 %	95.1 %	94.4 %	74.4 %	70.3 %	84.7 %	80.0 %
FP8 block, plain TP	18.5 GiB	182	1982	5620	265	5.4 ms	379 ms	95.5 %	94.0 %	95.1 %	94.2 %	73.9 %	71.0 %	84.5 %	80.6 %
FP8 per-channel, plain TP	18.7 GiB	140	1716	4729	-	7.0 ms	-	95.0 %	95.5 %	95.5 %	93.9 %	74.1 %	71.5 %	85.3 %	81.0 %
INT8 W8A8, plain TP	18.7 GiB	189	2147	5447	-	5.1 ms	-	95.5 %	94.0 %	94.7 %	94.8 %	73.8 %	71.3 %	84.2 %	80.3 %
NVFP4, plain TP	11.3 GiB	173	2227	6058	-	5.6 ms	-	96.0 %	93.0 %	95.0 %	94.8 %	74.3 %	70.6 %	84.4 %	80.1 %
INT4 W4A16, plain TP	10.4 GiB	177	2265	6253	-	5.5 ms	-	96.5 %	92.5 %	95.1 %	94.0 %	73.8 %	70.5 %	84.7 %	80.0 %

How it was made

Each safetensors shard of the BF16 checkpoint is read once; the targeted projections are quantized on the fly and written back into new shards together with their scales, and a compressed-tensors quantization_config is added to config.json. The INT4 packing was verified bit-for-bit against compressed_tensors.compressors.pack_quantized.pack_to_int32. Script: quantize_kolibri.py (in this repo).

Built by Maximilian Behr.