bibr-extract-qwen35-4b-FP8-Dynamic

This is an FP8 build of scienceverse/bibr-extract-qwen35-4b (revision 6ab47a4), the structured-extraction student model used by bibr. It is meant for vLLM on GPUs with FP8 support (Ada, Hopper, Blackwell). The weight files are 5.76 GB, against 9.32 GB for the bf16 model.

The memory figures below were emulated on a 32 GB card. No 8–11 GB card was run, and the CUDA context and any other GPU use come on top of the budgets shown.

Quantization

• Tool: llm-compressor 0.14.0.1 model_free_ptq (compressed-tensors 0.19.0). It needs no calibration data and works directly on the safetensors files.
• Scheme: FP8 dynamic, stored in the compressed-tensors float-quantized format.
  • Weights: FP8 E4M3 with one scale per output channel.
  • Activations: FP8 with a dynamic scale per token.
• What is quantized: 200 linear layers.
  • all 96 MLP projections;
  • the 32 full-attention q/k/v/o projections;
  • the 72 Gated DeltaNet (linear-attention) projections in_proj_qkv, in_proj_z and out_proj.
• Comparison with Qwen's own FP8 release: in the main decoder layers the layer choice is the same as in Qwen/Qwen3.5-27B-FP8. Its config.json (revision 97f5941) leaves every linear_attn.conv1d, in_proj_a and in_proj_b unconverted, along with lm_head, the embeddings, the vision tower and mtp.fc. Two things differ:
  • Qwen converts the rest of its MTP layer to FP8, while this build keeps the whole MTP head in bf16.
  • Qwen's release uses 128×128 block scales; this build uses per-channel weight scales.
• What is kept unquantized: 538 tensors, all byte-identical to the bf16 student, in the source's dtypes.
  • In bf16: the embeddings and the tied LM head, in_proj_a / in_proj_b, conv1d, dt_bias, the RMSNorm weights, the vision tower and the MTP head (mtp.*).
  • In F32, as in the source: the 24 linear_attn.A_log tensors and the 24 Gated DeltaNet norm weights (linear_attn.norm.weight).

Evaluation

The scores come from bibr's extraction harness of 9 Oct 2026: bibr 9621739, v5 scorer, 252 dev papers (gate192 + ci60), OCR replayed from cache.

• Comparisons are paired, with a paper bootstrap of 10,000 reps and a non-inferiority margin of −4 pp.
• One paper is 0.4 pp of the 252.

model	serving	passes / 252	this model vs it, Δ pp [95% CI]
bibr-extract-qwen35-4b, bf16: the merged checkpoint this build was made from, same card and harness setup	vLLM 0.31.0, RTX 5090	217	+0.4 [0.0, +1.2]; 1 gained, 0 lost
bibr-extract-qwen35-4b reference run: the unmerged LoRA on Qwen3.5-4B, from a separate run on another GPU	vLLM 0.31.0	216	+0.8 [0.0, +2.0]; 2 gained, 0 lost
this model	vLLM 0.31.0, RTX 5090	218	–

• Which row to compare with: the first row is like for like, with the same checkpoint, card and setup. The 216 reference is the run bibr's model comparisons have used since October.
• Subsets against the same-box bf16 model:
  • gate192 (n=192): +0.0 [0.0, 0.0];
  • ci60 (n=60): +1.7 [0.0, +5.0], 1 gained.
• Floor fields (title, DOI, authors, references), all 252 papers: no paper is lost on any field against either row.
  • Against the bf16 model, references gain one paper.
  • Against the reference run, title and references gain one paper each.
• Abstract: mean ROUGE-L 0.954, against 0.955 for the bf16 model.
• Noise: two identical runs of this harness agree on about 96% of LLM answers, because bibr picks a random fence boundary for each call. Differences of one or two papers are within run-to-run noise.

Speed

Measured on an RTX 5090, replaying the same 351 extraction requests. The bf16 numbers on vLLM 0.31 come from the same session and card.

	bf16 4B	this model
decode tok/s per request, concurrency 1, vLLM 0.31	164.5	233.5 (1.42×)
aggregate tok/s, concurrency 8, vLLM 0.31	705	1,059 (1.50×)
decode tok/s per request, concurrency 1, vLLM 0.32 + MTP k=3	320 *	411 (2.50×)
aggregate tok/s, concurrency 8, vLLM 0.32 + MTP k=3	1,214 *	1,608 (2.28×)
weights in GPU memory (--language-model-only)	7.99 GiB	4.73 GiB

* From an earlier run on the same card with the same 351 requests. The ratios in brackets are against the bf16 model on vLLM 0.31 without MTP.

• MTP settings: the MTP rows use --max-num-seqs 64 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --no-async-scheduling. The pass rates above were scored on vLLM 0.31 without MTP.
• MTP acceptance at concurrency 1, from vLLM's counters:
  • accepted / drafted tokens = 54,155 / 55,346 = 0.978. vLLM drafted 2.94 tokens per step on average, not always 3.
  • Mean acceptance length is 3.88 tokens per step (1 + accepted tokens per drafting step).
  • Acceptance per draft position: 99.5% for the first, 96.4% for the second and 92.2% for the third.

Memory

The budgets below were emulated on a 32 GB card by lowering --gpu-memory-utilization. No 8–11 GB card was run. The CUDA context and any other GPU use come on top of the budget:

• vLLM's own CUDA context took about 0.5 GiB outside its budget;
• vLLM refuses to start unless the whole budget is free once that context exists.

How the probes were run:

• budget = share × 31.45 GiB, the total that vLLM sees on this card;
• every probe used --max-model-len 32768 --max-num-seqs 64;
• one 32,768-token request needs 1.11 GiB of KV cache.

start	budget (share)	result	KV-cache tokens	vLLM process at ready
cold	7.11 GiB (0.226)	fails: room for 16,368 tokens	–	–
cold	7.86 GiB (0.25)	serves	39,891	7,272 MiB
warm	7.11 GiB (0.226)	serves	48,914	7,592 MiB
warm	10.69 GiB (0.34)	serves	153,867	11,316 MiB

• Cold: a first start. The vLLM, TorchInductor and Triton compile caches were empty, and another process held 1.6 GiB of the card (a 1 GiB buffer plus its own CUDA context).
• Warm: the compile cache from an earlier start was present.
• Why cold needs more: compiling during vLLM's memory profiling raises peak activation memory from 0.39 to 1.43 GiB. So a first start needs about 1 GiB more budget than a later start.

Usage

vllm serve scienceverse/bibr-extract-qwen35-4b-FP8-Dynamic --language-model-only \
  --default-chat-template-kwargs '{"enable_thinking": false}' --generation-config vllm --max-model-len 32768

Use the model with thinking off, as it was trained and scored.

Tested only on an RTX 5090 (Blackwell) with vLLM 0.31.0 and 0.32.0. It has not been run on Ampere, Ada or Hopper cards.

Licence

Apache-2.0, the same as the bf16 student and its base model Qwen/Qwen3.5-4B (Apache-2.0, revision 851bf6e).