flukethoughts/Behemoth-128B-v3-NVFP4

Behemoth-128B-v3 — NVFP4 (W4A4, GPTQ)

TheDrummer/Behemoth-128B-v3 quantized to NVFP4 W4A4 with GPTQ, in the compressed-tensors nvfp4-pack-quantized format, for vLLM on NVIDIA Blackwell.

What this is

Base	TheDrummer/Behemoth-128B-v3 (125 B dense, MistralForCausalLM, 88 layers)
Scheme	NVFP4 W4A4 — 4-bit float weights and activations, group 16, FP8 block scales
Algorithm	GPTQ, MSE weight observer
Quantized	all 88 layers × {q,k,v,o,gate,up,down}_proj = 616/616
Kept at BF16	lm_head, embed_tokens
Size	75.0 GB (69.8 GiB)
Tooling	llm-compressor 0.13.0 / compressed-tensors 0.18.0

⚠️ Recommended sampler settings

Like other low-bit quants of this model, occasional stray foreign-script tokens appear at high temperature with no min-p. min_p = 0.05 removes them. Measured on this artifact (non-Latin glyphs across 2 samples of creative prose):

setting	glitch tokens
temp 0.9, no min_p	4
temp 0.8 + min_p 0.05	0
temp 0.7 + min_p 0.05	0
temp 0.5 + min_p 0.05	0

Requirements

• Blackwell GPU. NVFP4 executes on 5th-gen FP4 tensor cores. Hopper (H100/H200) and older cannot run this — there is no FP4 kernel.
• vLLM. Verified on vLLM 0.28.0, RTX PRO 6000 Blackwell (sm_120).
• ~75 GB for weights, plus KV cache.

vllm serve <path> --served-model-name behemoth-128b-nvfp4 \
  --kv-cache-dtype fp8 --max-model-len 32768 \
  --gpu-memory-utilization 0.92 --max-num-batched-tokens 8192

Measured on a 96 GB card: GPU KV cache size: 77,680 tokens at --gpu-memory-utilization 0.92 with fp8 KV. --kv-cache-dtype fp8 roughly doubles usable context — at fp16 you get about half. Measured decode: 19.1 tok/s.

Calibration

256 sequences at up to 2048 tokens, through the model's own chat template:

• 55% Gryphe/Sonnet3.5-Charcard-Roleplay — multi-turn character RP, long-biased
• 25% HuggingFaceH4/ultrachat_200k — general instruct
• 20% Salesforce/wikitext (wikitext-2-raw-v1) — clean prose in fixed-length chunks

An explicit short system message is supplied so the chat template does not inject its stock 365-token "You are Mistral Medium 3.5…" prompt into every sample.

Verification

Functionally tested on Blackwell before publication — not just weight inspection:

• Control tokens: all 8 cases pass, including [INST] at position 0, [INST] as the final token, and the full documented [SYSTEM_PROMPT]…[MODEL_SETTINGS]…[INST] chat format (first-token logprob −0.833, coherent output).
• Long context / YaRN: needle retrieval at 740, 9,265 and 18,235 tokens — all found.
• Structure: 616/616 linears quantized across all 88 layers; lm_head and embed_tokens confirmed BF16; all scale tensors finite.
• Tokenizer: no truncation cap in either tokenizer file.

Note on weight-space metrics: GPTQ deliberately moves weights away from the originals to minimise output error, so cosine-vs-original (0.973–0.996 here) is lower than round-to-nearest PTQ would give and is not a fidelity indicator for this algorithm.

Known caveat

config.json uses the transformers-5.x nested rope_parameters block for YaRN. Runtimes that only read a top-level rope_scaling may silently drop YaRN rather than error, collapsing effective context. Verified working in vLLM 0.28.0 (needle test at 18k above). If long-context output degrades on another runtime, check rope parsing before suspecting the quantization.

Provenance

Quantization only — no abliteration, no fine-tuning, no modification of the base model's behaviour. The base was SHA256-verified against the Hub manifest (52/52 files) before quantization.