keptn/nemotron-3.5-bolt-e4b-bf16

NVIDIA Nemotron 3.5 Bolt E4B · bf16 · for radiance

nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 in one 16.4 GiB file for the radiance inference engine on AMD RDNA4. The weights are as NVIDIA ships them, with NVIDIA's DFlash drafter and the vision tower, on one card.

• Close to the original: KL divergence 0.0015 to the bf16 original on wikitext-2; top token the same in 98.8%
• Fast: 155 tok/s BetterBench combined on one card, 303 tok/s total with 8 streams
• Long context: 65,536 tokens; one card holds 8 sequences at full length
• Vision: images through the model's bf16 vision tower
• Thinking: on or off per request; the benchmarks below show both

File	nemotron-3.5-bolt-e4b-bf16.rad, 16.44 GiB
Needs	radiance 1.11.0 + PR #210 (Gemma 4 E2B/E4B form), built from source; one card with 32 GB (tested on Radeon AI PRO R9700) and 7 GiB of free host memory
Weights	bf16, as in the checkpoint
Activations	bf16
lm_head	bf16 (tied to the token table)
Speculation (DFlash)	NVIDIA's DFlash drafter, 5 layers, FP8 with one bf16 scale per 128×128 block, 7 draft tokens
Vision tower	the original, bf16, in pinned host memory
Context	65,536 tokens
Version	1.0 (2026-10-24)

Quick start

Until a radiance release contains PR #210, build the image from the pull request's branch. docker/build.sh tags it radiance:1.11.0 and radiance:latest:

git clone -b gemma4-e4b https://codeberg.org/keptn42/radiance.git && cd radiance && docker/build.sh

With the compose files from that checkout: nemotron-3.5-bolt-e4b-bf16.yaml. Set RADIANCE_IMAGE=radiance:1.11.0 in .env. Or directly:

hf download keptn/nemotron-3.5-bolt-e4b-bf16 nemotron-3.5-bolt-e4b-bf16.rad --local-dir ~/models
docker run -d --init --name nemotron-bolt --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e ROCR_VISIBLE_DEVICES=0 \
    radiance:1.11.0 --model /models/nemotron-3.5-bolt-e4b-bf16.rad --host 0.0.0.0 --port 8000 \
    --kv-cache-dtype bf16 --max-model-len 65536 --max-num-seqs 8 --num-speculative-tokens 7 \
    --mm-max-patches 2520 --vision-tower-placement ram \
    --temp 1.0 --top-p 0.95 --top-k 64 --chat-template-kwargs '{"enable_thinking":false}'

	change
fp8 KV	--kv-cache-dtype fp8: half the KV memory, KL divergence 0.014 instead of 0.0015
no drafter	--num-speculative-tokens 0
text only	--mm-max-patches 0, drop --vision-tower-placement
thinking	off by default, as in NVIDIA's template; per request "chat_template_kwargs": {"enable_thinking": true}

The drafter runs at 7 draft tokens only: it was trained on blocks of 8. Sampling is NVIDIA's (temperature 1.0, top_p 0.95, both thinking modes) and the checkpoint's top_k 64.

Quality

Distance to the bf16 original. KL divergence on wikitext-2 (test), llama.cpp method: 100 chunks of 512 tokens, the second half of each chunk scored, 25,500 tokens. Reference: the bf16 original in HF transformers, fp32 math.

	mean KL	median KL	99th percentile	top token the same	perplexity (reference: 117.18)
this file, bf16 KV	0.0015	0.00023	0.014	98.8%	117.56
this file, fp8 KV	0.0137	0.0021	0.18	96.3%	117.19

For scale: HF transformers with bf16 math has a KL divergence of 0.0012 to the same reference (first chunk, 255 tokens).

Benchmarks, greedy. The thinking-off column is the one to compare with other cards.

benchmark	thinking off	thinking on
GSM8K (math, 5-shot)	82.3%	89.6%
IFEval (following instructions, strict)	87.1%	91.5%
LiveCodeBench v6 (175 tasks, pass@1)	46.3%	55.4%

• LiveCodeBench v6 by difficulty (off and on): easy 40/43 and 43/43, medium 26/52 and 32/52, hard 15/80 and 22/80.
• Thinking on uses output limits of 8,192 (GSM8K), 16,384 (IFEval) and 65,536 tokens minus the prompt (LiveCodeBench; the context has 65,536 tokens).
• Thinking off, 137 GSM8K answers stop at the task's limit of 256 tokens, 5 IFEval answers at 1,280 and 5 LiveCodeBench answers at 8,192. Thinking on, 1 IFEval answer stops at 16,384. They count as wrong.

Speed

Radeon AI PRO R9700 at 215 W, radiance 1.11.0 + PR #210, bf16 KV, 7 draft tokens, thinking off.

BetterBench 0.6.1, full run: 8 categories, 20 runs each after 3 warm-up runs, temperature 0.7, one stream. Then prefill and parallel streams. Decode tok/s (median):

	1 card
combined	154.9
prose	77.3
code	167.4
JSON	241.6
summarization	137.5
prefill at 24k tokens	6,665
prefill at 47k tokens	5,581
4 streams, total	262
8 streams, total	303

The drafter gains most on code and JSON (4.1 to 4.7 tokens a step) and least on prose (2.0 tokens a step).

Vision

The model's vision tower (bf16, with the clamps the checkpoint stores) reads images in chat requests (image_url).

benchmark (lmms-eval 0.7.3, LMMs-Eval-Lite, greedy, thinking off)	result
DocVQA val lite (500 questions)	79.8 (ANLS)
AI2D lite (500 questions)	74.6%

Memory

KV cache, --max-model-len 65536, from radiance's startup report:

	2 sequences	4 sequences	8 sequences
1 card, bf16 KV	131,072 tokens	262,160 tokens	524,304 tokens
1 card, fp8 KV	131,104 tokens	262,160 tokens	524,336 tokens

Tokens of context the KV pool holds in total, shared by all sequences. Every sequence can have the full context. Only 24 of the 42 layers hold KV: the last 18 layers read the cache of layers 22 and 23. With 8 sequences the cache takes 18.2 GiB (bf16 KV) or 9.1 GiB (fp8 KV) of the card. The weights take 9.6 GiB. The token tables (6.5 GiB) and the vision tower (0.3 GiB) stay in pinned host memory.

How it was made

1. Downloaded nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 (revision 49843eed).
2. Converted NVIDIA's drafter from GGUF to safetensors with drafter/gguf_to_safetensors.py. The GGUF is nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash (revision 0949a2eb); config.json comes from nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4 (revision 913d13c4).
3. Compared every drafter tensor with the NVFP4 release: cosine 0.989 to 0.994 for the matrices (4-bit noise), 1.000 for the norms.
4. Built the container. The trunk and the vision tower stay as the checkpoint holds them; the drafter goes to block FP8.

python drafter/gguf_to_safetensors.py Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
    NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4 dflash-bf16
rad-convert NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 --recipe nemotron-3.5-bolt-e4b-bf16.recipe \
    --draft-model dflash-bf16 --set quantization_config.quant_method=compressed-tensors \
    -o nemotron-3.5-bolt-e4b-bf16.rad

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs radiance 1.11.0 + PR #210. Released radiance 1.11.0 refuses the E4B form of Gemma 4.
• One card only: radiance does not split the per-layer embeddings over cards. Two and four cards are not tested.
• NVIDIA lists text as the only input of this model. The vision tower is the tower of the checkpoint.
• The audio tower of the checkpoint is not in the file.
• The context is 65,536 tokens, the length NVIDIA trained.
• The wikitext-2 perplexity of the original is 117. Compare the KL divergence with the bf16 math value (0.0012), not with other cards.

Credits

• Model and drafter: NVIDIA, post-trained from Google's Gemma 4 E4B (Apache 2.0).
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
• Test sets: GSM8K, IFEval, LiveCodeBench, LMMs-Eval-Lite, wikitext-2.

License

OpenMDW License Agreement 1.1, the license of the base model. See LICENSE. The Gemma 4 base model is under the Apache License 2.0, see LICENSE.Apache-2.0. NOTICE names the sources and NVIDIA's terms.