onprem-ai/Apertus-v1.5-70B-NVFP4

Apertus v1.5 70B NVFP4

Structured two-pass quantization of swiss-ai/Apertus-v1.5-70B with 54% faster decode, 32% smaller, at 99% quality compared to FP8. Uses the NVFP4 scheme: NVFP4A16 on MLP projections, FP8_DYNAMIC on attention projections, and BF16 preserved on embeddings, lm_head, and norms.

Checkpoint	Size	Compression	Quality (MMLU)
BF16 (original)	135 GiB	1.0x	baseline
FP8 (onprem-ai/Apertus-v1.5-70B-FP8)	71 GiB	1.9x	no measurable loss
NVFP4 (this repo)	48 GiB	2.8x	-0.8 points

This is the recommended format for production deployment: 48 GiB leaves ample headroom on a single 96 GiB GPU for KV cache (up to ~230k context with fp8 KV cache) and multimodal encoder working memory.

Thanks to: Swiss AI, blancsw, Anunay-Yadav, Oleg, Cyrilvallez, AryanAhadinia, robmsmt, and Neural Magic. See Attribution for details.

Usage (vLLM)

Until Apertus 1.5 support is available in a native vLLM release, use the ghcr.io/swiss-ai/vllm_apertus_1.5_release image. The build recipe and patched Dockerfile are available at github.com/swiss-ai/model-launch/tree/main/images/vllm_apertus_1.5.

# Download the checkpoint
huggingface-cli download onprem-ai/Apertus-v1.5-70B-NVFP4 \
  --local-dir ./apertus-v1.5-70b-nvfp4

# Serve it
docker run --gpus all -v ./apertus-v1.5-70b-nvfp4:/model \
  ghcr.io/swiss-ai/vllm_apertus_1.5_release:latest-amd64 \
  --model /model \
  --served-model-name apertus-v1.5-70b \
  --host 0.0.0.0 --port 8080 \
  --dtype auto \
  --chat-template /model/chat_template.jinja \
  --tool-call-parser apertus --enable-auto-tool-choice \
  --reasoning-parser apertus

Performance: 54% Faster Single-User Decode Than FP8

Measured on NVIDIA RTX PRO 6000 Blackwell (96 GiB VRAM) with vLLM. Decode is memory-bandwidth bound, so the 48 GiB NVFP4 checkpoint outperforms the 71 GiB FP8 checkpoint while leaving 23 GiB more VRAM available.

Checkpoint	Weight Size	Single-User Decode	Relative Decode
FP8	71 GiB	20.0 TPS	baseline
NVFP4	48 GiB	30.7 TPS	54% faster

NVFP4 concurrency scaling was measured with llmapibenchmark using random prompts and 200-token prefill/decode workloads:

Concurrency	Decode TPS	Prefill TPS	TTFT
1	31	2500	0.3s
4	104	2500	1.3s
8	176	2500	2.7s
20 (peak)	294	2500	6.8s

Decode throughput scales up to 294 TPS at 20 concurrent requests. Prefill throughput remains near 2500 TPS through concurrency 20.

Quality: 99% of FP8 Quality yet 32% Smaller

Benchmarked via lm-evaluation-harness (MMLU, 5-shot, 100 samples) on NVIDIA RTX PRO 6000 Blackwell. The vLLM compressed-tensors backend loads the structured scheme natively with no decompression overhead.

Metric	BF16 (official)	NVFP4	Delta
MMLU overall	base	-0.8 pts	-1.1%
humanities	base	-0.3 pts	-0.4%
social sciences	base	-1.4 pts	-1.7%
STEM	base	-0.3 pts	-0.4%
other	base	-1.5 pts	-2.0%

All deltas within 1 standard error. At 2.8x compression versus the original BF16 checkpoint, this represents an excellent quality-efficiency trade-off.

Multimodal Support

All vision and audio capabilities from the original checkpoint are preserved:

• Image understanding (Emu3.5 vision encoder)
• Audio understanding (WavTokenizer audio encoder)
• Tool calling (native apertus tool call format)
• Reasoning (apertus reasoning parser)

The quantization respects the ignore list: vision tokenizer, audio tokenizer, and embedding layers remain in BF16.

Quantization Process

Two-pass calibration-free quantization using llm-compressor 0.12.0:

Pass 1: NVFP4A16 on MLP

scheme: NVFP4A16
targets: re:.*mlp\.up_proj$, re:.*mlp\.down_proj$
ignore: lm_head, model.language_model.embed_tokens,
        re:.*self_attn\.(q|k|v|o)_proj$,
        re:.*vision_tokenizer.*, re:.*audio_tokenizer.*

Pass 2: FP8_DYNAMIC on Attention

scheme: FP8_DYNAMIC
targets: re:.*self_attn\.(q|k|v|o)_proj$
ignore: lm_head, model.language_model.embed_tokens,
        re:.*mlp\.(up|down)_proj$,
        re:.*vision_tokenizer.*, re:.*audio_tokenizer.*

The generated quantization_config requires target group separation (fix_three_tier_quant_config.py in the build repo handles this).

Limitations

• vLLM only. SGLang does not support compressed-tensors structured checkpoints (see sgl-project/sglang#16276).
• Requires vLLM with Apertus1p5 arch support. Use the ghcr.io/swiss-ai/vllm_apertus_1.5_release image or build from upstream PR #50496.
• Recommened 96 GiB GPU. 48 GiB weights + multimodal encoder working memory
  • KV cache do not fit on 80 GiB cards for long contexts.

Supported Hardware

Hardware	Compatible	Notes
NVIDIA RTX PRO 6000 Blackwell (GB202, SM100)	Yes	Primary target
B100/B200/B300/GB200 (SM100)	Yes
H100/H200 (SM90)	Yes
DGX Spark (GB10, SM121)	No	FP4 kernels produce incorrect results on SM121 (no TMEM/WGMMA/DSMEM). Use the FP8 checkpoint instead.

Attribution

• Original model: swiss-ai/Apertus-v1.5-70B by Swiss AI (including Oleg Lavrovsky (loleg), Cyrilvallez), Apache 2.0
• GPU-native multimodal refactor: blancsw at Infomaniak -- foundational architecture work, Emu3.5 vision encoder, WavTokenizer audio encoder, apertus tool parser
• Upstream PR: vllm-project/vllm #50496 by Anunay-Yadav -- GPU-native Apertus 1.5 multimodal encoders for upstream vLLM
• Reasoning parser: AryanAhadinia
• Double BOS-token fix: robmsmt
• Quantization tooling: llm-compressor by Neural Magic
• Benchmarking: llmapibenchmark by onprem-ai