SenseNova-Loop-9B-FP8

FP8 weights of sensenova/SenseNova-Loop-9B for vLLM, including the MTP head.

Quantization

  • Every linear layer in the language model and the MTP head is stored as float8_e4m3fn with one per-tensor weight_scale = amax / 448. This is the same algorithm as vLLM's online --quantization fp8.
  • Activations are quantized dynamically. On Ampere GPUs vLLM runs these layers as Marlin W8A16.
  • Kept in BF16: the vision encoder, embeddings, lm_head, all norms, the mHC routers (FP32), loop_gate, and the linear-attention conv1d/A_log/dt_bias.
  • quantization_config: {"quant_method": "fp8", "activation_scheme": "dynamic", "ignored_layers": [vision linears]}
  • Size: 11.3 GiB (the BF16 original is about 19 GiB).
  • Export script: tools/export_fp8.py

Running

Stock vLLM does not support the Loop architecture yet. Use the patches at igerry/vllm-sensenova-loop (vLLM 0.33.0). You need commit e077110 or later: earlier versions of patch_fp8.py reject this checkpoint because of the attention k/v/q/prob_scale parameters.

./install.sh /path/to/venv/bin/python /models/SenseNova-Loop-9B   # applies the patches
QUANT= PY=/path/to/venv/bin/python ./serve.sh /models/SenseNova-Loop-9B-FP8

Results (2脳 RTX 3090 Ti, TP=2, MTP with 2 draft tokens)

This checkpoint, loaded as-is with --attention-backend FLASHINFER:

This checkpoint Online --quantization fp8
Decode tok/s, greedy (code and web generation, thinking on) 135.2 / 136.1 138.2 / 137.9
GSM8K, first 200 (greedy, thinking off) 190/200 188/200
Decode tok/s at 1k / 97k context 110.6 / 93.6 (sampled) 123.5 / 101.6 (sampled)

The sampled numbers vary by 卤5% from run to run, so compare the greedy rows. Earlier comparison against BF16 and AWQ, measured with online FP8 and the default FlashAttention 2 backend:

BF16 FP8
Decode tok/s (code and web generation, thinking on) 85.6 121.8
GSM8K, first 200 (greedy, thinking off) 189/200 189/200
Weights per GPU 8.9 GB 6.0 GB

Image input works.

License

Apache-2.0, same as the base model.

Downloads last month
5
Safetensors
Model size
10B params
Tensor type
BF16
路
F32
路
F8_E4M3
路
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for igerry/SenseNova-Loop-9B-FP8

Quantized
(44)
this model