SenseNova-Loop-9B-FP8
FP8 weights of sensenova/SenseNova-Loop-9B for vLLM, including the MTP head.
Quantization
- Every linear layer in the language model and the MTP head is stored as
float8_e4m3fnwith one per-tensorweight_scale = amax / 448. This is the same algorithm as vLLM's online--quantization fp8. - Activations are quantized dynamically. On Ampere GPUs vLLM runs these layers as Marlin W8A16.
- Kept in BF16: the vision encoder, embeddings, lm_head, all norms, the mHC routers (FP32), loop_gate, and the linear-attention conv1d/A_log/dt_bias.
quantization_config:{"quant_method": "fp8", "activation_scheme": "dynamic", "ignored_layers": [vision linears]}- Size: 11.3 GiB (the BF16 original is about 19 GiB).
- Export script: tools/export_fp8.py
Running
Stock vLLM does not support the Loop architecture yet. Use the patches at igerry/vllm-sensenova-loop (vLLM 0.33.0). You need commit e077110 or later: earlier versions of patch_fp8.py reject this checkpoint because of the attention k/v/q/prob_scale parameters.
./install.sh /path/to/venv/bin/python /models/SenseNova-Loop-9B # applies the patches
QUANT= PY=/path/to/venv/bin/python ./serve.sh /models/SenseNova-Loop-9B-FP8
Results (2脳 RTX 3090 Ti, TP=2, MTP with 2 draft tokens)
This checkpoint, loaded as-is with --attention-backend FLASHINFER:
| This checkpoint | Online --quantization fp8 |
|
|---|---|---|
| Decode tok/s, greedy (code and web generation, thinking on) | 135.2 / 136.1 | 138.2 / 137.9 |
| GSM8K, first 200 (greedy, thinking off) | 190/200 | 188/200 |
| Decode tok/s at 1k / 97k context | 110.6 / 93.6 (sampled) | 123.5 / 101.6 (sampled) |
The sampled numbers vary by 卤5% from run to run, so compare the greedy rows. Earlier comparison against BF16 and AWQ, measured with online FP8 and the default FlashAttention 2 backend:
| BF16 | FP8 | |
|---|---|---|
| Decode tok/s (code and web generation, thinking on) | 85.6 | 121.8 |
| GSM8K, first 200 (greedy, thinking off) | 189/200 | 189/200 |
| Weights per GPU | 8.9 GB | 6.0 GB |
Image input works.
License
Apache-2.0, same as the base model.
- Downloads last month
- 5
Model tree for igerry/SenseNova-Loop-9B-FP8
Base model
sensenova/SenseNova-Loop-9B