RakutenAI-3.0-NVFP4

NVFP4 quantization of Rakuten/RakutenAI-3.0 (671B MoE, DeepSeek-V3 architecture, MLA attention, 256 routed experts, MTP) with Four Over Six adaptive block scaling (arXiv:2512.02010).

• Weights: NVFP4 (FP4 values, FP8-e4m3 block scales, group size 16, FP32 global scale).
• KV cache: FP8. Attention MLA projections, MoE gates, lm_head and the MTP layer are kept in high precision.
• Standard NVFP4 checkpoint layout (ModelOpt format) — loads in SGLang (--quantization modelopt_fp4) and TensorRT-LLM on Blackwell GPUs without any code change.
• Calibration: a mixed Japanese/English calibration set including identity/safety prompts, instruction data, Wikipedia and news text, 820 samples.

Quantization method

Four Over Six only changes how the per-block weight scale is chosen. For every 16-element weight block the scale target is picked between amax/6 and amax/4 by minimum squared error; 44.8% of blocks selected the amax/4 target. This lowers weight quantization MSE by 16.6% compared to plain ModelOpt NVFP4 round-to-nearest, while the packed bytes, scale tensors and metadata stay a standard NVFP4 checkpoint — no custom kernel, loader or runtime patch is required.

How to run (sglang)

Two serving paths, both verified:

Blackwell (SM100+) — native NVFP4 (W4A4). Verified on 8×B300 with tensor parallel 8:

python3 -m sglang.launch_server \
  --model-path rakuten-junliu/RakutenAI-3.0-NVFP4 \
  --quantization modelopt_fp4 \
  --tp 8 \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --port 30000

Hopper/Ampere (SM80–SM90) — Marlin kernels (W4A16). Weights stay 4-bit packed; compute runs in BF16, so this is not native NVFP4 execution, but the same checkpoint loads unmodified. Requires a recent sglang build with the Marlin FP4 backends. Verified on 8×H100-80GB:

python3 -m sglang.launch_server \
  --model-path rakuten-junliu/RakutenAI-3.0-NVFP4 \
  --quantization modelopt_fp4 \
  --fp4-gemm-backend marlin \
  --moe-runner-backend marlin \
  --tp 8 \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --port 30000

Query the server

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "あなたは誰ですか?"}],
    "max_tokens": 256,
    "temperature": 0
  }'

With the OpenAI Python client:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Explain mixture-of-experts routing in two sentences."}],
    max_tokens=256,
    temperature=0,
)
print(resp.choices[0].message.content)

Hardware requirements

Item	Value
Native NVFP4 (W4A4)	Blackwell, SM100+ (B200 / B300)
Marlin fallback (W4A16)	SM80–SM90 (A100 / H100 / H200), verified on 8×H100-80GB
Checkpoint size	~368 GB
Tested topology	8 GPUs, --tp 8

Leave headroom for the KV cache: --mem-fraction-static 0.85 was used in all evaluation runs.

Evaluation

Measured with sglang TP8, greedy decoding.

Model	GSM8K (5-shot)	MMLU
FP8 baseline	94.39	85.27
RakutenAI-3.0-NVFP4 (Four Over Six)	95.00	84.69

License: Apache 2.0, same as the base model.