Index-Translate-2B-NVFP4

NVFP4 W4A4 quantization of IndexTeam/Index-Translate-2B for vLLM on Blackwell GPUs, built for high-volume batch translation.

• With 512 concurrent sequences it translates 268 short cells/s. The official FP8 tops out at about 205 cells/s under the same 11.5 GB budget.
• COMET-22 on WMT24++ EU pairs is 82.21, against 82.82 for BF16 and 82.24 for the official FP4.
• Weights take 1.8 GiB in vLLM with text-only serving. The official FP4 takes 2.3 GiB and BF16 takes 3.6 GiB.
• It keeps 97.0% of placeholders, tags and line breaks in Excel-style cells, the same as BF16.

The extra speed comes from memory. Each Gated DeltaNet layer, GDN for short, keeps a fixed-size state for every running sequence, so the number of sequences vLLM can run at once depends on the memory left after the weights. At vLLM's usual 128 sequences this build and the official FP8 run at the same speed, 144 and 148 cells/s. Above that the FP8 runs out of room and this one keeps going.

If you translate a few texts at a time, use the official Index-Translate-2B-FP8. It loses only 0.03 COMET.

What is quantized. Every language-model linear layer is NVFP4, which means FP4 weights and activations in groups of 16 with FP8 scales. That covers MLP, full attention, and the GDN in_proj_qkv, in_proj_z and out_proj. vLLM fuses in_proj_qkv and in_proj_z into one GEMM with a single global scale, so those two share one NVFP4 global scale here. The official FP4 keeps all GDN projections in BF16, which is why it is larger.

These stay in BF16:

• lm_head and embeddings
• the 32-wide GDN gate projections in_proj_a/b
• the vision tower and the MTP head, both copied unchanged from the base model

Calibration. 576 OPUS-100 pairs, EN→CS/DE/ES/FR/IT/NL/PL/PT and PL→EN, each rendered with the Index-Translate prompt followed by the reference translation. GPTQ chooses the weight rounding. The recipe and scripts are in quantization/ and target llm-compressor 0.15.

Quality

Test set. All non-canary segments of WMT24++, google/wmt24pp. That's 960 per direction and 8,640 in total, covering news, social, speech and literary text. EN→X uses the original English source. PL→EN reverses the EN→PL pair, so the Polish post-edited reference becomes the source and the original English becomes the reference.

Metrics. COMET-22 with Unbabel/wmt22-comet-da, scaled ×100, and corpus chrF from sacrebleu.

Setup. Greedy decoding on vLLM 0.26.0. One desktop RTX 5070 Ti 16 GB, capped at an 11.5 GB budget with gpu_memory_utilization ≈ 0.74 to mimic a 12 GB laptop GPU.

	EN→CS	EN→DE	EN→ES	EN→FR	EN→IT	EN→NL	EN→PL	EN→PT	PL→EN	All	chrF
BF16 reference	83.01	82.15	83.61	81.63	84.07	83.84	83.26	82.66	81.14	82.82	58.7
IndexTeam/Index-Translate-2B-FP8	82.94	82.15	83.66	81.57	83.99	83.83	83.23	82.61	81.15	82.79	58.6
IndexTeam/Index-Translate-2B-FP4	81.47	81.73	83.28	81.07	83.88	83.13	82.35	82.47	80.80	82.24	57.9
this model, NVFP4 GPTQ	81.39	81.69	83.52	81.02	83.91	83.15	82.15	82.40	80.71	82.21	58.0

Czech loses the most, 1.6 points, in both 4-bit builds.

Structure test. 175 Excel-style cells in each of PL, DE, FR, ES and CS, with 270 tokens that must survive verbatim: {placeholders}, %s, %1$s, {{var}}, ${var}, HTML tags, URLs, e-mails, SKUs, inline code and line breaks. BF16, the official FP8 and this build all keep 97.0% of protected tokens and leave 95.4% of cells fully intact. The official FP4 keeps 97.8%. The only misses are [PENDING]-style status labels, which the model translates.

Throughput

4,000 short cells, EN→X and 1 to 40 words each, submitted in one batch with an 11.5 GB budget.

build	128 sequences	256 sequences	512 sequences
BF16	142	not run	not run
IndexTeam/Index-Translate-2B-FP8	148	205	202
IndexTeam/Index-Translate-2B-FP4	134	not run	not run
this model	144	243	268

The 128-sequence column ran on one RTX 5070 Ti and the rest on a second one, which measured about 5% faster on the same workload.

On a 12 GB card, Hy-MT2-7B-NVFP4 scores 84.66 COMET-22 at 213 cells/s and is the better pick for the languages it covers. This model covers more languages and fits in less memory.

Usage with vLLM

Needs a Blackwell GPU for NVFP4. Tested with vLLM 0.26.0, text only.

vllm serve kurcontko/Index-Translate-2B-NVFP4 --language-model-only --max-model-len 4096 \
  --gpu-memory-utilization 0.9 --max-num-seqs 512

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
prompt = "请将以下English文本翻译为Polish，直接输出翻译结果，不要进行任何解释。\n\n" + "Hello {name}, your order has shipped."
r = client.chat.completions.create(model="kurcontko/Index-Translate-2B-NVFP4", temperature=0,
        messages=[{"role": "user", "content": prompt}],
        extra_body={"chat_template_kwargs": {"enable_thinking": False}})
print(r.choices[0].message.content)

The prompt template comes from bilibili/Index-Translate. Always pass enable_thinking: false. If vLLM reports that max_num_seqs exceeds the available Mamba cache blocks, lower --max-num-seqs.

If the engine fails at start-up while JIT-compiling FlashInfer's sampling kernels with CUDA compiler and CUDA toolkit headers are incompatible, set VLLM_USE_FLASHINFER_SAMPLER=0.

Limitations

• Evaluated on the 9 directions above only, with COMET-22 and chrF. No human evaluation.
• Image translation and MTP speculative decoding are untested with this checkpoint.

License

Apache-2.0, same as IndexTeam/Index-Translate-2B. Calibration data is OPUS-100.