Hy-MT2-7B-NVFP4

NVFP4 W4A4 quantization of tencent/Hy-MT2-7B for vLLM on Blackwell GPUs, calibrated on European-language translation pairs. It fits a 12 GB card such as the RTX 5070 Ti Laptop and leaves 5.5 GiB for KV cache under an 11.5 GB budget.

• Weights take 4.7 GB in vLLM. The official FP8 takes 7.6 GB and BF16 takes 15 GB.
• COMET-22 on WMT24++ EU pairs is 84.66, against 84.70 for BF16. That's a drop of 0.04.
• Batched short-cell translation runs at 213.4 cells/s with vLLM defaults, 1.7× the official FP8's 126.5 cells/s on the same GPU and 11.5 GB budget. Raising the batch limits gets 234.2 cells/s.

Of the models I tested for batch translation of spreadsheet cells on a 12 GB laptop, this is the one I'd use. On a 1,800-segment subset served through vLLM it scored 84.90 COMET-22, against 84.94 for GPT-6-luna. The next-best local model on quality, Index-Translate-9B-NVFP4, is 5× slower on the same cells.

Every attention and MLP linear layer is NVFP4, which means FP4 weights and activations in groups of 16 with FP8 scales. lm_head and embeddings stay in BF16.

Calibration. 576 OPUS-100 pairs, EN→CS/DE/ES/FR/IT/NL/PL/PT and PL→EN, 1 to 5 sentences each, rendered with the Hy-MT2 prompt followed by the reference translation. The recipe and scripts are in quantization/.

Quality

Test set. All non-canary segments of WMT24++, google/wmt24pp. That's 960 per direction and 8,640 in total, covering news, social, speech and literary text. EN→X uses the original English source. PL→EN reverses the EN→PL pair, so the Polish post-edited reference becomes the source and the original English becomes the reference.

Metrics. COMET-22 with Unbabel/wmt22-comet-da, scaled ×100, and corpus chrF from sacrebleu.

Setup. Greedy decoding on vLLM 0.26.0. One desktop RTX 5070 Ti 16 GB, capped at an 11.5 GB budget with gpu_memory_utilization ≈ 0.74 to mimic a 12 GB laptop GPU.

	EN→CS	EN→DE	EN→ES	EN→FR	EN→IT	EN→NL	EN→PL	EN→PT	PL→EN	All	chrF
BF16 reference on 2 GPUs	87.31	83.73	85.16	82.57	85.97	85.16	85.48	83.77	83.17	84.70	59.3
tencent/Hy-MT2-7B-FP8	87.25	83.71	85.20	82.55	86.00	85.13	85.39	83.79	83.09	84.68	59.3
this model, NVFP4	87.15	83.69	85.22	82.57	85.82	85.15	85.58	83.70	83.08	84.66	59.1
this model with keep-structure prompt	87.20	83.65	85.20	82.45	85.75	85.26	85.50	83.57	82.81	84.60	59.5

Structure test. 175 Excel-style cells in each of PL, DE, FR, ES and CS, with 270 tokens that must survive verbatim: {placeholders}, %s, %1$s, {{var}}, ${var}, HTML tags, URLs, e-mails, SKUs, inline code and line breaks. Each cell shows protected tokens kept / cells fully intact.

	card prompt	keep-structure prompt
tencent/Hy-MT2-7B-FP8	94.1% / 90.9%	not run
this model	94.1% / 90.9%	99.3% / 98.9%

With the base model card's prompt, Hy-MT2 tends to merge line breaks inside a cell. FP8 and NVFP4 both do it, so the cause is the base model and not the quantization. One extra sentence in the prompt fixes it. Fully intact cells go from 90.9% to 98.9%, and COMET-22 moves from 84.66 to 84.60. The usage example below includes that sentence.

Batched throughput

4,000 short cells, EN→X and 1 to 40 words each, submitted in one batch on a desktop RTX 5070 Ti with an 11.5 GB budget. Measured after warm-up.

	weights	KV cache left	cells/s
tencent/Hy-MT2-7B-FP8, vLLM defaults	7.6 GB	3.0 GiB	126.5
this model, --max-num-seqs 1024 --max-num-batched-tokens 32768	4.7 GB	5.5 GiB	234.2
this model, vLLM defaults	4.7 GB	5.5 GiB	213.4

The laptop RTX 5070 Ti has about 0.75× the memory bandwidth of the desktop card, so expect lower numbers there.

Usage with vLLM

Needs a Blackwell GPU, SM 10.x or 12.x, for NVFP4. Tested with vLLM 0.26.0.

vllm serve kurcontko/Hy-MT2-7B-NVFP4 --max-model-len 4096 --gpu-memory-utilization 0.9 \
  --max-num-seqs 1024 --max-num-batched-tokens 32768

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
prompt = ("Translate the following text into Polish. Keep placeholders, tags, URLs, codes, numbers and line breaks "
          "exactly as in the source. Note that you should only output the translated result without any additional "
          "explanation:\n\n" + "Warning: do not exceed 230 V.\nDisconnect power before cleaning.")
r = client.chat.completions.create(model="kurcontko/Hy-MT2-7B-NVFP4", temperature=0,
                                   messages=[{"role": "user", "content": prompt}])
print(r.choices[0].message.content)

Use a single user turn, no system prompt, and full language names, as the base model card does. The "Keep placeholders..." sentence is optional. Adding it gives the keep-structure prompt from the tables above.

The base card's recommended sampling also works: temperature 0.7, top_p 0.6, top_k 20, repetition_penalty 1.05. Every number on this card uses greedy decoding.

If the engine fails at start-up while JIT-compiling FlashInfer's sampling kernels with CUDA compiler and CUDA toolkit headers are incompatible, set VLLM_USE_FLASHINFER_SAMPLER=0.

Limitations

• Evaluated on the 9 directions above only. Hy-MT2 covers 33 languages and does not list Swedish, Danish, Finnish, Hungarian, Romanian, Greek, Bulgarian or Slovak.
• COMET-22 and chrF only. No human evaluation.
• The model sometimes translates status labels such as [PENDING].

License

Apache-2.0, same as tencent/Hy-MT2-7B. Calibration data is OPUS-100.