henry1477/translategemma-12b-it-NVFP4

NVFP4 quantization of google/translategemma-12b-it (revision d1b225e1caa17f1ddc7e62065d8637d0923f34e2), produced with llm-compressor and intended for vLLM on Blackwell.

NVFP4 (W4A4): 4-bit weights and 4-bit activations, both group size 16. Faster under load than the A16 variant, but see the structural-pass caveat below before using it for format-constrained generation.

What is different about this checkpoint

• Text-only. The SigLIP vision tower and multimodal projector are removed, not merely ignored: 12.19B -> 11.77B parameters, and the architecture is Gemma3ForCausalLM. Subtitle translation never uses the image path, and dropping it returns ~0.8 GiB of VRAM to the KV cache, which is the binding constraint on concurrency for this model on a 16 GB card.
• In-domain calibration (W4A4 variant only): activation scales are calibrated on real subtitle-translation prompts -- JSON arrays of dialogue cues in the exact shape the decoder receives in production -- rather than generic English chat data. A multilingual translation model calibrated on English assistant chat is calibrated on the wrong distribution. Note this predates the 30-language GPTQ recipe used for the A16 checkpoint in this collection; the calibration set here is Spanish-sourced only.
• lm_head is excluded from quantization (Gemma3 ties it to the embedding table).

Measured results

Corpus: 120 real Spanish subtitle cues (6 scenes of 20), es->en, on an RTX 5070 Ti (sm_120, 16 GB). "Structural pass" is the fraction of scenes whose output is a valid JSON array carrying every requested cue id -- the contract a subtitle pipeline needs in order to map translations back onto timings. "chrF++" is measured against a bf16-adjacent int8 decode of the same cues.

The transformers rows are single-stream (batch 1); the vLLM rows are aggregate throughput across a batched run (6 scenes for the W4A4 rows, 30 for the A16 rows). They are therefore not a like-for-like latency comparison -- the honest summary is that the vLLM+NVFP4 path turns a stage that was bottlenecked on fp32 CPU-offloaded layers into one that saturates the GPU.

Variant	tok/s	Structural pass	chrF++
int8 + fp32 CPU offload (transformers)	2.9	-	reference
NF4 all-on-GPU (transformers)	12.3	50%	80.8
NVFP4A16 (W4A16), GPTQ + 30-lang calibration	310	100%	84.6
NVFP4A16 (W4A16), data-free RTN	328	77%	84.4
NVFP4 (W4A4), this recipe, in-domain calibrated	97.6	33%	n/a
NVFP4 (W4A4), a public uncalibrated quant	97.5	0%	n/a

chrF++ is not reported for the W4A4 rows: with most scenes failing to parse there are too few recovered cues to score meaningfully. Where W4A4 output did parse, the translations themselves were good -- the failure is structural, not linguistic.

The W4A4 caveat matters. Both W4A4 variants produced fluent, accurate translations while dropping the requested JSON structure -- cue ids simply gone. A publicly available uncalibrated quant lost it on 6 of 6 scenes; the in-domain calibrated build in this collection recovers only part of the gap (2 of 6). Weight precision is identical across W4A4 and W4A16 and the throughput difference is under 5%, so quantizing activations to 4 bits buys almost nothing here and costs the output contract.

Calibration data is therefore worth something but is not a fix: it moved structural pass from 0% to 33% while W4A16 reaches 83% on the same corpus, same base revision, same remap. If your use depends on a specific output format, either constrain the decoder (see below) or use the A16 variant. A throughput-only benchmark will not catch this -- under free decoding W4A4 looks marginally faster right up until nothing parses.

Grammar-constrained decoding changes the structural picture

The structural-pass column above is measured with free decoding. vLLM can instead constrain generation to a JSON schema (structured_outputs=), which makes malformed output impossible rather than detected after the fact. Pinning every array position to its cue id -- so the decoder chooses only the translation text -- was measured on the same 30-scene Spanish track (581 cues), on both checkpoints, back-to-back in a single process:

Checkpoint	Decoding	Structural pass	Cues recovered	tok/s	chrF++ vs A16 free
NVFP4A16	free	30/30 (100%)	581	364	reference
NVFP4A16	pinned schema	30/30 (100%)	581	341	98.9
NVFP4 (W4A4)	free	10/30 (33%)	200	341	83.9
NVFP4 (W4A4)	pinned schema	30/30 (100%)	581	321	83.3

Three things follow:

• Constraining costs about 6% throughput and is essentially output-neutral on the A16 checkpoint: 563 of 581 cues come back byte-identical to the free decode (chrF++ 98.9). It is a guarantee, not a distortion.
• It removes the W4A4 structural failure entirely -- 33% -> 100%, with all 581 cues recovered instead of 200. That failure is a decoding-time formatting problem, not lost capability.
• Constraining does not paper over a quality gap, and does not close one either: W4A4's agreement with the A16 decode is about the same whether free (83.9, over only the 200 cues it managed) or pinned (83.3, over all 581), and both stay far below A16's 98.9 agreement with itself.

The free-decode structural numbers in the table above are therefore a property of the checkpoint and the decoding setup, not of the weights alone. Absolute tok/s in this table is not comparable with the one above -- the two runs differ in compile-cache warmth -- so compare within this table only.

For this checkpoint constrained decoding is the difference between usable and not, if you need structured output. The caveat above was measured with free decoding; constrain the decoder and it no longer applies. What remains is the quality gap against the A16 checkpoint, which constraining does not address.

Usage (vLLM)

from vllm import LLM, SamplingParams
llm = LLM(model="henry1477/translategemma-12b-it-NVFP4", max_model_len=4096, gpu_memory_utilization=0.85)

To constrain the output to a JSON schema (see the section above), pin each array position to the id you asked for so the decoder chooses only the text:

from vllm.sampling_params import StructuredOutputsParams

schema = {"type": "array", "minItems": len(ids), "maxItems": len(ids),
          "prefixItems": [{"type": "object", "additionalProperties": False,
                           "required": ["id", "text"],
                           "properties": {"id": {"const": i},
                                          "text": {"type": "string", "minLength": 1}}}
                         for i in ids],
          "items": False}
params = SamplingParams(max_tokens=1024, temperature=0.0,
                        structured_outputs=StructuredOutputsParams(json=schema))

On consumer Blackwell (sm_120: RTX 5070 Ti / 5080 / 5090, RTX PRO 6000) set these before importing vllm, or the FP4 GEMM raises an illegal instruction:

export FLASHINFER_CUDA_ARCH_LIST=12.0f
export FLASHINFER_FORCE_SM=120f
export FLASHINFER_DISABLE_VERSION_CHECK=1

Verify the engine log selects CutlassNvFp4LinearKernel. If it selects a Marlin NVFP4 path instead, output can be silently empty on sm_12x.

FlashInfer JIT-compiles kernels on first use, so the container needs ninja and a reachable nvcc; CUDA_HOME must point at vLLM's bundled CUDA (site-packages/nvidia/cu13) on a CUDA runtime base image.

Note that in plain transformers there is no FP4 kernel, so this checkpoint is slower than bf16 outside vLLM.

Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: NVFP4
      bypass_divisibility_checks: false
      requires_calibration_data: true

License

Governed by the Gemma Terms of Use; you must accept them on the base model before use. This is a derivative of google/translategemma-12b-it @ d1b225e1caa17f1ddc7e62065d8637d0923f34e2.