Gemma4-31B-Int8-RBLN-0.11.2

google/gemma-4-31B-it (multimodal, image-text-to-text), using the INT8 checkpoint from lokeshe09/gemma-4-31B-it-INT8, pre-compiled for Rebellions RBLN-CA25 NPUs with optimum-rbln and ready to serve with vllm-rbln.

This repository contains the compiled artifact only. RBLN NPUs and the RBLN SDK are required to run it.

Hardware and software

Component	Configuration
NPU	16× RBLN-CA25 (language model: 16 devices; vision tower and multimodal projector: device 0, shared)
Compiler	rebel-compiler==0.11.2
Runtime	optimum-rbln==0.11.2, vllm-rbln==0.11.2.post1, vllm==0.24.0, transformers==5.8.1
Quantization	INT8 weights / FP16 activations (W8A16, weight-only)

Compile configuration

Option	Value
max_seq_len	262,144
batch_size	8
decoder_batch_sizes	[1, 2, 4, 8]
attn_impl / kvcache_partition_len	flash_attn / 16,384
cache_impl / sliding_window	hybrid / 1,024
prefill_chunk_size	128
kvcache_num_blocks	36 (589,824 shared KV tokens)
Vision max_soft_tokens / image prefill	280 / image_prefill_384

The 36 KV blocks were selected by the SDK estimator with memory_budget=0.9. The shared KV capacity is approximately 2.25 full 262K contexts; shorter requests can use the compiled dynamic batches up to batch 8.

Serving

export VLLM_PLUGINS=rbln
export VLLM_RBLN_USE_VLLM_MODEL=0
export VLLM_RBLN_NUM_DEVICES_PER_LOCAL_RANK=16
export VLLM_RBLN_SUB_BLOCK_CACHE=false

vllm serve <this-repo-or-local-path> \
  --dtype float32 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 128 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

Notes:

• Do not pass --tensor-parallel-size; the 16-device split is baked into the compiled language model and vllm-rbln runs it with TP=1.
• Keep --max-num-seqs 8 and --max-num-batched-tokens 128 aligned with the compiled decoder and prefill graphs.
• Prefix caching must remain disabled. Gemma4 uses multi-group hybrid KV caches, which sub-block prefix caching does not support in this SDK version.
• Thinking is controlled per request with chat_template_kwargs={"enable_thinking": true}. Parsed reasoning is returned separately from the final response content.
• The artifact accepts one image per prompt with the compiled vision profile.

License

Derived from google/gemma-4-31B-it. Use is subject to the Gemma Terms of Use.