iAcloud/gemma-4-31B-it-rbln-tp8

gemma-4-31B-it — RBLN (ATOM CA25) compiled

google/gemma-4-31B-it (multimodal, image-text-to-text) pre-compiled for Rebellions RBLN-CA25 NPUs with optimum-rbln, ready to serve with vllm-rbln. This repo contains the compiled artifact only — you need RBLN NPUs and the RBLN SDK to run it.

Hardware & software

NPU	8× RBLN-CA25 (language model: 8 devices, vision tower: 8 devices, shared)
Compiler	rebel-compiler==0.11.0
Runtime	optimum-rbln==0.11.0.post1, vllm-rbln==0.11.0, transformers==5.8.1

Compile configuration

Option	Value
max_seq_len	131,072
batch_size	4
decoder_batch_sizes (dynamic batch)	[1, 2, 4]
attn_impl / kvcache_partition_len	flash_attn / 16384
cache_impl / sliding_window	hybrid / 1024
prefill_chunk_size	128
kvcache_num_blocks	23 (≈377k shared KV tokens → ~2 concurrent full-131k sequences, or 4× ~94k)
vision max_soft_tokens / image prefill	280 / image_prefill_384

kvcache_num_blocks is 23, the maximum that actually loads on 8 devices: auto sizing fills every device to ~15.7 GiB with the language model alone, leaving no room for the vision tower + multimodal projector runtimes on the same devices at serve time. 23 was determined with the SDK's own per-chiplet estimator against a budget reduced by 512 MB/device reserved for the vision-side runtimes.

Serving

vllm serve <this-repo-or-local-path> \
  --max-num-seqs 4 \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

Notes:

• --max-num-seqs 4 is required — vllm-rbln defaults it to 1, and the compiled decoder tops out at batch 4.
• Do not pass --tensor-parallel-size; the 8-device split is baked into the compiled model (vllm-rbln forces TP=1).
• Reasoning (thinking) is per-request: send chat_template_kwargs={"enable_thinking": true}; reasoning text is returned in message.reasoning_content.

License

Derived from google/gemma-4-31B-it; use is subject to the Gemma Terms of Use.