Qwen3-4B-Batch8-RBLN-0.11.0.post1

Qwen/Qwen3-4B (bf16) pre-compiled for Rebellions RBLN-CA25 NPUs with optimum-rbln, ready to serve with vllm-rbln. This repository contains the compiled artifact only. RBLN NPUs and the RBLN SDK are required to run it.

Hardware and software

Component	Configuration
NPU	4× RBLN-CA25 (tensor parallel 4)
Compiler	rebel-compiler==0.11.0.post1
Runtime	optimum-rbln==0.11.0.post1, vllm-rbln==0.11.0, transformers==5.8.1, torch==2.11.0+cpu

Compile configuration

Option	Value
max_seq_len	40,960
batch_size	8
num_devices	4
attn_impl / kvcache_partition_len	flash_attn / 8,192

Compiled from the bf16 checkpoint. The official Qwen/Qwen3-4B-FP8 stores 128×128 block FP8 scales with dynamic activation scaling, which optimum-rbln==0.11.0.post1 cannot load.

These options are the RBLN Model Zoo (v0.11.0.post1) example for Qwen/Qwen3-4B with batch_size raised to 8.

This repository and iAcloud/Qwen3-4B-RBLN-0.11.0.post1 differ only in batch_size (an A/B pair for dynamic batching throughput tests).

genia_compile_manifest.json records the exact compile options and package versions.

Measured throughput (dynamic batching A/B)

Measured on 4× RBLN-CA25 through an OpenAI-compatible gateway: non-streaming, 8 concurrent clients for 600 s per variant, max_tokens 256, temperature 0, the same prompt for every request (every response reached the 256-token limit). Throughput = output tokens of successful requests ÷ measured time.

Variant	Requests (failed)	Output tokens	Measured time	Throughput
Qwen3-4B-RBLN-0.11.0.post1	241 (0)	61,696	620.2 s	99.5 tok/s
Qwen3-4B-Batch8-RBLN-0.11.0.post1	1,040 (0)	266,240	601.5 s	442.6 tok/s

Batch 8 delivers 4.45× the batch-1 throughput (+345.0%).

Serving

vllm serve <this-repo-or-local-path>

vllm-rbln reads the batch size, maximum length and device count from rbln_config.json. Do not pass --tensor-parallel-size; the 4-device split is already baked into the compiled model. The artifact requires the runtime versions above (vllm-rbln==0.11.0).

License

Derived from Qwen/Qwen3-4B (Apache-2.0); use is subject to the original model's license and terms.