neural-nova/Qwen3-235B-A22B-Instruct-2507-optimized

---

Qwen3-235B-A22B — Tuned vLLM Serving Recipe

Docker build, launch script, and benchmark-reproduction script for serving Qwen/Qwen3-235B-A22B with vLLM.

This repository does not contain model weights. vllm serve pulls them directly from the upstream repo above (and caches them) when the server starts.

Model Overview

• Sparse MoE causal LM, 235B total parameters / 22B active per token
• 128 experts total, 8 active per token
• 94 layers, hidden size 4,096, 64 query heads / 4 KV heads (GQA)
• Native context 32,768 tokens (131,072 with YaRN)

Environment

Hardware	8x NVIDIA H100-80GB, single node
Input Length	409–3,686 tokens · nominal 2,048 · uniform
Output Length	51–460 tokens · nominal 256 · uniform
Dataset	random (synthetic) · range ratio 0.8
Base Docker image	vllm/vllm-openai:v0.21.0
vLLM version	v0.21.0
Parallelism	Tensor-parallel 8, mp distributed executor
Features	Chunked prefill, prefix caching, Triton MoE backend
Prompts Measured	1,000 · warm-ups excluded: 10
Request Rate	inf · max concurrency is the only throttle
Tested Concurrency	512
Precision	BF16

Optimization Changes

• Capped --max-num-batched-tokens at 8,192 to convert unused context budget into KV-cache slots
• --gpu-memory-utilization 0.90
• --max-num-seqs 512 for deeper batching
• Enabled chunked prefill and prefix caching
• Triton MoE backend with the mp distributed executor
• VLLM_WORKER_MULTIPROC_METHOD=spawn, CUDA_DEVICE_MAX_CONNECTIONS=1

1. Docker

Installation

Build the image:

docker build -t your-org/qwen3-235b-a22b-vllm:latest .

Run it (weights download into a cache volume on first start):

docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  your-org/qwen3-235b-a22b-vllm:latest optimized

Pass baseline instead of optimized as the final argument to run the unoptimized configuration used for the comparison below.

Baseline

vllm serve Qwen/Qwen3-235B-A22B
--host 0.0.0.0 --port 8000
--trust-remote-code
--tensor-parallel-size 8

Optimized

export VLLM_WORKER_MULTIPROC_METHOD=spawn export CUDA_DEVICE_MAX_CONNECTIONS=1

vllm serve Qwen/Qwen3-235B-A22B
--tensor-parallel-size 8
--gpu-memory-utilization 0.90
--max-num-batched-tokens 8192
--max-num-seqs 512
--enable-chunked-prefill
--enable-prefix-caching
--max-model-len 8192
--distributed-executor-backend mp
--moe-backend triton

Validation

Once the container is up, confirm the server is healthy and serving:

curl -sf http://localhost:8000/health && echo "server is healthy"

curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-235B-A22B", "prompt": "The capital of France is", "max_tokens": 16}'

The completions call should return generated text once the server has finished loading and warming up (this is a 235B model — first load can take several minutes).

Files in this repository

File	Purpose
Dockerfile	vllm/vllm-openai:v0.21.0-based image with the two scripts below baked in
launch_server.sh	Starts vllm serve with the baseline or optimized flags
run_benchmark.sh	Starts baseline + optimized servers, benchmarks each, and prints the comparison table below — fully automated
README.md	This model card

2. Shell Script

Launch Server

./launch_server.sh optimized   # or: ./launch_server.sh baseline

This is also the image's default command — docker run ... <image> optimized runs the same thing. Inside a running container, invoke it with docker exec -it <container> ./launch_server.sh <mode>.

Reproduce Benchmark

This is a separate container invocation from "Launch Server" above — it overrides the entrypoint so run_benchmark.sh (not launch_server.sh) drives the container. It starts the baseline server itself, benchmarks it, shuts it down, starts the optimized server, benchmarks that, shuts it down, and prints the comparison table — nothing to copy in by hand:

docker run --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint ./run_benchmark.sh \
  your-org/qwen3-235b-a22b-vllm:latest both

That prints the full table in the terminal when it finishes (large models — expect this to take a while: two model loads plus two 1,000-prompt runs). Pass baseline or optimized instead of both to run just one side. The raw per-run JSON (benchmark_baseline.json, benchmark_optimized.json) and each server's startup log (server_baseline.log, server_optimized.log) are left in the container's /workspace if you want them too:

docker cp <container>:/workspace/benchmark_optimized.json .

Methodology (vllm bench serve):

• Dataset: random (synthetic), range ratio 0.8
• Input length: 409–3,686 tokens (nominal 2,048, uniform)
• Output length: 51–460 tokens (nominal 256, uniform)
• 1,000 measured prompts, 10 warm-up prompts (discarded, not saved)
• Request rate: unbounded (inf) — max concurrency is the only throttle
• Max concurrency: 512 (override with -e MAX_CONCURRENCY=...)

Results

8x H100-80GB, vLLM v0.21.0. This is exactly the table run_benchmark.sh both prints — running it is the only step required to reproduce it:

Metric	Baseline	Optimized
Request throughput (req/s)	2.92	6.98
Output tok/s	759.0	1812.0
TTFT mean (ms)	4304.4	3786.8
TTFT p99 (ms)	16379.8	105526.9
TPOT mean (ms)	146.3	53.9
TPOT p99 (ms)	195.1	72.6
ITL mean (ms)	144.3	53.2
ITL p99 (ms)	535.7	240.6
E2E mean (ms)	41766.3	17589.2
E2E median (ms)	40450.2	17168.5
Duration (s)	342.1	143.3

At max concurrency 512, TTFT will reflect queuing delay on top of the model's actual first-token latency — expect it to be much higher than per-request TTFT under light load.