neural-nova/GLM-5.2-Instruct-Optimized

GLM-5.2 — Tuned vLLM Serving Recipe (ROCm)

Docker build, launch script, and benchmark-reproduction script for serving zai-org/GLM-5.2 with vLLM on AMD Instinct GPUs.

This repository does not contain model weights. vllm serve pulls them directly from the upstream repo above (and caches them) when the server starts.

Model Overview

• MoE causal LM using MLA (multi-head latent attention) with dynamic sparse attention
• Served at 65,536 max context; requires --trust-remote-code
• Workload is prefill-dominated and compute-bound

Environment

Hardware	8x AMD Instinct MI325X 256GB, single node
Software	ROCm 7.2.3 · vLLM v0.25.1+rocm723
Input Length	409–3,686 tokens · nominal 2,048 · uniform
Output Length	51–460 tokens · nominal 256 · uniform
Framework	vllm bench serve
vLLM	0.25.1
Dataset	random (synthetic) · range ratio 0.8
Base Docker image	vllm/vllm-openai-rocm:v0.25.1
Parallelism	Tensor-parallel 8, pipeline-parallel 1, mp distributed executor
Attention	ROCM_AITER_MLA_SPARSE decode backend, ROCM_AITER_FA MLA prefill
Features	AITER kernels, FP8 KV cache, prefix caching, async scheduling, CUDA graphs
Prompts Measured	1,000 · warm-ups excluded: 10
Request Rate	inf
Tested Concurrency	512

Optimization Changes

• FP8 (e4m3) KV cache
• --gpu-memory-utilization lowered 0.95 → 0.90 to leave headroom for graph capture
• --max-num-seqs 256, --max-num-batched-tokens 16384
• --long-prefill-token-threshold 4096 so long prefills stop blocking decode
• Enabled prefix caching and async scheduling
• --compilation-config with cudagraph_mode FULL_AND_PIECEWISE, max capture size 256
• VLLM_ROCM_USE_AITER=1, VLLM_WORKER_MULTIPROC_METHOD=spawn
• Speculative decoding deliberately omitted — MTP is the only speculator shipped with GLM-5.2, and it deadlocks on gfx942 at TP8 (vllm-project/vllm#48568)

1. Docker

Installation

Build the image:

docker build -t your-org/glm-5.2-vllm:latest .

Run it (ROCm containers use --device/--group-add, not --gpus):

docker run \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --ipc=host --shm-size 16g \
  --security-opt seccomp=unconfined \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  your-org/glm-5.2-vllm:latest optimized

Pass baseline instead of optimized as the final argument to run the unoptimized configuration used for the comparison below.

Baseline

export VLLM_ROCM_USE_AITER=1 vllm serve zai-org/GLM-5.2 \ --tensor-parallel-size 8 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len 65536

Optimized

xport VLLM_ROCM_USE_AITER=1 export VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve zai-org/GLM-5.2 \ --host 0.0.0.0 --port 8000 \ --tensor-parallel-size 8 \ --pipeline-parallel-size 1 \
--distributed-executor-backend mp \ --max-model-len 65536 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8_e4m3 \ --max-num-seqs 256 \ --max-num-batched-tokens 16384 \ --long-prefill-token-threshold 4096 \
--enable-prefix-caching \ --async-scheduling \ --compilation-config ‘{“cudagraph_mode”: “FULL_AND_PIECEWISE”, “max_cudagraph_capture_size”: 256}’ \ --trust-remote-code

Validation

Once the container is up, confirm the server is healthy and serving:

curl -sf http://localhost:8000/health && echo "server is healthy"

curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "zai-org/GLM-5.2", "prompt": "The capital of France is", "max_tokens": 16}'

Files in this repository

File	Purpose
Dockerfile	vllm/vllm-openai-rocm:v0.25.1-based image with the two scripts below baked in
launch_server.sh	Starts vllm serve with the baseline or optimized flags
run_benchmark.sh	Starts baseline + optimized servers, benchmarks each, and prints the comparison table below — fully automated
README.md	This model card

2. Shell Script

Launch Server

./launch_server.sh optimized   # or: ./launch_server.sh baseline

This is also the image's default command — docker run ... <image> optimized runs the same thing. Inside a running container, invoke it with docker exec -it <container> ./launch_server.sh <mode>.

Reproduce Benchmark

This is a separate container invocation from "Launch Server" above — it overrides the entrypoint so run_benchmark.sh (not launch_server.sh) drives the container. It starts the baseline server itself, benchmarks it, shuts it down, starts the optimized server, benchmarks that, shuts it down, and prints the comparison table — nothing to copy in by hand:

docker run \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --ipc=host --shm-size 16g \
  --security-opt seccomp=unconfined \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint ./run_benchmark.sh \
  your-org/glm-5.2-vllm:latest both

That prints the full table in the terminal when it finishes (large model, 8-GPU node — expect this to take a while: two model loads plus two 1,000-prompt runs). Pass baseline or optimized instead of both to run just one side. The raw per-run JSON (benchmark_baseline.json, benchmark_optimized.json) and each server's startup log (server_baseline.log, server_optimized.log) are left in the container's /workspace if you want them too:

docker cp <container>:/workspace/benchmark_optimized.json .

Methodology (vllm bench serve):

• Dataset: random (synthetic), range ratio 0.8
• Input length: 409–3,686 tokens (nominal 2,048, uniform)
• Output length: 51–460 tokens (nominal 256, uniform)
• 1,000 measured prompts, 10 warm-up prompts (discarded, not saved)
• Request rate: unbounded (inf) — max concurrency is the only throttle
• Max concurrency: 512 (best command tuned at this concurrency; override with -e MAX_CONCURRENCY=...)

Results

8x AMD Instinct MI325X 256GB, vLLM v0.25.1+rocm723. This is exactly the table run_benchmark.sh both prints — running it is the only step required to reproduce it:

Metric	Baseline	Optimized
Request waiting (peak)	491	478
Request waiting (avg)	169.5	95.7
Output tok/s	1019.0	1292.2
TTFT mean (ms)	49744	23744
TTFT median (ms)	51841	11448
TTFT p99 (ms)	104700	77224
TPOT mean (ms)	269.7	294.2
TPOT p99 (ms)	524.6	744.4
ITL mean (ms)	258.1	272.2
ITL p99 (ms)	738.9	1079.2
E2E median (ms)	110457	85579
KV cache peak	100%	100%
Duration (s)	342.1	143.3