Instructions to use zai-org/GLM-5.3-Flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zai-org/GLM-5.3-Flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zai-org/GLM-5.3-Flash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("zai-org/GLM-5.3-Flash")
model = AutoModelForMultimodalLM.from_pretrained("zai-org/GLM-5.3-Flash", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zai-org/GLM-5.3-Flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zai-org/GLM-5.3-Flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zai-org/GLM-5.3-Flash

SGLang

How to use zai-org/GLM-5.3-Flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zai-org/GLM-5.3-Flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zai-org/GLM-5.3-Flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zai-org/GLM-5.3-Flash with Docker Model Runner:
```
docker model run hf.co/zai-org/GLM-5.3-Flash
```

GLM-5.3-Flash FP8 running on 2×L40S (sm_89) + EPYC 9845 — setup notes, patches and benchmarks

#38

by goldvet - opened 2 days ago

Discussion

goldvet

2 days ago

GLM-5.3-Flash FP8 on 2×NVIDIA L40S (Ada, sm_89) + AMD EPYC 9845 — it works, numbers inside

Since day-0 support targets Hopper/Blackwell, here are working notes for running the
native FP8 checkpoint (no quantization) on Ada GPUs with CPU expert offload,
fully offline. As far as I know these are the first published numbers for this combo.

Hardware

2× NVIDIA L40S (48 GB, sm_89, PCIe Gen4, no NVLink), TP=2
AMD EPYC 9845 (160C Zen 5c, SMT off, AVX-512, no AMX), NPS1
768 GB DDR5-6400 RDIMM running at 5200 MT/s (12×64 GB, 1DPC — platform max on
Gigabyte R263-ZG0-AAL2 / MZG3-GU0)
Debian 13 (glibc 2.41), driver-only CUDA + side-by-side toolkit 12.8

Stack

pip install "ktransformers[sglang]" in Python 3.11 → kt-kernel 0.7.0.post1,
sglang-kt 0.7.0.post1, transformers-kt 5.6.0.post4
--kt-method FP8 (CPU experts on AVX512-BF16 backend), PLE/N-gram table in host RAM
Launch: TP2, --kt-cpuinfer 152, --kt-max-deferred-experts-per-token 4,
--kt-gpu-prefill-token-threshold 2048 (layerwise GPU prefill),
--chunked-prefill-size 16384, --max-mamba-cache-size 16,
--mem-fraction-static 0.55, context 524k, KV fp8_e4m3

Patches needed on sm_89 (all one-liners, .bak kept)

config.json: index_share_for_mtp_iteration: true → false (KPool validator
rejects IndexShare-for-MTP; harmless without speculative decoding)
nsa_indexer_kpool.py ~L1109: disable the short-sequence skip path — both
fast_topk_transform_fused and fast_topk use Hopper features
(cluster launch) and crash with set_up_kernel_once failed: invalid argument
CUDA 12.8 math_functions.h: add noexcept (true) to sinpi/cospi/sinpif/cospif
(glibc 2.41 conflict, breaks all JIT compiles otherwise)
nvidia-cudnn-cu12==9.16.0.29 (torch 2.9.1 + cuDNN 9.10 Conv3d guard)
Optional: server_args.py cuda_graph_bs [1,2,4] → [1,2,4,8,16,24]
(decode without graphs collapses: 3.8 tok/s/user at n=5)

Also tuned Triton configs for L40S (6 block-FP8 GEMM shapes + fused MoE
E=288,N=1024) — 12h of tuning for +6.6% hot prefill; marginal, skippable.

Measured performance (single-run, `bench` via OpenAI API)

Metric	Value
Decode, 1 stream	18.3 tok/s (15.6 without deferred experts ×4; 9.7 initial config)
Decode, 8 streams	~47 tok/s aggregate (5.5/user)
Decode, 16–24 streams	saturates ~64 tok/s aggregate
Decode at 200k ctx	no degradation (linear attention + sparse holds)
Prefill 68k, warm	936 tok/s steady chunks, 2,226 tok/s peak, 85 s total
Prefill global budget	~900 tok/s regardless of concurrency (n=1→24: 857→926)
KV pool	2.11M tokens (fp8_e4m3), 524k per request

The prefill ceiling is structural: the generic FP8 layerwise path uses a single
GPU full-layer slot (gpu_full_layer_slots=1), so concurrent prefills serialize
FIFO. The MXFP4 path (DeepSeek V4 Flash) already has dual slots — generalizing
that to the FP8/GLM path could ~2× long-context prefill. Filed as a feature
request on the KTransformers repo.

Known limits on this fork/GPU: no MTP (KPool sm89 = non-speculative only),
radix/prefix cache force-disabled (KPool live-tail restore not implemented yet),
so multi-turn agents re-prefill history every turn (~70 s per 100k at 900 tok/s).

Quality note: everything is native FP8 weights; the only approximation is
--kt-max-deferred-experts-per-token 4 (+15% decode). Drop it if you want
bit-exact expert application.

Happy to share the full launch script / patch list.

goldvet

about 14 hours ago

•

edited about 14 hours ago

Update (Sept 4): prefill tuning results, two failure modes documented, and ecosystem notes

Follow-up after another optimization session on the same 2×L40S + EPYC 9845 setup.

Chunked prefill: 32k is the measured optimum on sm_89

Swept --chunked-prefill-size on the FP8 layerwise route (68k-token prompt, best of repeated runs):

Chunk	68k total	Status
16,384	85 s	baseline
24,576	69.8 s	ok (3 expert passes, same as 32k)
32,768	69.7 s	new default (−17%)
40,960	crash	OOM in the mHC fp32 fallback (~2.5 GiB transient)
40,960 + `max_split_size_mb:512`	120 s	stable but allocator thrashing — worse than 16k

Total time is quantized by ⌈prompt/chunk⌉ expert passes over PCIe, so 24k and 32k tie at 3 passes for 68k. The 40k wall is the Manifold-Constrained Hyper-Connections pre-mix: no fused MHC kernel appears to be available on sm_89, so each chunk pays a float32 (pre * residual) transient that scales with chunk size. Mitigating the OOM with max_split_size_mb trades the crash for cudaFree/retry cycles. A fused MHC kernel for Ada would remove both.

Transport comparison: native vs legacy (both at chunk=16,384)

Measured before switching the chunk default, so both transports ran the
then-default 16,384 chunks — apples to apples:

Transport	Steady chunks	68k total
native	936 tok/s	85 s
legacy (`SGLANG_KT_GLM5_NEXT_FP8_TRANSPORT=legacy`)	716 tok/s	105 s (−23%)

(The 32k-chunk numbers above are a separate, later improvement on the native path; per-chunk rates aren't comparable across chunk sizes, totals are.)
The native transport's overlap (H2D of layer N+1 over attention of layer N, per the docstring in kt_ep_wrapper.py) is real and worth keeping. Legacy also OOM'd under concurrent load where native holds.

Two operational gotchas worth knowing

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True breaks CUDA graph capture in TP=2 (invalid argument in custom_all_reduce.cuh — VMM memory can't be IPC-exported for the custom all-reduce buffers). Don't enable it on multi-GPU KT setups.
A desktop session (GNOME/Xwayland) was silently holding 0.5–0.7 GiB on GPU 0 — on a 2-GPU TP setup the most loaded card sets the limit, so headless (systemctl set-default multi-user.target) is worth the check.

Current state of the box

18.3 tok/s single-stream decode, ~47 tok/s aggregate at 8 streams, 68k prefill in 69.7 s solo / ~80 s under 8 concurrent decodes, ~1.9M-token KV pool, validated OOM-free under combined prefill+decode load.

Ecosystem notes for other sm_89 users

The community EXL3-K2 pack (2-bit routed experts, 91 GiB) runs through vLLM with native MTP and was smoke-tested on 4×L40 — proof that GLM-5.3-Flash + MTP + Ada work together in vLLM. The KT-side MTP lock is fork-specific, not a model/hardware limitation. (2-bit experts are a real quality trade-off for long-horizon agentic work, though.)
Encouraging precedent: KTransformers v0.6.4 shipped Ampere support with BF16 fallbacks, FP8 MoE Marlin for GPU experts/prefill, and RAWINT4 prefill acceleration for DeepSeek V4 Flash ~1 month after its day-0. I hope we can get the same treatment for GLM-5.3-Flash on sm_89 (MTP, radix cache tail-restore, dual GPU full-layer slots)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment