Instructions to use shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts")
model = AutoModelForMultimodalLM.from_pretrained("shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts

SGLang

How to use shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts with Docker Model Runner:
```
docker model run hf.co/shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts
```

GLM-5.3-Flash expert-only NVFP4 W4A4

This is a community conversion of zai-org/GLM-5.3-Flash-BF16 at revision b1967181a3917ae70a437f4884748f6b8e3a1f4d.

It was built for native NVFP4 execution on four NVIDIA GB10/DGX Spark nodes with vLLM TP4+EP4. It is not currently a drop-in checkpoint for an unmodified upstream vLLM installation: the tested deployment uses gated compatibility patches for GLM-5.3's rope-free sparse MLA/KDA path on SM121a, ProcessGroupNCCL collectives, mixed-precision lm_head, and deterministic batch execution.

The complete conversion pipeline, vLLM compatibility patches, Docker/Compose configuration, four-node launchers, acceptance tools, and pinned offline dependencies are preserved in shankinson/glm53-nvfp4-team. The accepted source snapshot is commit 95e08fd, tagged glm53-nvfp4-w4a4-accepted-20260829.

Quantization boundary

NVIDIA Model Optimizer 0.46.0, commit 43fd41a58d52c4e6e5dec1d1ff5989ecc737ae1a
Base routed-expert layers 3–44 only
288 experts per layer
gate_proj, up_proj, and down_proj: 36,288 weights total
NVFP4 E2M1 weights, group size 16, with calibrated W4A4 activations
MTP layer 45 remains BF16: 864 expert projection weights
Vision, attention/KDA, sparse indexers, routers, shared experts, dense layers, embeddings, lm_head, and norms remain BF16

The activation-scale policy uses a 5% safety margin with:

p99 across experts by default
p98 for w2 at layers 3–8
p98.9 for w2 at layers 24–44

The conversion and active-scale hashes are recorded in conversion_manifest.json. The installed activation-scale shard SHA-256 is caacc401233a4d6bcae6fc4f830c1754b12f28d0f6c5b513ceb1e586b40f5eb1.

Validation

The final clean-cold four-node run used vLLM 0.1.dev20051+g487ecf187, FlashInfer 0.6.17, CUDA sm_121a, native FLASHINFER_CUTLASS NVFP4 MoE, and FLASHINFER_MLA_SPARSE_SM120.

Constrained 400-item MMLU/ScienceQA regression evaluation:

Checkpoint	Overall	Text	Image-text
Official FP8 reference	89.25%	84.5%	94.0%
W4A16 activation reference	88.5%	83.0%	94.0%
This W4A4 checkpoint	89.0%	84.0%	94.0%

Unprofiled W4A4 phase throughput on four GB10 nodes:

Concurrent width	Prefill prompt tok/s	Generation output tok/s
1	1,279.01	24.86
2	1,332.22	40.63
4	1,334.71	65.73
8	1,383.55	99.77
16	1,382.60	173.77

The long-prompt prefill case generated one token. The decode-dominant case used a short prompt and 256 forced output tokens. Exact deterministic output, reasoning, tools, image, four-image, 32-frame video, streaming, 7,820-token and 31,020-token prompts passed. A 901.5-second mixed-width soak completed 2,495 requests without failures or earlyoom intervention.

Download

hf download shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts \
  --local-dir /path/to/GLM-5.3-Flash-NVFP4-W4A4-Experts

The checkpoint is approximately 191 GiB (205 GB decimal). The repository uses 120 primary safetensors shards plus a separate activation-scale shard.

Important runtime note

Use this checkpoint only with a runtime that recognizes ModelOpt MIXED_PRECISION metadata and native NVFP4 expert layers. The validated GB10 deployment requires the companion vLLM compatibility patches and launch settings; vanilla vLLM behavior has not been claimed or validated. The checkpoint retains the upstream multimodal processor, tokenizer, chat template, and MIT license.

This conversion is not affiliated with Z.AI, NVIDIA, vLLM, or FlashInfer.

Downloads last month: -

Safetensors

Model size

169B params

Tensor type

F32

BF16

Model tree for shankinsonhf/GLM-5.3-Flash-NVFP4-W4A4-Experts

Base model

zai-org/GLM-5.3-Flash-BF16

Quantized

(20)

this model