Instructions to use intel-ai/GLM-5.3-MXFP4-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use intel-ai/GLM-5.3-MXFP4-AutoRound with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="intel-ai/GLM-5.3-MXFP4-AutoRound")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("intel-ai/GLM-5.3-MXFP4-AutoRound")
model = AutoModelForCausalLM.from_pretrained("intel-ai/GLM-5.3-MXFP4-AutoRound", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use intel-ai/GLM-5.3-MXFP4-AutoRound with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "intel-ai/GLM-5.3-MXFP4-AutoRound"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "intel-ai/GLM-5.3-MXFP4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/intel-ai/GLM-5.3-MXFP4-AutoRound

SGLang

How to use intel-ai/GLM-5.3-MXFP4-AutoRound with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "intel-ai/GLM-5.3-MXFP4-AutoRound" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "intel-ai/GLM-5.3-MXFP4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "intel-ai/GLM-5.3-MXFP4-AutoRound" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "intel-ai/GLM-5.3-MXFP4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use intel-ai/GLM-5.3-MXFP4-AutoRound with Docker Model Runner:
```
docker model run hf.co/intel-ai/GLM-5.3-MXFP4-AutoRound
```

Configuration Parsing Warning:In UNKNOWN_FILENAME: "quantization_config.config_groups.group_0.format" must be a string

Model Details

This model is an MXFP4 (W4A4, group size 32) build of zai-org/GLM-5.3 produced with intel/auto-round (--model_free RTN), with module coverage identical to RedHatAI/GLM-5.3-MXFP4. Please follow the license of the original model.

Quantized (MXFP4, W4A4) — routed expert and MTP expert projections, MLA attention (q_a_proj, q_b_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj), shared expert projections, the leading dense MLPs (layers 0-2) and the DSA indexer's wq_b.
Kept in BF16 — DSA indexer wk and weights_proj, MoE router (mlp.gate), the MTP fusion projection (eh_proj), token embedding, LM head, LayerNorms.
KV cache — not quantized in the checkpoint; vLLM on SM10x selects FP8 KV cache for this architecture automatically.
Size — 403.34 GB of weight safetensors (141 shards, 118 607 tensors), byte-for-byte the same total as the RedHatAI artifact (403,337,136,288 bytes, total_size 403,321,862,144). The RedHatAI repo's whole-repo figure is 403,368,910,311 bytes (403.37 GB) — the extra 31.8 MB is config/tokenizer/README, not weights.

How to Use

Environment

# produced and validated with:
#   vllm 0.32.1rc1.dev253+g30aded62d  (registers GlmMoeDsaForCausalLM via vllm.models.deepseek_v32)
#   torch 2.14.0+cu130 | auto-round 0.16.1 | compressed-tensors 0.17.0 | lm-eval 0.4.13
uv pip install "vllm==0.32.1rc1.dev253"

vLLM Usage

vllm serve {org}/GLM-5.3-MXFP4-CT-RTN-AutoRound \
    --tensor-parallel-size 4 \
    --dtype bfloat16 \
    --reasoning-parser glm45

--tensor-parallel-size 4 is what this artifact was validated at (403 GB → ~101 GB/GPU). Other topologies are fine as long as moe_intermediate_size / (TP * 32) stays a multiple of 4 (2048 → TP=1/2/4/8 all satisfy it).

Evaluation

lm_eval 0.4.13 + vLLM 0.32.1rc1.dev253, TP=4, enable_thinking=False, seed 42. PIQA / HellaSwag / MMLU loglikelihood, GSM8K chat template with 5-shot; full test sets, no --limit.

| Benchmark | This model | |---|---:|---:| | PIQA (acc) | 83.46 | | HellaSwag (acc) | 71.50 | | MMLU (acc) | 84.23 | | GSM8K (strict-match) | 97.04 | | Average | 84.06 |

Generate the Model

Two ingredients: the auto-round CLI invocation below, and a small opt-in patch that disables auto-round's built-in GLM ignore rule (auto_round/special_model_handler.py) so the dense MLPs and indexer.wq_b are quantized like RedHatAI does.

# 1) patch auto-round in a private overlay (shared venv is left untouched)
SITE=<venv>/lib/python3.12/site-packages
mkdir -p /path/to/ar_overlay && cp -r "$SITE/auto_round" /path/to/ar_overlay/
patch --batch --fuzz=0 -p1 -d /path/to/ar_overlay -i patches/autoround_glm_moe_dsa_rh_strict.patch

# 2) quantize
PYTHONPATH=/path/to/ar_overlay AUTO_ROUND_GLM_RH_STRICT=1 auto-round \
    --model_name zai-org/GLM-5.3 \
    --model_free \
    --scheme MXFP4 \
    --ignore_layers "mlp.gate$,lm_head,embed_tokens,eh_proj,indexer.weights_proj,indexer.wk" \
    --format llm_compressor \
    --device_map auto \
    --output_dir <output_dir>

Scheme: single group mxfp4-pack-quantized — weights E2M1, group size 32, E8M0 scales; activations FP4 with dynamic per-group scaling (W4A4).
Targets: ["Linear"] (same as RedHatAI).
Ignore: exactly RedHatAI's six items — mlp.gate$ (MoE router; the $ anchor matters because the dense layers have mlp.gate_proj), lm_head, embed_tokens, eh_proj, indexer.weights_proj, indexer.wk. Expands to 123 module names.
Why the patch: auto-round ships ignore_layers=[get_glm_flash_ignore_layers, "weights_proj", "indexer"] for the glm_moe_dsa family, which keeps the leading dense MLPs and the entire DSA indexer in BF16. Those workarounds target the tuning/forward path (torch.compile AOTAutograd with the indexer's torch.no_grad(), and a vLLM loading issue); with --model_free nothing is loaded or compiled, so the narrower ignore is safe. Without the patch the artifact is 31 tensors larger in BF16 terms and ~1.3 GB bigger.
Frozen recipe: scripts/glm53_quant_rh_style.sh (aqa quant.command_override).

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to provide factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs. Therefore, before deploying any applications of this model, developers should perform safety testing.

Caveats and Recommendations

MXFP4 weights with dynamic MXFP4 activations is the main source of quantization error; the routed experts carry ~97.5 % of the parameters.
The four benchmark numbers above come from a single run each; treat ±0.5 pt as noise.
No same-harness FP8 baseline: the source checkpoint is 756 GB and does not fit TP=4 on 183 GB cards (it needs TP=8), so quality deltas versus FP8/bf16 are not measured here.

Users (both direct and downstream) should be made aware that the risks of this model involve hallucinations, biased or offensive content, and factual errors.

Cite

@article{cheng2023optimize,
  title={Optimize weight rounding via signed gradient descent for the quantization of large language models},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties using this model. Please consult an attorney if you need legal advice about your use of this model, and follow the license of zai-org/GLM-5.3.

Signed-off-by: Kaokao, Lv kaokao.lv@intel.com