Instructions to use soyrsoyr/GLM-5.3-Flash-NVFP4-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use soyrsoyr/GLM-5.3-Flash-NVFP4-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="soyrsoyr/GLM-5.3-Flash-NVFP4-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("soyrsoyr/GLM-5.3-Flash-NVFP4-MTP")
model = AutoModelForMultimodalLM.from_pretrained("soyrsoyr/GLM-5.3-Flash-NVFP4-MTP", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use soyrsoyr/GLM-5.3-Flash-NVFP4-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "soyrsoyr/GLM-5.3-Flash-NVFP4-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "soyrsoyr/GLM-5.3-Flash-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/soyrsoyr/GLM-5.3-Flash-NVFP4-MTP

SGLang

How to use soyrsoyr/GLM-5.3-Flash-NVFP4-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "soyrsoyr/GLM-5.3-Flash-NVFP4-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "soyrsoyr/GLM-5.3-Flash-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "soyrsoyr/GLM-5.3-Flash-NVFP4-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "soyrsoyr/GLM-5.3-Flash-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use soyrsoyr/GLM-5.3-Flash-NVFP4-MTP with Docker Model Runner:
```
docker model run hf.co/soyrsoyr/GLM-5.3-Flash-NVFP4-MTP
```

GLM-5.3-Flash NVFP4 with NVFP4 MTP

This checkpoint keeps the NVFP4 backbone from RedHatAI/GLM-5.3-Flash-NVFP4 and replaces its FP8 block MTP layer with a weight-only NVFP4 MTP layer produced from zai-org/GLM-5.3-Flash.

The MTP expert and shared-expert projections are packed as NVFP4. MTP attention, router, fusion, normalization, and shared-head tensors remain BF16. MTP activations are not statically quantized because Transformers does not construct the MTP module during calibration.

Recipe

The backbone recipe is included as recipe.yaml:

default_stage:
  default_modifiers:
    QuantizationModifier:
      config_groups:
        experts:
          targets: ["re:.*mlp\\.experts\\..*(gate|up|down)_proj$"]
          weights:
            num_bits: 4
            type: float
            symmetric: true
            group_size: 16
            strategy: tensor_group
            dynamic: false
            scale_dtype: torch.float8_e4m3fn
            observer: memoryless_minmax
          input_activations:
            num_bits: 4
            type: float
            symmetric: true
            group_size: 16
            strategy: tensor_group
            dynamic: local
            scale_dtype: torch.float8_e4m3fn
            observer: static_minmax
      targets: [Linear]
      ignore:
      - "re:.*visual.*"
      - lm_head
      - "re:.*mlp\\.gate$"
      - "re:.*self_attn\\.indexer\\..*"

With MTP quantization support from llm-compressor PR #3118, the corresponding oneshot call is:

oneshot(
    model=model,
    processor=processor,
    recipe="recipe.yaml",
    dataset=dataset,
    max_seq_length=4096,
    num_calibration_samples=512,
    moe_calibrate_all_experts=True,
    data_collator=data_collator,
    output_dir="/workspace/models/GLM-5.3-Flash-NVFP4-MTP",
    mtp_scheme="NVFP4",
)

mtp_scheme="NVFP4" applies model-free weight quantization to the unloaded MTP layer after the calibrated backbone is saved.

Validation

867 MTP projections packed as NVFP4
18 MTP weights retained in BF16 by the GLM architecture policy
2,623 MTP tensors indexed in model_mtp.safetensors
No stale native FP8 inverse scales
All NVFP4 global scales finite and positive

vLLM

vllm serve soyrsoyr/GLM-5.3-Flash-NVFP4-MTP \
  --tensor-parallel-size 4 \
  --no-enable-flashinfer-autotune \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --gpu-memory-utilization 0.85 \
  --disable-custom-all-reduce \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'