Instructions to use local-inference-lab/GLM-5.3-Flash-NVFP4-Spark with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use local-inference-lab/GLM-5.3-Flash-NVFP4-Spark with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="local-inference-lab/GLM-5.3-Flash-NVFP4-Spark")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("local-inference-lab/GLM-5.3-Flash-NVFP4-Spark")
model = AutoModelForMultimodalLM.from_pretrained("local-inference-lab/GLM-5.3-Flash-NVFP4-Spark", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use local-inference-lab/GLM-5.3-Flash-NVFP4-Spark with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/local-inference-lab/GLM-5.3-Flash-NVFP4-Spark

SGLang

How to use local-inference-lab/GLM-5.3-Flash-NVFP4-Spark with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use local-inference-lab/GLM-5.3-Flash-NVFP4-Spark with Docker Model Runner:
```
docker model run hf.co/local-inference-lab/GLM-5.3-Flash-NVFP4-Spark
```

2x 6kpro

by mtcl - opened 7 days ago

Discussion

mtcl

7 days ago

You said that it fits on it. How much context we can get here please?

This is so exciting. Thank you!

mvaranka

2 days ago

Currently running GLM-5.3-Flash with llama.cpp from unsloth with IQ4_XS gguf. During token generation GPUs are taking turns and load is about 50%, so tg stays on about 50 tok/s without mtp.
So looking recipe for vLLM.

Got it finally running with following commands but it is very tight on VRAM and 32k context does not fit. It is much faster (tg ~80 tok/s without mtp):

Model loading took 86.08 GiB per GPU
Available KV cache memory: 1.3 GiB
GPU KV cache size: 26,437 tokens, Maximum concurrency for 16,384 tokens: 1.61x
Generation: 87-90 tok/s, no speculator

IMAGE=voipmonitor/vllm@sha256:488ddf752938b5ab17e3083dd7d5bb84f418bc3f8856f93cc514c8b66abbe4c6

docker run --name jovian-mtp3-tp2 --init --gpus '"device=0,1"' --network host --ipc host --shm-size 32g -v jovian-judgement-vllm-cache:/cache -v ~/.cache/huggingface:/root/.cache/huggingface -e HF_HUB_OFFLINE=1 -e MODEL=local-inference-lab/GLM-5.3-Flash-NVFP4-4p67 -e PORT=8080 -e HOST=0.0.0.0 -e TP=2 -e DCP=1 -e MAX_MODEL_LEN=16384 -e MAX_NUM_SEQS=4 -e MAX_NUM_BATCHED_TOKENS=1024 -e MAX_CUDAGRAPH_CAPTURE_SIZE=48 -e GPU_MEMORY_UTILIZATION=0.97 -e B12X_PCIE_ALLREDUCE=1 -e NCCL_MIN_NCHANNELS=32 -e NCCL_MAX_NCHANNELS=32 -e NCCL_CUMEM_ENABLE=0 -e NCCL_IB_DISABLE=1 -e NCCL_P2P_LEVEL=SYS -e NCCL_PROTO=LL,LL128,Simple -e OMP_NUM_THREADS=2 -e INSTANTTENSOR_BUFFER_SIZE=67108864 -e INSTANTTENSOR_IO_DEPTH=3 -e INSTANTTENSOR_CONCURRENCY=1 -e INSTANTTENSOR_CHUNK_SIZE=8388608 "$IMAGE" --kv-cache-memory 1400000000

vLLM print following notification: "WARNING kv_cache_utils.py:1253] Add 10 padding layers, may waste at most 29.41% KV cache memory
Keeping split GLM-5.3 cache groups with physical page sizes [287232, 2170880]"

mvaranka

1 day ago

Got it running with 512k context and 2 parallel request using sglang and model :
https://github.com/ormandj/sglang-glm53-flash-sm120
https://huggingface.co/ormandj/GLM-5.3-Flash-W4A16-NVFP4-K32-Experts-FP8-WO

Now it flies 100-200tok/s, both GPUs at 100% when processing

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment