Instructions to use willfalco/GLM-5.2-EXL3-TR3-3.40bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use willfalco/GLM-5.2-EXL3-TR3-3.40bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="willfalco/GLM-5.2-EXL3-TR3-3.40bpw")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("willfalco/GLM-5.2-EXL3-TR3-3.40bpw")
model = AutoModelForCausalLM.from_pretrained("willfalco/GLM-5.2-EXL3-TR3-3.40bpw", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Trellis

How to use willfalco/GLM-5.2-EXL3-TR3-3.40bpw with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use willfalco/GLM-5.2-EXL3-TR3-3.40bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "willfalco/GLM-5.2-EXL3-TR3-3.40bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.40bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/willfalco/GLM-5.2-EXL3-TR3-3.40bpw

SGLang

How to use willfalco/GLM-5.2-EXL3-TR3-3.40bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "willfalco/GLM-5.2-EXL3-TR3-3.40bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.40bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "willfalco/GLM-5.2-EXL3-TR3-3.40bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.40bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use willfalco/GLM-5.2-EXL3-TR3-3.40bpw with Docker Model Runner:
```
docker model run hf.co/willfalco/GLM-5.2-EXL3-TR3-3.40bpw
```

GLM-5.2 EXL3 TR3 3.40 bpw

This was superseded by 3.42bpw vesrion

Might still be used with compose from version 3.42bpw for slightly better overall KLD as 3.42bpw is Coder version, inheriting less busier Coder experts at higher bpw.

This is a TP4, rank-sliced EXL3 Trellis build of zai-org/GLM-5.2, optimized for four NVIDIA Blackwell workstation GPUs. Routed MoE experts in layers 3-78 use EXL3 Trellis weights targeting 3.0/4.0 bits per weight, including the MTP (layer 78) routed experts using malaiwah's calibration-capture. Accuracy-sensitive and dense components remain in BF16 but can be used in mxfp8 or EXL3 Trellis 6bpw format (see below). The repository payload is 329 GiB. This format requires the custom vLLM + Sparkinfer runtime below; it is not a drop-in Transformers model. The routed weights are EXL3 Trellis and the required launch flag is --quantization exl3. NVFP4 in the supplied runtime refers to the KV cache, not the routed-expert weight format.

Weights        | KV format                 | KLD
───────────────────────────────────────────────────────────────────────
NF3            | Dynamic NVFP4 + RoPE8     | 0.139036 ± 0.002010
NF3            | Standard FP8 + BF16 RoPE  | 0.1263†
EXL3 3.0-bpw   | Dynamic NVFP4 + RoPE8     | 0.119525
EXL3 3.0-bpw   | Standard FP8 + BF16 RoPE  | 0.102508
EXL3 3.25-bpw  | Dynamic NVFP4 + RoPE8     | 0.095971
EXL3 3.25-bpw  | Standard FP8 + BF16 RoPE  | 0.087711
EXL3 3.36-bpw  | Dynamic NVFP4 + RoPE8     | 0.077767
EXL3 3.36-bpw  | Standard FP8 + BF16 RoPE  | 0.068458
EXL3 3.40-bpw  | Dynamic NVFP4 + RoPE8     | ...
EXL3 3.40-bpw  | Standard FP8 + BF16 RoPE  | ...
EXL3 3.40-bpw  | FP8 + Dynamic EXL3 6bpw   | ...

FP8 Context 454,656 tok with partial online MXFP8 quant of dense layers, trading more KV for a bit of accuracy:

KLD 0.06862 - '--quantization-config={"linear":{"weight":"mxfp8"},"ignore":["re:.*\\.q_a_proj$$","re:.*kv_a_proj_with_mqa"]}'

KLD 0.06958 - '--quantization-config={"linear":{"weight":"mxfp8"},"shared_experts":{"weight":"mxfp8"},"ignore":["re:.*\\.fused_qkv_a_proj$","re:.*\\.q_a_proj$","re:.*kv_a_proj_with_mqa","re:.*\\.mlp\\.gate$","model.layers.78.eh_proj","lm_head"]}'

KLD ....... ONLINE_QUANT=exl3-b6

Mind that reasoning_effort:high, set to reasoning_effort:max

services:
  g52h:
    image: voipmonitor/vllm:gilded-gnosis-v20-vllmf5981f1-si978cdb3-fi801d57a-cu132-20260803-r25
    container_name: g52h
    ports:
      - "0.0.0.0:8000:8000"
    gpus: all
    shm_size: "32g"
    ipc: "host"
    ulimits:
      memlock: -1
      nofile: 1048576
    environment:
      - CUDA_VISIBLE_DEVICES=2,3,4,5
      - CUDA_DEVICE_MAX_CONNECTIONS=32
      - CUTE_DSL_ARCH=sm_120a
      - OMP_NUM_THREADS=16
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - SAFETENSORS_FAST_GPU=1
      - NCCL_IB_DISABLE=1
      - NCCL_P2P_LEVEL=SYS
      - NCCL_PROTO=LL,LL128,Simple
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - VLLM_USE_B12X_FP8_GEMM=0  # +kld
      - VLLM_USE_B12X_SPARSE_INDEXER=1
      - VLLM_USE_V2_MODEL_RUNNER=1
      - VLLM_ENABLE_PCIE_ALLREDUCE=1
      - VLLM_PCIE_ALLREDUCE_BACKEND=b12x
      - VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE=64KB
      - VLLM_PCIE_ONESHOT_FUSED_ADD_RMS_NORM_MAX_SIZE=84KB
      - B12X_PCIE_DMA_FP8=0  # +kld
      - B12X_DENSE_SPLITK_TURBO=1
      - B12X_W4A16_TC_DECODE=1
      - B12X_MOE_FORCE_A16=1
      - VLLM_USE_AOT_COMPILE=1
      - VLLM_USE_BREAKABLE_CUDAGRAPH=0
      - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
      - VLLM_USE_B12X_MHC=1
      - B12X_MHC_MAX_TOKENS=16384
      - VLLM_USE_B12X_WO_PROJECTION=1
      - B12X_MLA_SM120_UNIFIED=1
      - VLLM_CACHE_DIR=/cache/jit/vllm
      - TRITON_CACHE_DIR=/cache/jit/triton
      - TORCH_EXTENSIONS_DIR=/cache/jit/torch_extensions
      - TORCHINDUCTOR_CACHE_DIR=/cache/jit/torchinductor
      - FLASHINFER_WORKSPACE_BASE=/cache/jit/flashinfer
      - XDG_CACHE_HOME=/cache/jit
      - TVM_FFI_CACHE_DIR=/cache/jit/tvm-ffi
      - GLOO_SOCKET_IFNAME=lo
      - NCCL_SOCKET_IFNAME=lo
      - VLLM_WORKER_MULTIPROC_METHOD=spawn
      - VLLM_PCIE_DMA_MIN_BYTES=6MB
      - VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE=0  # +pp +kld
      - VLLM_B12X_MLA_SPEC_DECODE_MAX_Q=8
      - VLLM_USE_B12X_DCP_A2A=1
      - VLLM_DCP_A2A_MAX_TOKENS=16
      - VLLM_DCP_A2A_LARGE_BACKEND=ag_rs
      - VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_B12X_MLA_CKV_GATHER_MIN_TOKENS=512  # for VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS=16384  # for VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_DCP_QUERY_SPLIT=1  # r14
      - VLLM_MEMORY_PROFILE_INCLUDE_ATTN=1
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
      - TORCH_CUDA_ARCH_LIST=12.0a
      - FLASHINFER_CUDA_ARCH_LIST=12.0f
      - FLASHINFER_DISABLE_VERSION_CHECK=1
      - VLLM_USE_B12X_MOE=1
      - VLLM_CPP_AR_1STAGE_NCCL_CUTOFF=56KB
      - VLLM_CPP_AR_IGNORE_CUTOFF_MAX_ROWS=0
      - VLLM_RTX6K_FUSED_ALLREDUCE_ADD=0
      - VLLM_RTX6K_FUSED_ALLREDUCE_ADD_END_BARRIER=0
      - VLLM_DISABLE_SHARED_EXPERTS_STREAM=0  # v20 
      - VLLM_DISABLED_KERNELS=MarlinFP8ScaledMMLinearKernel
      - VLLM_DCP_GLOBAL_TOPK=1
      - VLLM_DCP_SHARD_DRAFT=1
      - VLLM_DCP_QUERY_SPLIT=0
      - VLLM_EXL3_TRELLIS_MIN_M=1
      - VLLM_EXL3_TRELLIS_MAX_M=48
      - VLLM_EXL3_TRELLIS_BLOCK_M=8
      - VLLM_EXL3_PREFILL_CHUNK=128
      - KV_FP8_ROPE=0  # +kld
      - VLLM_B12X_ABSORB_BMM=0
      - ONLINE_QUANT=exl3-b6
      - VLLM_EXL3_ONLINE_TRELLIS_BITS=6
      - VLLM_EXL3_ENCODER_SOURCE=/opt/exllamav3-python/exllamav3
      - VLLM_EXL3_ONLINE_CACHE_DIR=/cache/exl3-online
      - VLLM_EXL3_ONLINE_CACHE_MODE=readwrite
    volumes:
      - /data1/GLM-5.2-EXL3-TR3-3.40bpw:/model:ro
      - /data1/GLM-5.2-EXL3-TR3-3.40bpw.cache:/cache:rw
      - /data1/GLM-5.2-EXL3-TR3-3.40bpw.cache:/root/.cache:rw
      - /data1/GLM-5.2-EXL3-TR3-3.40bpw.cache:/container-tmp:rw
    entrypoint:
      - /bin/sh
      - -c
      - "unset NCCL_GRAPH_FILE NCCL_GRAPH_DUMP_FILE VLLM_B12X_MLA_EXTEND_MAX_CHUNKS && exec vllm serve \"$@\""
      - --
    command:
      - /model
      - --served-model-name=g52h
      - --trust-remote-code
      - --tensor-parallel-size=4
      - --decode-context-parallel-size=4
      - --dcp-comm-backend=a2a
      - --dcp-kv-cache-interleave-size=1
      - --quantization=exl3
      - --kv-cache-dtype=fp8
      - --attention-backend=B12X_MLA_SPARSE
      - --moe-backend=b12x
      - --load-format=safetensors
      - '--compilation-config={"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"],"pass_config":{"fuse_allreduce_rms":true}}'
#      - --gpu-memory-utilization=0.98  # without ONLINE_QUANT=exl3-b6
      - --gpu-memory-utilization=0.971
      - '--quantization-config={"linear":{"weight":"mxfp8"},"ignore":["re:.*\\.q_a_proj$$","re:.*kv_a_proj_with_mqa"]}'  # KLD 0.06862
#      - '--quantization-config={"linear":{"weight":"mxfp8"},"shared_experts":{"weight":"mxfp8"},"ignore":["re:.*\\.fused_qkv_a_proj$","re:.*\\.q_a_proj$","re:.*kv_a_proj_with_mqa","re:.*\\.mlp\\.gate$","model.layers.78.eh_proj","lm_head"]}'  # KLD 0.06958
      - --max-model-len=128128
      - --max-num-seqs=16
      - --max-num-batched-tokens=2048
      - --max-cudagraph-capture-size=64
      - --enable-auto-tool-choice
      - --tool-call-parser=glm47
      - --reasoning-parser=glm45
      - --enable-prefix-caching
      - --enable-chunked-prefill
      - --no-async-scheduling
      - --enable-flashinfer-autotune
      - '--default-chat-template-kwargs={"reasoning_effort":"high"}'
      - '--hf-overrides={"use_index_cache":true,"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}'
      - '--speculative-config={"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton","draft_sample_method":"greedy"}'
#      - '--override-generation-config={"top_p":0.95,"repetition_penalty":1.18}'  # for temp=0.1 MMLU-Pro
      - --host=0.0.0.0
      - --port=8000

Source

License

The model and this derivative are released under the MIT license. See LICENSE and the upstream model card for attribution and usage terms.

Downloads last month: 62

Safetensors

Model size

176B params

Tensor type

BF16

F32

F16

I16

Model tree for willfalco/GLM-5.2-EXL3-TR3-3.40bpw

Base model

zai-org/GLM-5.2

Quantized

(146)

this model

Paper for willfalco/GLM-5.2-EXL3-TR3-3.40bpw

GLM-5: from Vibe Coding to Agentic Engineering

Paper • 2602.15763 • Published Feb 17 • 221