GLM-5.2-W4A16

4-bit (W4A16) weight-quantized version of zai-org/GLM-5.2.

Weights are quantized to INT4 (group size 128); activations run in BF16. The result is a 388 GB checkpoint — about 3.9× smaller than the 1.5 TB BF16 original — that fits on a single 8×A100 (80 GB) node while preserving full-precision quality on reasoning and knowledge benchmarks.

Because compute stays in BF16 and only the weights are INT4, this model runs on NVIDIA Ampere (A100) and newer — it does not require Hopper/Blackwell FP8 support.

Highlights

• Quality on par with the FP8 release — no measurable degradation (see below).
• ~3.9× smaller than BF16: 388 GB vs. 1.5 TB.
• Runs on A100 (Ampere) — no FP8 hardware needed.
• Serves out of the box with vLLM (compressed-tensors format).

Evaluation

Evaluated against the reference FP8 deployment of GLM-5.2. Greedy decoding (temperature 0); reasoning traces stripped and the final answer graded.

Benchmark	This model (W4A16)	Reference (FP8)
GSM8K (n=200), exact-match	96.5%	94.5%
MMLU (n=200), accuracy	86.5%	80.0%

W4A16 matches the FP8 reference within evaluation noise (n=200, standard error ≈ 2 pts). The takeaway is parity — 4-bit quantization retains GLM-5.2's reasoning and knowledge capability.

Model details

Base model	zai-org/GLM-5.2
Architecture	GlmMoeDsaForCausalLM (MoE, 78 layers, 256 routed + 1 shared expert, top-8)
Weight precision	INT4, group size 128, symmetric
Activation precision	BF16
Format	compressed-tensors (pack-quantized)
Checkpoint size	388 GB (8 shards)
Context length	up to 1,048,576 tokens

The sparse-attention (DSA) indexer, the MoE router, and the LM head are kept in BF16; the large linear and expert weights carry the INT4 quantization.

Serving on A100 (8× A100 80 GB, vLLM)

The full INT4 checkpoint fits on one 8×A100-80GB node with room for KV cache.

pip install "vllm>=0.24.0"

# A100 (Ampere) note: use BF16 compute paths and skip Hopper-only kernels.
export VLLM_USE_FLASHINFER_SAMPLER=0   # avoid FlashInfer sampler JIT on some CUDA toolkits
export VLLM_USE_DEEP_GEMM=0            # DeepGEMM (FP8 block-scale) is not needed on A100

vllm serve lowbitcoffee/GLM-5.2-W4A16 \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --served-model-name glm-5.2-w4a16 \
  --trust-remote-code

vLLM auto-detects the quantization from the checkpoint — no --quantization flag is required. Increase --max-model-len toward the model's 1M limit only if you have KV-cache headroom; lower it to raise concurrency.

On 8× A100 40 GB, the weights alone (388 GB) exceed the 320 GB of aggregate VRAM — use two nodes (--tensor-parallel-size 16) or the 80 GB SKU.

Query it (OpenAI-compatible)

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "glm-5.2-w4a16",
        "messages": [{"role": "user", "content": "What is 84 * 3 / 2?"}],
        "max_tokens": 1024,
        "temperature": 0
      }'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="glm-5.2-w4a16",
    messages=[{"role": "user", "content": "Explain MoE routing in two sentences."}],
    max_tokens=1024,
    temperature=0.6,
)
print(resp.choices[0].message.content)

GLM-5.2 is a reasoning model: responses may include a <think>…</think> block before the final answer. Strip it client-side, or configure a reasoning parser in your serving stack if you want the fields separated.

License

Released under the MIT license, inheriting the license of the base model zai-org/GLM-5.2.