Acknowledge the Responsible Use Agreement to access this repository
Access is granted automatically after you agree to the terms below and submit the form.
Responsible Use Agreement
This model has had safety refusals removed. That makes it useful for red-teaming, security research, evaluation, and unfiltered assistant tasks — and also removes guardrails a user must therefore supply themselves.
Prohibited uses (you must agree before access is granted):
- Anything involving the sexual exploitation or endangerment of minors.
- You must be of age 18 years or older to use and download this model.
- You agree any information generated that can cause harm in terms of generating recipe, knowledge to make any materials/substances is your own input and responsibility. You will be accountable for any harm/damage caused by your action/input.
- Content promoting self-harm or suicide.
- Generation of material that is illegal in your jurisdiction, or that targets real individuals for harassment, doxxing, or fraud.
- Any use prohibited by the upstream Z.AI / GLM MIT license.
You are responsible for adding appropriate safety filtering, human review, and access controls for your deployment. The weights are provided as-is, with no warranty. The license is inherited from the upstream Z.AI GLM-5.3-Flash MIT license — review and comply with it before use or redistribution.
Log in or Sign Up to review the conditions and access this model content.
keys-GLM-5.3-EXL3-Abliterated — full GLM-5.3 (753B MoE), 3-bit EXL3, derisked/abliterated
This is the abliterated build of drowzeys/keys-GLM-5.3-EXL3 — same quant, same serving stack, with the Blackfrost "derisk" edit baked into the weights. Everything below the Abliteration section is identical to the base card.
Abliteration (what was edited, and how)
Credit: the BF16 derisk projections are Blackfrost AI's (Blackfrost-Research, GLM-5.3-DERISKED-BF16). We adapted them to this EXL3 routed-expert quant with the MTP block and the DFlash2 draft checkpoints preserved.
The reference edit, measured tensor-by-tensor against zai-org/GLM-5.3-BF16, is a rank-1 orthogonal projection applied in bf16 before quantization:
W' = W − α · r (rᵀ W), α = 3.0, one unit direction r per layer, on self_attn.o_proj and every residual-writing down_proj (dense, shared experts, routed experts), layers 2–49 only (layers 0–1, 50–78 and the MTP block untouched).
What is baked into these shards (see ABLIT_BAKE_REPORT.json for the per-tensor relative change):
| Tensor | Layers | Edit | Fidelity |
|---|---|---|---|
self_attn.o_proj |
2–49 | replaced by the exact DERISKED-BF16 tensor (cast to fp16) | exact |
mlp.down_proj (dense) |
2 | projected with the recovered r₂, α=3.0 | rank-1 fit residual 2–4% |
mlp.shared_experts.down_proj |
2–49 | projected with r_L, α=3.0 | same |
mlp.experts.*.down_proj (routed) |
2–49 | not edited — these are the 3-bit EXL3 trellis tensors; editing them requires re-quantizing from DERISKED-BF16 | — |
| MTP block (layer 78), layers 0–1, 50–77 | — | stock, as in the reference | — |
The directions were recovered by SVD of (W_derisked − W_stock) per layer (s1/s2 ≈ 300–850, α fits 2.999–3.001 on every layer). A load-time variant of the same edit (Mia's ablit_runtime.py transplant + our down_proj projection hook) is in serving/; it produces the same weights this repo ships pre-applied, so serve this repo with ABLIT=0.
Coherence checks on the served model (greedy): factual/arithmetic, code, multi-step reasoning, long creative prose and benign "refusal-shaped" prompts all answered correctly and fully. Speed is unchanged versus the base build (MTP acceptance per step identical).
Preserved speculators: the bf16 MTP block (layer 78) is in the shards and serves as the MTP k=3 speculator; the full-model DFlash2 draft (GLM-5.3-DFlash2) is kept separately and works without DCP.
What this is: the complete zai-org/GLM-5.3 (78 layers, 256 routed experts, DeepSeek-style sparse MLA attention with indexer, one MTP block) with the routed experts quantized to 3-bit EXL3 trellis and everything else kept bf16/fp16. 308 GB in 41 safetensors shards. It serves on four NVIDIA DGX Spark (GB10) with TP=4 + decode-context-parallel 4 at a 1,000,000-token context, inside Mia AI Lab's vLLM container for the GLM-5.3-Flash EXL3 (ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3) plus the small patch set shipped in serving/.
| Source | zai-org/GLM-5.3 (FP8 block-scaled, 756 GB, 141 shards) |
| Quantizer | exllamav3 1.4.5 (convert.py, -hb 16 -mb 16 -cpi 0, calibration 250 rows × 2048) |
| Routed experts | 57,600 matrices (75 MoE layers × 256 experts × gate/up/down) at 3.00 bpw, out_scales: always |
| Codebook | layer 3 = mcg, layers 4–77 = mul1 (see Codebook note); the serving overlay handles both per layer |
| Everything else | attention (q_a/q_b/kv_a/o_proj/indexer), dense MLP (layers 0–2), shared experts, router, norms, embed, lm_head: bf16/fp16 (unquantized) |
| MTP block (layer 78) | bf16, kept — usable as an MTP speculator |
| Per-rank footprint (TP=4) | |
| Context verified | needle-in-haystack PASS at 8.5K / 25K / 50K / 105K / 263K / 979K tokens (DCP=4 sharded KV) |
Why we made it this way
Every previous EXL3 GLM-5.3 we tried to serve on DGX Spark cost hours-to-days of patching: the published quants were made for TabbyAPI/exllamav3 (full-model EXL3, 2-bit heads, mul1 everywhere) and vLLM has no full-model EXL3 — Mia's container only implements EXL3 for the routed experts, with attention/dense/head native. So this time we quantized to the container's ABI first:
- scope = routed experts only (the only thing the overlay's
Exl3Configquantizes), uniform integer bits in the overlay's allowed set {3,4,5,6} → 3-bit, because 4-bit experts (≈101 GB/rank) do not fit four 128 GB GB10s next to a 1M KV cache; 3-bit ≈ 78 GB/rank does. -hb 16 -mb 16: head and MTP stay bf16 so the stock vLLM paths load them.- suffix layout
trellis/suh/svh/<codebook marker>exactly as the overlay's loader expects,quantization_config.jsonwithquant_method=exl3, bits=3, scope=glm53_routed_experts_only.
It still needed a handful of patches (all in serving/), but they are small, mechanical, and now documented — the point of this repo.
How it was made (4× DGX Spark "encode farm")
One 753B model does not fit a single 128 GB Spark for quantization, and exllamav3 has no multi-GPU convert. We built a rolling pipeline (quant/orchestrate.py):
- Rolling source shards. The FP8 source is 756 GB and not layer-ordered; the orchestrator keeps a 5-shard download window (sha-checked), runs
convert.pyfor two layers at a time in a fresh container (--max_module L+1, then-rresume), and after each step rewrites finished shards dropping onlymodel.layers.L.mlp.experts.*(norms/attention stay — the compile step needs them). Peak disk ≈ 660 GB; permanent shards = embed/lm_head, final norm, and the layer-78 MTP shard (must exist at every resume). - Remote expert-encoding farm (
quant/exl3_farm.py+quant/patch_farm.py). exllamav3's per-layer H (Hessian) capture is serial, but the 768 expert matrices per layer are independent once H is final. The patch adds remote work threads toquantize_linears_parallel: it ships each matrix + its finalized H over TCP (length-prefixedtorch.save) to worker containers on the other three Sparks, which returntrellis/suh/svh/marker. Verified bit-identical to local encoding. One MoE layer: 49 min single-GPU → ~20–22 min on four. Whole model: ~27 wall-clock hours including a power-cut restart. - Resume-safe. A power interruption at 60/78 lost nothing: exllamav3 checkpoints after every layer; the in-flight layer is simply redone.
- Compile.
compile.pyneedsmarisa_trie(not in the base image) —pip install marisa-triein the quant container.
quant/recipe.yaml (58,167 tensor entries) is included; quant/build.sh + quant/patch_aarch64.py build exllamav3 1.4.5 on aarch64/CUDA 13 inside the Mia container image (AVX all-reduce/CPU-MoE stubs + pause intrinsic).
Codebook note (read before you quantize with exllamav3 ≥ 1.4)
convert.py -r (resume) lets the command-line default win over the stored args for the codebook (("codebook", True, "mul1") in the arg table). We passed -cb mcg on the first run only; every resumed 2-layer run silently quantized with mul1. Result: layer 3 is mcg, layers 4–77 are mul1. Quality is identical (both are 3-bit trellis codebooks), but Mia's overlay was mcg-only — which is why serving/exl3.py now detects the codebook per layer and drives the kernels' (mcg, mul1) flag pairs. If you resume an exllamav3 job, re-pass every quant flag.
Serving on 4× DGX Spark (what serving/ contains and why)
Base: ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3 + its start-tp4.sh launcher (github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks). Image layers we add (Dockerfiles included, build order dcp → mul1 → v145 → keys → keys3):
| Patch | Why |
|---|---|
exl3.py (overlay) |
+ mul1 marker tensors and per-layer codebook detection; markers may be 0-d scalars; NullConfig/InferParams stub for exllamav3 ≥ 1.x's LinearEXL3; MTP block exempt from EXL3 so SPEC_METHOD=mtp loads the bf16 MTP experts through the stock path |
exllamav3 1.4.5 swap (Dockerfile.v145) |
the image's exllamav3 0.0.43 exl3_moe kernel hard-rejects mul1 (TORCH_CHECK); 1.4.5 has the same 29-arg signature (+num_active) and accepts it. Built against the same base torch, so the .so drops in |
sparse_attn_indexer.py top-k fallback (Dockerfile.keys) |
prefill persistent_topk for top-k 2048 oversubscribes the GB10's 48 SMs; Mia's runtime patch only covers the decode ("kpool") path |
sm120_dcp.py + mla_attn_dcp.py (Dockerfile.dcp) |
decode-context-parallel port of the SM120 sparse-MLA backend (KV sharded across TP ranks → 1M context on four GB10s; the stock stack replicates KV and tops out ~300K) |
patch_ablit_dsv2.py + ablit_runtime.py |
optional load-time o_proj transplant hook for the full-model classes (DeepseekV2Model/DeepSeekMTP); off by default |
patch_tp4_launcher.py |
adds LOCAL_MODEL_DIR/LOCAL_DFLASH_DIR (serve from an NFS dir) and ABLIT_HOST_DIR to start-tp4.sh |
keys-launch.sh, env.tp4.example |
our launch wrapper (drops page caches on all ranks first) and the working .env.tp4 |
Launcher-level gotchas we hit (all in env.tp4.example comments): --ulimit nofile=1048576 (NCCL "Too many open files" at 4 ranks), a rank whose fabric IP sits on the second CX7 port needs its own WORKER_CX7_IF/IB, the NFS server node must bind-mount its own export at the shared path, gpu_memory_utilization above ~0.82 is refused by vLLM's startup guard (the worker already holds ~14 GiB of CUDA/NCCL state at check time) — use --kv-cache-memory-bytes to size the KV cache explicitly.
Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV, MTP k=3)
Production config: MTP k=3 speculator (the checkpoint's own bf16 MTP block), 1M context, idle serve, 256-token outputs, temperature 0.6 / top_p 0.95, thinking off. tok/s is decode only (completion tokens ÷ (wall − TTFT)); aggregate is summed across streams.
| Task | C1 per-stream | C2 agg | C4 agg | C8 agg | accepted tok/step |
|---|---|---|---|---|---|
| prose | 11.1 | 15.4 | 21.0 | 30.5 | 1.8 |
| code | 15.0 | 24.4 | 28.4 | 41.7 | 2.5 |
| list | 13.5 | 21.7 | 28.2 | 42.9 | 2.4 |
| essay | 15.0 | 20.6 | 26.5 | 37.9 | 2.3 |
TTFT 0.6–2.0 s across C1–C8. Decode rises with depth as the fixed per-step DCP cost amortizes (8→12 tok/s from short context to 1M). Prefill ≈ 320–410 tok/s is the limiter at depth: a 979K prompt is ~52 min to first token. Speculative gain is acceptance-bound — code/list/essay accept 2.3–2.5 draft tokens/step, prose ~1.8 — so 25–30 tok/s single-stream is not reachable on prose with this MTP head.
DFlash2 draft (parked): the full-model DFlash2 draft (incoai/GLM-5.3-DFlash2) can be made to boot at DCP=4 (see serving/ patches: replicate the sliding-window KV per rank, --prefix-match-unit 16), but vLLM has no correct decode-context-parallel path for sliding-window attention, so it generates garbage. Full-model DFlash2 at DCP needs a DCP-aware SWA attention kernel (upstream), not a config change. MTP k=3 is the shipping speculator.
Quick start (4 Sparks)
git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks glm53-mia && cd glm53-mia
python3 serving/patch_tp4_launcher.py start-tp4.sh # LOCAL_MODEL_DIR + ABLIT_HOST_DIR knobs
# build the image chain on EVERY rank (see serving/Dockerfile.* ; ~1 min each on top of the Mia base)
cp serving/env.tp4.example .env.tp4 # set rank IPs / NIC pins / LOCAL_MODEL_DIR=<this repo on shared storage>
SKIP_PULL=1 SKIP_SHIP=1 SKIP_BUILD=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 SKIP_OVERLAY_VERIFY=1 ./start-tp4.sh
OpenAI-compatible API on :8888, served model name GLM-5.3-EXL3. Sampling: temperature 1.0, top_p 0.95, thinking on (GLM defaults).
Lessons for "quantize with the serving container in mind"
- Read the serving overlay's loader first: allowed scope, bits set, tensor suffixes, marker dtype/shape, codebook — and quantize to that ABI, not to the quantizer's defaults.
- Keep anything the container serves natively (attention, dense, head, MTP) unquantized; the memory cost is small next to the experts and it removes whole classes of patches.
- Verify tensor suffixes after the first resumed step, not at the end (see the codebook note).
- Budget memory per rank for weights + KV + the ~14 GiB the process holds before vLLM measures anything; size KV explicitly.
- A per-layer encode farm turns a multi-day single-GPU convert into an overnight job on the same boxes that will serve the model.
Credits
Zhipu / zai-org for GLM-5.3 · turboderp for exllamav3 · Mia AI Lab for the GB10 EXL3 vLLM container and launcher · Blackfrost AI (Blackfrost-Research) for the BF16 derisk projections (GLM-5.3-DERISKED-BF16) that this build bakes into o_proj/down_proj, adapted by keys to EXL3 with the MTP/DFlash2 checkpoints preserved · Quantized and served by @drowzeys on a 4× DGX Spark cluster, 2026-09.
License: the weights inherit the zai-org/GLM-5.3 license (MIT). Patches in serving/ are MIT; they modify Apache-2.0 vLLM/SGLang/exllamav3 code — see their headers.
- Downloads last month
- 13
Model tree for drowzeys/keys-GLM-5.3-EXL3-Abliterated
Base model
zai-org/GLM-5.3