jlnsrk/GLM-5.2-colibri-int4

GLM-5.2 — colibrì int4 container (~370 GB)

Pre-converted weights for colibrì — the pure-C engine that runs GLM-5.2 (744B MoE) on a consumer machine with ~25 GB of RAM by streaming routed experts from disk.

This is the output of colibrì's coli convert (convert_fp8_to_int4.py --ebits 4 --io-bits 8, including the MTP head for native speculative decoding), uploaded so you don't have to download the 756 GB FP8 checkpoint and spend a day converting it.

⚠️ This is NOT a GGUF / AWQ / GPTQ / MLX model. It is colibrì's own container: for each quantized weight, name (U8, packed int4 nibbles) + name.qs (F32 per-row scales), quantized with math bit-identical to the engine's C kernels. It only works with the colibrì engine.

Usage

# get the engine
git clone https://github.com/JustVugg/colibri && cd colibri/c && ./setup.sh

# download this repo to a FAST local disk (NVMe, ext4 — never a network/9p mount)
hf download jlnsrk/GLM-5.2-colibri-int4 --local-dir /nvme/glm52_i4

# chat (RAM budget, expert cache and MTP auto-detected)
COLI_MODEL=/nvme/glm52_i4 ./coli chat

Requirements: Linux (or WSL2), gcc + OpenMP, AVX2, ≥16 GB RAM, ~400 GB free NVMe.

What's inside

file	contents
out-*.safetensors	dense weights (attention/MLA, shared experts, embeddings) + 21,504 routed experts, int4 per-row scales; router/norms kept F32
MTP shard	GLM-5.2's multi-token-prediction head (layer 78) — enables lossless speculative decoding (~2 tok/forward)
config.json, tokenizer*.json, generation_config.json	copied from the base repo

Conversion: FP8 (e4m3, 128×128 block scales) → f32 → int4 with np.rint matching the engine's lrintf — token-identical to converting locally.

Provenance & license

Converted from zai-org/GLM-5.2-FP8 (MIT). This derivative is likewise MIT. Conversion performed with colibrì's official converter, unmodified.