icecubetr/GLM-5.2-504B-Code-GGUF

Model Description

This is a quantized version of the GLM-5.2-REAP-504B-GGUF model using llama.cpp's IQ2_XXS quantization (2-bit). The model has been converted from BF16 to GGUF format with importance matrix optimization.

Provided Quants

file	bits	size	notes
GLM-5.2-504B-Code-IQ2_XXS	2.29	~140 GB	smallest

Other Files

file	notes
GLM-5.2-REAP-504B-imatrix	imatrix file created with code.txt dataset, saved after 80/146 chunks of calibration data
llama-quant.cpp	modified llama.cpp file for leaving indexer tensors unquantized

Run it

llama-cli -m GLM-5.2-504B-Code-IQ2_XXS.gguf -p "Hello"
llama-server -m GLM-5.2-504B-Code-IQ2_XXS.gguf --host 0.0.0.0 --port 8080

Recommended serving — recover most of the loop gap for free

• Anti-loop (recommended):

  • min_p=0.05, repetition_penalty=1.05
  • min_p=0.05, repetition_penalty=1.10

  Start at 1.05; go to 1.10 if you see loops — a higher repetition penalty trades a little risk of over-penalizing legitimate repetition (e.g. in code) for near-zero looping.

• Conciseness: a brevity system prompt — "Be concise. Think only as much as the task needs, then answer and stop." — halves median length (1267 → 507 tokens). Note it does not reduce looping (that's the sampler's job); combine the two for short, low-loop output.

Quantization Details

This model has been quantized using llama.cpp's llama-quantize tool with importance matrix (--imatrix) for optimal quantization. The quantization target was IQ2_XXS (2-bit quantization).

Quantization Summary

Metric	Value
Original Model Size (BF16)	956,419.32 MiB (16.00 BPW)
Quantized Model Size	136,813.06 MiB (2.29 BPW)
Compression Ratio	~7:1
Total Tensors	1809
Tensors with Fallback	79

Tensor Type to Quantization Method Mapping

Tensor Pattern	Quantization Method	Original Format	Notes
*attn_k_b.weight	iq4_nl	bf16	Fallback from IQ2_XXS (ncols=192 not divisible by 256)
*attn_kv_a_mqa.weight	iq2_xxs	bf16
*attn_output.weight	iq2_xxs	bf16
*attn_q_a.weight	iq2_xxs	bf16
*attn_q_b.weight	iq2_xxs	bf16
*attn_v_b.weight	iq2_xxs	bf16
*ffn_down_exps.weight	iq2_xxs	bf16
*ffn_down_shexp.weight	iq2_xxs	bf16
*ffn_gate_exps.weight	iq2_xxs	bf16
*ffn_gate_shexp.weight	iq2_xxs	bf16
*ffn_up_exps.weight	iq2_xxs	bf16
*ffn_up_shexp.weight	iq2_xxs	bf16
blk.0.ffn_down.weight	q2_K	bf16	Dense layers (blocks 0-2)
blk.1.ffn_down.weight	q2_K	bf16	Dense layers (blocks 0-2)
blk.2.ffn_down.weight	q2_K	bf16	Dense layers (blocks 0-2)
output.weight	q5_K	bf16	Output projection
token_embd.weight	q2_K	bf16	Token embeddings

Tensors Kept in Original Format (Not Quantized)

f32 (Float 32-bit) - Kept for Numerical Stability

• All *_norm.weight and *_norm.bias tensors
• All *_gate_inp.weight tensors (e.g., ffn_gate_inp.weight)
• All exp_probs_b.bias tensors
• blk.78.nextn.enorm.weight
• blk.78.nextn.hnorm.weight
• blk.78.nextn.shared_head_norm.weight

bf16 (bfloat16) - Not Quantized (Missing Importance Matrix Data)

• All indexer.attn_k.weight tensors
• All indexer.attn_q_b.weight tensors
• All indexer.proj.weight tensors
• blk.78.attn_kv_a_mqa.weight
• blk.78.attn_output.weight
• blk.78.attn_q_a.weight
• blk.78.attn_q_b.weight
• blk.78.attn_v_b.weight
• blk.78.ffn_down_exps.weight
• blk.78.ffn_down_shexp.weight
• blk.78.ffn_gate_exps.weight
• blk.78.ffn_gate_shexp.weight
• blk.78.ffn_up_exps.weight
• blk.78.ffn_up_shexp.weight
• blk.78.nextn.eh_proj.weight

Quantization Method Details

Method	Bits	Description
IQ2_XXS	~2	Extremely low-bit quantization with importance matrix optimization
IQ4_NL	~4	Non-linear 4-bit quantization with higher quality
Q2_K	~2	K-quant 2-bit quantization for dense layers
Q5_K	~5	K-quant 5-bit quantization (used for output layer)

Important Notes

1. Fallback Quantization: 79 tensors required fallback quantization because:

   • attn_k_b.weight tensors have ncols=192 (not divisible by 256, required for IQ2_XXS), falling back to IQ4_NL
   • Other tensors missing importance matrix data

2. Indexer Tensors: All indexer.* tensors were kept in BF16 format as they are critical for the model's routing/indexing mechanism and no importance data was available.

3. Final Layer: blk.78 (the final transformer block) was largely left unquantized to preserve output quality, as importance data was missing for these tensors.

4. Dense Layers: The first three blocks (blk.0 to blk.2) use dense FFN layers instead of MoE, which were quantized with Q2_K.

5. Importance Matrix: Quantization used an importance matrix from 80 chunks of calibration data, optimizing which tensors receive more bits based on their importance to model performance.