GLM-4.7-Flash-abliterated-FP8-Dynamic

FP8 (W8A8, dynamic) quantization of huihui-ai/Huihui-GLM-4.7-Flash-abliterated, itself an abliterated finetune of zai-org/GLM-4.7-Flash (a 30B-A3B glm4_moe_lite MoE model).

Quantized by mlavkin.

What this is

• Weights: FP8 E4M3, per-channel scales.
• Activations: FP8 dynamic, per-token (W8A8).
• Kept in BF16 (not quantized): all MoE router gates (*.mlp.gate) and lm_head — standard for MoE to preserve routing quality.
• Format: compressed-tensors (float-quantized).
• Size: ~29 GB (vs ~59 GB BF16).

Why

FP8 halves weight memory traffic. On memory-bandwidth-bound hardware this yields a large decode speedup with negligible quality loss. Measured on an NVIDIA GB10 (DGX Spark), single-stream decode vs BF16:

Context	BF16	FP8	Speedup
~0.6k	25.0	41.3	1.65×
~13k	21.7	33.6	1.55×
~33k	18.2	25.6	1.41×

Quality spot-checks (arithmetic, factual, code, needle-in-haystack retrieval up to ~47k) matched BF16.

Usage (vLLM)

Requires a recent vLLM + Transformers with glm4_moe_lite support.

vllm serve operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --max-model-len 202752 \
  --host 0.0.0.0 --port 8000

The model's native context is 202,752 tokens. RoPE scaling (e.g. YaRN) can extend it further but noticeably degrades coherence — native length is recommended.

Quantization recipe

Produced with llm-compressor:

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = AutoModelForCausalLM.from_pretrained(SRC, dtype="bfloat16", device_map="cpu")
recipe = QuantizationModifier(
    targets="Linear", scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*mlp.gate$", "re:.*eh_proj$"],
)
oneshot(model=model, recipe=recipe, output_dir=OUT)

FP8_DYNAMIC needs no calibration data (data-free).

License

MIT, inherited from the base models. Abliteration by huihui-ai; original model by Z.ai (zai-org).