qtum/Hy4-preview-GGUF

Hy4-preview GGUF

GGUF quantizations of tencent/Hy4-preview, made with llama.cpp.

Chinese version: README_zh.md

780B total parameters, 78 layers. DeepSeek-style MLA attention (q-LoRA rank 2048, kv-LoRA rank 512) with a gated output path and learnable attention sinks, wrapped in iHC hyper-connections on every attention and MLP block. 256 routed MoE experts with top-8 routing plus one shared expert. DSA sparse attention: 21 of the 78 layers own a full indexer, the rest share the preceding full layer's top-k. Vocabulary 120,832.

Quantized from the official BF16 weights. Every tier is imatrix-calibrated and ships as 39 shards.

Quantizations

Tier	Size	Shards	BPW	PPL (wikitext-2)	Mean KLD	Same top-1
master (BF16, not in this repo)	1435.7 GiB	39	16.02	2.9361 ± 0.0253	—	—
IQ4_XS	388.3 GiB	39	4.33	3.0789 ± 0.0157	0.0793 ± 0.0008	90.50 %
Q2_K	269.6 GiB	39	3.01	3.8898 ± 0.0217	0.3451 ± 0.0030	80.35 %
IQ2_XS	218.9 GiB	39	2.44	4.9648 ± 0.0294	0.5988 ± 0.0044	73.44 %

The master row is not a file in this repo. It is listed so the numbers above have a reference point — the 1.4 TiB BF16 master was actually loaded and measured on the same machine, so every tier is compared against a real baseline rather than an estimate.

Pick IQ4_XS unless you cannot fit it. It keeps the same top-1 token as the BF16 master on 90 % of positions and its KL divergence is 4.4× lower than the next tier down. Q2_K is the balanced choice and the best option for CPU-only inference. IQ2_XS exists for when the footprint is hard-capped.

What the imatrix is worth

Every tier here is calibrated with an importance matrix. To quantify what that is worth, Q2_K was built twice — once with the imatrix and once without — at byte-identical size (276,118 MiB both times), so the only variable is the calibration:

	PPL	Mean KLD	Same top-1	RMS Δp
Q2_K without imatrix	4.2988 ± 0.0238	0.4543 ± 0.0034	76.98 %	24.82 %
Q2_K as shipped	3.8898 ± 0.0217	0.3451 ± 0.0030	80.35 %	21.36 %
improvement	−9.5 %	−24.0 %	+3.4 pts	−13.9 %

Measured against this repo's own size-quality curve, that 24 % KL reduction is worth +0.257 BPW — i.e. reaching the same quality without an imatrix would need a file 8.5 % larger (about 23 GiB more on the Q2_K tier). The imatrix itself costs nothing in the shipped weights.

Where the gain lands, by KL percentile:

Percentile	without	with	gain
10 %	0.00155	0.00058	−62.8 %
median	0.16232	0.09597	−40.9 %
90 %	1.2245	0.9229	−24.6 %
99 %	3.7199	3.3953	−8.7 %
99.9 %	6.5440	6.4235	−1.8 %

The improvement is concentrated in the body of the distribution, not the tail: an imatrix makes the typical token markedly closer to the original model, while the rare catastrophic divergences are a property of low-bit quantization that calibration cannot undo. This is exactly what per-channel importance weighting is expected to do.

Calibration corpus: 476 KiB of real-world text, 220 chunks at n_ctx=512, run over the full BF16 master (6.7 h on 8× H100).

What is protected

The routed experts are 96.5 % of this model's weights, so everything else can be kept at high precision for almost no size cost. It is kept:

Tensors	Type	Reason
ffn_gate_inp / exp_probs_b	F32	MoE router; compressing it routes tokens to the wrong experts
ffn_*_shexp	Q8_0	the shared expert runs on every token
hc_attn_* / hc_ffn_* / output_hc_*	F32	iHC hyper-connection streams, every layer
indexer.k_norm / indexer.proj	F32	DSA top-k selection; a wrong pick discards the whole attention span
indexer.attn_k / indexer.attn_q_b	Q8_0	DSA indexer projections
attn_kv_a_mqa / attn_k_b / attn_v_b / attn_q_a	Q8_0	MLA low-rank paths are already narrow; compressing them damages the KV cache
attn_gate / attn_sinks	Q8_0 / F32	gated-MLA output path and learnable sinks
ffn_gate / ffn_up / ffn_down	Q6_K	the single dense MLP layer before the MoE stack
token_embd / output	Q6_K	a global type would otherwise squeeze these hard

Usage

# Point at the first shard; llama.cpp finds the rest on its own.
llama-cli -m Hy4-preview-IQ4_XS-00001-of-00039.gguf

Do not pass -ngl manually. llama.cpp fits layers to free VRAM by itself, and an explicit -ngl — including 0 — aborts that fitting and can fail to allocate compute buffers even when the weights would have fit.

Every tier ships as 39 shards. Download all 39 into one directory — you only ever name -00001-of-00039 on the command line.

Requires a llama.cpp build with hy_v4 support (merged upstream 2026-09-04, PR #28127). Earlier builds will not load these files.

About the numbers

wikitext-2 test, n_ctx=512, 655 chunks (~335 K tokens), every tier through the exact same command on 8× H100 80GB. KL divergence and top-1 agreement are computed against logits saved from the BF16 master over 200 chunks.

These numbers are only comparable within this table. Do not compare them against PPL figures published by other repos — different corpora and chunk counts make absolute values meaningless across setups.

Two notes on why the error bars here are small enough to be useful: the PPL runs use 655 chunks rather than the handful common in quantization write-ups, and KL divergence is reported alongside PPL because perplexity is a mean statistic that hides where a quantized model actually breaks. The percentile table above is only readable because both were done.

License

Apache-2.0, inherited from tencent/Hy4-preview — see LICENSE for terms. Quantized by qtum.