mesmertech/Breeze-TTS-2-int4-hqq-g64

Breeze-TTS-2 int4 (HQQ g64, pre-packed)

Weight-only int4 derivative of BreezeBlue/Breeze-TTS-2 (revision c1c8ca18b70b, Apache-2.0). This is not a drop-in transformers checkpoint -- it needs the breeze-tts-runpod loader (int4_tinygemm.py + load_packed.py).

What was quantised

280 bias-free nn.Linears: the 84 in depth_decoder.model.layers[0..11] and the 196 in backbone_model.layers[0..27] (self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj).

Never quantised: the T5Gemma2 text encoder, lm_head, the codebooks head, every embedding, the two projectors, codec_model, and the Qwen3-TTS audio_tokenizer/. Those tensors are here verbatim, in their original dtype and under their original names.

3.25 GiB of bf16 decoder weights become 0.91 GiB of int4 + bf16 scales.

Recipe

item	value
method	HQQ proximal solver (hqq 0.2.8.post1), nbits=4, axis=1, channel_wise, optimize=True, round_zero=False, no calibration data
group size	64, along K
solver arithmetic	float32 on the build host. HQQ picks float16 only when device.type == "cuda" (hqq/core/optimize.py:231); this checkpoint was built on an Apple M1 Pro, so HQQ's own non-CUDA float32 branch ran. The resulting per-Linear weight errors match the CUDA/float16 gate run to four decimals (0.0889 depth / 0.0879 backbone).
dequant	w = (q - 8) * scale + zero, q in [0, 15] (torch int4mm.cu convention)
<m>.qweight	uint8 [N, K/2], byte j = (q[:, 2j] << 4) | q[:, 2j+1]
<m>.scales_and_zeros	bfloat16 [K/64, N, 2], [..., 0] = scale, [..., 1] = zero
kernel	torch._weight_int4pack_mm(x_bf16, packed, 64, scales_and_zeros), sm_80+
weight error (Frobenius, per Linear, with the bf16-rounded scales the kernel really sees)	depth mean 0.0889 / max 0.0982; backbone mean 0.0879 / max 0.0932

Why "pre-packed"

qweight is stored in the portable nibble layout -- exactly the tensor that torch._convert_weight_to_int4pack takes as input -- not in torch's tensor-core-tiled output layout, which is architecture- and version-specific (innerKTiles semantics changed at torch 2.5) and has no CPU equivalent. So a loader does:

packed = torch._convert_weight_to_int4pack(qweight.cuda(), 2)   # milliseconds
mod = Int4Linear(packed, scales_and_zeros.cuda(), K, N, 64)

instead of re-running the quantiser at every boot (0.6 s for RTN g128, 3.4 s for HQQ g64 on an RTX 4090). Shipping HQQ pre-packed is the only reason HQQ is practical here: it has the lower weight error of the two validated variants but was too slow to redo per process.

Quality

The recipe (not this artefact) passed a 320-clip distributional gate on an RTX 4090: 5 texts x {design, clone} x 8 seeds x 4 variants, scored with UTMOS22, DNSMOS, Whisper WER, resemblyzer speaker cosine, pyin F0 and words/s. No int4 variant was distinguishable from bf16 beyond the bf16 seed-to-seed noise floor; HQQ g64 had the lowest clone WER. Speed on a 4090: 47.7 -> 33.5 ms per codec frame (design), 41.3 -> 31.5 (clone); VRAM after warmup 7.53 -> 5.20 GiB.

Files

file	bytes
model-int4.safetensors	4,461,414,090
audio_tokenizer/model.safetensors	682,293,092
tokenizer.json	33,386,945
tokenizer_config.json	1,157,960
model.safetensors.index.json	123,791
quant_stats.json	54,108
int4_manifest.json	19,875
config.json	10,161
README.md	4,976
audio_tokenizer/config.json	2,336
special_tokens_map.json	886
generation_config.json	251
audio_tokenizer/preprocessor_config.json	234
audio_tokenizer/configuration.json	76

model.safetensors.index.json is regenerated and maps every tensor to the single model-int4.safetensors. config.json is the upstream one, verbatim -- it still says text_encoder_config.preferred_attn_implementation = "flash_attention_2", which breeze-tts-runpod/model_setup.py:ensure_attn_patch rewrites to sdpa at boot.

Loading

<m>.weight does not exist for the 280 quantised modules, so transformers will report them as missing and report .qweight / .scales_and_zeros as unexpected. See load_packed.py in breeze-tts-runpod/optimization-research/J-packed-checkpoint/ for the two construction routes and the load_packed_int4(model, packed_dir, device) reference implementation. The swap must happen after load_runtime and before Engine.warmup(), which is the CUDA-graph capture.

Licence

Apache-2.0, inherited from BreezeBlue/Breeze-TTS-2. This repository contains only a derived (quantised) form of those weights plus the upstream tokenizer/config files verbatim.