felipesztutman/MiniMax-H3-W4A4

MiniMax H3 W4A4 (SVDQuant, nunchaku)

The 50 transformer blocks of MiniMax H3 quantized to int4, weights and activations, in the SVDQuant lineage. As far as the public record shows this is the first working weights-and-activations int4 of H3 on pre-Blackwell GPUs. It runs on one RTX 4090, and the venue target is an RTX 3090.

H3 is a 33B omni transformer that generates video and stereo audio in the same denoising pass. Its published paths want a 128GB unified-memory box. The other 4-bit ports on the Hub require a Blackwell GPU and quantize weights only, or mark their W4A4 variants experimental. This one quantizes weights and activations with real int4 GEMM kernels through nunchaku, calibrated on activations collected during real generation.

Numbers (RTX 4090, 124-frame 832x480 clips with stereo audio, same seed as the bf16-path reference)

build	denoise step	clip	VRAM resident	VRAM peak
int8 weight-only + block offload (baseline)	~14 to 15 s	~950 s	streaming	~20 GB
W4A4, AdaLN streamed	8.0 s	624 s	13.0 GB	18.6 GB
W4A4, AdaLN collapsed to a table	3.65 s	408 s	14.0 GB	19.6 GB

Calibration of all 50 blocks took 24 minutes on the same card. Quality holds by eye across an 8-scene bench, trajectories diverge from the reference as any 49-step rollout does under perturbation, coherence and detail do not drop.

Files

transformer_blocks.safetensors (10.58 GB) holds the 300 quantized linears in the standard nunchaku two-file block format, qweight int4 group 64, wscales, rank-32 low-rank branch, smooth factors. Rows follow the diffusers layout of the root transformer/ of the H3 repo, to_q, to_k, to_v, to_out.0, ff.net.0.proj, ff.net.2 per block. Everything the checkpoint does not carry, AdaLN, norms, embedders, token refiner, heads and VAEs, is read from the original H3 repo at load time.

How to run

Loader, runtime, calibration pipeline and all the documented traps live in the companion repo, sztlink/h3-consumer-bench. The short version. Build the diffusers modular pipeline of H3 (PR 14355 branch), load the transformer int8, swap the 300 block linears for SVDQW4A4Linear from this checkpoint behind a 2D wrapper, precompute the AdaLN table from the deterministic sigma schedule, leaf-offload the 9.8GB video VAE. nunchaku 1.3.0.dev wheels cover torch 2.11.

One trap deserves the front page. The H3 repo ships two transformers. FL2VA/transformer is the reference sibling that never executes in the t2va and i2va paths. The runtime uses the root transformer/, different weights, same shapes. Calibrate against the wrong twin and every diagnostic stays green while the output is pure noise.

Lineage

Krea 2 Turbo W4A4 port, then krea-realtime-bench, then h3-consumer-bench, then this. Quantized weights are a derivative of MiniMax H3 and carry the MiniMax H3 Community License.

This checkpoint powers Arquivo Terra Viva, a machine that writes a novel, prints it line by line on a dot matrix printer, and paints a 4.8m wall with it in Rio de Janeiro, no cloud, on the GPUs we already own. heretica.xyz