diffusers-modular/MiniMax-H3-Pruned-Ref-Delta-Fused-r1024

MiniMax-H3, one transformer for references and keyframes

MiniMax-H3 ships two 37.5 GB transformer partitions: fl2va for first/last-frame conditioning and ref2va for reference conditioning. This is one checkpoint that serves both, so a deployment holds 37.5 GB instead of 75 GB. It is the pruned fl2va partition with a rank-1024 approximation of the ref2va − fl2va weight delta fused into the weights.

Experimental and mechanically derived — the delta was extracted from two released checkpoints by SVD, not trained. This is not transformer_ref: against the real thing it reaches video-latent cosine 0.875 / 0.897 / 0.691 on three matched reference requests, short of the 0.99 that would make it a drop-in replacement. Use it when one-partition deployment is worth that gap.

Governed by the MiniMax H3 Community License Agreement, which carries territory exclusions and redistribution conditions. The upstream LICENSE is included and this card is the modification notice.

Inference

Both workflows resolve to the weights in this repo, and each loads only its own slot:

import torch
from diffusers import ComponentsManager, ModularPipeline

REPO = "diffusers-modular/MiniMax-H3-Pruned-Ref-Delta-Fused-r1024"

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained(REPO, workflow="ref2va", components_manager=manager,
                                      trust_remote_code=True)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
manager.enable_auto_cpu_offload(device="cuda")   # 37.5 GB DiT + 62 GB conditioner + two VAEs

References (workflow="ref2va") — image, video and/or audio, in the order they are passed:

from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3ImageReference

out = pipe(
    prompt="The woman from the reference stands in a snowy park at dusk, catching snowflakes and laughing.",
    references=[MiniMaxH3ImageReference.from_file("subject.png")],
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)

Keyframes (workflow="fl2va") — a first and/or last frame:

from PIL import Image

out = pipe(                       # same repo, reloaded with workflow="fl2va"
    prompt="She lifts the teacup and takes a slow sip, then smiles; quiet kitchen room tone.",
    image=Image.open("first.png"), last_image=Image.open("last.png"),
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)
video, audio, rate = out["videos"][0], out["audio"][0], out["sampling_rate"]

One generation carries one shape: the blocks dispatch on their inputs and references wins, so passing both silently drops the keyframes.

Few-step generation — keep the turbo LoRA live

pipe.load_lora_weights("larryvrh/MiniMax-H3-Turbo-Lora", adapter_name="turbo",
                       load_into_transformer_ref=True)   # `load_into_transformer_ref` only for ref2va

Do not fuse it. A distill LoRA's deltas are 2–5e-4 of the weights they modify, and folding them into bf16 keeps only 0.67–0.79 — a third of turbo's AdaLN modulation is lost to rounding. Left live it costs ~7–15% wall time. For the same reason, an AoT-compiled transformer cannot be combined with a live adapter: the graph is captured from the base modules and runs straight past it.

Requires diffusers with MiniMaxH3LoraLoaderMixin (PR #14408) and trust_remote_code=True — the pruned layout ships a MiniMaxH3PrunedTransformer3DModel whose AdaLN is 8 wide, where the stock class expects 2688.

How it was made

1. Take the difference between the twins. ref2va − fl2va, tensor by tensor. Both partitions are architecturally identical, so this is what makes one able to use references.
2. Compress it. ethanfel's randomized-SVD extraction at rank 1024 (9.4 GB), applied at strength 1.0.
3. Apply the parts that cannot be compressed exactly. 267 patches: 211 RMSNorm deltas, 56 biases, and adaln_t_table — the timestep coordinate table, which differs between the partitions and which the earlier rank-256 extractions omit.
4. Fuse. fuse_lora, then unload the adapter, leaving an ordinary checkpoint. The delta is 0.02–1.9 of the weights it lands in, so it survives a bf16 fold at 1.00 (unlike a distill LoRA — see above).
5. Keep the AdaLN affine map. adaln_basis / adaln_mean ship as buffers, so LoRAs trained on the released 2688-wide AdaLN still project onto this 8-wide one.

Verified by re-downloading this repo and generating: bit-identical (torch.equal on video and audio latents) to the local build it was made from.

Measured

Video-latent cosine against the true ref2va partition, generated on the same GPU (identical weights on a different GPU only agree to 0.959–0.987, so cross-machine anchors are not usable at this precision). Three seed-matched requests: photoreal image reference, stylized image reference, video reference.

applied to pruned FL2VA	size	photoreal	stylized	video ref
rank-1024 delta (this repo)	9.4 GB	0.875	0.897	0.691
rank-256 delta	2.4 GB	0.772	0.798	0.527
AdaLN-only, exact	95 MiB	0.678	0.803	0.505
nothing	0	0.668	0.795	0.480

Rank is what matters, and the bulk is load-bearing: the exact AdaLN half — the tempting 95 MiB shortcut — lands on the no-delta baseline, so the trunk attention and MLP deltas are carrying the result.

Keyframes still work. Against the pristine fl2va partition on the same first+last-frame request, this checkpoint reaches video cosine 0.930 and is visually indistinguishable. Audio is the weaker half (0.662, and it runs louder). Few-step, with turbo live: 64–179 s per 5 s clip against 174–478 s for 20 steps on the true partition, at the VRAM of a single partition.

Credit

Delta extraction ethanfel; approach established by Kijai; pruned repack Comfy-Org; turbo LoRA larryvrh; base model and license MiniMaxAI. Related community work on the same question: lihaoyun6 (exact-diff-only patch), smhfacct (AdaLN block swap), PulpCut (turbo merged into Ref2VA).