gladia/qat-transfer-vae

qat-transfer checkpoints: Stable Diffusion KL-f8 VAE decoders

Decoder checkpoints for the "Beyond classification" experiment (Appendix, Further Variants of QV Transfer) of Zero-Shot Quantization via Weight-Space Arithmetic (NeurIPS 2026). Code: github.com/gladia-research-group/qat-transfer · arXiv · OpenReview · Research graph

The other model families are in gladia/qat-transfer-timm, gladia/qat-transfer-open_clip, gladia/qat-transfer-text and gladia/qat-transfer-vae (SD KL-f8 VAE decoders). The staged-schedule QAT/QAD ViT-B/16 donors of the appendix are in gladia/qat-transfer-timm, checkpoints/000_finetune/.

Contents

84 decoder checkpoints of the SD-1 KL-f8 autoencoder (stable-diffusion-v1-5/stable-diffusion-v1-5@451f4fe, vae/), finetuned decoder-only on 12 image domains: celebahq, chartqa, chestxray, coco, floorplan, food101, lsun_bedroom, lsun_church, pixelart, publaynet, resisc45, rico.

Role	Weight format	Runs
FP	full precision	12
QAT	int3, symmetric, per-channel	12
QAD	int3, symmetric, per-channel	12
QAT	fp4 (e2m1), symmetric, per-channel	12
QAD	fp4 (e2m1), symmetric, per-channel	12
QAT	fp4 (e2m1), NVFP4 block scaling	12
QAD	fp4 (e2m1), NVFP4 block scaling	12

Shared configuration: 256 px, seed 2038, bf16_autocast_tf32, AdamW lr=1e-4, wd=0, gradient clipping 1.0, warmup 250, batch 32 × 2 accumulation, 5000 total steps, L1 + LPIPS loss. QA runs (weight-only, all decoder-side Linear and Conv2d layers) start from the FP run's step-1500 checkpoint and train for the remaining 3500 steps with the optimizer and schedule carried over. QAD uses the final FP decoder of the same domain as a frozen teacher (kd_pix·L1 + kd_lpips·LPIPS, no ground-truth term).

The quantization vector is QV = QA_final - FP_final for a donor domain; the patched receiver is FP_receiver + lambda * QV_donor, evaluated after weight PTQ in the same format.

Files

Each run directory mirrors the code repository's layout (the directory names are the run identity):

checkpoints/000_finetune/002_vae/000_sd1_klf8/finetune_fp/model=sd15_klf8/dataset={domain}/.../step=5000/model.pt
checkpoints/000_finetune/002_vae/000_sd1_klf8/finetune_qa/model=sd15_klf8/dataset={domain}/.../method={qat|qad},.../w_fmt=...,w_scheme=...,w_gran=.../.../step=5000/model.pt

• model.pt: a flat state dict of 140 fp32 tensors, decoder.* and post_quant_conv.* only (49.5 M params). QA checkpoints hold the trained full-precision latent weights with the fake-quantization wrappers removed. The encoder and quant_conv are the unchanged upstream weights.
• metrics.json, run_id.json, run_config.json: the run's held-out metrics and its full configuration.
• manifest.json at the root lists every run with its role, domain, format, path and parameters.

Loading

import torch
from diffusers import AutoencoderKL

vae = AutoencoderKL.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", subfolder="vae", revision="451f4fe")
sd = torch.load("model.pt", map_location="cpu")
missing, unexpected = vae.load_state_dict(sd, strict=False)
assert not unexpected and all(k.startswith(("encoder.", "quant_conv.")) for k in missing)

License

Derived from the Stable Diffusion v1.5 VAE (CreativeML Open RAIL-M). The finetuning datasets carry their own terms.

Citation

@inproceedings{solombrino2026zeroshot,
  title     = {Zero-Shot Quantization via Weight-Space Arithmetic},
  author    = {Solombrino, Daniele and Gargiulo, Antonio Andrea and Zirilli, Alessandro and Zhou, Luca and Minut, Adrian Robert and Rodol{\`a}, Emanuele},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://openreview.net/forum?id=wrUEnnSgXa}
}