leffff/diffusion-alignment-workshop-adapters

Diffusion Alignment Workshop: Sana-600M aligned for text rendering

LoRA adapters that align Sana-600M-512px to render the text asked for in the prompt (a chalkboard with the word "Recycling" written on it), trained with six methods, plus two more reward experts and their on-policy-distilled merge. Every adapter was trained for 5 minutes on one H100 with the workshop notebook. Data: leffff/diffusion-alignment-workshop.

Rewards: OCR = 1 − (character edits to find the target text in the Qwen3-VL-2B transcription) / its length; PickScore; aesthetic (LAION predictor v2); CLIP score (held out, never trained on). Evaluation: 64 held-out prompts × 2 seeds, 20 Euler steps, CFG 4.5.

file	method	data	training (5 min)	OCR ↑	PickScore	aesthetic	CLIP
—	Sana-600M (reference)	—	—	0.569	22.41	5.05	32.2
adapter_sft.pt	SFT on the best of 5 samples (OCR ≥ 0.5)	offline	1018 it × 32, lr 3e-4	0.712	22.45	4.91	33.3
adapter_rwr.pt	reward-weighted regression, group softmax τ = 0.1	offline	1052 it × 6 prompts × 5, lr 3e-4	0.696	22.42	4.95	33.0
adapter_dpo.pt	Diffusion-DPO, best vs worst of 5, β = 500	offline	463 it × 32 pairs, lr 3e-4	0.677	22.51	4.99	32.9
adapter_online_dpo.pt	online Diffusion-DPO, β = 500, 4 updates / rollout	online	40 it × 16 prompts × 4, lr 1e-3	0.612	22.40	5.04	32.5
adapter_grpo.pt	Flow-GRPO, SDE window of 2 steps, branching groups	online	28 it × 8 prompts × 8, lr 1e-3	0.599	22.40	5.03	32.3
adapter_nft.pt	DiffusionNFT (OCR expert), β = 1	online	38 it × 8 prompts × 8, lr 1e-3	0.746	22.35	4.79	32.3
adapter_nft_pick.pt	DiffusionNFT on PickScore (expert)	online	40 it × 8 × 8, lr 1e-3	0.687	22.54	4.95	32.6
adapter_nft_aes.pt	DiffusionNFT on the aesthetic score (expert, reward-hacked: drops the text)	online	39 it × 8 × 8, lr 1e-3	0.259	21.88	5.56	29.9
adapter_opd.pt	OPD: on-policy distillation of $v_\text{ref}+\sum_k(v_k-v_\text{ref})$ over the three experts	online, no reward	101 it × 32, lr 1e-4	0.642	22.52	5.01	32.9

For comparison, merging the three experts without training: LoRA sum 0.558 / 22.32 / 5.13 / 31.3, LoRA mean 0.581 / 22.42 / 5.07 / 32.2, and the composed teacher (4 networks per step) 0.620 / 22.49 / 5.04 / 32.6. The OPD student matches the teacher at one network per step.

Run-to-run spread at 5 minutes (OCR, several seeds): online DPO 0.646 ± 0.03, GRPO 0.632 ± 0.02, DiffusionNFT 0.736 ± 0.01. The adapters here are from the single reference run of the notebook.

Format

Each .pt file is a dict produced by the notebook's save_adapter: a peft LoRA state dict (get_peft_model_state_dict, fp32, r = 32, alpha = 32) for the targets to_q, to_k, to_v, to_out.0, conv_inverted, conv_point, proj_out of SanaTransformer2DModel. All adapters are used with CFG 4.5 (the policy is the guided velocity).

Usage (in the workshop notebook)

from huggingface_hub import hf_hub_download
import shutil
for m in ["sft", "rwr", "dpo", "online_dpo", "grpo", "nft", "nft_pick", "nft_aes", "opd"]:
    shutil.copy(hf_hub_download("leffff/diffusion-alignment-workshop-adapters", f"adapter_{m}.pt"), f"{CACHE_DIR}/adapter_{m}.pt")
    load_adapter(m)
evaluate("DiffusionNFT", pi("nft"))