Switch transformer & unconditional_transformer to split Q/K/V
#2
by multimodalart HF Staff - opened
Updates transformer/ and unconditional_transformer/ from fused attention.qkv/attention.o to split attention.to_q/to_k/to_v/to_out.0, matching the diffusers loader change in huggingface/diffusers#13859 (commit fbe4750).
The q/k/v weights are contiguous row-slices of the old fused weight, so the conversion is lossless (verified bit-exact after nf4 dequant). Configs are unchanged.
Source of truth: diffusers-internal-dev/ideogram-4-nf4-diffusers.
multimodalart changed pull request title from Switch transformer & unconditional_transformer to split Q/K/V to Match diffusers keys
multimodalart changed pull request title from Match diffusers keys to Switch transformer & unconditional_transformer to split Q/K/V
multimodalart changed pull request status to closed