latentdivergence/instruct-pix2pix-npu-8gen3

InstructPix2Pix for the Hexagon NPU (Snapdragon 8 Gen 3, HTP v75)

InstructPix2Pix (Brooks, Holynski, Efros, 2023) edits a photo from a typed instruction ("make it sunset", "turn it into a watercolour", "add snow"), changing the whole image while keeping its layout. This repo is the Snapdragon 8 Gen 3 (SM8650, Galaxy S24) build used by Latent Studio's "Edit with words": precompiled QNN context binaries (Qualcomm AI Hub, QAIRT 2.45) driven through ONNX Runtime's QNN execution provider, with fp32 inputs and outputs, at 512x512.

Files

Folder	Runs on	Graph	Precision	AI Hub ms/run	cosine vs fp32 (real Galaxy S24)
unet/	NPU	8-channel UNet, batch 1: sample [1,8,64,64] (scaled noisy latent 4 + input-image latent 4), timestep [1] f32, encoder_hidden_states [1,77,768] -> out_sample [1,4,64,64] (eps)	fp16	296.3	0.999921
vae_encoder_npu/	NPU	SD 1.5 VAE encoder: sample [1,3,512,512] in [-1,1] -> latent [1,4,64,64], posterior mean, unscaled	fp16	138.3	0.999976
vae_decoder_npu/	NPU	SD 1.5 VAE decoder: latent_sample [1,4,64,64] (= latent / 0.18215) -> sample [1,3,512,512] in [-1,1]	fp16	320.8	1.000000
text_encoder/	CPU	CLIP ViT-L/14 text: input_ids int64 [1,77] -> last_hidden_state [1,77,768], pooler_output	fp16 weights, fp32 compute		1.000000
vae_encoder/, vae_decoder/	CPU	fallback if the NPU VAE will not start (same IO as the NPU halves)	fp16 weights, fp32 compute
tokenizer/		CLIP BPE (vocab.json, merges.txt)

edit_defaults.txt: steps=10, sampler=dpmpp_2m_karras, text guidance=7.5, image guidance=1.5. UNet + VAE time per edit on AI Hub's Galaxy S24: about 9.3 s.

How to run one edit

1. Resize the photo to 512x512, RGB to [-1,1] CHW; vae_encoder_npu -> img (use it unscaled).
2. Encode the instruction and the negative prompt ("") with the text encoder -> c, u.
3. Start from x = N(0,1) * sigma_0 (dpmpp_2m_karras, Karras sigmas, SD 1.x scaled-linear schedule, 10 steps).
4. Each step, with xs = x / sqrt(sigma^2 + 1), run the UNet three times: e_full = U([xs, img], t, c), e_img = U([xs, img], t, u), e_unc = U([xs, 0], t, u) and combine eps = e_unc + s_img * (e_img - e_unc) + s_txt * (e_full - e_img); then take the sampler step.
5. vae_decoder_npu(x / 0.18215) -> image.

Build

Exported from timbrooks/instruct-pix2pix (diffusers, fp32) at a static 512px, batch 1. fp16 everywhere the HTP can hold it; the 8 Gen 2 UNet is w8a16 (AI Hub's Galaxy S23 cannot hold the fp16 UNet at inference), calibrated on 15 real edit trajectories (all three guidance branches, every step) over photos and instructions not used for evaluation. Each graph was run on a real AI Hub device and compared with fp32 PyTorch.

Chip-specific builds

Repo	Chip	SoC	Hexagon
instruct-pix2pix-npu-8gen2	Snapdragon 8 Gen 2	SM8550	v73
instruct-pix2pix-npu-8gen3	Snapdragon 8 Gen 3	SM8650	v75
instruct-pix2pix-npu-8elite	Snapdragon 8 Elite	SM8750	v79
instruct-pix2pix-npu-8elitegen5	Snapdragon 8 Elite Gen 5	SM8850	v81

Licence

See LICENSE.txt. The upstream InstructPix2Pix weights are MIT-licensed by their authors; because they are fine-tuned from Stable Diffusion v1.5, this repo also carries the CreativeML OpenRAIL-M licence and its use restrictions. Both permit commercial use. Credit: Tim Brooks, Aleksander Holynski, Alexei A. Efros (InstructPix2Pix); CompVis, Runway and Stability AI (Stable Diffusion); OpenAI (CLIP).