znasif/embeddinggemma-2-qnn-v69

EmbeddingGemma 2 (+ Whisper base.en) for the Galaxy XR Hexagon V69 NPU

ExecuTorch .pte exports of google/embeddinggemma-2 (text, vision and audio encoders) and openai/whisper-base.en, lowered with the Qualcomm QNN backend (QAIRT 2.46.0, fp16, HTP V69 = Samsung Galaxy XR / Snapdragon XR2+ Gen 2, SoC SXR2230P). There are 0 CPU-fallback ops. A small native server (npu_server) runs them. These files are used by the xr-panosearch app; its tools/setup_device.sh downloads this repo and pushes npu/ to /data/local/tmp/npu/ on the headset.

The folder npu/ mirrors the device layout exactly.

File	What	Input → output	V69 latency
npu/embgemma2/embgemma2_text_S128_qnn_fp16.pte	text backbone, ≤128 tokens	inputs_embeds [1,128,512], mask [1,128] → L2-normalized [1,768]	50 ms
npu/embgemma2/embgemma2_text_S512_qnn_fp16.pte	text backbone, ≤512 tokens	[1,512,512], [1,512] → [1,768]	251 ms
npu/embgemma2/embgemma2_vision_landscape_qnn_fp16.pte	vision tower + embed_vision	RGB [1,3,672,960] in [0,1] → 280 soft tokens [1,280,512]	776 ms
npu/embgemma2/embgemma2_vision_portrait_qnn_fp16.pte	same, portrait	[1,3,960,672] → [1,280,512]	767 ms
npu/embgemma2/embgemma2_audio_5s_qnn_fp16.pte	audio tower + embed_audio	log-mel [1,500,128], mask [1,500] → [1,125,512]	67 ms
npu/embgemma2/embgemma2_audio_11s_qnn_fp16.pte	same, 11.2 s	[1,1120,128], [1,1120] → [1,280,512]	213 ms
npu/embgemma2/embed_tokens_262144x512_f16.bin	token embedding table (CPU side, ×22.627417)	row-major fp16	—
npu/embgemma2/tokenizer.json, embed_meta.json	tokenizer (from the base model), table metadata
npu/embgemma2/gemma4_mel_filters_128x257.f32	mel filterbank for the audio front end
npu/whisper_{encoder,decoder}_qnn_fp16.pte, whisper_mel_filters.bin, whisper_vocab.bin	Whisper base.en (encoder 99 ms, decoder 9 ms/token)
npu/npu_server	arm64 Android server binary (stripped) that runs everything above	stdin/stdout frames
npu/libqnn_executorch_backend.so	ExecuTorch QNN delegate runtime

Latencies were measured on device with the headset off (asleep). While the headset is worn the NPU runs about 1.7× slower; the cause is likely the headset's own tracking sharing the DSP, but that is not confirmed.

Accuracy on device vs PyTorch fp32: final 768-d embedding cosine ≥ 0.999997 (text), ≥ 0.999984 (image), ≥ 0.999997 (audio). Retrieval rankings were identical to fp32 on the test sets.

fp16 note: stock fp16 overflows inside RMSNorm (activations up to |x| ≈ 2200). The exports rescale each RMSNorm input exactly (text: by max|x|; vision and audio: by fixed powers of two) before squaring.

Images: stretch to 960×672 (landscape) or 672×960 (portrait) with an antialiased bicubic resize, PIL BICUBIC style. Bilinear or area resizing costs up to 0.035 cosine on thin text.

Not included: the Qualcomm QNN runtime libraries (libQnnHtp*.so, libQnnSystem.so). Get them from Maven Central com.qualcomm.qti:qnn-runtime:2.46.0 (Qualcomm license). The setup script does this.

Licenses: EmbeddingGemma 2 derivatives are Apache-2.0 (base model license), Whisper derivatives are MIT, and ExecuTorch is BSD-3-Clause.