experimentalmachines/gemma-3-1b-it-ExecuTorch

gemma-3-1b-it for ExecuTorch

ExecuTorch exports of google/gemma-3-1b-it (revision dcc83ea841ab) for phone NPUs only: each file needs the runner its section below names, not a generic ExecuTorch runtime.

Files

Windows in this repo: Qualcomm QNN (HTP) SM8750 (Snapdragon 8 Elite) at 2k to 32k; MediaTek NeuroPilot MT6991 (Dimensity 9400) at 512 to 32k. The window is fixed inside the file: the runtime allocates the whole KV cache at load, so pick the largest window the device can hold (fits_phone_budget in each folder's config.json is the estimate against a 5 GB budget).

Backend	Target	File	Window	Size	Smoke test
Qualcomm QNN (HTP)	SM8750 (Snapdragon 8 Elite)	qnn/sm8750/gemma-3-1b-it-qnn-hybrid-2k.pte	2,048 tokens	1.15 GB	structure checked (no host NPU runtime)
Qualcomm QNN (HTP)	SM8750 (Snapdragon 8 Elite)	qnn/sm8750/gemma-3-1b-it-qnn-hybrid-4k.pte	4,096 tokens	1.15 GB	structure checked (no host NPU runtime)
Qualcomm QNN (HTP)	SM8750 (Snapdragon 8 Elite)	qnn/sm8750/gemma-3-1b-it-qnn-hybrid-8k.pte	8,192 tokens	1.16 GB	structure checked (no host NPU runtime)
Qualcomm QNN (HTP)	SM8750 (Snapdragon 8 Elite)	qnn/sm8750/gemma-3-1b-it-qnn-hybrid-16k.pte	16,384 tokens	1.17 GB	structure checked (no host NPU runtime)
Qualcomm QNN (HTP)	SM8750 (Snapdragon 8 Elite)	qnn/sm8750/gemma-3-1b-it-qnn-hybrid-32k.pte	32,768 tokens	1.21 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-512-chunk1of2.pte	512 tokens	0.38 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-512-chunk2of2.pte	512 tokens	0.69 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-2k-chunk1of2.pte	2,048 tokens	0.38 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-2k-chunk2of2.pte	2,048 tokens	0.69 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-4k-chunk1of2.pte	4,096 tokens	0.38 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-4k-chunk2of2.pte	4,096 tokens	0.69 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-8k-chunk1of2.pte	8,192 tokens	0.38 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-8k-chunk2of2.pte	8,192 tokens	0.69 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-16k-chunk1of2.pte	16,384 tokens	0.38 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-16k-chunk2of2.pte	16,384 tokens	0.69 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-32k-chunk1of2.pte	32,768 tokens	0.38 GB	structure checked (no host NPU runtime)
MediaTek NeuroPilot	MT6991 (Dimensity 9400)	mtk/mt6991/gemma-3-1b-it-neuropilot-a16w8-32k-chunk2of2.pte	32,768 tokens	0.69 GB	structure checked (no host NPU runtime)

MediaTek folders also hold the token embedding table the NeuroPilot runner reads from disk, shared by every window: mtk/mt6991/gemma-3-1b-it-neuropilot-embedding-fp32.bin.

Tokenizer: tokenizer.json, copied unchanged from the source repo. Each backend folder has a config.json listing every window as a variant with the metadata the .pte reports, and an export-report-<window>.json per file with the full export record.

Memory

How it was made

• Qualcomm QNN (HTP) SM8750 (Snapdragon 8 Elite): ExecuTorch 1.5.1 Qualcomm static LLM (examples/qualcomm/oss_scripts/llama, --decoder_model gemma3-1b): the quantization recipe ExecuTorch registers for this model, calibrated on wikitext (1 sample), hybrid prefill (128 tokens per step) and decode graphs compiled with QAIRT 2.37.0.250724 for SM8750 (Snapdragon 8 Elite).
• MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.5.1 MediaTek LLM export (examples/mediatek, gemma.py): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's alpaca.txt prompts (9 of 9) in the gemma.json chat template, cut into 2 chunks, with a 128-token prompt graph and a one-token generation graph over a 512-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400).
• MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.5.1 MediaTek LLM export (examples/mediatek, gemma.py): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's alpaca.txt prompts (9 of 9) in the gemma.json chat template, cut into 2 chunks, with a 128-token prompt graph and a one-token generation graph over a 2048-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400).
• MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.5.1 MediaTek LLM export (examples/mediatek, gemma.py): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's alpaca.txt prompts (9 of 9) in the gemma.json chat template, cut into 2 chunks, with a 128-token prompt graph and a one-token generation graph over a 4096-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400).
• MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.5.1 MediaTek LLM export (examples/mediatek, gemma.py): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's alpaca.txt prompts (9 of 9) in the gemma.json chat template, cut into 2 chunks, with a 128-token prompt graph and a one-token generation graph over a 8192-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400).
• MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.5.1 MediaTek LLM export (examples/mediatek, gemma.py): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's alpaca.txt prompts (9 of 9) in the gemma.json chat template, cut into 2 chunks, with a 128-token prompt graph and a one-token generation graph over a 16384-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400).
• MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.5.1 MediaTek LLM export (examples/mediatek, gemma.py): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's alpaca.txt prompts (9 of 9) in the gemma.json chat template, cut into 2 chunks, with a 128-token prompt graph and a one-token generation graph over a 32768-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400).

License

A quantized derivative of google/gemma-3-1b-it, distributed under the same terms (gemma). See NOTICE for the attribution the license requires.

The qnn/ folders hold QNN HTP context binaries compiled with the Qualcomm AI Runtime SDK (QAIRT) 2.37.0.250724 from Qualcomm Technologies, Inc., used under its AI Stack License. No Qualcomm SDK or runtime library is included; running them needs the matching QNN runtime (for example executorch-android-qnn 1.5.1, which depends on qnn-runtime 2.37.0).

The mtk/ folders hold model binaries compiled with the MediaTek NeuroPilot Express SDK 8.0.8-build20250925 (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) from MediaTek Inc., used under MediaTek's license terms for that SDK. No MediaTek SDK or runtime library is included. They run on MediaTek's LLM runner from ExecuTorch (examples/mediatek/executor_runner) with the device's NeuroPilot runtime. That runner takes its settings as command-line flags and does not read config.json; the values to pass it are in each folder's config.json, under each variant's runner.

Variants whose runner lists rot_emb_inputs: [global, local] take two rotary-embedding inputs, in that order after the masks: the global layers' table at rot_emb_base (1e+06) and the sliding layers' at rot_emb_local_base (10000), with a sliding-window mask of window_size (512) beside the global one. MediaTek's runner as ExecuTorch 1.5.1 ships it builds one table and cannot run them correctly, and neither app feeds the second table yet; a runner has to build both. With one table for every layer this model was measured far from Hugging Face's (perplexity 22.7 against 9.6, fp32); with both it matches exactly.