experimentalmachines/Llama-3.2-3B-Instruct-ExecuTorch

Built with Llama

Llama-3.2-3B-Instruct for ExecuTorch

ExecuTorch exports of meta-llama/Llama-3.2-3B-Instruct (revision 0cb88a4f764b) for on-device inference with the openweights Android app or any ExecuTorch 1.4.0 runtime.

Files

Every backend is exported at every context window the runner could build (2k to 32k). The window is fixed inside the file: the runtime allocates the whole KV cache at load, so pick the largest window the device can hold (fits_phone_budget in each folder's config.json is the estimate against a 5 GB budget).

Backend	Target	File	Window	Size	Smoke test
XNNPACK (CPU)	any arm64	xnnpack/Llama-3.2-3B-Instruct-8da4w-2k.pte	2,048 tokens	2.21 GB	passed ("Paris")
XNNPACK (CPU)	any arm64	xnnpack/Llama-3.2-3B-Instruct-8da4w-4k.pte	4,096 tokens	2.21 GB	passed ("Paris")
XNNPACK (CPU)	any arm64	xnnpack/Llama-3.2-3B-Instruct-8da4w-8k.pte	8,192 tokens	2.21 GB	passed ("Paris")
XNNPACK (CPU)	any arm64	xnnpack/Llama-3.2-3B-Instruct-8da4w-16k.pte	16,384 tokens	2.22 GB	passed ("Paris")

Tokenizer: tokenizer.json, copied unchanged from the source repo. Each backend folder has a config.json listing every window as a variant with the metadata the .pte reports, and an export-report-<window>.json per file with the full export record.

Memory

• XNNPACK (CPU) at 2,048 tokens: the KV cache costs 229,376 bytes per token (fp32), 469,762,048 bytes for the whole window, allocated in full when the model loads.
• XNNPACK (CPU) at 4,096 tokens: the KV cache costs 229,376 bytes per token (fp32), 939,524,096 bytes for the whole window, allocated in full when the model loads.
• XNNPACK (CPU) at 8,192 tokens: the KV cache costs 229,376 bytes per token (fp32), 1,879,048,192 bytes for the whole window, allocated in full when the model loads.
• XNNPACK (CPU) at 16,384 tokens: the KV cache costs 229,376 bytes per token (fp32), 3,758,096,384 bytes for the whole window, allocated in full when the model loads.

How it was made

• XNNPACK (CPU) any arm64: ExecuTorch 1.4.0 export_llm: 8-bit dynamic activations and 4-bit weights in groups of 32, int8 per-channel embeddings, XNNPACK with extended ops, prefill chunk 2048, fp32 KV cache. Built by run 1.

License

A quantized derivative of meta-llama/Llama-3.2-3B-Instruct, distributed under the same terms (llama3.2). The upstream license files are included unchanged: LICENSE.txt, USE_POLICY.md. See NOTICE for the attribution the license requires.