experimentalmachines/Qwen2.5-3B-ExecuTorch

Qwen2.5-3B for ExecuTorch

ExecuTorch exports of Qwen/Qwen2.5-3B (revision 3aab1f1954e9) for on-device inference with the openweights Android app or any ExecuTorch 1.4.0 runtime.

Files

Every backend is exported at every context window the runner could build (2k to 32k). The window is fixed inside the file: the runtime allocates the whole KV cache at load, so pick the largest window the device can hold (fits_phone_budget in each folder's config.json is the estimate against a 5 GB budget).

Backend	Target	File	Window	Size	Smoke test
XNNPACK (CPU)	any arm64	xnnpack/Qwen2.5-3B-8da4w-2k.pte	2,048 tokens	2.05 GB	passed ("Paris")
XNNPACK (CPU)	any arm64	xnnpack/Qwen2.5-3B-8da4w-4k.pte	4,096 tokens	2.06 GB	passed ("Paris")
XNNPACK (CPU)	any arm64	xnnpack/Qwen2.5-3B-8da4w-8k.pte	8,192 tokens	2.07 GB	passed ("Paris")
XNNPACK (CPU)	any arm64	xnnpack/Qwen2.5-3B-8da4w-16k.pte	16,384 tokens	2.08 GB	passed ("Paris")

Tokenizer: tokenizer.json, copied unchanged from the source repo. Each backend folder has a config.json listing every window as a variant with the metadata the .pte reports, and an export-report-<window>.json per file with the full export record.

Memory

• XNNPACK (CPU) at 2,048 tokens: the KV cache costs 73,728 bytes per token (fp32), 150,994,944 bytes for the whole window, allocated in full when the model loads.
• XNNPACK (CPU) at 4,096 tokens: the KV cache costs 73,728 bytes per token (fp32), 301,989,888 bytes for the whole window, allocated in full when the model loads.
• XNNPACK (CPU) at 8,192 tokens: the KV cache costs 73,728 bytes per token (fp32), 603,979,776 bytes for the whole window, allocated in full when the model loads.
• XNNPACK (CPU) at 16,384 tokens: the KV cache costs 73,728 bytes per token (fp32), 1,207,959,552 bytes for the whole window, allocated in full when the model loads.

How it was made

• XNNPACK (CPU) any arm64: ExecuTorch 1.4.0 export_llm: 8-bit dynamic activations and 4-bit weights in groups of 32, int8 per-channel embeddings, XNNPACK with extended ops, prefill chunk 2048, fp32 KV cache. Built by run 1.

License

A quantized derivative of Qwen/Qwen2.5-3B, distributed under the same terms (qwen-research). The upstream license files are included unchanged: LICENSE.