software-mansion/Llama-3.2-ExecuTorch

llama-3.2

This repository hosts the llama-3.2 models exported for the React Native ExecuTorch library as ExecuTorch .pte programs, ready to run on device.

Variants

Path	Backend	Precision	Context length
1b/mlx/llama_3_2_1b_mlx_int4.pte	mlx	int4	16,384
1b/xnnpack/llama_3_2_1b_xnnpack_spinquant.pte	xnnpack	spinquant	16,384
3b/mlx/llama_3_2_3b_mlx_int4.pte	mlx	int4	4,096
3b/xnnpack/llama_3_2_3b_xnnpack_spinquant.pte	xnnpack	spinquant	4,096

Repository structure

1b/mlx/config.json                             1.3 KiB
1b/mlx/llama_3_2_1b_mlx_int4.pte               1.1 GiB
1b/xnnpack/config.json                         1.3 KiB
1b/xnnpack/llama_3_2_1b_xnnpack_spinquant.pte  1.0 GiB
3b/mlx/config.json                             1.3 KiB
3b/mlx/llama_3_2_3b_mlx_int4.pte               2.3 GiB
3b/xnnpack/config.json                         1.3 KiB
3b/xnnpack/llama_3_2_3b_xnnpack_spinquant.pte  2.2 GiB
config.json                                    31 B
tokenizer.json                                 9.4 MiB
tokenizer_config.json                          53.2 KiB

Compatibility

These files are published for the ExecuTorch v1.4.1 runtime. ExecuTorch gives no forward compatibility guarantee, so an older runtime may fail to load them.

To use them in React Native ExecuTorch, pass the model constant shipped in the library's model registry to the corresponding task pipeline. See the documentation.

To load these files in your own ExecuTorch runtime, read the compatibility note first.

Model details

Upstream model: Llama 3.2 collection.

Both the 1B and 3B sizes are published, each as bf16 and as SpinQuant, the quantization Meta trained for on-device use.

Model details

Upstream model: Llama 3.2 collection.

Both the 1B and 3B sizes are published in three forms: XNNPACK bf16, XNNPACK SpinQuant, the quantization Meta trained for on-device use, and MLX int4.

The MLX builds quantize from the bf16 weights rather than loading a pre-quantized checkpoint, so they are independent of SpinQuant rather than derived from it.