litert-community/Phi-3-mini-4k-instruct

See the original model card for training, intended uses, limitations, and citation information. This conversion uses checkpoint revision f39ac1d28e925b323eae81227eaba4464caced4e.

Available artifact

File	Quantization	Context	Size
phi3_mini_4k_instruct_int8.litertlm	INT8 weights and embedding, float KV	Up to 4,096 tokens; load-time configurable	3.58 GiB

The bundle is self-contained; no separate tokenizer or weights file is needed.

Conversion Notes

The .litertlm artifact was produced from the original Hugging Face checkpoint with a TorchAO INT8 weight-only quantization recipe. Linear projection weights, including the output head, use per-channel INT8 with FP32 scales and the symmetric [-127, 127] weight range. Token embeddings use per-row INT8. Normalization paths and public KV cache tensors remain FP32; internal activation precision depends on the runtime backend.

The model uses the odml.swiglu composite for its existing packed gate/up projections. Prefill executes fixed 256-token chunks, and decode processes one token at a time. Both signatures include all KV cache tensors. The graph preserves the checkpoint's 2,047-token sliding attention window within its full 4,096-token context, with cache capacity selected at model loading time.

The bundle includes the tokenizer and a text chat template supporting system/user/assistant messages and cached multi-turn conversations. It accepts string content or lists of text parts, adds no BOS token, and uses stop IDs 32000, 32001, and 32007.

Runtime and context sizing

The same file was tested with 2,048 tokens on Android and 2,048 / 4,096 tokens on Linux. LiteRT-LM replaces the exported placeholder dimension, 4,099, with the selected capacity when loading the model. Keep the capacity within the checkpoint's trained 4,096-token limit.

With a compatible LiteRT-LM CLI and its GPU libraries on the library search path:

litert_lm_advanced_main \
  --model_path=phi3_mini_4k_instruct_int8.litertlm \
  --backend=gpu --sampler_backend=gpu \
  --max_num_tokens=2048 --prefill_batch_sizes=256 \
  --gpu_external_tensor_mode=false --sampler_handles_input=true \
  --input_prompt="What is the capital of France?" --async=false

Use --max_num_tokens=4096 for the full context when sufficient memory is available. Public FP32 KV tensors alone total 1.5 GiB per cache bank at 2,048 tokens, or 3 GiB at 4,096, in addition to weights and scratch memory. GPU execution can allocate multiple cache banks. These tensor sizes are not measured peak memory or a minimum device-RAM specification.

Android GPU performance

This model file is too slow for practical interactive use on Android phones, based on the measurements below. For Android apps, we recommend using Gemma models through Android AICore on supported devices instead.

Measured on 2026-09-20 with litert_lm_advanced_main over ADB. Both devices fully delegate the prefill and decode graphs to the OpenCL GPU backend.

Workload: 2,048-token context, 256 prefill tokens, and 256 decode tokens. Values are medians of three consecutive measured iterations after one warm-up iteration in the same process. TTFT means time to first token with the model already loaded; model loading is excluded.

Hardware benchmark disclosure: Results were measured by us on retail devices purchased through normal channels. These results are not affiliated with, sponsored by, endorsed by, or verified by Samsung, vivo, Qualcomm, MediaTek, Google, MLCommons, or Hugging Face. Results depend on device SKU, OS build, thermal state, battery mode, backend, model quantization, runtime version, and benchmark settings.

Device	Backend	Prefill (tokens/s)	Decode (tokens/s)	TTFT (s)
TECNO LJ9	GPU OpenCL	81.58	3.88	3.396
Samsung SM-S937U1	GPU OpenCL	329.19	7.34	0.891

Benchmark settings

The measured runs used GPU sampling and these additional flags:

--benchmark=true
--benchmark_prefill_tokens=256
--benchmark_decode_tokens=256
--num_iterations=4
--enable_profiling=false

The first iteration is excluded from the table. The input prompt was Write a short explanation of what a compiler does.; benchmark mode enforces the token counts above.

Try It

Install uv and run:

uv tool install litert-lm
uvx litert-lm run \
  --from-huggingface-repo=litert-community/Phi-3-mini-4k-instruct \
  phi3_mini_4k_instruct_int8.litertlm \
  --prompt="What is the capital of France?"

Artifact identity and license

• File size: 3,848,159,232 bytes.
• SHA-256: f997586cf635d3dbadde8aa46109e2c50cab530eecea9feb507abb60891cf4c6.
• License: MIT, inherited from the source checkpoint.