tonythethompson/Phi-4-Mini-Instruct-CPU-INT4-ONNX

Phi-4 Mini Instruct — CPU INT4 (ONNX Runtime GenAI)

This repository repackages the cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4 configuration from Microsoft's official ONNX release microsoft/Phi-4-mini-instruct-onnx. It is not a newly trained model. This is an unofficial community repackage; Microsoft is the original author and license holder.

Source

Field	Value
Upstream model	microsoft/Phi-4-mini-instruct-onnx
Upstream source revision	fc04c8f93df696602fd9f300a30d1bf2e3081347
Export tool/script	Microsoft ONNX Runtime GenAI model builder (upstream Phi ONNX bundle)
Quantization recipe	ONNX Runtime GenAI RTN INT4 (cpu-int4-rtn-block-32-acc-level-4)

Files

All files are under cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/:

File	Size	Description
model.onnx	~50 MB	INT4 ONNX graph
model.onnx.data	~5 GB	External weights
genai_config.json	~1 KB	ONNX Runtime GenAI session config
config.json	~2 KB	Model config
configuration_phi3.py	~11 KB	Phi-3/4 config class
tokenizer.json + tokenizer_config.json	~15 MB	Tokenizer
vocab.json + merges.txt	~6 MB	Vocabulary / merges
added_tokens.json + special_tokens_map.json	<1 KB	Special tokens

Intended Use

A 4-bit, CPU-targeted build of Phi-4-Mini-Instruct for local text generation and translation on CPU-only or low-VRAM hardware via ONNX Runtime GenAI. A GPU variant is in tonythethompson/phi-4-mini-instruct-gpu-int4-onnx.

Runtime Notes

• Designed for ONNX Runtime GenAI compatible runtimes.
• Execution provider: CPU (onnxruntime-genai).
• Context length: 128K tokens (inherited from Phi-4-Mini-Instruct).
• Microsoft's published throughput examples: ~16-24 tokens/sec on server-class Xeon CPUs; ~3-5 tokens/sec on laptop CPUs. Validate on your hardware before production use.

Precision and Packaging

Export tooling, precision, and quantization are recorded in the Source table above. This packaging mirror does not publish independent parity benchmarks; validate on your target execution provider before production use.

Limitations

• CPU inference is substantially slower than GPU inference; not suitable for real-time use on most hardware.
• INT4 quantization introduces approximation error relative to the base FP16/FP32 model.
• No repository-specific translation quality evaluation is documented here.
• Validate on your target language pair before production use.

License

MIT — inherited from microsoft/Phi-4-mini-instruct-onnx. This packaging repo adds no new license terms.