PP-OCRv5-OpenVINO

PP-OCRv5 for Intel NPUs, GPUs and CPUs: ppocr_det_fp16 and ppocr_rec_fp16 from litert-community/PP-OCRv5-LiteRT as ONNX (tf2onnx 1.17.0, opset 17), OpenVINO IR with fp32 weights, and precompiled Intel NPU blobs. tf2onnx 1.17.0 drops the bias of 2 FULLY_CONNECTED layers of ppocr_rec_fp16; these files were converted with a one-line fix of that handler (convert/patch_tf2onnx.py in the GitHub repo), and their outputs match the .tflite. On the NPU of a Core Ultra 9 386H the blob runs in 8.29 ms (ppocr_det_fp16) and 4.95 ms (ppocr_rec_fp16) per run, numpy input to numpy output.

Scripts, raw records, and the same table for all 52 graphs of LiteRT-Models: https://github.com/john-rocky/windows-zoo

Files

file	bytes	sha256
ppocr_det_fp16.onnx	17,678,543	6a6a9e4b650cb0cf9cdbce27febbef80da03a03cff04f228fb98daced51b0dc5
openvino/ppocr_det_fp16.xml	2,976,282	1d57a87f4645798d3fdbc737eeeab0c5c6033ec6bed900e3acc82bd7b30e1963
openvino/ppocr_det_fp16.bin	15,956,512	7a4307b58a89aa82d02deebd1e464d2ef922ccd3c4466ad30c994b4c59c2fa70
npu/ppocr_det_fp16.blob	12,914,817	46eb1c4d7a99a9c9c2490487eb2d033e69d7b38f2d014b0259053d4713ff8914
ppocr_rec_fp16.onnx	33,763,275	0b7dd3e87499ed80dab5ed25234b177e3058fa51bbfb7a9c5b7e593c4ad21e90
openvino/ppocr_rec_fp16.xml	2,865,171	7f7afb1aeda9757efeee195a238c6e67e8d7f719724fdbf718efd6ea62e6a711
openvino/ppocr_rec_fp16.bin	32,025,508	f6c3f1c8f6364865705ebff7c65c4a56d51fd26e1af01d65fed9e9848bdfba73
npu/ppocr_rec_fp16.blob	19,460,225	b15f014caaa85f430475a67a79ebcbadc9018990c132e817e1db186d7f8f5430

The NPU blobs were compiled with OpenVINO 2026.4.1, Intel NPU driver 32.0.100.4841 (OpenVINO reports NPU_DRIVER_VERSION 1004841, NPU_COMPILER_VERSION 524292). If import_model refuses a blob on another driver or OpenVINO version, compile the IR instead (the commented line in the snippet).

How it was measured

MSI Stealth 16 AI+ (Core Ultra 9 386H, RTX 5080 Laptop GPU at a 95 W power limit, Windows 11 build 26200.9711, on AC power), 2026-11-06 to 2026-11-07. On the laptop, each row is one process: load, one run, 5 warmup runs, then the median of 30 timed runs (2 warmup + 10 timed runs when the first run takes more than 3 s), numpy input to numpy output, every runtime at its default settings, one process at a time, each started after the CPU, the RTX and the Intel GPU had gone idle (the thresholds are in METHOD.md of the GitHub repo). Output check: each output against the CPU output in fp32 on the same input (ONNX Runtime's CPU EP); PASS = cosine ≥ 0.99, no NaN or Inf, and top-1 agreement ≥ 0.99 for class scores. The Snapdragon rows are Qualcomm AI Hub profile jobs of the same ONNX files (their own timer). No Snapdragon rows for ppocr_rec_fp16: Qualcomm AI Hub measured the file before the bias fix (its numbers are in the GitHub table).

ppocr_det_fp16

processor	runtime	file	ms	load s	output check
Intel AI Boost NPU	OpenVINO 2026.4.1	npu/ blob (import_model)	8.29	0.01	PASS (cos ≥ 0.9996)
Intel AI Boost NPU	ONNX Runtime 1.31.1 + OpenVINO EP 1.8.1	.onnx, compiled when the session loads	8.35	311.11	FAIL: output_0_output: cos 0.2421 < 0.99
Intel Graphics (iGPU)	ONNX Runtime 1.31.1 + OpenVINO EP 1.8.1	.onnx	10.92	2.06	FAIL: output_0_output: cos 0.9848 < 0.99
RTX 5080 Laptop (95 W limit)	ONNX Runtime 1.31.1, CUDA EP	fp16 copy of the .onnx (not in this repo)	18.47	1.40	FAIL: output_0_output: 409600 NaN, 0 Inf; output_0_output: cos 0.0000 < 0.99
RTX 5080 Laptop (95 W limit)	ONNX Runtime 1.31.1, CUDA EP	.onnx	9.12	1.37	PASS (cos ≥ 0.9999)
Core Ultra 9 386H CPU	ONNX Runtime 1.31.1, CPU EP	.onnx	62.66	0.33	reference
Snapdragon X2 Elite NPU	Qualcomm AI Hub, precompiled QNN ONNX	.onnx compiled by AI Hub	7.24	0.39	not compared (AI Hub profile job, 2026-10-30)
Snapdragon X2 Elite NPU	Qualcomm AI Hub, ONNX Runtime QNN EP	.onnx, compiled when it loads	32.32 (8 layers on the CPU)	13.86	not compared (AI Hub profile job, 2026-10-30)
Galaxy S26 NPU	Qualcomm AI Hub, precompiled QNN ONNX	.onnx compiled by AI Hub	7.05	0.17	not compared (AI Hub profile job, 2026-10-30)

The ONNX file against the source .tflite (LiteRT interpreter on the CPU, the same input): PASS, cosine ≥ 0.9999 for every output.

ppocr_rec_fp16

processor	runtime	file	ms	load s	output check
Intel AI Boost NPU	OpenVINO 2026.4.1	npu/ blob (import_model)	4.95	0.02	PASS (cos ≥ 0.9999)
Intel AI Boost NPU	ONNX Runtime 1.31.1 + OpenVINO EP 1.8.1	.onnx, compiled when the session loads	4.36	7.39	PASS (cos ≥ 0.9999)
Intel Graphics (iGPU)	ONNX Runtime 1.31.1 + OpenVINO EP 1.8.1	.onnx	2.55	1.43	PASS (cos ≥ 0.9999)
RTX 5080 Laptop (95 W limit)	ONNX Runtime 1.31.1, CUDA EP	fp16 copy of the .onnx (not in this repo)	17.28	1.25	PASS (cos ≥ 0.9999)
RTX 5080 Laptop (95 W limit)	ONNX Runtime 1.31.1, CUDA EP	.onnx	8.29	1.24	PASS (cos ≥ 0.9999)
Core Ultra 9 386H CPU	ONNX Runtime 1.31.1, CPU EP	.onnx	17.68	0.24	reference

The ONNX file against the source .tflite (LiteRT interpreter on the CPU, the same input): PASS, cosine ≥ 0.9999 for every output.

Run it on the Intel NPU

Download this repo, then run the snippets from its folder:

pip install huggingface_hub
hf download mlboydaisuke/PP-OCRv5-OpenVINO --local-dir PP-OCRv5-OpenVINO
cd PP-OCRv5-OpenVINO

OpenVINO (pip install openvino==2026.4.1):

import numpy as np
import openvino as ov

core = ov.Core()
if "NPU" not in core.available_devices:
    raise SystemExit(f"no Intel NPU visible to OpenVINO: {core.available_devices}")
with open("npu/ppocr_det_fp16.blob", "rb") as f:
    compiled = core.import_model(f.read(), "NPU")  # precompiled: no compile at load
# or compile the IR on this machine: compiled = core.compile_model("openvino/ppocr_det_fp16.xml", "NPU")
# replace the zeros with your preprocessed input (float32, the shapes printed by compiled.inputs)
feeds = {i.get_any_name(): np.zeros(list(i.shape), np.float32) for i in compiled.inputs}
outputs = compiled(feeds)
print({o.get_any_name(): outputs[o].shape for o in compiled.outputs})

ONNX Runtime with the OpenVINO EP (pip install onnxruntime==1.31.1 onnxruntime-ep-openvino==1.8.1):

import numpy as np
import onnxruntime as ort
import onnxruntime_ep_openvino as ov_ep

ort.register_execution_provider_library(ov_ep.get_ep_name(), ov_ep.get_library_path())
npu = [d for d in ort.get_ep_devices()
       if d.ep_name == ov_ep.get_ep_name() and d.ep_metadata.get("ov_device") == "NPU"]
if not npu:
    raise SystemExit("no Intel NPU visible to the OpenVINO EP")
so = ort.SessionOptions()
so.add_provider_for_devices(npu, {})
sess = ort.InferenceSession("ppocr_det_fp16.onnx", sess_options=so)  # compiles for the NPU here
# replace the zeros with your preprocessed input (float32)
feeds = {i.name: np.zeros(i.shape, np.float32) for i in sess.get_inputs()}
outputs = sess.run(None, feeds)
print([o.shape for o in outputs])

The same lines run ppocr_rec_fp16 with its own file names.

Licence

Apache-2.0, the licence of litert-community/PP-OCRv5-LiteRT. The upstream models: PaddlePaddle/PP-OCRv5_mobile_det and PaddlePaddle/PP-OCRv5_mobile_rec.