MiniCPM-OCR-1.0 — LiteRT-LM (OCR)

On a Galaxy S26 this model reads a 448×448 text region in 10 s per request on the CPU, engine creation included, and it read all ten test regions with a known answer without an error (the VL_int8vis file, 2026-10-29). This repository is a conversion of openbmb/MiniCPM-OCR-1.0 to LiteRT-LM (.litertlm), made with hf-to-litertlm for Google's LiteRT-LM 0.18.1 and 0.19.0.

openbmb/MiniCPM-OCR-1.0 (OpenBMB, 2026-10-21) is a 1.3B OCR model trained on top of MiniCPM-V 4.6: a 27-layer vision encoder and a Qwen3.5-style hybrid text decoder (18 GatedDeltaNet + 6 attention layers). It reads text, tables, formulas, seals and charts, and extracts named fields. One 448×448 picture becomes 256 image tokens; a larger picture is cut into an overview plus up to 9 slices.

File	What is inside	Size	sha256
MiniCPM-OCR-1.0-VL_int8.litertlm	text decoder int8 (dynamic, per channel; the 36 GatedDeltaNet decay/gate projections stay float) with fp32 activations declared for the GPU, int8 embedding table, vision encoder fp16, vision adapter fp16	2.00 GB	f233e3b991b4b76abec8910295ddc0660c8f93df30708a6709f27dd770fe03cc
MiniCPM-OCR-1.0-VL_int8vis.litertlm	the same decoder, embedding table and adapter; vision encoder int8 (dynamic)	1.59 GB	d16e301515208650f055b7807555d1ea8ac5b26ff79af9fb633e8ada1ab1f0a6

Which file.

• Phones: MiniCPM-OCR-1.0-VL_int8vis.litertlm with the decoder and the vision encoder on the CPU. It is the fastest setup on both phones and read every test picture with a known answer correctly; one test formula comes back with extra spaces (\frac{n (n + 1) (2 n + 1)}{6}, the same LaTeX; see Correctness). Galaxy S26: 10 s a region and 25–26 s an A4 page (the next fastest setups: 11 s and 34 s). Pixel 8a: 24–25 s and 74–75 s (VL_int8: 32 s on the CPU, 88 s a page with its vision encoder on the GPU). At most 2.9 GB.
• Mac: VL_int8vis on the CPU for a region (12.9 s against 14.7 s). For a page, MiniCPM-OCR-1.0-VL_int8.litertlm with its vision encoder on the GPU is faster (20.0 s against 22.2 s).
• Prefer MiniCPM-OCR-1.0-VL_int8.litertlm (fp16 vision encoder) when the vision encoder must run on a GPU, or when the LaTeX must come out spaced exactly as the original model writes it. On the Pixel 8a it is also the lowest-memory setup, with its vision encoder on the GPU (2.0 GB a region against 2.3 GB, at 36 s instead of 24–25 s).
• Keep VL_int8vis's vision encoder on the CPU: on the Mac GPU with a cache directory it answered [Non-Text] (see Known runtime issues).

How to use

The runtime resizes each picture to 448×448 without keeping its aspect ratio (and without EXIF orientation), so prepare the pictures yourself:

• A region (a line, a paragraph, a table, a formula, a receipt): paste it centred on a white square, then resize the square to 448×448.
• A page (larger than about 2× 448×448 pixels): cut it on the model processor's grid — an overview of the page padded to a white square, then up to 9 slices (at most 10 pictures in all), each resized to 448×448. Send them in one user message with the extra-context value slice_cols set to the grid's number of columns. The chat template then marks them as slices, the layout the model was trained on.

ocr_page.py does both (Pillow + litert-lm). With this repo's sample pictures:

pip install litert-lm pillow huggingface_hub
hf download litert-community/MiniCPM-OCR-1.0 --include "MiniCPM-OCR-1.0-VL_int8.litertlm" --include "samples/*" --local-dir mcpocr
curl -LO https://raw.githubusercontent.com/john-rocky/hf-to-litertlm/main/minicpmocr_work/ocr_page.py
python ocr_page.py mcpocr/MiniCPM-OCR-1.0-VL_int8.litertlm mcpocr/samples/page_en.png
python ocr_page.py mcpocr/MiniCPM-OCR-1.0-VL_int8.litertlm mcpocr/samples/table.png --prompt "Table Recognition:"
python ocr_page.py mcpocr/MiniCPM-OCR-1.0-VL_int8.litertlm mcpocr/samples/receipt.png --prompt 'Extract the "store_name", "date", "total". Format the output as a JSON object with keys: "store_name", "date", "total". If an item is not in the image, use "###".'

--backend gpu puts the decoder and the vision encoder on the GPU, --vision-backend gpu only the vision encoder, and --cache-dir keeps the runtime's caches away from the model file.

With the litert-lm CLI, give it a picture that is already 448×448 (padded as above):

litert-lm run mcpocr/MiniCPM-OCR-1.0-VL_int8.litertlm --prompt "Text Recognition:" --attachment mcpocr/samples/line_en_448.png

In Python (litert-lm 0.18.1 and 0.19.0), send the overview and the slices as the images of one message, and the column count as extra context:

import litert_lm
from ocr_page import pictures, png_item      # the grid, the padding and the resizing of ocr_page.py
from PIL import Image

pics, slice_cols = pictures(Image.open("mcpocr/samples/page_en.png").convert("RGB"))   # 9 pictures, 2 columns
engine = litert_lm.Engine("mcpocr/MiniCPM-OCR-1.0-VL_int8.litertlm", backend=litert_lm.interfaces.Backend.CPU(),
                          vision_backend=litert_lm.interfaces.Backend.CPU(), max_num_images=len(pics))
conv = engine.create_conversation(extra_context={"slice_cols": slice_cols})
reply = conv.send_message({"role": "user", "content": [png_item(p) for p in pics] +
                                                      [{"type": "text", "text": "Text Recognition:"}]})
print(reply["content"][0]["text"])

In Kotlin, pass the same value with Conversation.sendMessage(contents, extraContext = mapOf("slice_cols" to 2)) (LiteRT-LM 0.18.1's signature; we have not run the Kotlin path). Create one Engine per picture or page (see Known runtime issues). On a phone, use MiniCPM-OCR-1.0-VL_int8vis.litertlm with the CPU backend for the decoder and the vision encoder.

Prompts (from the model card): Text Recognition: (text, paragraphs, titles, pages; inline formulas come back as LaTeX), Table Recognition: (OTSL tokens: <fcel> cell, <nl> row end), Formula Recognition: (LaTeX), Seal Recognition:, Chart to Table:, and field extraction: Extract the "field", .... Format the output as a JSON object with keys: "field", .... If an item is not in the image, use "###".

Correctness

The bar (the reading bar below): on each test picture, a setup's character error rate against the truth may be at most 0.01 above that of VL_int8 with its vision encoder on the CPU on the same device (0.005 for an A4 page), and every field of the receipt must be right; LaTeX spacing does not count. We use the 11 test pictures the original model reads well at 448×448 (character error rate ≤ 0.05 against the truth, or against its own full-size reading where there is no truth): 10 synthetic regions with a known answer (a text line, a title, a paragraph in three forms, Chinese, a table, two formulas, a receipt with three fields) and a receipt photo, judged against the original's full-size reading. Until 2026-10-30 the bar was the fp32 original model's exact text (transformers 2df500f3, greedy); the first two points below are that bar's counts, the third gives both bars for the phone setups the two judge differently. A whole A4 page or a form photo in one 448×448 picture is unreadable for the original model too (character error rate 0.136 / 0.305), so those are judged as tiled pages. Under the old bar, where the original's own 448×448 and full-size readings differ, a different reading there did not count (parity_bar.py); that happened on one picture, a printed "Rounded" in the receipt photo. The reading bar counts it: it makes the difference between the receipt photo's two character error rates below (0.023 against 0.016).

• Mac M4 Max (litert-lm 0.18.1): VL_int8 11/11 on the CPU, 11/11 with the vision encoder on the GPU, and 11/11 on the GPU (the first build with the same model sections; the ship file's own GPU runs are two tiled pages); VL_int8vis 11/11 on the CPU. Tiled pages match the original's run on the same tiles (CPU 4/4, GPU 2/2, split 2/2). ocr_page.py from a fresh pip install litert-lm pillow (litert-lm 0.19.0) matched in 6 of 6 runs (an A4 page on the CPU and the GPU, a form photo, a receipt photo, a text line, a receipt's three fields). VL_int8 gave the same text on litert-lm 0.15.0, 0.16.0, 0.17.0, 0.17.1, 0.18.0, 0.18.1 and 0.19.0 (Mac CPU: two regions and a tiled paragraph).
• Phones (LiteRT-LM v0.18.1 litert_lm_advanced_main, one process per picture): VL_int8 with the vision encoder on the CPU read all 11 pictures like the original on the Galaxy S26 (15/15 runs) and the Pixel 8a (14/14). The whole model on the Galaxy S26 GPU: the 7 pictures it ran (9/9 runs). The vision encoder on the GPU: 11 of 11 on the Galaxy S26 (15/15 runs), 10 of 11 on the Pixel 8a (13/14 runs). The A4 page as 9 pictures matched the original's tiled run on every form that ran it.
• The one picture that splits the forms is the second formula. The original writes \frac{n(n+1)(2n+1)}{6}. VL_int8vis on both phones' CPUs (Galaxy S26 4 of 4 processes, Pixel 8a 3 of 3) and VL_int8 with the vision encoder on the Pixel 8a's Mali GPU write \frac{n (n + 1) (2 n + 1)}{6}, also with fp32 activations (--force_f32=true). Every Mac form writes the original's text. The original model itself is close to the spaced form here: 78 % against 22 % at that token. Under the reading bar, VL_int8vis on the CPU of both phones (Galaxy S26 18 of 18 runs, Pixel 8a 16 of 16) and VL_int8 with the vision encoder on the Pixel 8a's Mali GPU (14 of 14) pass: no error on the 10 known-answer pictures or the A4 page, and a character error rate of 0.023 on the receipt photo against the original's full-size reading (VL_int8 on the CPU: 0.016).
• Parts: the text decoder's logits equal the full model's (20 tokens, max |Δ| 0.0); the vision tower matches the original get_image_features at token cosine ≥ 0.99999989 in fp32 and ≥ 0.9998 with the shipped fp16 weights (17 pictures); the template gives the processor's token ids in 9 of 9 chat cases and 6 of 6 tiled pictures.

On-device demo

Galaxy S26, MiniCPM-OCR-1.0-VL_int8.litertlm on the CPU (LiteRT-LM 0.18.1 litert_lm_advanced_main, one process per picture). Each answer is the phone's output and also the original fp32 model's output, character for character. The GPU and MiniCPM-OCR-1.0-VL_int8vis.litertlm on the CPU printed the same four answers.

Picture (the phone got it padded to a white 448×448 square)	Prompt	Answer on the phone
	Text Recognition:	Invoice 48213 was issued on 12 March 2025 and is due within 30 days.
	Table Recognition:	<otsl><fcel>Station<fcel>Samples<fcel>Nitrate (mg/L)<fcel>Phosphate (mg/L)<nl><fcel>Kessel<fcel>212<fcel>3.1<fcel>0.42<nl><fcel>Arden<fcel>185<fcel>2.7<fcel>0.38<nl><fcel>Morrow<fcel>164<fcel>4.0<fcel>0.51<nl><fcel>Lisle<fcel>143<fcel>1.9<fcel>0.29<nl></otsl>
	Formula Recognition:	\[ E = \frac{1}{2} m v^{2} + \int_{0}^{h} m g \, dz \] (on three lines)
	Extract the "store_name", "date", "total". …	{"store_name": "GREENLEAF MARKET", "date": "2025-03-14", "total": "17.60"} (indented JSON)

Performance

One request = a fresh process (a new Engine; this decoder needs one per conversation): engine creation, vision encode, prefill and decode. Mac: timed inside the process; phones: the process wall clock. "Region" = a 448×448 paragraph (277 tokens in, 100 out). "Page" = an A4 page as 9 pictures, 210 tokens out (2,344 tokens in on the Mac with the slice marks; the phones' command-line runner has no extra-context flag, so there 2,373 tokens go in unmarked, with the same answer). The phones ran LiteRT-LM v0.18.1's litert_lm_advanced_main (Android tag build) with the runtime caches from an earlier process. Bold = the fastest setup on that device that passes the reading bar in Correctness.

Mac M4 Max, litert-lm 0.18.1 (Python API)	Region request (s)	Engine and conversation (s)	First text (s)	Decode (tok/s)	A4 page request (s)
VL_int8vis, CPU	12.9 (12.5–13.5)	9.4–9.9	1.14–2.07	48.8–51.1	22.2 (20.7–23.2)
VL_int8, CPU	14.7 (12.9–16.2)	9.7–12.0	1.28–2.68	46.9–51.9	24.0 (23.5–24.5)
VL_int8, CPU decoder, GPU vision	14.3 (13.3–16.1)	10.6–12.6	0.89–1.53	46.1–53.4	20.0 (19.8–21.5)
VL_int8, GPU	28.9 (28.5–29.3)	27.1–28.1	0.46–0.67	128.4–143.1	30.7

Median (range) of the warm processes over four timing windows. A cold first process took 14.1–17.4 s with the decoder on the CPU and 27.4 s on the GPU. The GPU decodes about three times as fast but compiles its large programs again in every process.

Galaxy S26 (SM8850, Android 16)	Region request (s)	Engine + vision executor (s)	Vision encode (s)	First token after the encode (s)	Decode (tok/s)	Peak (MB)	A4 page request (s)
VL_int8, CPU	11 / 11 (cold 12 / 14)	5.9–6.9 + 0.4–1.6	1.35–2.05	0.59–0.86	42.7–47.2	3,783–4,401	35 (peak 4,815 MB)
VL_int8vis, CPU	10 / 10 (cold 12)	6.1 + 0.06–0.07	0.68	0.58–0.60	43.5–45.1	2,449	25 / 26 (peak 2,869 MB)
VL_int8, CPU decoder, GPU vision	19 / 19 (cold 32)	6.0–6.2 + 7.8–8.0	0.85–0.86	0.75–0.96	38.3–39.0	3,491	34 (peak 3,909 MB)
VL_int8, GPU	71 / 77 (60–84 over 9 runs, cold 66)	51.3–53.9 + 10.9–13.3	0.92–0.96	0.60–0.61	30.7–32.2	4,531–4,847	86

On the Galaxy S26, VL_int8vis on the CPU is the fastest setup (10 s a region, 25–26 s a page, up to 2.9 GB); one test formula comes back with extra spaces. With VL_int8's vision encoder on the GPU a page takes 34 s (VL_int8 with it on the CPU: 35 s, one process each) and a region 19 s, because the GPU vision executor takes 7–8 s to create. A GPU decoder spends 44–65 s creating its engine in every process.

Pixel 8a (Tensor G3, 7.75 GB, Android 16)	Region request (s)	Engine + vision executor (s)	Vision encode (s)	First token after the encode (s)	Decode (tok/s)	Peak (MB)	A4 page request (s)
VL_int8, CPU	32 / 35 (cold 53)	13.5 + 1.1–1.4	8.83–9.57	1.86–2.58	17.2–19.3	4,141–4,223	177 (peak 4,356 MB)
VL_int8, CPU decoder, Mali GPU vision	36 / 36 (cold 44)	13.5–13.6 + 10.2–10.9	2.70–2.83	2.0–2.4	18.7–19.7	2,038–2,053	88 (peak 2,372 MB)
VL_int8vis, CPU	24 / 25 (cold 35)	12.8–13.9 + 0.1	2.80–2.84	1.77–1.84	19.6–19.8	2,281–2,286	74 / 75 (peak 2,612 MB)

On the Pixel 8a's Arm Mali GPU the decoder's GPU engine does not build within our test's memory line: our test stops a run below 400,000 kB of free memory, and here it read 396,436 kB while the fourth of the seven signatures was being built (one try). Use the CPU backend on 8 GB phones for the decoder. VL_int8vis on the CPU is the fastest setup (24–25 s a region, 74–75 s a page, up to 2.6 GB). With VL_int8's vision encoder on the GPU a page takes half the time and 2 GB less memory than VL_int8 on the CPU (88 s and 2.4 GB against 177 s and 4.4 GB). Both write the same test formula with extra spaces.

Conversion notes

• Source: openbmb/MiniCPM-OCR-1.0 at revision 165531f9 (the minicpm_ocr code is on transformers main at 2df500f3; v5.21.0 does not have it).
• Text decoder: a text-only checkpoint derived from the VLM weights (qwen3_5_text config, tied head; logits equal to the full model's), exported with a litert-torch checkout at commit 115a136 (version string 0.10.0) plus the Qwen3.5 hybrid patch of hf-to-litertlm, with litert-converter 0.3.0, ai-edge-quantizer 0.8.0, ai-edge-litert 2.1.6 and transformers 5.14.1: 7 signatures (prefill 1024, 256, 64, 16, 4, 1 and decode), a 4096-token KV budget, the embedder outside the decoder for the fast_vlm path.
• Vision: the 4× tower (27 layers over all 1,024 patches of a 448×448 picture, then a 2×2 merger) re-written in plain PyTorch from the checkpoint weights — the position table as a constant, LayerNorm inputs scaled by powers of two so fp16 holds — and exported with litert-torch 0.9.3 / litert-converter 0.4.0 (no GATHER, FLEX or custom ops).
• Quantization: dynamic int8 on the decoder's linears and the embedding; the 36 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) stay float (their run-time int8 activation quantization on the CPU broke the Qwen3.5 siblings). The vision encoder is fp16 in VL_int8 and dynamic int8 in VL_int8vis (worst token cosine 0.59 against upstream's features, 0.9998 for fp16). The GPU runs the decoder with fp32 activations, declared in the file; forcing fp16 activations printed ! until the output cap in 6 of 6 Mac GPU processes.
• Template: ChatML with the model's empty <think>\n\n</think>\n\n block (thinking off, upstream's default), stops <|im_end|> and <|endoftext|>, no start token; slice_cols adds the tile marks.
• Packing: litert-lm-builder 0.15.0 plus the ExecutorMetadata section the hybrid runtime binds its 48 state buffers with.
• Reproduction: REPRODUCE.md · machine-readable facts: litertlm_manifest.json.

Known runtime issues

• One Engine per picture or page. A Conversation opened on an Engine that already served another one continues from that conversation's recurrent state on this hybrid decoder (LiteRT-LM #3165), so every request in the tables above creates its own Engine.
• The GPU decoder compiles again every time. On the Galaxy S26 GPU every engine creation after the first compiles the decoder's five large prefill programs again (OpenCL, LiteRT-LM v0.18.1): reading them from the GPU program cache fails with DATA_LOSS, so a GPU request spends about 50 s creating its Engine, and the program cache file next to the model grows by about 231 MB at each creation (241 → 473 → 704 → 935 MB over four creations) until you delete it (LiteRT-LM #4178). The Mac GPU recompiles the same way (engine creation 26–28 s in every process). iOS and the iPhone were not measured with this file.
• VL_int8vis with its vision encoder on the GPU. MiniCPM-OCR-1.0-VL_int8vis.litertlm with its vision encoder on the Mac GPU (WebGPU, litert-lm 0.18.1) answered [Non-Text] in every process that had a cache directory (3 of 3 with the text recorded, the first with an empty directory; 4 timing processes returned 10 characters each). With the cache off (cache_dir=":nocache", litert-lm run --cache no) it read the test pictures correctly; litert-lm run --cache memory failed ("An error occurred"). MiniCPM-OCR-1.0-VL_int8.litertlm reads correctly with its vision encoder on the GPU and a cache directory. Keep VL_int8vis's vision encoder on the CPU; we have not run it on a phone GPU.
• Tool calls are not parsed. The template renders no tool definitions, this file's metadata carries no tool-call config, and LiteRT-LM up to 0.18.1 has no parser for the Qwen3.5-style XML tool call.
• Later turns. From the second turn of a conversation on, LiteRT-LM feeds the previous answer's stop token into the KV cache with a zero embedding (this file keeps its embedding table in its own section), so a later answer can differ from the answer to the same messages in one prompt. Read in the runtime source 0.16.0–0.18.1 and measured with 0.18.1 on two other bundles (WeVisOne-VL-2B, Qwen2.5-Coder-1.5B-Instruct), not on this file. Ask one question per conversation, or start a new conversation with the earlier messages and the new question.

License and changes

Apache-2.0, inherited from openbmb/MiniCPM-OCR-1.0. Changes from the original work: exported to the LiteRT-LM format; the text decoder quantized to int8; the vision encoder and adapter cast to fp16 (the vision encoder quantized to int8 in the _int8vis file); a ChatML chat template with thinking off and the slice_cols tile marks; stop tokens <|im_end|> and <|endoftext|>. The sample pictures in samples/ were drawn for this conversion (DejaVu fonts). Problems with these files: open a discussion in this repo.