mlboydaisuke/OvisOCR2-LiteRT

OvisOCR2 — LiteRT-LM

ATH-MaaS/OvisOCR2 (0.85B, apache-2.0) converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15.

OvisOCR2 is a page-level document-parsing model built by post-training Qwen3.5-0.8B: given a document page image it emits a Markdown transcription in reading order — text as Markdown, tables as HTML <table>, formulas as LaTeX. This package wires the checkpoint's own 12-layer ViT to LiteRT-LM's fast_vlm contract at a static 512×512 input, on the same rail as our litert-community/Qwen3.5-0.8B VL build (the two checkpoints share every config byte; only the weights differ).

File	Recipe	Size
OvisOCR2_int8.litertlm	int8 decoder (dynamic on linears + embedding; convs and the delta rule stay float, fp32 activations declared) + fp16 vision encoder / int8 adapter, static 512×512, six-signature prefill ladder	1.30 GB

What it does on-device (measured)

Transcription of a rendered table page (512×512), CPU backend, greedy — output shown verbatim:

Quarterly Sales Report 2025

This report summarizes unit sales for the first three quarters. Phones remain the
strongest product line across all quarters.

<table border="1"><tr><td>Product</td><td>Q1</td>...</table>

Table 1: Unit sales by product and quarter.

• Table page: every one of the 20 table cells, the title, the paragraph and the caption are transcribed exactly (verified against the page's source text), and the output matches the HF fp32 reference at similarity 1.0000 on the same 512×512 input.
• The GPU backend produces byte-identical output to CPU on this page.
• Formula page: the LaTeX is exact (an inverted Planck law renders symbol-for-symbol) — similarity 1.0000 against the 1-D-position fp32 reference, 0.9968 against the full M-RoPE reference (a one-token $B$/B difference, the position-contract cost).
• Dense academic pages (9 pt text) exceed what a 512×512 input can resolve: the opening is transcribed correctly, then the model invents and eventually repeats. The HF fp32 original does the same on the same input (measured: every variant loops on this page; this build matches the fp32 reference byte-for-byte through the entire legible region and diverges only inside the invented continuation). Upstream's own reference inference code ships a repeat-cleanup post-processor for exactly this behavior. Use the model for pages whose text is legible at 512×512, or tile.
• A note for anyone comparing against stock transformers: the upstream repo has no generation_config.json, so HF's eos is only <|endoftext|> and fp32 generation runs through the turn end and rambles. This bundle declares both <|endoftext|> and <|im_end|> as stop tokens and stops cleanly; add 248046 to eos_token_id on the HF side before comparing.

Scope: this is a document parser, not a chat model

The OCR post-training visibly erodes general chat ability: on a generic 8-question sanity gate the model tends to transcribe the prompt instead of answering it. The fp32 original on the stock HF stack fails the same questions the same way — echoing 17 + 25 back, looping Tokyo, drifting into a numbered list on a French-vocabulary question — measured question-by-question against this build. It is the checkpoint's behavior, not a conversion artifact. Feed it document pages with the transcription prompt below.

Usage

litert-lm run ./OvisOCR2_int8.litertlm \
  --attachment page_512.png \
  --prompt "
Extract all readable content from the image in natural human reading order and output the result as a single Markdown document. For charts or images, represent them using an HTML image tag: <img src=\"images/bbox_{left}_{top}_{right}_{bottom}.jpg\" />, where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). Format formulas as LaTeX. Format tables as HTML: <table>...</table>. Transcribe all other text as standard Markdown. Preserve the original text without translation or paraphrasing." \
  --backend cpu --cache no --temperature 0

The prompt is the upstream reference prompt verbatim. The runtime resizes the attachment to 512×512; pre-rendering your page at 512×512 keeps the aspect ratio under your control.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	2175 tok/s	140.9 tok/s	0.13 s
CPU	659 tok/s	48.9 tok/s	0.43 s

On device (iPhone 17 Pro, iOS 27, litert-lm v0.16.0 vendored build, cold start, single runs):

Backend	Decode	TTFT	Peak memory
GPU (Metal)	48.5 tok/s	1.5 s	3.8 GB
GPU (Metal), image leg	64.7 tok/s	1.5 s	3.8 GB
CPU	18.0 tok/s	1.0 s	0.9 GB

The six-signature ladder loads Metal cleanly (no jetsam, no maxNumTokens override), the same envelope as the Qwen3.5-0.8B VL build. On-device image legs are grounded: shown a poster, it starts transcribing the poster's title; shown a photo without text, it emits its picture-region <img> tag — exactly its document convention.

Conversion notes

• The decoder is the shipped Qwen3.5-0.8B gated-delta hybrid rail (18 linear-attention + 6 full-attention layers, 48 state buffers); the MTP head is dropped.
• The vision tower's fp16-safe LayerNorm scales are calibrated per checkpoint; OvisOCR2's differ from base 0.8B in magnitude (absmax 2803.9 vs 2390.9) though not in shape.
• Vision quantization was gated on document fixtures (a table page, a formula page and an arXiv page), not photographs: the ship shape (fp16 encoder + int8 adapter) holds end-to-end correlation 0.9998+ on all three.
• Stop tokens are <|endoftext|> (248044) and <|im_end|> (248046); the upstream repo has no generation_config.json, so both were taken from config.json + the tokenizer and verified against the tokenizer at bundle build.

Conversion scripts and a step-by-step reproduction: hf-to-litertlm (qwen35vl_work/).