mlboydaisuke/manga-ocr-ExecuTorch

manga-ocr — ExecuTorch (Japanese text out of a picture of Japanese text)

Hand one line of Japanese to this and get the characters back. Vertical or horizontal, furigana and all, printed or laid over artwork. It generates the characters rather than classifying them: a ViT-base encoder reads the crop once, and a two-layer BERT decoder emits one character at a time until it stops.

The shelf's other OCR is docTR, which reads Latin script in two stages — find the boxes, then read each crop. This one is the second stage for Japanese, and it is the only shelf model that reads vertical text.

manga_ocr_encoder  pixel_values (1, 3, 224, 224) fp32
    -> encoder_hidden (1, 197, 768) fp32
manga_ocr_decoder  input_ids (1, L) int64, encoder_hidden (1, 197, 768) fp32
    -> logits (1, 6144) fp32      # the next character, at the last position only

• Source: kha-white/manga-ocr-base, trained on Manga109s — 111.0M parameters, 6,144-character vocabulary
• License: Apache-2.0
• Reference implementation: kha-white/manga_ocr

Files

Pick one row. The two halves must match: the encoder's hidden state is what the decoder was measured against.

build	encoder	decoder	encoder speed	decoder speed
Core ML + XNNPACK int8	manga_ocr_encoder_coreml_all.pte 172.1 MB	manga_ocr_decoder_xnnpack_int8.pte 45.0 MB	4.2 ms	2.5 ms
XNNPACK int8	manga_ocr_encoder_xnnpack_int8.pte 88.9 MB	manga_ocr_decoder_xnnpack_int8.pte 45.0 MB	25.5 ms	2.5 ms
XNNPACK fp32	manga_ocr_encoder_xnnpack_fp32.pte 343.3 MB	manga_ocr_decoder_xnnpack_fp32.pte 117.4 MB	29.5 ms	3.2 ms
XNNPACK fp16	manga_ocr_encoder_xnnpack_fp16.pte 173.1 MB	manga_ocr_decoder_xnnpack_fp16.pte 58.8 MB	71.2 ms	3.1 ms

Host numbers, on a loaded machine, decoder timed at a 3-character prefix — read them as a lower bound and as ratios rather than as device latency. Eager fp32 on the same host is 29.9 ms for the encoder and 4.6 ms for the decoder. XNNPACK takes 82.9% of the encoder and 51.9% of the decoder; Core ML takes all of the encoder.

fp16 is the row nothing should pick. Its encoder is the size of the Core ML build and less than half the speed of fp32, because XNNPACK inserts a cast at every boundary it cannot fuse. It is listed because it was built and measured, not because it wins anything.

int8 is the row to pick on Android: half of fp16 on the encoder and three-quarters of it on the decoder, faster than eager PyTorch on both, and no character different from eager anywhere it was measured. What that costs is margin, not characters — see below.

Running it

1. Preprocess — the checkpoint's recipe, not PIL's defaults.

image = image.convert("L").convert("RGB").resize((224, 224), processor.resample)
mean = np.array(processor.image_mean, dtype=np.float32)   # 0.5, 0.5, 0.5
std = np.array(processor.image_std, dtype=np.float32)     # 0.5, 0.5, 0.5
x = (np.asarray(image, dtype=np.float32) / 255.0 - mean) / std
pixel_values = torch.from_numpy(np.ascontiguousarray(x.transpose(2, 0, 1)))[None]

Three details that are each load-bearing. Greyscale and back to RGB — the reference pipeline does this, so a colour page and a scan of the same panel give the same tensor. resample from the config, which is bilinear — Image.resize defaults to bicubic, and a filter the model was not trained under changes what it reads. The aspect ratio is squashed to a square on purpose: 224x224 regardless of whether the line is a wide horizontal strip or a tall vertical one, which is what the model saw in training.

2. The encoder, once per crop. The decoder, once per character.

hidden = encoder.execute([pixel_values])[0]
ids = torch.tensor([[2]])                  # decoder_start_token_id
for _ in range(300):
    token = int(decoder.execute([ids, hidden])[0][0].argmax())
    if token == 3:                         # eos_token_id
        break
    ids = torch.cat([ids, torch.tensor([[token]])], dim=1)

3. Decode with vocab.txt — there is no tokenizer to install. The checkpoint names BertJapaneseTokenizer, whose MeCab step exists only to split text into words, and this model never reads text. subword_tokenizer_type is character and the vocabulary holds no ## continuations, so a character is a line of vocab.txt and decoding is a lookup and a join:

vocab = open("vocab.txt", encoding="utf-8").read().splitlines()
text = "".join(vocab[i] for i in ids[0, 1:].tolist()
               if vocab[i] not in {"[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"})

Worth knowing because transformers 5.x cannot instantiate that tokenizer class at all — it has dropped the slow tokenizers. Reading the vocabulary directly is not a workaround, it is what an app on the device does anyway.

Why there is no KV cache

The decoder is two layers over a 6,144-entry vocabulary. Re-running the whole prefix each step is cheap enough that a cache would buy less than it costs: it would turn one method with a dynamic length into a cache protocol the caller has to hold, allocate and reset, and it would pin a maximum length into the file.

The decoder also returns only the last row of logits. The other L-1 rows are the model re-deriving characters the caller already has; dropping them keeps 6,144 floats crossing the boundary per step instead of L times that.

Greedy, not beam search

The checkpoint's own generation config asks for 4 beams. Greedy is what the graph is shaped for and what every number here was measured with; on the horizontal gate image greedy and beam-4 read the same string, and the 84-line set never needed more than greedy to match eager. Beam search remains possible from outside — the decoder is a pure function of (prefix, hidden), so a caller can run it once per beam per step — but what it buys on harder input than this has not been measured.

Core ML: the encoder yes, the decoder no

The encoder is the whole speed story: 100% delegated, 4.2 ms against 29.9 ms for eager on the same host, at 172.1 MB. Worst-output correlation against fp32 eager is 0.999984.

The decoder is not shipped for Core ML, and the reason is specific. It converts, it delegates 100%, and — once eight zero-element constants that transformers' BERT attention leaves behind are pruned — it loads. What it will not do is run at a length other than the one it was exported at: execute() failed with error 0x32. A decoder whose prefix grows by one character per step is exactly the case that needs the dynamic axis, so the decoder stays on XNNPACK, where it runs at every length from 1 to 300. This is not "Core ML rejects dynamic shapes" — it accepts the export and pins the shape.

What was measured

The gate is the string that came out, not correlation. The encoder's output is a hidden state nobody reads, and the decoder's output is one row of logits whose argmax is all that survives — a correlation of 0.999 and a different character are the same file.

84 generated lines: 12 sentences x 4 typefaces (Hiragino gothic W3 and W6, rounded gothic, mincho) x both orientations x 7 degradations, from clean through gaussian blur, sensor noise, low contrast, heavy smear, heavy grain, and a near-contrastless JPEG at quality 12.

The set is generated deterministically, so the numbers below reproduce run to run.

build	lines identical to eager	mean CER vs eager
XNNPACK fp32	84 / 84	0.0000
XNNPACK fp16	84 / 84	0.0000
XNNPACK int8	84 / 84	0.0000

Eager itself gets 82 of 84 exactly right (CER 0.0022 against the rendered truth), so the model is not perfect on this set and every build reproduces it character for character anyway — its two mistakes included.

How close it came

84 lines agreeing is 84 samples. What generalises is how much room each decision had. A character changes only if a build moves the winning logit past its best rival, so this is measured at every one of the 1,008 decoding steps: the winner's lead in the build's own logits, as a fraction of the lead the fp32 reference left it. 1.0 is untouched, 0.0 is a tie, below 0.0 is a changed character.

build	median	worst step	steps changed
XNNPACK fp32	1.0000	1.0000	0 / 1008
XNNPACK fp16	0.9999	0.9820	0 / 1008
XNNPACK int8	0.9999	0.6513	0 / 1008

int8's worst moment still kept 65% of the margin, with the median step untouched at 0.9999. That is the number behind the recommendation, not the 84 identical strings.

The same steps also carry a worst-case bound — the reference gap over twice the largest error anywhere in the 6,144 logits, which assumes that error landed exactly where it would do the most damage. Its minimum is 1.16 for int8, above 1.0, so on this set a changed character is not merely absent but arithmetically impossible. Both numbers are printed by convert/check_manga_ocr.py.

What a whole line costs

The decoder re-reads the prefix each step, so its cost grows with the characters already emitted. Measured per call, median of 20:

prefix	fp32	fp16	int8
1	2.36 ms	2.27 ms	2.18 ms
8	3.37 ms	3.45 ms	2.90 ms
32	3.94 ms	4.88 ms	4.26 ms
64	5.31 ms	6.94 ms	4.99 ms

Sixty-four times the prefix costs 2.3 times the step, which is what makes the missing cache a fair trade. Summed over the prefixes a real line walks through, the decoder side of a 12-character line is 33 ms on int8 (39 ms fp32); add one encoder run — 4.2 ms on Core ML, 25.5 ms on XNNPACK int8 — for the whole read.

What this does not say

• These are rendered lines, not photographed manga. The set varies what a renderer can vary; it has no screentone, no handwriting, no speech-bubble clutter, no page curl. Read it as how far a build has drifted from eager, not as an accuracy score for the model.
• The degradations never broke eager. Its two mistakes are both on the clean and lightly-noised rows; heavy smear, heavy grain and the quality-12 JPEG produced none. So the set never reached a regime where the model itself starts failing, and cannot say what the builds do there.
• Nothing here ran on a phone. Every number is host CPU, and the machine was busy.

Source

Converted with executorch-convert — convert/export_manga_ocr.py for the two files, convert/check_manga_ocr.py for everything measured above. ExecuTorch 1.4.0, PyTorch 2.13.0.