nara-lightonocr-3-4b

lightonai/LightOnOCR-3-4B fine-tuned for line-by-line transcription of English-language archival documents — United States National Archives Revolutionary War pension files: printed forms completed by hand, catalogue cards, and 18th–19th-century American cursive. One line of output per physical line, printed and handwritten text together in reading order.

LightOnOCR-3 is LightOn's document-OCR model, built on the Qwen3.5 vision-language architecture. This fine-tune keeps that architecture unchanged and retrains it on 38,756 pages of this collection.

Results

762 held-out pages from 39 documents, none of which appear among the 1,407 training documents. Documents are held out whole: ~27 pages of one document share a hand and a printed form, so a page-level split would measure memorisation. NFD normalisation is a no-op on English ASCII, so plain CER is quoted.

model	CER	macro	WER	line recall	CER excl. degenerate	truncated
LightOnOCR-3-4B, zero-shot (same job)	0.2057	0.3259	0.2794	0.1956	0.2057	0.1%
this model	0.2312	0.3531	0.2964	0.1927	0.1857	0.7%
LightOnOCR-3-0.8B, full FT	0.2550	0.4133	0.3297	0.1867	0.2230	1.0%
nara-qwen-3.5-2b (published)	0.2688	0.4607	0.3205	0.1911	0.2688	0.9%
nara-qwen-3.5-0.8b (published)	0.2868	0.3822	0.3379	0.1901	0.2328	1.3%

Greedy decoding, 3,072 max new tokens, 2,048 visual tokens per page, eval batch 4. Treat differences under 0.02 CER as ties — four seeds of one configuration in this project span 0.021.

Read this first: this fine-tune is slightly worse than the model it started from. Untrained LightOnOCR-3-4B given this repo's prompt.txt scores 0.2057; after three epochs on 38,756 NARA pages it scores 0.2312 (+0.0255). Both were measured in the same job on the same pages, so the comparison is exact. For the best aggregate CER on this material, use stock lightonai/LightOnOCR-3-4B with prompt.txt.

It is published anyway because the aggregate hides a real improvement. On pages where decoding does not degenerate, the fine-tune reads better — CER excluding degenerate pages falls from 0.2057 to 0.1857 — and it has learned the archive's conventions: Rev War appears 21 times in its output against 19 in the references (zero-shot: 3).

What it lost is stopping, and one learned convention is the trap. It writes [illegible] 2,289 times against 357 in the references (zero-shot: 8) — but 3 pages account for 1,722 of those (75%), each looping [illegible] [illegible] … to the token ceiling where its reference has 2, 3, 1. Across the other 759 pages it uses the marker 1.6× as often as the references (567 vs 351). 0.7% of pages now loop against 0.0% before, and on micro-averaged CER those few pages outweigh every ordinary page it improved.

The cause is the learning rate. The base was already close to the target, and at the schedule's peak (5e-5) training loss rose from 0.80 to 1.02 before recovering; evaluated mid-run the model collapsed and then climbed back (CER 1.2148 at 0.62 epochs, 0.5984 at 0.83, 0.2312 at the end). A lower learning rate (~1e-5) for a start this strong is the obvious next experiment and has not been run.

Comparing across GPUs

This model was evaluated on an H200; most comparison rows on an RTX PRO 6000. That matters more than it sounds: untrained LightOnOCR-3-4B scored 0.2332 on one GPU type and 0.2057 on the other — the same weights, pages and code, 0.0275 apart. bf16 arithmetic differs between architectures and greedy decoding amplifies it. Treat gaps under ~0.03 between this model and RTX-scored rows as ties. The zero-shot row above was scored in the same job as this model, so that one comparison is exact.

Usage

import torch
from PIL import Image
from huggingface_hub import hf_hub_download
from transformers import AutoModelForImageTextToText, AutoProcessor

REPO = "wjbmattingly/nara-lightonocr-3-4b"
model = AutoModelForImageTextToText.from_pretrained(REPO, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(REPO)

# The prompt is part of the model: it was trained to answer this exact instruction.
# Use it verbatim -- LightOnOCR's own empty transcription prompt is a different task.
prompt = open(hf_hub_download(REPO, "prompt.txt"), encoding="utf-8").read()

image = Image.open("page.jpg").convert("RGB")
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False,
                                     add_generation_prompt=True, enable_thinking=False)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=3072, do_sample=False,
                     repetition_penalty=1.1, use_cache=True)
print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

• prompt.txt is required. This model was trained on a 1,539-character instruction, shipped in this repo; LightOnOCR-3 itself was trained on an empty prompt and emits block-labelled markdown with bounding boxes, which is not this task.
• enable_thinking=False. Passed explicitly so the generation prompt closes the thinking block regardless of which template variant is loaded.
• The image budget is in the repo. processor_config.json is capped at 2,048 visual tokens, the budget used in training and scoring. LightOnOCR-3 ships uncapped (~16k tokens); loading a stock processor shows the page at a resolution this model never trained on.
• repetition_penalty=1.1 is this project's measured serving setting: it removes most looping on sparse pages at a small cost in line recall. The table above is greedy (penalty 1.0); set 1.0 to reproduce it.
• generation_config.json sets use_cache: true. LightOnOCR-3-0.8B ships false (a training artefact), which makes generation recompute the whole sequence at every step; every number above was produced with the cache on.

Training

Full fine-tune of all 4,539,265,536 parameters (vision tower included), 3 epochs, 14,535 steps, learning rate 5e-05, cosine schedule, effective batch 8, bf16, 2,048 visual tokens per page, on H200. It exceeded the cluster's 48-hour job limit and was trained across chained jobs, each resuming optimizer, scheduler and RNG state from the previous one's last full checkpoint (the final job from checkpoint-14000).

Data: wjbmattingly/si-test (first 40,000 rows).

Limits

A research artifact for cultural-heritage and digital-humanities work, not a production OCR service. Trained on one collection, it should be expected to degrade on other hands, languages, genres and centuries. It reproduces what is written, including historical spelling and language; transcribing a record is not an endorsement of its contents. Verify any name, date or figure before relying on it.