zosmaai/zorro-ocr

Zorro OCR — Hindi/English OCR (MiniCPM-V 4.7 1B LoRA)

A LoRA adapter that turns openbmb/MiniCPM-V-4.7-1B (open-weight ~1.5B VLM, Apache-2.0) into a focused document-OCR model for Devanagari (Hindi) + English text, including the conjuncts and mixed-script layouts where the base model is weakest.

• Base model: openbmb/MiniCPM-V-4.7-1B
• Adapter: PEFT LoRA r=16, alpha=32 on q/k/v/o/gate/up/down projections of the LLM (no vision-tower retraining)
• Training data: 20,000 synthetic document images (Table 1) generated deterministically (seed 20261013) — pure-Devanagari, mixed Hindi+English+digits, conjunct-heavy vocabulary (क्षेत्र, ज्ञान, त्रिभुज, …)
• Recipe: bf16, 2× RTX 5060 Ti (DDP), LR 1e-4 cosine w/ 3% warmup, 2 epochs, length-batched, gradient checkpointing, NaViT variable-shape slicing (4× downsample, ≤36 slices)

Results

Measured with eval/run_eval.py (NF4, seed 0) + eval/metrics.py (NFC-normalized, whitespace-stripped Levenshtein CER).

eval set	base weighted CER	+ LoRA weighted CER	Δ	base word acc	+ LoRA word acc
held-out synthetic val (n=300, same distribution as training)	93.51%	3.17%	−90.34 pp	7.6%	96.8%
50-image Hi/En test set (dev/eng/messy/mixed buckets)	29.88%	20.65%	−9.23 pp	60.1%	72.6%

Per-bucket mean CER on the 50-image set (base → LoRA):

bucket	n	base	LoRA
dev (Devanagari)	15	37.33%	37.06%
eng (English)	15	0.71%	0.26%
messy	5	27.83%	26.68%
mixed (Hi+En)	15	57.94%	33.21%

The 50-image set is genuinely out-of-distribution (real fonts/layouts vs synthetic training), so the honest headline there is word accuracy 60.1% → 72.6%; mixed-script improves most, pure-Devanagari is already the base model's strong bucket. The 300-image val set shares distribution with training — it measures fit, not generalization.

Usage

from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

base = "openbmb/MiniCPM-V-4.7-1B"
adapter = "zosmaai/zorro-ocr"

proc = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base,
        torch_dtype="bfloat16", device_map="auto").eval()
from peft import PeftModel
model = PeftModel.from_pretrained(model, adapter)

msgs = [{"role": "user", "content": [
    {"type": "image", "image": Image.open("doc.jpg").convert("RGB")},
    {"type": "text", "text": "Extract all the text in this image exactly as it appears."}]}]
inputs = proc.apply_chat_template(msgs, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt",
    downsample_mode="4x", max_slice_nums=36, enable_thinking=False)
out = model.generate(**inputs.to(model.device), max_new_tokens=1024)
text = proc.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)

Notes:

• Keep downsample_mode="4x" and enable_thinking=False to match training.
• Works with any HF transformers ≥ the MiniCPM-V 4.7 release; flash-linear-attention is used automatically for the gated delta-net layers.

Training details

param	value
data	19,700 train / 300 val (length-sorted batching)
batch (per GPU)	2, grad-accum 2, world size 2 → effective 8
max seq len	1664 (amounting to ~1,560 tokens + image slice features)
optimizer	AdamW (β 0.9/0.95, wd 0)
LR	1e-4 cosine, 3% linear warmup
LoRA	r=16 α=32 dropout=0, 7 target modules
ckpt	LLM gradient checkpointing + vision encoder layers wrapped
run	4850 steps in 16,421 s (~4.6 h), final val loss 0.0299
dates	trained 2026-10-12 → 2026-10-13 (UTC)

Limitations

• Trained on synthetic rendered documents only (no scans, no photos, no low-quality print); real-world performance should be validated on your own data before relying on it.
• Optimized for horizontal document text; handwritten Devanagari, vertical text, and Indic scripts other than Devanagari are not covered.
• Audio modalities and general VQA performance of the base model are out of scope for this adapter.

License

Adapter weights: Apache-2.0 (matching the base model). Synthetic training data: generated in-house.