NetraOCR-1.0 (1.27B Parameters)

NetraOCR-1.0 is a unified, single-checkpoint 1,273,967,936-parameter multimodal OCR model built on the Qwen3.5-0.8B hybrid Gated-DeltaNet + full-attention architecture. It combines IndicOCR (bodhan-ai/indic-ocr) and OvisOCR3 (ATH-MaaS/OvisOCR3) into a single model via Branch-Train-MiX (BTX, arXiv:2403.07816) and Dual Interleaved Vision Token Fusion (Leo, arXiv:2501.06986).

Architecture Highlights

1. Dual Frozen Vision Encoders with 2-Frame M-RoPE Interleaving:
   • Retains both model.visual (OvisOCR3 ViT + merger) and model.visual_indic (IndicOCR ViT + merger).
   • For each 32×32 spatial cell $(h, w)$, visual tokens are interleaved as [indic(h,w), ovis(h,w)] with 3D M-RoPE temporal coordinates $t=0$ (Indic) and $t=1$ (Ovis).
   • Both ViT backbones stay 100% frozen (max_abs_drift = 0.0), while both 2-layer MLP mergers (visual.merger and visual_indic.merger) are trained to project into a shared latent space.
2. 24-Layer Decoder with Soft-Routed MoE FFN Experts (NetraMoEFFN):
   • Token mixers (18 linear-attention GatedDeltaNet layers + 6 full-attention layers) and RMSNorms are initialized by exact arithmetic weight averaging (M0_avg) and trained end-to-end in Stage M2.
   • Each decoder layer's FFN is a NetraMoEFFN containing Expert 0 (IndicOCR FFN, frozen) and Expert 1 (OvisOCR3 FFN, frozen) weighted by a learned token-level linear router (router.weight $\in \mathbb{R}^{2 \times 1024}$).
3. Lossless Unified Vocabulary (303,101 tokens, padded to 303,104):
   • Preserves all 248,070 OvisOCR3/Qwen3.5 tokens at identical IDs (0..248,069) and appends 55,031 pure-Indic-script tokens from Sarvam's tokenizer (248,070..303,100), initialized via orthogonal Procrustes alignment ($s = 1.1483$, median cosine similarity $0.830$ across $85,892$ shared tokens).
4. Two-Pass Region-Crop Mode for Dense Handwritten Ledgers:
   • Includes split_page_regions(img) in processing_netra.py to automatically segment high-resolution multi-line handwritten ledger pages into horizontal text bands prior to area_clamp.

---

Quickstart Usage

import torch
from PIL import Image
from modeling_netra import NetraForConditionalGeneration
from processing_netra import NetraProcessor, area_clamp, split_page_regions

model_id = "dipankar500/NetraOCR-1.0"
model = NetraForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda().eval()
proc = NetraProcessor.from_pretrained(model_id, frames=2)

img = area_clamp(Image.open("sample.png").convert("RGB"))
prompt = proc.prompt("Extract all text from the image.")
batch = {k: v.cuda() for k, v in proc([prompt], [img]).items()}

with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    out = model.generate(**batch, max_new_tokens=512, do_sample=False)
text = proc.decode(out[0, batch["input_ids"].shape[1]:])
print(text)