qandeelasim13/urdu-ocr-trocr-si26

Urdu OCR — Fine-Tuned TrOCR (SI-26)

A TrOCR model fine-tuned end-to-end to read printed Urdu (Nastaliq-style) text from images and output editable Unicode text.

Try the live demo: Streamlit app

---

Model Description

Optical Character Recognition for Urdu lags far behind Latin-script OCR — Urdu's cursive, context-dependent Nastaliq script and the scarcity of labeled datasets make off-the-shelf tools like Tesseract perform poorly out of the box (a baseline test on this project's own data measured only a 2.1% word-recovery rate from Tesseract). This model fine-tunes TrOCR specifically on Urdu text so it can read real-world Urdu images — for example, digitizing scanned documents, signboards, or book pages.

TrOCR pairs a vision encoder (which "looks" at the image) with a text decoder (which "writes out" what it reads). Its decoder's tokenizer is a byte-level BPE tokenizer that can already represent any Unicode text — including Urdu — via UTF-8 byte-level fallback tokens. This model fine-tunes the full pretrained checkpoint, encoder and decoder together, rather than training a decoder from scratch, so it only has to learn Urdu rather than language modeling from zero.

• Base model: microsoft/trocr-base-printed
• Fine-tuning: two-phase — decoder-only warm-up, then the full model unfrozen at a lower learning rate, 15 epochs total
• Language: Urdu (ur)
• Task: image-to-text (OCR)
• License: Apache 2.0

Intended Use

Extracting Urdu text from clean, printed, single-line images — scanned documents, signboards, book pages, or screenshots of printed text. Not intended for handwriting, multi-line paragraphs, or heavily degraded/noisy images.

How to Use

from transformers import VisionEncoderDecoderModel, TrOCRProcessor
from PIL import Image
import torch

repo_id = "qandeelasim13/urdu-ocr-trocr-si26"
processor = TrOCRProcessor.from_pretrained(repo_id)
model = VisionEncoderDecoderModel.from_pretrained(repo_id)
model.eval()

image = Image.open("your_urdu_image.png").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values

with torch.no_grad():
    generated_ids = model.generate(pixel_values, max_length=319, num_beams=4)

text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(text)

Training Data

• Size: 3,160 labeled images (~1,348 unique source images plus augmented rotation/blur/brightness copies)
• Sources: UTRSet-Real (real printed Urdu word images, ICDAR 2023), synthetic sentences rendered from Urdu Wikipedia text in 3 different Urdu fonts, augmented variants (blur/brightness/rotation), and manual screenshots from Dawn Urdu, BBC Urdu, Jang, and Wikipedia.
• Split: leakage-safe, grouped by parent image so augmented copies of the same source image never appear in both train and test.

Evaluation Results

Metric	Score
Character Error Rate (CER)	0.52
Character-level accuracy	47.66%
Training loss	5.00 → 0.18 (avg per epoch, 15 epochs)

Sample predictions showed exact or near-exact matches on shorter sentences, with more errors on longer, more complex sentences.

Limitations

• Dataset size (~1,348 unique source images) is the primary bottleneck — the training notebook's own working estimate suggests 80–95% accuracy would be achievable with substantially more data.
• Performs best on clean, printed, single-line Urdu text; accuracy drops on handwriting, multi-line paragraphs, low-resolution scans, and heavy noise/blur.
• Character Error Rate of 0.52 means roughly half of characters may be misread on average — always spot-check output for anything beyond casual/exploratory use.

Training Procedure

1. Loaded the pretrained microsoft/trocr-base-printed checkpoint (encoder + decoder + processor together).
2. Phase 1: froze the encoder, trained the decoder only (warm-up).
3. Phase 2: unfroze the full model, continued training at a lower learning rate.
4. 15 epochs total, evaluated on a held-out, leakage-safe test split using Character Error Rate.

Frameworks: PyTorch, Hugging Face transformers (VisionEncoderDecoderModel, Seq2SeqTrainer), evaluate + jiwer for CER, trained on a Google Colab GPU runtime (Tesla T4).

Credit

Qandeel Asim — built during the Code Saviours ML/AI Internship, Batch SI-26.

• GitHub: URDU-OCR-PROJECT-CODE-SAVIOURS-SI-2026-QANDEEL-ASIM
• Live demo: Streamlit app