tsan1024/tocr

TeleOCR Layout LoRA (text_with_formula)

This model is a layout-detection LoRA fine-tune of TeleOCR (Qwen2.5-VL 1.2B), merged back into the base weights for direct inference.

This model is provided for personal amateur study and research only; production availability is not guaranteed.

Purpose

Built on top of the original TeleOCR layout detection, this model adds a text_with_formula label to distinguish:

• text: ordinary text blocks
• text_with_formula: blocks whose text contains inline formulas ($...$)

The output format is identical to native TeleOCR:

<box:xmin ymin xmax ymax><label:text_with_formula><up>

Coordinates are relative to a 1036×1036 layout image (integers in 0~1000).

Training Details

• Base model: TeleOCR (Qwen2.5-VL, ~1.2B)
• Data: OmniDocBench (1651 pages; 1568 train / 83 validation)
• Method: LoRA (r=32, alpha=64, dropout=0.05), training only the language-model projection layer
• Training: 3 epochs, lr=2e-4, batch=1×8 (grad accumulation), bfloat16, cosine schedule
• Data cleaning: strict inline-formula detection (distinguishes currency $5 from real $...$)

Evaluation (83-page validation set, Hungarian full-category matching + NMS)

Category	Precision	Recall	F1
text	0.767	0.723	0.744
text_with_formula	0.837	0.692	0.757
micro overall	0.779	0.713	0.744

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
from TeleOCR.vlm_utils.TeleOCR_client import TeleOCRClient

model_path = "<path to this model>"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, fix_mistral_regex=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_path, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
).cuda().eval()

client = TeleOCRClient(backend="transformers", model=model, processor=processor, model_path=model_path)
blocks = client.layout_detect(image)  # each block contains type / bbox / angle

Note: this model includes custom code modeling_naviocr.py, so trust_remote_code=True is required when loading.

Acknowledgements

Many thanks to the TeleOCR team for open-sourcing the excellent TeleOCR model and code. This model was fine-tuned and studied on top of its base model and toolchain.