thealper2/LaTeX_OCR_rec-im2latex

LaTeX_OCR_rec fine-tuned on AlFrauch/im2latex

PaddleOCR LaTeX_OCR_rec fine-tuned for printed formula image → LaTeX recognition on a 200,000-sample subset of AlFrauch/im2latex. Architecture, tokenizer, preprocessing and decoding are identical to PaddleOCR v3.7.0; only the weights changed.

Model

Architecture	LaTeX-OCR: HybridTransformer encoder (ResNetV2 stem, layers [2, 3, 7] → 4 ViT blocks, dim 256, 8 heads, patch 16) + LaTeXOCRHead Transformer decoder (4 layers, dim 256, 8 heads, cross-attention, attention-on-attention, GLU feed-forward)
Parameters	25.56 M
Base weights	official PaddleOCR rec_latex_ocr_train/best_accuracy.pdparams
Input	grayscale image, width ≤ 672 px, height ≤ 192 px, padded to multiples of 16; normalization mean 0.7931, std 0.1738
Output	LaTeX string, at most 512 tokens including [BOS]/[EOS]
Tokenizer	PaddleOCR latex_ocr_tokenizer.json, byte-level BPE, 8,000 tokens (unmodified)
Decoding	official: top-k filter (threshold 0.9) + sampling at temperature 0.333, no KV cache

Files

File	Size	Description
inference.json, inference.pdiparams, inference.yml	101 MB	inference model exported with PaddleOCR tools/export_model.py
best_accuracy.pdparams	100 MB	training checkpoint (dygraph weights) for further fine-tuning with PaddleOCR
latex_ocr_tokenizer.json	24 KB	tokenizer used for training and decoding
config.yaml	4 KB	full training configuration
predict.py	3 KB	standalone inference script (Paddle Inference)

Usage

pip install paddlepaddle-gpu==3.3.1 -i https://www.paddlepaddle.org.cn/packages/stable/cu129/
pip install albumentations==1.4.24 tokenizers pyyaml opencv-python
git clone --depth 1 --branch v3.7.0 https://github.com/PaddlePaddle/PaddleOCR.git
python predict.py --paddleocr ./PaddleOCR --image formula.png

Output (tab-separated):

formula.png	\mathcal{E}\mathcal{B}(\Delta)\simeq\left(\frac{\Delta}{T}\right)^{1-p}

Use --cpu for CPU inference. The exported files follow the same format as PaddlePaddle/LaTeX_OCR_rec (Global.model_name: LaTeX_OCR_rec). Note that tools/infer/predict_rec.py in PaddleOCR v3.7.0 rejects LaTeX_OCR_rec inference directories.

To continue training, set Global.pretrained_model in a PaddleOCR LaTeX_OCR_rec config to best_accuracy (without extension).

Training data

Source	AlFrauch/im2latex, revision 403adaf00649332f7065d33034da51f79356916a, 1,586,584 pairs
Filtering	1,461,238 pairs kept. Rejected: 121,984 image larger than 672×192; 594 image width or height < 32 px; 2,636 label with characters outside the tokenizer vocabulary (mostly U+00A0); 130 label longer than 510 tokens; 2 empty label. No label was truncated.
Label normalization	leading/trailing whitespace stripped; LaTeX otherwise unchanged
Split	grouped by identical label text or identical image bytes (no shared label or image across splits): train 1,432,047 / validation 14,726 / test 14,465
Used for this model	200,000 pairs sampled uniformly from the train split (seed 0)

Training procedure

Framework	PaddlePaddle 3.3.1 (CUDA 12.9), PaddleOCR v3.7.0 tools/train.py
Hardware	1× NVIDIA GeForce RTX 5060 Ti 16 GB
Epochs / steps	2 / 7,120
Training time	42 min
Batching	each batch holds images of the same padded size and a similar label length (tiers ≤64, ≤128, ≤256, ≤512 tokens); batch size 26–64 (mean 56), gradient accumulation 1
Optimizer	AdamW, β = (0.9, 0.999), weight decay 0.01, gradient clipping (global norm 1.0)
Learning rate	cosine decay from 5e-5, linear warmup over the first 1.25 % of an epoch
Precision	automatic mixed precision O1 (float16), dynamic loss scaling
Augmentation	official LatexTrainTransform (shift/scale/rotate, grid distortion, RGB shift, Gaussian noise, brightness/contrast, JPEG compression, binarization p = 0.04)
Seed	42
Final training loss	0.082

Evaluation

Validation subset: 2,000 images sampled from the validation split (seed 0). Metrics as in ppocr.metrics.rec_metric.LaTeXOCRMetric: prediction and reference are both decoded with LaTeXOCRDecode (whitespace not between two letters is removed) before comparison.

• ExpRate: exact string match of the whole expression
• BLEU: mean sentence-level BLEU
• Edit distance: character-level normalized Levenshtein distance (lower is better)

Model	Step	ExpRate	BLEU	Edit distance
this model	2,000	0.4525	0.8419	0.1326
this model	4,000	0.4915	0.8648	0.1158
this model (released)	6,000	0.5085	0.8691	0.1115

A separate evaluation of the released checkpoint on the same 2,000 images (fp32, seed 42) gave ExpRate 0.5035, BLEU 0.8687, edit distance 0.1108; the difference comes from the sampling decoder. The full validation and test splits were not evaluated.

Limitations

• Inputs larger than 672×192 px are cropped to the ink bounding box and downscaled by MinMaxResize; such images were excluded from training.
• Trained on 14 % of the filtered training split for 2 epochs; validation accuracy was still improving at the end.
• The source dataset contains identical renderings with different LaTeX strings (e.g. with/without \displaystyle, {\rm d} vs \mathrm{d}), which bounds exact-match accuracy.
• Decoding samples from the output distribution, so repeated runs can produce slightly different strings.
• Characters outside the tokenizer vocabulary (non-ASCII) cannot be produced.
• Inference is autoregressive without KV cache; runtime grows quadratically with output length.

Checksums (SHA-256)

File	SHA-256
inference.json	e6a1a69107231c68483b5e029ebdfdea83cb17a97731fac67e62a3a580c8244c
inference.pdiparams	e307368424e6ff1e220d4d2aa88ef6608dcd3bc2a93d1e20ffbe1fbe48d787b5
inference.yml	31e9f4fa65e903fff3f32ea6c2016a0ae84e69de34eb28f32c2764093d84f9cd
best_accuracy.pdparams	e4e5f43ff1f0160381aee1a8d966b9afdca768fc9e701fcda4b073375923d223
latex_ocr_tokenizer.json	50aff4b7f753cc27388cde5c4fda7cc8e19c5904ca1d993cb332187a6d3eb060
config.yaml	2c63748f2c0429ce9884ecc4cdb7ae593b2afc84693b6c7849f2fc34097ca384