MonkeyOCRv2: A Visual-Text Foundation Model for Document AI

News

2026.07.11 🚀 We release MonkeyOCRv2, including MonkeyOCRv2 vision encoder, MonkeyOCRv2-Parsing for multilingual document parsing, MonkeyOCRv2-Und for efficient document understanding.

Introduction

MonkeyOCRv2 is a text-centric visual foundation model that unifies fine-grained text modeling, cross-task representation learning, and cross-lingual generalization in a single encoder. MonkeyOCRv2 generalizes effectively across a broad range of OCR and document intelligence tasks, including multilingual document parsing, document understanding, text recognition, formula recognition, document tampering detection, scene text detection, and overlapping text segmentation.

Model Zoo

1. Vision Encoder

Model	Backbone	Params	Pretraining Resolution	Applicable Tasks	Checkpoint Link
Monkey OCRv2-S	ViT-S	28M	12802828	Recognition / Parsing / Understanding	🤗HuggingFace 🤖ModelScope
Monkey OCRv2-B	ViT-B	113M	12802828	Recognition / Parsing / Understanding	🤗HuggingFace 🤖ModelScope
Monkey OCRv2-AS	ViTAEv2-S	21M	17603232	Detection / Segmentation	🤗HuggingFace 🤖ModelScope

2. Document Parsing Model

Model	Link	Total Params	ViT	LLM	All	Digit.	Photo.	Latin Avg.	DE	EN	ES	FR	ID	IT	NL	PT	VI	Non-Latin Avg.	AR	HI	JP	KO	RU	TH	ZH	ZH-T
MonkeyOCRv2-S-Parsing	HuggingFace ModelScope	0.6B	0.03B	0.6B	82.5	87.9	80.7	83.2	87.3	83.6	76.8	73.6	85.4	87.2	85.5	87.4	81.9	81.7	91.2	87.1	69.9	88.7	78.0	79.8	84.4	74.7
MonkeyOCRv2-B-Parsing	HuggingFace ModelScope	0.7B	0.1B	0.6B	83.3	88.1	81.7	84.2	87.7	84.5	75.2	78.4	86.5	88.6	86.1	87.9	83.2	82.1	90.7	87.2	71.9	87.6	80.1	80.8	83.6	75.3

3. Document Understanding Model

Model	Link	Total Params	Overall	DocVQA	InfoVQA	DF	KLC	WTQ	ChartQA	DT-VQA	OCRBench
MonkeyOCRv2-S-Und	HuggingFace ModelScope	1.7B	55.9	79.3	44.5	65.1	37.6	43.0	62.0	63.1	52.2
MonkeyOCRv2-B-Und	HuggingFace ModelScope	1.8B	57.2	79.3	46.3	65.8	38.2	43.2	62.0	64.3	58.1

Quick Start

Vision Encoder

1. Install

Install transformers and flash attention:

conda create -n MonkeyOCRv2 python=3.10
conda activate MonkeyOCRv2
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126
pip install transformers==4.57.6
pip install flash-attn==2.7.4.post1 --no-build-isolation
pip install accelerate
pip install qwen_vl_utils

2. Download Model Weights

Download our model from Huggingface.

python download_model.py -n MonkeyOCRv2-B # or MonkeyOCRv2-S / MonkeyOCRv2-AS

You can also download our model from ModelScope.

pip install modelscope
python download_model.py -t modelscope -n MonkeyOCRv2-B # or MonkeyOCRv2-S / MonkeyOCRv2-AS

3. Extract Image Feature

cd vision
# For MonkeyOCRv2-B and MonkeyOCRv2-S
python extract_feature.py
# For MonkeyOCRv2-AS
python extract_feature_vitae.py

Document Parsing

1. Install

Install vLLM following its official guide:

conda create -n MonkeyOCRv2Parsing python=3.10
conda activate MonkeyOCRv2Parsing
pip install uv
uv pip install vllm==0.11.2 --torch-backend=auto -i https://pypi.tuna.tsinghua.edu.cn/simple requests
pip install -r parsing/requirements.txt

2. Download Model Weights

Download our model from Huggingface.

python download_model.py -n MonkeyOCRv2-B-Parsing # or MonkeyOCRv2-S-Parsing

You can also download our model from ModelScope.

pip install modelscope
python download_model.py -t modelscope -n MonkeyOCRv2-B-Parsing # or MonkeyOCRv2-S-Parsing

3. Inference

Parse a single document or a directory containing PDFs or images:

cd parsing
python parse.py \
    -i ../images_test/ar.JPEG \
    -o output/test \
    -m ../model_weight/MonkeyOCRv2-B-Parsing \
    -g 500 \
    --draw-layout \
    --skip-processed
# Show help messages
python parse.py -h

4. Web Demo

Start gradio web demo:

cd parsing
python demo/gradio_demo.py \
    --model-path ../model_weight/MonkeyOCRv2-B-Parsing \
    --output-dir output/demo_outputs

Document Understanding

1. Install

See install part of MonkeyOCRv2.

2. Download Model Weights

Download our model from Huggingface.

python download_model.py -n MonkeyOCRv2-B-Und # or MonkeyOCRv2-S-Und

You can also download our model from ModelScope.

pip install modelscope
python download_model.py -t modelscope -n MonkeyOCRv2-B-Und # or MonkeyOCRv2-S-Und

3. Inference

cd understanding
python infer.py \
    -m ../model_weight/MonkeyOCRv2-B-Und \
    -i ../images_test/vqa.png \
    -q 'What is the serving size?'
# Show help messages
python infer.py -h

Visualization

Our model supports robust document parsing in real-world scenarios across 17 languages, including Simplified Chinese (ZH), Traditional Chinese (ZH-T), English (EN), Arabic (AR), German (DE), Spanish (ES), French (FR), Hindi (HI), Indonesian (ID), Italian (IT), Japanese (JP), Korean (KO), Dutch (NL), Portuguese (PT), Russian (RU), Thai (TH), and Vietnamese (VI).

Evaluation Results

1. Text recognition results on Common Benchmarks, Union14M-Benchmark, OST, and Chinese Benchmarks. We follow the training and evaluation protocols of OpenOCR.

Model	Overall	Union14M-Benchmark								Chinese Benchmarks					Occlusion SceneText
Model	Overall	Avg	Artistic	Context less	Curve	General	Multi Oriented	Multi Words	Saliency	Avg	Scene	Web	Document	Hand writing	Occlusion SceneText
ABINet	73.7	75.7	71.7	74.7	80.4	79.8	69.0	76.8	77.6	70.3	66.6	63.2	98.2	53.1	75.0
MAERec	81.6	85.2	79.0	84.2	89.1	84.6	87.1	85.9	86.3	83.1	84.4	83.0	99.5	65.6	76.4
CPPD	80.4	81.9	76.5	82.9	86.2	83.5	78.7	81.9	83.5	81.7	82.7	82.4	99.4	62.3	79.6
IGTR-AR	81.0	84.9	77.0	82.4	90.4	84.4	91.2	84.0	84.7	81.7	82.0	81.7	99.5	63.8	76.3
SMTR	80.4	85.0	76.8	83.9	89.1	83.7	87.7	89.3	84.6	82.7	83.4	83.0	99.3	65.1	73.5
SVTRv2	83.1	86.1	79.3	86.1	90.6	85.1	89.0	86.7	86.2	83.3	83.5	83.3	99.5	67.0	80.0

CRNN (ResNet)	58.7	49.2	51.2	62.3	48.1	68.2	13.0	60.4	41.4	68.8	63.8	68.2	97.0	46.1	58.0
CRNN (MonkeyOCRv2-S)	67.3	65.2	63.7	73.0	71.1	74.5	28.6	72.1	73.4	74.2	73.0	74.9	96.9	51.8	62.4
PARSeq (ViT)	82.2	84.3	76.5	83.4	87.6	84.9	88.8	84.3	84.4	82.4	84.2	82.8	99.5	63.0	79.9
PARSeq (MonkeyOCRv2-S)	84.3	87.6	78.6	86.4	92.1	85.4	93.9	88.7	87.7	83.7	84.6	83.2	99.5	67.3	81.5

2. Formula recognition results on OmniDocBench 1.6, MathWriting, and UniMER-Test.

Model	Params	Overall		OmniDocBench 1.6		MathWriting		SPE		CPE		HWE		SCE
Model	Params	CDM	ExpRate	CDM	ExpRate	CDM	ExpRate	CDM	ExpRate	CDM	ExpRate	CDM	ExpRate	CDM	ExpRate
Pix2tex	25.5M	53.8	23.3	69.4	27.0	0.4	0.0	96.2	72.4	64.9	7.1	24.5	0.6	67.6	32.8
Texify	312M	67.3	40.4	76.5	46.4	26.6	2.0	98.5	91.0	70.4	28.2	52.7	23.6	79.3	51.3
UniMERNet-B	325M	89.5	64.5	90.4	59.5	63.8	12.3	99.1	93.3	96.0	80.5	94.0	64.3	93.7	77.0
UniMERNet-S	202M	89.8	64.0	90.1	59.1	65.9	12.7	99.1	93.4	95.9	77.7	93.7	63.9	94.1	76.9

UniMERNet-T (Swin)	107M	89.4	61.8	89.9	57.2	65.6	12.9	99.1	92.3	94.9	69.9	93.3	61.9	93.8	76.6
UniMERNet-T (MonkeyOCRv2-S)	110M	90.9	66.4	90.8	61.1	70.8	16.2	99.2	93.8	96.1	79.2	94.3	69.5	94.0	78.6

3. Text detection results on Total-Text, CTW1500, ICDAR2015 and ArT. We follow the training and evaluation protocols of MMOCR and DPText-DETR.

4. Document tampering detection results on DocTamper benchmark.

Method	Params	Overall		DocTamper-Test				DocTamper-FCD				DocTamper-SCD
Method	Params	IoU	F	IoU	P	R	F	IoU	P	R	F	IoU	P	R	F
PSCC-Net	5M	13.7	31.3	17.0	25.0	83.0	39.0	13.0	19.0	82.0	30.0	11.0	15.0	83.0	25.0
UperNet	67M	49.3	54.0	70.0	66.0	60.0	62.0	30.0	57.0	35.0	43.0	48.0	57.0	58.0	57.0
CAT-Net	114M	67.3	71.0	78.0	75.0	69.0	72.0	66.0	85.0	70.0	76.0	58.0	65.0	65.0	65.0
Swin-UPer	81M	66.7	71.7	79.0	75.0	72.0	73.0	64.0	80.0	70.0	75.0	57.0	66.0	68.0	67.0
SegFormer	85M	70.3	74.0	81.0	77.0	74.0	75.0	69.0	82.0	74.0	78.0	61.0	68.0	70.0	69.0
Mask2Former	69M	69.7	78.0	84.0	82.0	83.0	82.0	66.0	81.0	75.0	78.0	59.0	70.0	79.0	74.0
ConvNext	122M	69.7	75.3	84.0	81.0	78.0	79.0	62.0	76.0	71.0	74.0	63.0	71.0	74.0	73.0
ConvNextV2	121M	72.7	77.7	86.0	82.0	79.0	81.0	65.0	79.0	75.0	77.0	67.0	74.0	76.0	75.0
InternImage	128M	73.3	77.7	84.0	81.0	77.0	79.0	72.0	83.0	79.0	81.0	64.0	73.0	74.0	73.0
ASC-Former	80M	68.2	80.8	81.5	91.8	87.8	89.8	61.3	74.9	77.1	76.0	61.9	78.0	75.0	76.5
DTD	66M	77.0	79.7	84.0	81.0	77.0	79.0	79.0	88.0	82.0	85.0	68.0	75.0	76.0	75.0

FFDN* (ViTAEv2)	69M	70.7	82.7	69.4	76.2	88.7	82.0	79.0	92.5	84.4	88.3	63.6	79.1	76.5	77.8
FFDN (MonkeyOCRv2-AS)	71M	78.2	87.5	87.4	94.8	91.8	93.3	79.9	90.4	87.4	88.9	67.2	81.0	79.8	80.4

* denotes models trained with the ViTAEv2 pretrained by DeepSolo

5. Overlapping text segmentation results on MOT dataset.

Model	mIoU_Text	IoU_Occ	IoU_Occd	IoU_Ov
Unet	62.2	80.2	65.7	40.7
Deeplab v3	67.9	83.2	71.2	49.3
OCRNet	65.8	81.0	68.5	47.8
Segformer	69.0	83.6	74.1	49.3
MaskFormer	68.4	83.5	70.3	51.4
TexRNet	68.9	84.2	73.2	49.3
EAFormer	69.1	83.8	74.2	50.5
WASNet	70.8	84.8	74.4	53.1

Mask2Former (ResNet)	70.3	84.7	73.3	52.8
Mask2Former (MonkeyOCRv2-AS)	76.6	88.6	83.4	57.7
MOTS (ResNet)	72.6	85.2	77.5	54.9
MOTS (MonkeyOCRv2-AS)	76.9	88.6	82.6	59.4

6. Document parsing results on MDPBench, a comprehensive multilingual benchmark for real-world document parsing.

Model	Total Params	ViT	LLM	All	Digit.	Photo.	Latin Avg.	DE	EN	ES	FR	ID	IT	NL	PT	VI	Non-Latin Avg.	AR	HI	JP	KO	RU	TH	ZH	ZH-T
Closed-source VLMs
ChatGPT-5.2-2025-12-11	-	-	-	68.6	85.6	63.0	75.2	70.8	79.4	71.4	60.0	77.7	78.5	71.6	85.0	82.1	61.1	64.9	63.4	55.8	65.4	60.7	63.8	56.3	58.7
Claude-Sonnet-4.6	-	-	-	73.1	85.0	69.3	79.2	79.8	80.6	72.8	66.5	82.3	83.3	76.7	88.0	83.1	66.2	67.8	71.7	63.4	64.3	70.8	65.2	61.3	65.1
Doubao-2.0-pro	-	-	-	74.2	78.9	72.8	75.7	82.8	74.4	69.0	70.0	73.3	82.0	69.9	83.4	76.5	72.5	81.3	75.7	65.8	74.7	63.3	71.9	71.9	75.2
Gemini-3-pro	-	-	-	86.4	90.4	85.1	88.4	91.2	90.6	83.4	82.7	91.5	91.6	87.7	91.4	85.9	84.1	89.4	90.4	74.8	85.5	84.9	80.6	85.1	82.1
Open-source VLMs
InternVL-3.5-8B	8.3B	0.3B	8B	42.7	59.7	37.0	53.4	39.8	64.2	47.5	42.7	53.8	60.6	52.2	63.2	57.0	30.6	8.2	9.0	45.6	30.3	26.1	10.8	55.3	59.3
MinerU-2.5	1.2B	0.7B	0.5B	46.3	61.9	40.8	63.0	68.8	78.4	54.7	57.3	67.5	75.2	60.4	58.8	46.0	27.4	1.3	9.0	39.1	14.7	8.6	11.3	72.9	62.2
DeepSeek-OCR	3.4B	0.4B	3B	51.8	80.7	42.2	54.5	55.0	58.3	44.1	43.2	60.9	69.3	52.4	53.0	54.1	48.9	56.9	52.2	49.1	28.2	36.2	49.4	59.7	59.2
MonkeyOCR-pro-3B	3.7B	0.7B	3B	52.2	68.0	47.0	65.1	71.7	77.9	55.9	62.1	66.2	74.5	66.3	71.1	40.2	37.6	4.6	4.2	55.2	60.5	42.6	9.1	72.2	52.4
Nanonets-OCR-s	4.7B	0.7B	4B	63.7	78.8	58.7	71.3	75.1	78.5	61.2	62.5	70.3	81.0	69.6	75.9	67.5	55.0	59.5	61.8	55.9	51.2	43.5	39.5	67.4	61.5
Nanonets-OCR2-3B	3.7B	0.7B	3B	64.2	79.2	59.3	71.4	76.7	76.4	61.8	66.1	68.4	78.5	74.1	74.2	66.0	56.2	60.2	59.2	52.1	54.7	45.5	44.6	68.3	65.1
Qwen3.5-Instruct-9B	9.7B	0.7B	9B	65.7	74.8	62.7	72.5	72.8	72.0	72.0	64.4	66.2	77.6	74.5	79.1	74.0	58.2	53.4	56.2	55.7	60.3	54.7	56.7	60.8	67.5
GLM-OCR	0.9B	0.4B	0.5B	67.3	77.9	63.7	78.7	82.7	84.5	75.8	76.2	79.7	82.8	80.2	77.4	69.2	54.3	21.7	39.6	65.5	61.2	64.2	27.4	78.5	76.7
Qwen3-VL-Instruct-8B	8.3B	0.3B	8B	68.3	78.4	65.0	73.6	73.7	71.4	69.3	66.2	68.5	79.1	78.3	82.2	73.4	62.5	63.1	58.4	59.9	61.9	57.9	62.0	62.6	73.8
HunyuanOCR	1B	0.4B	0.6B	68.3	80.2	64.3	72.4	75.0	73.1	63.0	66.1	69.9	80.3	61.4	81.9	80.6	63.7	68.3	73.1	55.6	68.9	52.2	60.7	66.8	64.2
PaddleOCR-VL	0.9B	0.6B	0.3B	69.6	87.6	63.6	72.1	78.2	79.3	62.9	66.0	77.4	78.4	67.9	72.0	66.6	66.7	65.8	68.4	59.9	77.8	56.9	57.8	78.2	68.5
olmOCR2	7.7B	0.7B	7B	70.4	79.9	67.2	76.7	75.7	77.3	72.5	68.9	70.6	81.0	72.0	88.0	84.0	63.3	59.0	60.8	59.4	70.6	65.8	59.2	68.6	63.4
MinerU-2.5-Pro	1.2B	0.7B	0.5B	71.0	86.2	66.1	74.6	78.3	79.5	63.4	67.4	78.0	79.7	72.1	78.6	74.2	67.0	56.6	72.2	59.1	77.6	62.6	61.8	76.5	69.7
PaddleOCR-VL-1.6	0.9B	0.6B	0.3B	75.0	82.8	72.6	78.0	84.1	79.7	69.2	74.8	81.6	82.0	74.7	76.4	79.3	71.6	69.4	65.6	68.7	82.5	70.7	62.3	78.0	75.7
HunyuanOCR-1.5	1B	0.4B	0.6B	76.8	86.2	73.6	79.7	79.6	80.4	74.2	70.0	81.5	84.5	78.4	86.4	82.4	73.5	71.8	71.6	65.5	75.7	67.4	77.7	80.8	77.2
Kimi-K2.5	1T	0.4B	1T	77.5	85.0	75.0	81.6	85.9	86.2	72.7	71.0	80.6	86.6	77.4	87.6	86.2	72.9	75.8	74.5	72.5	70.9	61.8	67.0	81.7	78.6
PaddleOCR-VL-1.5	0.9B	0.6B	0.3B	78.3	87.4	75.2	81.2	84.8	83.0	75.7	78.1	83.9	85.2	80.6	80.2	78.9	74.9	71.3	67.7	69.5	86.0	76.0	68.4	84.8	75.7
chandra-ocr-2	5.3B	0.5B	4.8B	79.7	87.8	77.1	82.7	86.6	86.5	69.7	70.3	84.6	87.4	82.7	90.7	85.6	76.4	78.2	81.1	68.8	80.3	74.0	78.5	73.8	76.3
dots.mocr	3B	1.2B	1.8B	80.5	90.5	77.2	81.7	82.6	87.4	71.3	70.1	84.5	89.3	83.2	86.8	79.9	79.2	83.3	83.6	75.0	78.7	71.2	77.9	84.6	79.6
MonkeyOCRv2-S-Parsing🤗	0.6B	0.03B	0.6B	82.5	87.9	80.7	83.2	87.3	83.6	76.8	73.6	85.4	87.2	85.5	87.4	81.9	81.7	91.2	87.1	69.9	88.7	78.0	79.8	84.4	74.7
MonkeyOCRv2-B-Parsing🤗	0.7B	0.1B	0.6B	83.3	88.1	81.7	84.2	87.7	84.5	75.2	78.4	86.5	88.6	86.1	87.9	83.2	82.1	90.7	87.2	71.9	87.6	80.1	80.8	83.6	75.3

7. Document understanding performance comparison across different vision foundation models. The evaluation benchmarks are selected following TextMonkey and DT-VQA.

Model	Params	Overall	DocVQA	InfoVQA	DF	KLC	WTQ	ChartQA	DT-VQA	OCRBench
CLIP-B	86M	16.0	20.1	24.2	2.3	13.8	12.8	22.2	22.3	10.6
SigLIP2-B	93M	24.9	27.0	23.5	3.1	16.7	17.4	35.0	41.5	35.1
RADIOv2.5-B	98M	37.5	60.3	31.2	29.9	30.4	29.7	51.1	44.2	23.1
OpenVision-B	87M	44.0	63.3	30.7	19.8	33.1	31.1	58.3	62.6	52.9
DINOv3-B	86M	16.1	26.5	20.8	5.6	13.2	14.0	28.9	15.8	3.9
SAM-B	90M	25.2	37.8	22.2	4.7	17.5	17.6	46.5	33.3	21.9
SAM2-B	69M	22.3	32.5	21.9	2.7	15.8	16.6	40.2	30.3	18.4
oCLIP	24M	12.4	14.8	19.5	1.4	7.4	11.4	17.9	19.2	7.4
DiT	86M	8.9	11.3	20.9	0.9	5.2	9.9	12.0	9.2	1.9
*MonkeyOCRv2-S🤗Link**	28M	55.9	79.3	44.5	65.1	37.6	43.0	62.0	63.1	52.2
*MonkeyOCRv2-B🤗Link**	113M	57.2	79.3	46.3	65.8	38.2	43.2	62.0	64.3	58.1

Expert Model Labeling Toolchain

We adopt a multi-expert labeling pipeline to obtain reliable annotations for documents. The pipeline includes the following steps:

Structure Detection
We use dots.mocr for document structure detection and reading-order prediction. The detected regions, including text blocks, tables, formulas, and other layout elements, are cropped from the original page image for subsequent recognition.
Content Recognition
Each cropped block is independently recognized by three expert models: dots.mocr, PaddleOCR-VL, and Qwen3-VL. These complementary models provide multiple annotations for the same block, reducing reliance on any single OCR system.
Block-Level Agreement Filtering
We compare the recognition results from the three expert models and filter out blocks with low agreement. For retained blocks, we select the prediction that has the highest average agreement with the other two predictions as the final block-level annotation.
Page-Level Quality Control
Pages containing any filtered block are discarded. In addition, we use Qwen3 to verify whether the predicted reading order is reasonable, and Qwen3-VL to check whether document regions are missed during structure detection. This multi-expert agreement strategy reduces model-specific annotation errors and improves the reliability of the generated annotations.

References

dots.mocr: https://github.com/rednote-hilab/dots.mocr
PaddleOCR-VL: https://github.com/PaddlePaddle/PaddleOCR
Qwen3-VL: https://github.com/QwenLM/Qwen3-VL
Qwen3: https://github.com/QwenLM/Qwen3

Copyright

We warmly welcome your feedback, suggestions, and contributions, which are essential to the continued development and improvement of our framework. Note: This model is intended for academic research and non-commercial use only. For any questions, please contact us at xbai@hust.edu.cn or ylliu@hust.edu.cn.

Downloads last month: 16

Safetensors

Model size

0.9B params

Tensor type

BF16

Dataset used to train zenosai/MonkeyOCRv2-B-Parsing

Space using zenosai/MonkeyOCRv2-B-Parsing 1

Collection including zenosai/MonkeyOCRv2-B-Parsing

MonkeyOCRv2

Collection

The collection of the paper `MonkeyOCRv2: A Visual-Text Foundation Model for Document AI`. • 9 items • Updated 1 day ago • 1

Papers for zenosai/MonkeyOCRv2-B-Parsing