KITAB -Bench: A Comprehensive Multi-Domain Benchmark for A rabic OCR and Document Understanding (2025)

Abstract: With the growing adoption of Retrieval-Augmented Generation (RAG) in document processing, robust text recognition has become increasingly critical for knowledge extraction. While OCR (Optical Character Recognition) for English and other languages benefits from large datasets and well-established benchmarks, Arabic OCR faces unique challenges due to its cursive script, right-to-left text flow, and complex typographic and calligraphic features. We present KITAB-Bench, a comprehensive Arabic OCR benchmark that fills the gaps in current evaluation systems. Our benchmark comprises 8,809 samples across 9 major domains and 36 subdomains, encompassing diverse document types including handwritten text, structured tables, and specialized coverage of 21 chart types for business intelligence. Our findings show that modern vision language models (such as GPT-4o, Gemini, and Qwen) outperform traditional OCR approaches (such as EasyOCR, PaddleOCR, and Surya) by an average of 60% in the character error rate (CER). Furthermore, we highlight significant limitations of current Arabic OCR models, particularly in PDF-to-Markdown conversion, where the best model Gemini-2.0-Flash achieves only 65% accuracy. This underscores the challenges of accurately recognizing Arabic text, including issues with complex fonts, numeral recognition errors, word elongation, and table structure detection. This work establishes a rigorous evaluation framework that can drive improvements in Arabic document analysis methods and bridge the performance gap with English OCR technologies.

## introduction with the upsurge in adoption of retrieval-augmented generation (rag) based systems for document processing, the quality of document ingestion pipelines has become increasingly critical. optical character recognition (ocr) plays a crucial role in this pipeline, enabling the conversion of physical documents into machine-readable text and databases for enabling effective knowledge retrieval. although significant progress has been made in the multilingual ocr @xcite @xcite , with comprehensive datasets like publaynet @xcite , docbank @xcite , m6doc @xcite , and doclaynet @xcite , arabic ocr continues to lag behind. this gap is largely due to the unique challenges of the arabic script, including its cursive nature, complex typography, and right-to-left text orientation. ## related work the development of robust optical character recognition (ocr) systems has been extensively studied across document layout analysis @xcite @xcite @xcite , table detection @xcite @xcite @xcite , and document understanding @xcite @xcite . while english ocr benefits from rich table recognition, image to text, diagram to json, and layout detection) showing successful and failed cases for different models including ground truth, easyocr, gpt-4, qwen, surya, tesseract, yolo, and detr on arabic document benchmark data. domain total samples pdf to markdown 33 layout 2,100 line detection 378 line recognition 378 table recognition 456 image to text 3,760 charts to dataframe 576 diagram to json 226 vqa 902 total 8,809 ## kitab-bench our methodology offers a novel approach to benchmarking arabic ocr systems via a comprehensive data collection strategy and a systematic evaluation framework. we gather curated samples from existing arabic document datasets, manually collected and annotated pdfs, and employ a five-phase llmassisted human-in-the-loop pipeline (figure 4 ) to generate diverse supplementary content. our evaluation framework spans nine specialized tasks, enabling thorough assessment of ocr performance across various document processing challenges and providing a robust benchmark for arabic document understanding tasks. ## experiments our experimental evaluation comprehensively assesses the capabilities of current ocr systems and state-of-the-art vision-language models (vlms) across different arabic and multilingual document understanding tasks. figure 2 illustrates the nine distinct tasks in our evaluation framework. we evaluate three categories of systems: vlms, traditional ocr systems, and specialized document processing tools. for vlms, we include both closed-source models like gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18 @xcite , and gemini-2.0-flash @xcite , as well as open-source alternatives such as qwen2-vl-7b @xcite , qwen2.5-vl-7b @xcite , and the ain-7b @xcite . traditional ocr approaches in our evaluation include surya @xcite , tesseract @xcite , easyocr (jaidedai, 2020), and paddleocr @xcite . for specialized document processing tasks, we employ systems like docling @xcite , and marker @xcite . layout detection capabilities are evaluated using methods implemented in surya-layout @xcite , yolo-doclaynet @xcite from mineru @xcite , and rt-detr @xcite based method in docling @xcite . ## results and discussion in this section, we present a comprehensive evaluation of different models across different tasks of our framework. the results provide a clear distinction between the performance of closed-source models, open-source models, and framework-based solutions, revealing both their strengths and limitations. we observe very clear performance gap between closed and open-source solutions. while closedsource models like gemini-2.0-flash consistently outperform other models almost all the tasks. table [7] presents model performance across different chart and diagram understanding tasks, evaluated using scrm and chartex (for charts), and vqa-based accuracy metrics. among closedsource models, gemini-2.0 achieves the highest performance on chart understanding metrics, scoring 71.4% on scrm and 56.28% on chartex. the performance gap between gemini-2.0 and gpt-4o is particularly pronounced in chartex evaluation (10.33%) compared to scrm (2.8%). open-source models shows a significant limitation in complex chart understanding. while their scrm scores remain competitive, both qwen variants score below 23% on chartex evaluation. the visual questionanswering results reveal an important exception to the general closed-source advantage. ain achieves 87% on patdvqa, surpassing gemini-2.0 by 11.5%. ain also shows competitive performance on mtvqa (31.50%), which is similar to gpt-4o and 4% better than gpt-4o-mini. this shows that open-source models can be competitive with closed-source alternatives. ## conclusion we introduce a comprehensive benchmark for arabic ocr that fills the gap in standardized evaluation frameworks for arabic document processing. our dataset of 8,809 samples across nine major domains is the most diverse collection assembled for ocr evaluation, incorporating handwritten, scanned, synthetic, and scene text, as well as complex tables, charts, and end-to-end pdf-tomarkdown. this framework extends beyond simple text recognition to include structural document analysis and enables systematic assessment of ocr performance across various fonts, styles, and layouts.