Benchmarking Table Extraction: Multimodal LLM s vs Traditional OCR (2025)

Abstract: This paper compares two approaches for table extraction from images: deep learning computer vision and Multimodal Large Language Models (MLLMs). Computer vision models for table extraction, such as the Table Transformer model (TATR), have enhanced the extraction of complex table structural layouts by leveraging deep learning for precise structural recognition combined with traditional Optical Character Recognition (OCR). Conversely, MLLMs, which process both text and image inputs, present a novel approach by potentially bypassing the limitations of TATR plus OCR methods altogether. Models such as GPT-4o, Phi-3 Vision, and Granite Vision 3.2 demonstrate the potential of MLLMs to analyze and interpret table images directly, offering enhanced accuracy and robust extraction capabilities. A state-of-the-art metric like Grid Table Similarity (GriTS) evaluated these methodologies, providing nuanced insights into structural and text content effectiveness. Utilizing the PubTables-1M dataset, a comprehensive and widely used benchmark in the field, this study highlights the strengths and limitations of each approach, setting the stage for future innovations in table extraction technologies. Deep learning computer vision techniques still have a slight edge when extracting table structural layout, but in terms of text cell content, MLLMs are far better.

## introduction with the increasing volume of digital documents, such as records, manuals, and scientific papers, processing and transforming them into representations that allow proper extraction of information has become highly challenging @xcite . many of these documents contain tables, as they help represent data in an organized, readable, and straightforward manner. however, automatically identifying and extracting structural layout and content information becomes more complex, which can be crucial in scientific and business applications @xcite . this work explores and compares two strategies for extracting tables contained in images in a structured manner: (a) a deep learning computer vision model, table transformer (tatr), combined with optical character recognition (ocr) and (b) the novel multimodal large language models (mllms). these approaches were evaluated using metrics that capture how well they extract the tables' structural and text content. the remainder of this paper is structured as follows. section 2 provides an overview of existing and related work. section 3 outlines the followed methodology and experiment details and section 4 discusses the obtained results. finally, conclusions and limitations are drawn in sections 5 and 6. ## related work this section reviews key literature on optical character recognition (ocr) and table extraction, and llms. ## methodology this section first explains the methodology used to retrieve table outputs from large language models. then, in the second subsection, the evaluation metrics used are briefly detailed. finally, in the third subsection, the specific details of the experiment's execution are provided. ## results the results of the evaluation comparison, depicted in figure 3 , provide a comprehensive overview of the performance of the five models. these models are tatr-ocr, granite, phi-3-vision, gpt-4omini, and the standout performer gpt-4o. the analysis begins in figure 3(a) with the grits f1 score, where gpt-4o stands out with an impressive 89.6%, closely followed by tatr-ocr at 87.8%. gpt-4o-mini and granite yield more moderate scores at 74.6% and 76.3%, respectively, while phi-3-vision records a relatively lower score of 65.2%. the structural layout f1 score in figure 3 (a) further differentiates the models, with tatr-ocr achieving a remarkable 98.2% and gpt-4o also performing strongly at 94.9%. granite and gpt-4o-mini demonstrate similar mid-tier performance levels at 81.5% and 81.9%, respectively, and phi-3-vision lay behind at 70.8%. in terms of table shape accuracy, the pattern is similar: gpt-4o and tatr-ocr excel with accuracies of 95.2% and 98.4%, respectively, while granite and gpt-4o-mini maintain comparable scores of 82.3% and 82.6%. phi-3-vision again underperforms with only 71.3% accuracy. beyond these overall metrics, the analysis delves into structural layout errors in figure 3(b) . these errors refer to discrepancies in the arrangement of elements within the document, examining both missing and extra rows and columns. gpt-4o performs the best, with only 0.3% missing rows. tatr-ocr, phi-3-vision, and gpt-4o-mini show moderate performance, missing around 2.7-3.2% of rows. granite is the least reliable, missing 7.9% of table rows, which could lead to major data loss. for the extra rows, tatr-ocr and granite are tied as best performance with a percentage of 0.7%. followed by phi-3-vision with almost 2%, the worst performers are gpt-4o with 5.2% and gpt-4omini with 8.2%. looking at the missing columns gpt-4o and tatr-ocr both have a 0.3%. phi-3-vision and gpt-4o-mini are in mid-performance 1-1.5%, respectively. the worst is granite with 4.2% missing columns. regarding the extra columns percentage granite is again the worst with almost 15%, phi-3-vision and gpt-4o-mini are in mid-performance 4.7 -5.4%. tatr-ocr is close to gpt-4o with 0%. overall, the analysis of structural layout errors highlights significant differences in performance among the models. gpt-4o consistently demonstrates the best overall accuracy, only worse by a high percentage of extra rows. tatr-ocr also performs well, particularly in handling columns. phi-3-vision and gpt-4o-mini exhibit moderate performance, showing errors in missing and extra elements. however, granite proves to be the least reliable, with the highest percentage of missing rows and columns and a substantial number of extra columns. these discrepancies in structural layout accuracy can significantly impact data integrity, reinforcing the importance of selecting the most precise model for document processing tasks. the analysis of string matching thresholds in figure 3 (c) reveals distinct performance variations among the five models. these thresholds are crucial as they determine the level of similarity required for a match, with a higher threshold indicating a stricter matching condition. gpt-4o consistently achieves the highest scores across all thresholds, demonstrating superior accuracy in string matching. tatr-ocr follows closely behind, maintaining a competitive performance. phi-3-vision and gpt-4o-mini exhibit moderate results, while granite consistently underperforms, scoring the lowest in most cases. as the matching threshold increases from 75 to 100, all models experience a decline in performance, indicating that stricter criteria lead to greater difficulty in identifying matches. despite this trend, gpt-4o remains the most reliable, maintaining high accuracy even at the strictest threshold. these findings highlight the varying ro-bustness of different models and emphasize the importance of selecting the most suitable one based on the required level of precision. in conclusion, the analysis underscores the importance of selecting the most suitable model for the task at hand. gpt-4o consistently outperforms the other models across all key metrics, exhibiting superior structural accuracy and text recognition capabilities. gpt-4o-mini, on the other hand, emerges as a strong alternative, showcasing its potential with slightly lower performance. tatr-ocr and phi-3-vision deliver similar midrange results, while granite shows the least favorable performance with significant structural errors and lower text accuracy. these findings suggest that gpt-4o and gpt-4o-mini are the most reliable choices for table extraction tasks, whereas the other models may benefit from additional postprocessing steps to improve their accuracy. ## conclusions tatr outperforms the llms when only the table structural layout is considered based on the metric results. if the exact content of the cell is not a top priority, using tatr with ocr is a viable choice, keeping in mind that the ocr may not correctly identify all the cells. however, when the text content of the cells is considered, the llms perform better than tatr with ocr. gpt-4o is by far the best among the llms tested, but is also the largest and most expensive model. smaller models can be a good option depending on the specific use case and the volume of data to be processed.