TIU -Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding (2025)

Abstract: Text-rich images are ubiquitous in real-world applications, serving as a critical medium for conveying complex information and facilitating accessibility.Despite recent advances driven by Multimodal Large Language Models (MLLMs), existing benchmarks suffer from limited scale, fragmented scenarios, and evaluation protocols that fail to fully capture holistic image understanding.To address these gaps, we present TIU-Bench, a large-scale, multilingual benchmark comprising over 100,000 full-image annotations and 22,000 rigorously validated question-answer (QA) pairs that span 18 subtasks across diverse real-world scenarios.TIU-Bench introduces a novel full-image structured output format that jointly models geometric, textual, and relational information, enabling fine-grained evaluation of perception and reasoning capabilities. Furthermore, we propose a two-stage understanding framework named T2TIU, which first generates a structured representation of the entire image and subsequently conducts reasoning on this representation to address complex visual-textual queries.Extensive experiments on 10 state-of-the-art generative models highlight the challenges and opportunities in advancing text-rich image understanding.Our benchmark and framework provide a comprehensive platform for developing and evaluating next-generation multimodal AI systems.

## introduction text-rich images play a pivotal role in real-world scenarios by efficiently conveying complex information and improving accessibility @xcite . accurate interpretation of such images is essential for automating information extraction, advancing ai systems, and optimizing user interactions. to formalize this research domain, we define text-rich image understanding (tiu) as consist-* corresponding author. ing of two core capabilities: perception and understanding. the perception dimension encompasses visual recognition tasks such as text detection @xcite ), text recognition @xcite , formula recognition @xcite , and document layout analysis @xcite . in contrast, the understanding dimension involves semantic reasoning for downstream applications such as key information extraction and document-based visual question answering (e.g., docvqa @xcite , chartqa @xcite , and textvqa @xcite ). recently, mllms have been proposed, which integrate large language models (llms) with visual encoders to jointly process visual tokens and linguistic elements through unified attention mechanisms, enabling end-to-end sequence modeling. within the tiu domain, mllms have demonstrated impressive results in both perception and understanding. nevertheless, despite recent advances, there are still two key challenges in current tiu research paradigms. dataset limitations. tiu tasks currently face challenges related to data diversity, scale, and quality. existing datasets such as docvqa and chartqa focus on isolated scenarios (e.g., documents, tables, or charts), and their fragmented objectives and scenario-specific designs hinder the vi su a l t e x t u n de rstandin g relat io n e x t r a c ti o n g lo b al recogn it io n reading order prediction visual relationship extraction text-rich image understanding comprehensive evaluation of perceptual and interpretative capabilities. recent benchmarks such as ocrbench and its v2 version assess line-level text recognition in mllms but overlook holistic image understanding challenges and suffer from limited language coverage and limited scale. figure 1 presents a quantitative comparison, illustrating the significantly larger scale and greater diversity of our benchmark compared to existing ones. ## task formulation we formally define the task of text-rich image understanding (tiu), which aims to analyze images containing rich textual information comprehensively. this task is composed of two interrelated sub-tasks: full-image parsing and visual question answering (vqa). full-image parsing. given an input image i that contains dense textual content, the objective of full-image parsing is to extract a structured representation s of the image. specifically, s includes a set of text paragraphs {p 1 , p 2 , . . . , p m }, each associated with spatial coordinates and orientation angles. formally, the output can be expressed as: ## dataset construction in this section, we present the construction details of tiu-bench. the process of creating datasets in different domains can be divided into three stages: (1) data selection and preprocessing, (2) qa generation and refinement, (3) data quality check. the overview of this process is shown in figure 3 . ## framework in this section, we present our framework, which consists of two stages: (1) holistic image parsing, and (2) the generation of answers based on the parsing results, utilizing a foundational generative model. ## evaluation metrics to achieve a fine-grained evaluation of holistic image understanding, our benchmark adopts a comprehensive structured output format that simultaneously captures: (1) the geometric locations of all text segments via bounding box coordinates, (2) recognition results including confidence scores, and (3) inter-paragraph relationships modeled as a spatial semantic graph. this structured representation supports our multi-level evaluation protocol: global evaluation: missing text detection to identify unrecognized paragraphs. in this part, we generate a structured output for the entire image and then compare it against the globally annotated ocr in the dataset to determine the proportion of missed paragraphs. this metric is referred to as ocr-complete. local evaluation: (a) visual alignment is measured by the intersection over union (iou) between predicted and ground truth bounding boxes to assess coordinate accuracy. we compute recall, precision, and f1 scores for both paragraph coordinates and recognized text, denoted as para-recall, para-precision, and para-f1, respectively; (b) text accuracy is quantified at the character level using the levenshtein distance to measure consistencydenoted as avg-edit-distence. ## experiments 6.1 experimental baselines and settings baselines we evaluate 2 closed-source models and 8 open-source models. specifically, we select 2 popular closed-source multimodal large models: gpt-4o @xcite , gpt-4o-mini @xcite . for the open-source models, we choose 9 multimodal large models: qwen2.5-vl-3b-instruct @xcite , qwen2.5-vl-7b-instruct, qwen2.5-vl-32b-instruct, qwen2.5-vl-72b-instruct @xcite , llava-next-8b @xcite , llava-ov-7b @xcite , internvl2-8b @xcite and internvl2-26b @xcite . experiment details for open-source large models with fewer than 40b parameters, we adopt supervised fine-tuning. specifically, we split the full-image parsing and visual question answering datasets into training, validation, and test sets with an 8:1:1 ratio. the models are trained on the training set, the best model parameters are selected based on validation performance, and a final evaluation is conducted on the test set. for larger models such as qwen-2.5-vl-72b, closed-source gpt-4o, and gpt-4o-mini, we em- ## related work to evaluate lmms, developing a comprehensive benchmark is essential. previous effort has focused on creating scenario-specific benchmarks to assess lmms in particular contexts. for example, docvqa @xcite is designed to evaluate the document comprehension abilities of lmms, while chartqa @xcite is tailored to chart interpretation skills. similarly, infographics vqa @xcite is dedicated to assessing the understanding of infographic images. additionally, textvqa @xcite aims to evaluate text comprehension in real-world scenes. to further investigate the robustness of the model, some methods expand the scope of evaluation scenarios. ocrbench @xcite introduces a holistic evaluation framework that covers five core ocr tasks. contextual @xcite is developed with contextsensitive instructions. seed-bench-2-plus @xcite encompasses a wide spectrum of textrich images from various sources, including web content, maps, and charts. to provide a more thor-ough assessment, some benchmarks design multiple evaluation tasks within a specific scenario. tablevqa-bench @xcite first focuses on vqa tasks in the table domain. mmtab @xcite and comtqa @xcite then extend the task scope, including table detection, structure recognition, and table querying. moreover, charty @xcite , chartx @xcite , and mmc @xcite evaluate lmms in chart understanding through tasks such as chart information extraction and reasoning. in this work, we focus on establishing a new benchmark called tiubechmarks, which contains more tasks than previous benchmarks and provides a systematic evaluation framework to reveal the limitations of lmms in diverse text-rich environments. 8 conclusion we present tiu-bench, a large-scale multilingual benchmark addressing key limitations of existing datasets by providing over 100,000 full-image annotations and 22,000 validated qa pairs across 18 diverse subtasks. our novel structured output format enables fine-grained evaluation of perception and reasoning in text-rich images. alongside, we propose a two-stage framework that generates structured image representations and performs reasoning to answer complex queries. experiments on 10 state-of-the-art models reveal significant challenges, underscoring tiu-bench's value as a comprehensive platform for advancing multimodal ai research. ## limitation while tiu-bench significantly advances the evaluation of text-rich image understanding by providing large-scale, multilingual, and diverse annotations with a novel structured output format, several limitations remain. first, despite the extensive coverage of 18 subtasks, the benchmark may not fully encompass all possible real-world scenarios, especially those involving highly specialized or domainspecific text-image interactions. second, the twostage understanding framework, although effective, relies on accurate, structured representation generation as a prerequisite, which may propagate errors and limit end-to-end performance.