Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review (2025)

Abstract: The recent emergence of Multi-modal Large Language Models (MLLMs) has introduced a new dimension to the Text-rich Image Understanding (TIU) field, with models demonstrating impressive and inspiring performance. However, their rapid evolution and widespread adoption have made it increasingly challenging to keep up with the latest advancements. To address this, we present a systematic and comprehensive survey to facilitate further research on TIU MLLMs. Initially, we outline the timeline, architecture, and pipeline of nearly all TIU MLLMs. Then, we review the performance of selected models on mainstream benchmarks. Finally, we explore promising directions, challenges, and limitations within the field.

## introduction text-rich images play a pivotal role in real-world scenarios by efficiently conveying complex information and improving accessibility @xcite . accurately interpreting these images is essential for automating information extraction, advancing ai systems, and optimizing user interactions. to formalize this research domain, we term it text-rich image understanding (tiu), which encompasses two core capabilities: perception and understanding. the perception dimension focuses on visual recognition tasks, such as text detection @xcite ), text recognition @xcite , formula recognition @xcite , and document layout analysis @xcite . the understanding dimension, conversely, requires semantic reasoning for applications like key information extraction and document-based visual question answering (e.g., docvqa @xcite , chartqa @xcite , and textvqa @xcite ). historically, perception and understanding tasks were handled separately through specialized models or multi-stage pipelines. recent advances in vision-language models have unified these tasks within visual question answering (vqa) paradigms, driving research towards the development of end-to-end universal models. scene document chart table gui visual encoder ocr-free encoder ocr-based encoder mixture of encoders text encoder modality connector embedding token compression token reducer llm decoder llama series qwen series vicuna series internlm series q modality alignment instruction alignment preference alignment training pipeline …… figure 2: the general model architecture of mllms and the implementation choices for each component. this paradigm evolution addresses two critical limitations of earlier methods. first, the emergent mllm framework eliminates modality-specific inductive biases through homogeneous token representation, enabling seamless multi-task integration. second, the linguistic priors encoded in llms empower unprecedented zero-shot generalization and allow direct application to diverse tasks without task-specific tuning. although these mllms present impressive and inspiring results, their rapid evolution and broad adoption have made tracking cutting-edge advancements increasingly challenging. therefore, a systematic review that is tailored for documents to summarize and analyse these methods is in demand. however, existing surveys on text-rich image understanding often exhibit narrow focus: they either analyze domain-specific scenarios (e.g., tables and figures @xcite , charts @xcite , forms @xcite ) or emphasize unified deep learning frameworks @xcite . our systematic survey addresses the gap by providing the first comprehensive analysis of nearly all tiu mllms in four dimensions: model architectures (section 2), training pipeline (section 3), datasets and benchmarks (section 4), challenges and trends (section 5). this holds both academic and practical significance for advancing the field @xcite . ## model architecture tiu mllm methods typically leverage pre-trained general visual foundation models to extract robust visual features or employ ocr engines to capture text and layout information from images. a modality connector is then used to align these visual features with the semantic space of the language features from the llm. finally, the combined visuallanguage features are fed into the llm, which utilizes its powerful comprehension capabilities for semantic reasoning to generate the final answer. as illustrated in figure 2 , the framework of tiu mllms can be abstracted into three core components: visual encoder, modality connector, and llm decoder. ## training pipeline the training pipeline of mllm for tiu can be delineated into three main stages: 1) modality alignment (ma); 2) instruction alignment (ia); and 3) preference alignment (pa). ## datasets and benchmarks the rapid advancements in tiu tasks have been fundamentally driven by the proliferation of specialized datasets and standardized benchmarks. as illustrated in specifically, some datasets are derived by converting training data from traditional tasks @xcite into visual question answering (vqa) formats, such as text detection, text spotting, table recognition, and etc.. these datasets are typically utilized for modality alignment in the first stage of training, enabling models to bridge the gap between textual and visual information effectively. other datasets are specif-ically designed in vqa formats for certain scenarios, such as docvqa @xcite , infovqa @xcite , chartqa @xcite , and textvqa @xcite . these datasets have played a pivotal role in advancing the field of tiu by providing structured and domain-specific challenges. their introduction has significantly accelerated progress in tasks like document understanding, chart interpretation, and natural scene text comprehension. consequently, published papers frequently report these metrics, as they not only contribute to instruction alignment in the second stage of training but also serve as essential benchmarks for evaluating model performance. in addition to training datasets, there is a distinct category of datasets that are exclusively designed for evaluating specific capabilities of mllms. examples include tablevqa-bench @xcite , chartbench @xcite , and mmlongbench-doc @xcite . these datasets are tailored to assess advanced functionalities such as long-context understanding, cross-modal reasoning, and domain-specific comprehension. by providing targeted evaluation frameworks, they enable researchers to identify strengths and weaknesses in mllms, driving further innovation and refinement in the field. ## limitations this paper provides a systematic review of multimodal large language models (mllms) in the field of text-rich image understanding (tiu). while the research team has conducted comprehensive retrieval and integration of core literature prior to the submission deadline, certain minor studies may still remain uncovered. it should be particularly noted that due to publisher formatting requirements, the exposition of existing technical approaches and benchmark datasets in this work maintains essential conciseness. for complete algorithmic implementation details and experimental parameter configurations, researchers are strongly recommended to consult the original publications.