R eal B ench: A C hinese Multi-image Understanding Benchmark Close to Real-world Scenarios (2025)

Abstract: While various multimodal multi-image evaluation datasets have been emerged, but these datasets are primarily based on English, and there has yet to be a Chinese multi-image dataset. To fill this gap, we introduce RealBench, the first Chinese multimodal multi-image dataset, which contains 9393 samples and 69910 images. RealBench distinguishes itself by incorporating real user-generated content, ensuring high relevance to real-world applications. Additionally, the dataset covers a wide variety of scenes, image resolutions, and image structures, further increasing the difficulty of multi-image understanding. Ultimately, we conduct a comprehensive evaluation of RealBench using 21 multimodal LLMs of different sizes, including closed-source models that support multi-image inputs as well as open-source visual and video models. The experimental results indicate that even the most powerful closed-source models still face challenges when handling multi-image Chinese scenarios. Moreover, there remains a noticeable performance gap of around 71.8% on average between open-source visual/video models and closed-source models. These results show that RealBench provides an important research foundation for further exploring multi-image understanding capabilities in the Chinese context. Our datasets will be publicly available.

## introduction in recent years, multimodal large language models (mllms) have seen remarkable advancements. both closed-source models like gpt-4v (ope-nai, 2023), gemini @xcite , qwen-vl-max @xcite , and claude-3.5-sonnet @xcite , as well as open-source models such as llava @xcite and minigpt-4 @xcite , and blip @xcite , have exhibited impressive visual-language understanding in single-image tasks such as textvqa @xcite and pope @xcite . however, real-world applications, particularly in social media platforms, predominantly involve multiple images presented simultaneously. for instance, many platforms feature posts containing 4-19 images to comprehensively convey information. this multi-image context poses unique challenges that extend beyond traditional single-image understanding, prompting a shift in research focus towards multi-image multimodal models @xcite to better align with human cognitive processes in processing complex visual information. to systematically evaluate multi-image understanding capabilities, researchers have developed various evaluation benchmarks, which can be categorized into three main approaches. specifically, the majority of existing benchmarks @xcite @xcite @xcite @xcite @xcite @xcite are constructed by first sampling images from existing vision datasets (e.g., iconqa @xcite , nlvr2 @xcite ), then generating corresponding multi-image tasks through either rulebased templates or llm-assisted generation. a small portion of multi-image benchmarks are obtained from the english data on wikipedia with the assistance of closed-source models like gpt-4o. the generated text is then manually checked to ensure its relevance to the associated images, as seen in datasets like mmdu @xcite . additionally, some multi-image benchmarks are entirely manually annotated, such as slidevqa @xcite . despite containing manually annotated qa pairs, slidevqa is limited to slide-based content. previous benchmarks, while cost-effective, may not fully capture the complexity and diversity of real-world multi-image scenarios. 200 400 600 800 1000 width 0 20 40 60 80 frequency 0 1000 2000 3000 height 0 50 100 150 200 250 frequency (b) distribution of width and height. 0 1 2 3 4 5 6 7 8 entropy 0 2000 4000 6000 8000 10000 samples (c) distribution of image entropy. from the chart, we can see that the widths ranges from 200 to 1080, while the heights ranges from 137 to 3500, with a relatively scattered distribution, indicating the diversity of image resolutions. figure (c ) presents the distribution of image entropy, where image entropy is an important metric for measuring the amount of information in an image. higher entropy values suggest that the image contains more details or complex structures, and thus has a greater information content. the entropy value ranges from 0 to 8, and as seen in figure beyond the construction issue, existing datasets are predominantly english-centric, lacking chinese multi-image understanding capabilities despite chinese being widely used on social media. to address this, we introduce realbench, the first chinese multi-image benchmark built from public user-generated content. realbench features three key characteristics (shown in figure 1 ): (1) diversity of scenarios: the dataset encompasses 36 distinct categories drawn from everyday life scenarios, including fields such as travel experiences, food tutorials, home decoration, with a balanced distribution 1 , fully reflecting the diversity of the real world; (2) diversity of resolutions: due to users capturing images with different devices or under various shooting conditions in real-world scenarios, the images present natural variations in quality and resolution (ranging from 200 × 137 to 1080 × 3500 pixels); (3) diversity of image structures: given that the realbench is directly derived from real-world environments, the image structures also exhibit diversity, including images with special fonts, images with mixed tables and text, and images containing nested sub-figures, all of which contribute to the increased complexity of multi-image understanding. realbench contains 9393 data samples and 69910 images, with an average of 7.4 images per sample. to systematically evaluate the model's multi-image understanding capability in real-world chinese scenarios, we carefully design four com1 the 36-categories distribution is shown in appendix a. we conduct a comprehensive evaluation of re-albench using 21 multimodal llms with varying capabilities, including closed-source models that support multi-image inputs, as well as open-source visual and video models. our evaluation reveals significant challenges in multi-image understanding, even for leading models. specifically, we observe that: (1) performance degradation in complex tasks: even top-performing models like gpt-4o and claude-3.5-sonnet achieve only 28.55% and 39.32% accuracy in one-to-many retrieval tasks, respectively; (2) task difficulty progression: model performance consistently decreases as tasks become more complex, with reasoning tasks showing the largest gaps. for example, open-source visual model minicpm-v achieves an average rough-l score of 0.51 in the multi-image extraction task, but this drops to around 0.24 in multi-image reasoning tasks, indicating that complex reasoning tasks place higher demands on the model's capabilities; ## realbench to evaluate the multi-image understanding capability of the model in real-world chinese scenarios, we design four complementary tasks: retrieval, ranking, extraction, and reasoning. these tasks are commonly found in real-world scenarios, closely aligned with practical application needs, and are highly representative. among them, the retrieval task ensures the efficient localization of relevant information, enhancing the speed of information acquisition; the ranking task optimizes the ordering of results to highlight key information; the extraction task accurately distills core content, improving the intuitiveness of the information; and the reasoning task performs in-depth analysis in complex situations to ensure the reliability of the results. through these tasks, we can systematically evalu- task2: multi-image ranking task3: multi-image extraction task4: multi-image reasoning instruction: extract cross-image information to answer question question: which attractions are open all day？ output: extracted answer (observation deck, heipaijiao beach, gaoyangwei beach) instruction: synthesize multi-image information to answer question question: what is the total calories of the three lowest-calorie foods in 711 convenience stores？ output: reasoned answer (534 kcal) instruction: choose the images that are most relevant to the text text: a giant and hearty egg tart weighing 180g. except for the charcoal-grilled original flavor, all the others contain a chunk of mochi inside. overall, it's neither too sweet nor too greasy. output: selected images (1, 2) image 1 image 2 image 3 image 4 instruction: reorder image sequences based on text description text: the style of bronze ware varies in different periods. zenghouyi chime bells, early warring states period. zuocezhegong, early western zhou dynasty. xiangzun, late shang dynasty. options: a: [1, 3, 2]; b: [2, 1, 3]; c: [3, 1, 2]; d: [3, 2, 1] output: ordered sequence (b: [2, 1, 3]) image 1 image 2 image 3 task1: multi-image retrieval basic semantic understanding sequential understanding information integration complex inference ate the model's capabilities from multiple dimensions. below is a detailed description of the task design and data collection process. ## conclusion in this paper, we present the first chinese multiimage dataset, realbench. this dataset is based on real user inputs, making it highly relevant to the real world. realbench covers four distinct tasks: multiimage retrieval, ranking, extraction, and reasoning. we conduct a comprehensive evaluation of real-bench using 21 mllms. the experimental results indicate that even the most powerful closed-source models encounter limitations when handling multiimage scenarios in chinese. additionally, there is a substantial performance gap between open-source models and closed-source models. these findings highlight the importance of realbench and encourage the community to explore multi-image understanding capabilities within the chinese context.