Muffin or C hihuahua? Challenging Multimodal Large Language Models with Multipanel VQA (2024)

Abstract: Multipanel images, commonly seen as web screenshots, posters, etc., pervade our daily lives. These images, characterized by their composition of multiple subfigures in distinct layouts, effectively convey information to people. Toward building advanced multimodal AI applications, such as agents that understand complex scenes and navigate through webpages, the skill of multipanel visual reasoning is essential, and a comprehensive evaluation of models in this regard is important. Therefore, we introduce Multipanel Visual Question Answering (MultipanelVQA), a novel benchmark comprising 6,600 triplets of questions, answers, and multipanel images that specifically challenge models in comprehending multipanel images. Our evaluation shows that questions in the MultipanelVQA benchmark pose significant challenges to the state-of-the-art Multimodal Large Language Models (MLLMs) tested, even though humans can attain approximately 99% accuracy on these questions. Distinctively, the MultipanelVQA benchmark features synthetically generated multipanel images specifically crafted to isolate and assess the impact of various factors, such as the layout, on MLLMs’ multipanel image comprehension abilities. As a result, in addition to benchmarking the capabilities of MLLMs in understanding multipanel images, we analyze various factors of the multipanel image that affect MLLMs’ performance with synthetic data and offer insights for enhancement.

## introduction multimodal large language models (mllms) have become a significant leap in the integration of visual and textual data processing, enabling more nuanced understanding and generation of content that blends both visual and linguistic elements. being trained on extensive data, advanced no grid style layouts splashed style layouts does this image show a muffin? single-panel image multipanel image (poster) does the middle subfigure in the bottom row show a muffin? outputs of gpt-4v gpt-4v: it appears to be a dog. gpt-4v: it appears to be a muffin. the second image in the top row is showing a muffin. is this image showing a dog or a muffin? is this image showing a dog or a muffin? is the second image in the top row showing a dog or a muffin? pareidolia: muffins on the table? mllms @xcite @xcite @xcite have shown remarkable proficiency in various tasks (e.g., image captioning and visual question answering) that require natural language understanding, visual-language grounding, visual reasoning, etc. as mllms become more competent, there is a trend of establishing increasingly challenging benchmarks that are often arduous for average humans to achieve @xcite . however, this raises a pertinent question: have mllms advanced to the stage where elementary benchmarks easily handled by average humans pose little challenge to them? to answer this question, we target multipanel images, each involving a series of subfigures. these subfigures are presented together in certain layouts, such as web screenshots capturing multiple thumbnail images and posters utilizing multipanel formats to present a cohesive narrative or argument. we observe that while humans typically find interpreting multipanel images to be a straightforward task, mllms struggle with this challenge when presented with the entire multipanel image as input, as shown in figure 1 . this study aims to holistically evaluate mllms in understanding multipanel images. we introduce the multipanelvqa benchmark with 6,600 triplets of multipanel images, questions and answers, challenging models to answer each question based on the multipanel image. there are three questions with distinct types for each multipanel image: identifying common or unique contents across subfigures, pinpointing content in specific subfigures through positional descriptions, and locating subfigures via visual grounding in a multi-choice format. especially, the first type of question mainly tests the mllms' ability to reason about contents and the other two question types also assess the mllms' understanding of multipanel image layouts in addition to the content reasoning ability. uniquely, the multipanel images in the multi-panelvqa benchmark features a diverse mix of real-world web screenshots, posters and synthetic multipanel images, categorized into real-world data and synthetic data subsets. unlike the real-world data that requires human annotation, the synthetic multipanel images are automatically generated by scripts with subfigures from two existed datasets. the script ensures the generated synthetic multipanel images have even distribution of various attributes such as the number of subfigures, their sizes, and the complexity of layouts, etc. as a result, based on the synthetic data, we are able to precisely isolate and pinpoint the impact of their attributes on the performance of mllms. we then benchmark popular open-sourced and proprietary mllms on the multipanelvqa benchmark and conduct thorough error analysis with the help of the synthetic data, which delves into the reasons behind mllms' difficulties in interpreting multipanel images. as a result, our main findings are 1) mllms are susceptible to content interference caused by the occurrence of multiple subfigures within the multipanel image. 2) the layout for subfigures has an impact on the mllms' performance on multipanel images. mllms tend to be more successful in understanding multipanel images with layouts with fewer subfigures and larger subfigure sizes. 3) adding sequential numbers for subfigures as visual prompt can benefit some mllms that are sensitive to embedded texts in the input multipanel images. last but not least, we explore how adding sequential numbers to subfigure captions in multipanel images, akin to the set-of-mark visual prompting method @xcite , improves mllms' understanding of these images. we test mllms on multipanel images with and without sequential number captions for each subfigure. as a result, we observed that only gpt-4v (openai, 2023b) and minigpt-v2 @xcite show a notable improvement when the sequential number is not only embedded in the image but also explicitly mentioned in the question. in conclusion, the contributions of this study are listed as follows: ## related work multimodal large language models the development of multimodal large language models (mllms) has been propelled by advances in largelanguage models (llms) @xcite and vision-and-language learning @xcite , merging visual comprehension with llms for multi-modal tasks in a zero-shot manner @xcite @xcite . instruction tuning, using visual instruction data derived from open-source datasets and pre-trained llms, enhances mllms' zero-shot performance on complex tasks @xcite ; dai question 1 (q1): do all the subfigures contain the same food item? question 2 (q2): what food is shown in the top right subfigure? question 3 (q3): which subfigure contains cake? a) second in the middle row b) first in the bottom row c) top right d) bottom left synthetic multipanel images … … real-world multipanel images multipanelvqa questions for one multipanel image (example) background added visual text hint added et al., 2023; ye et al., 2023a ). further advancements include grounding and multilingual training to expand mllms' capabilities @xcite @xcite . evaluations for mllms with the advancement of mllms, there's a growing need for comprehensive multi-modal benchmarks to assess their capabilities. traditional tasks like image captioning @xcite and vqa @xcite @xcite , along with text recognition and knowledge-based reasoning @xcite @xcite , have been key in evaluating mllms. newer benchmarks aim to assess models more holistically @xcite @xcite . recently, more holistic and comprehensive benchmarks have been proposed, which evaluate models' comprehensive capabilities from multiple perspectives @xcite @xcite . unlike former evaluation benchmarks, we propose the mul-tipanelvqa benchmark that not only identifies a distinguished practical challenge in real life, multipanel image understanding, but also statistically analyzes the mllms' capability through the synthetic data. synthetic data synthetic data, recognized for its scalability, diversity, cost-effective annotations, etc, has been widely explored for enhancing model training, especially in vision-related tasks like semantic segmentation, object detection, and image classification @xcite @xcite . additionally, synthetic data's role extends beyond training to include model performance evaluation and analysis. kortylewski et al. (2019) use synthetic faces to analyze neural network generalization across different poses, finding deeper architectures perform better. van breugel et al. (2023) propose the 3s testing framework to generate synthetic test sets that evaluate models under distributional shifts. in this work, we introduce the multipanelvqa benchmark, enriched with synthetic data to conduct error analysis, exploring the factors influencing the performance of mllms on multipanel image understanding. ## experiments we first evaluate eight popular multimodal large language models (mllms) on multipanelvqa. then, based on the evaluation results, we conduct a thorough error analysis. ## discussion and conclusion in this study, we introduce the multipanelvqa benchmark, designed to evaluate the capability of multimodal large language models (mllms) in interpreting multipanel images. this benchmark, comprising both real-world and synthetic data, enables a detailed analysis of mllms on their multipanel image understanding abilities. our results highlight a significant performance gap between mllms and humans, especially since humans achieve nearly perfect scores in this benchmark. this gap highlights the current limitations of mllms in interpreting highly structured visual information and pinpoints the specific need where further model training and refinement are necessary. moreover, by analyzing mllms' abilities to effectively interpret multipanel images, we believe our benchmark can facilitate the development of specialized algorithms, such as severing as a tool to select strong mllms in tasks related to graphic user interface (gui) understanding @xcite . also, as multipan-elvqa identifies the key factors in multipanel images that affect model performance, it can inspire and guide related applications, for example, presenting lengthy sequences of images as subfigures in multipanel images @xcite . last but not least, the synthetic data of multipan-elvqa helps isolate specific performance factors and ensures that the test images were not part of the models' training datasets. this is essential for large-scale mllms with undisclosed training data. the creation method for these synthetic images is replicable, enabling ongoing generation of new test images and potentially aiding broader ai evaluation efforts.