MIB ench: Evaluating Multimodal Large Language Models over Multiple Images (2024)

Abstract: Built on the power of LLMs, numerous multimodal large language models (MLLMs) have recently achieved remarkable performance on various vision-language tasks. However, most existing MLLMs and benchmarks primarily focus on single-image input scenarios, leaving the performance of MLLMs when handling realistic multiple images underexplored. Although a few benchmarks consider multiple images, their evaluation dimensions and samples are very limited. In this paper, we propose a new benchmark MIBench, to comprehensively evaluate fine-grained abilities of MLLMs in multi-image scenarios. Specifically, MIBench categorizes the multi-image abilities into three scenarios: multi-image instruction (MII), multimodal knowledge-seeking (MKS) and multimodal in-context learning (MIC), and constructs 13 tasks with a total of 13K annotated samples. During data construction, for MII and MKS, we extract correct options from manual annotations and create challenging distractors to obtain multiple-choice questions. For MIC, to enable an in-depth evaluation, we set four sub-tasks and transform the original datasets into in-context learning formats. We evaluate several open-source and closed-source MLLMs on the proposed MIBench. The results reveal that although current models excel in single-image tasks, they exhibit significant shortcomings when faced with multi-image inputs, such as limited fine-grained perception, multi-image reasoning and in-context learning abilities. The annotated data of MIBench is available at https://huggingface.co/datasets/StarBottle/MIBench.

## introduction recently, leveraging the powerful comprehension and reasoning abilities of llms, many mllms such as llava-1.5 (liu et al., 2024) † corresponding authors. and mplug-owl2 @xcite have demonstrated outstanding performance across various vision-language tasks (e.g. image captioning, vqa and visual grounding). concurrently, numerous benchmarks like mme @xcite , mm-bench @xcite and seed-bench @xcite ) have been proposed to evaluate the abilities of mllms in terms of different perspectives such as recognition, localization and reasoning. however, most existing mllms focus on singleimage scenarios. accordingly, previous benchmarks primarily evaluate mllms based on singleimage inputs. in contrast, real-world multimedia information, such as web pages and social media, generally contains multiple images and corresponding text in interleaved forms. therefore, multi-image scenarios have greater practical value than single-image scenarios, which makes it worth exploring whether existing single-image mllms possess emergent abilities for multi-image inputs. moreover, some methods like sparkles @xcite and mantis @xcite explore multi-image scenarios but have not comprehensively evaluated the models' multi-image abilities. as shown in ## conclusion while mllms have shown strong performance in various vision-language tasks, their abilities with multi-image inputs remain underexplored. to address this, we introduce mibench in this paper, a benchmark that evaluates mllms across three multi-image scenarios: multi-image instruction, multimodal knowledge-seeking and multimodal incontext learning, covering 13 tasks with 13k annotated samples. the evaluation results reveal that existing models, despite excelling in single-image tasks, face significant challenges with multi-image inputs. the annotated data is publicly available to facilitate further research. we hope this work can spur progress in improving the multi-image abilities of mllms.