CODIS : Benchmarking Context-dependent Visual Comprehension for Multimodal Large Language Models (2024)

Abstract: Multimodal large language models (MLLMs) have demonstrated promising results in a variety of tasks that combine vision and language. As these models become more integral to research and applications, conducting comprehensive evaluations of their capabilities has grown increasingly important. However, most existing benchmarks fail to consider that, in certain situations, images need to be interpreted within a broader context. In this work, we introduce a new benchmark, named as CODIS, designed to assess the ability of models to use context provided in free-form text to enhance visual comprehension. Our findings indicate that MLLMs consistently fall short of human performance on this benchmark. Further analysis confirms that these models struggle to effectively extract and utilize contextual information to improve their understanding of images. This underscores the pressing need to enhance the ability of MLLMs to comprehend visuals in a context-dependent manner.

## introduction recent years have witnessed a rapid advancement in multimodal large language models @xcite @xcite . they have achieved remarkable results on various downstream tasks, such as image captioning @xcite @xcite , visual question answering @xcite @xcite figure 1 : interpretation of images can be significantly influenced by contextual information. in this instance, the determination of whether the photographer was ascending or descending a staircase remains ambiguous without supplementary context (a). however, when additional information is provided, indicating the position of the greenery relative to the observer, the direction of movement of the observer becomes clear (b). in the responses, words originating from the image and the two pieces of context are highlighted in purple, blue, and green, respectively. 2015 ) and visual reasoning @xcite @xcite . as the performance of mllms continues to improve, the comprehensive assessment of their capabilities becomes increasingly important. previous research highlights a fascinating and vital human ability: understanding visual elements within broader contexts @xcite . consider the example in figure 1 , where simply looking at an image does not reveal whether the stairs go up or down. however, introducing the context of whether the greenery is above or below the viewpoint of the photographer enables humans to clarify the ambiguity and accurately interpret the image. in this study, we argue that mllms should also be evaluated from the perspective of context-dependent visual comprehension. from a scientific perspective, it is important to understand the extent to which mllms can leverage context to enhance their visual comprehension. practically speaking, as mllms are increasingly applied in real-world scenarios, assessing their reliability in interpreting visual data within context is essential for ensuring they provide accurate responses. however, the capability for context-dependent visual understanding has not been fully assessed by current benchmarks for mllms. to address this challenge, we introduce a new benchmark, named as codis, designed to evaluate the capability of mllms in context-dependent image disambiguation. codis utilizes the visual question answering (vqa, @xcite format for this purpose, as shown in figure 1 @xcite . it stands out from existing benchmarks in three main aspects: first, each image in codis contains inherent ambiguity that can only be resolved with additional context; second, the questions are de-liberately designed to highlight these ambiguities, requiring external context for accurate interpretation; third, for every image-question pair, we provide two contexts in a free-form text format. these contexts are subtly different yet lead to different interpretations of the image and, consequently, different answers. we have carefully curated all images, questions, and contexts by hand to maintain a high standard of quality and diversity. our assessment of 14 widely-used mllms with codis indicates that the performance of these models in understanding context-dependent visuals significantly falls short of human capabilities. further analysis demonstrates that the models struggle with identifying crucial contextual cues and extracting relevant visual features. to summarize, our contributions are three-fold: ## codis codis is proposed for evaluating the capability of mllms in context-dependent image disambiguation. figure 2 presents several examples from our benchmark, highlighting the diversity of contexts covered. in this section, we first elaborate our taxonomy of context. then, we delve into the instruction design and evaluation method of codis. finally, we introduce procedures of data collection. ## conclusion in this work, we introduce a new benchmark codis to evaluate capability of models to use context to enhance visual comprehension. results suggest that mllms significantly fall behind human in context-dependent visual comprehension. further analysis suggests that shortcomings in visual information extraction and bias in model outputs also account for low scores of mllms on codis. in summary, we pave a path for assessing capability of mllms on utilizing contextual information, which is a vital ability in real-world scenarios.