Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences (2024)

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks. However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated. To address this challenge, this paper introduces Mementos, a new benchmark designed to assess MLLMs’ sequential image reasoning abilities. Mementos features 4,761 diverse image sequences with varying lengths. We also employ a GPT-4 assisted method to evaluate MLLM reasoning performance. Through a careful evaluation of nine recent MLLMs on Mementos, including GPT-4V and Gemini, we find that they struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects and their corresponding behaviors. Our quantitative analysis and case studies identify three key factors impacting MLLMs’ sequential image reasoning: the correlation between object and behavioral hallucinations, the influence of co-occurring behaviors, and the compounding impact of behavioral hallucinations.

## introduction the recent emergence of multimodal large language models (mllms) such as gpt-4v (ope-nai, 2023b) and gemini @xcite has shown strong visual-language understanding and generation capabilities in many areas, like image captioning and visual question answering. despite the notable performance of existing mllms, they often suffer from hallucination (a phenomenon where mllms produce inaccurate descriptions of the given images) due to insufficient reasoning capabilities, generating inaccurate responses in visual inference @xcite . thus, monitoring the reasoning capability is of great importance in understanding the ability and the limitations of mllms and applying mllms in the real world. previous benchmarks, such as @xcite and @xcite , have primarily addressed evaluating reasoning in each individual image, relying on static and object-centric knowledge. however, they are insufficient to comprehensively assess the reasoning capabilities of mllms due to a lack of time-varying object behaviors or events. to investigate the capabilities of multi-modal language models (mllms) in dynamic reasoning across image sequences, we present a new benchmark, mementos. this benchmark focuses on the complex task of monitoring and deciphering the positional changes of objects within an image sequence, followed by the inference of behavioral patterns and logical connections among them. such an endeavor requires the interpretation of the overarching context based on time-variant visual elements, posing a greater challenge than the analysis of static scenes. concretely, mementos consists of 4,761 image sequences with varying episode lengths, encompassing diverse scenarios from everyday life, robotics tasks, and comic-style storyboards. an episode refers to a specific event or series of events depicted in the image sequence. each sequence is paired with a human-annotated description of the key objects and their behaviors. to assess the reasoning capability of mllms on mementos, we employ a gpt-4-assisted evaluation procedure: after an mllm produces a description for an image sequence, we extract behavior and object keywords from both ai-generated and human-annotated descriptions using gpt-4. we then use keyword matching to assess the degree of behavioral and object hallucinations. to refine the correctness of this evaluation, we have developed behavior and object synonym graphs for each domain. these graphs facilitate more precise keyword matching, ensuring a thorough and nuanced analysis of the mllms' reasoning abilities. besides, we also provide the comparison with human evaluation to demonstrates that the gpt-4-assisted evaluation procedure is very reliable. we evaluated the reasoning proficiency of nine leading-edge mllms on mementos, encompassing both black-box and open-source models. our findings indicate that mementos poses a considerable challenge to these current mllms. for instance, as depicted in figure 1 , gpt-4v exhibits notable behavioral and object hallucinations in various domains during image sequence reasoning. behavioral hallucinations are defined as the mllms' erroneous interpretations or predictions of entity actions, while object hallucinations pertain to the inaccurate identification or creation of objects. notably, behavioral hallucinations were more frequent than object hallucinations, highlighting a significant deficiency in mllms' capability to deduce events from image sequences. furthermore, our research pinpoints three principal factors that lead to the reasoning failures of mllms: (1) the interconnectedness of object and behavioral hallucinations, (2) the impact of cooccurring behaviors, and (3) the cumulative effect of behavioral hallucinations. the objective of our proposed benchmark and analyses is to shed light on innovative approaches to augment the reasoning abilities of mllms and to reduce hallucinations in their subsequent advancements. ## mementos in this section, we introduce mementos, a novel and challenging benchmark designed to test the reasoning capability of multimodal large language model (mllm) under sequential image input. initially, we detail the data gathering and annotation methodology for mementos, alongside an overview of data distribution. subsequently, we outline the procedure and the metric employed to evaluate the reasoning capabilities of mllms on mementos. ## experiments in our experimental section, we delve into two key questions: (a) we examine the reasoning capabilities of current mllms on mementos. specifically, we assess the severity of object and behavioral hallucinations in these models. (b) we investigate the underlying causes of reasoning failures in mllms when interpreting image sequences. ## conclusion in this paper, we present mementos, a novel and challenging benchmark designed to assess the reasoning abilities of multimodal large language models (mllms) in interpreting image sequences. we conduct evaluations on nine most recent mllms using gpt-4-assisted evaluation procedure. our findings indicate that all tested mllms struggle with significant behavioral and object hallucinations in generating descriptions for image sequences. through a mix of quantitative analysis and case studies, we identify three primary factors contributing to these reasoning failures.