MM - SAP : A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception (2024)

Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in visual perception and understanding. However, these models also suffer from hallucinations, which limit their reliability as AI systems. We believe that these hallucinations are partially due to the models’ struggle with understanding what they can and cannot perceive from images, a capability we refer to as self-awareness in perception. Despite its importance, this aspect of MLLMs has been overlooked in prior studies. In this paper, we aim to define and evaluate the self-awareness of MLLMs in perception. To do this, we first introduce the knowledge quadrant in perception, which helps define what MLLMs know and do not know about images. Using this framework, we propose a novel benchmark, the Self-Awareness in Perception for MLLMs (MM-SAP), specifically designed to assess this capability. We apply MM-SAP to a variety of popular MLLMs, offering a comprehensive analysis of their self-awareness and providing detailed insights. The experiment results reveal that current MLLMs possess limited self-awareness capabilities, pointing to a crucial area for future advancement in the development of trustworthy MLLMs. Code and data are available at https://github.com/YHWmz/MM-SAP.

## introduction recently, breakthrough advances in large language models (llms) have greatly reshaped the artificial intelligence landscape @xcite @xcite @xcite . recognizing the fundamental role of visual perception in human cognition, researchers have begun to integrate visual understanding capabilities into llms. ## self-awareness in perception self-awareness refers to a model's ability to recognize its information limitations, encompassing their capabilities to discern 'knowns' and 'unknowns'. for llms, we can categorize their knowledge using the knowledge quadrant framework to evaluate their self-awareness. however, this framework encounters great complexity when applied to mllms due to the inclusion of visual inputs. in this work, we narrow our focus to self-awareness in image perception, namely, the ability of mllms to recognize the information they can and cannot perceive from images. ## conclusion in this paper, we introduce mm-sap, a novel benchmark designed to evaluate self-awareness in perception for mllms. by innovatively integrating image information with knowledge quadrants, we have developed a modified quadrant specifically tailored for mllms. building on this, we present the mm-sap benchmark, which comprises three distinct sub-datasets. we conducted evaluations of various mllms using this benchmark and analyzed their results to gain insights into the selfawareness capabilities of these models. we believe that the mm-sap benchmark offers a nuanced 8 and detailed perspective on the self-awareness of mllms, contributing significantly to the development of more trustworthy and reliable ai systems.