MIKE : A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing (2024)

Abstract: Multimodal knowledge editing represents a critical advancement in enhancing the capabilities of Multimodal Large Language Models (MLLMs). Despite its potential, current benchmarks predominantly focus on coarse-grained knowledge, leaving the intricacies of fine-grained (FG) multimodal entity knowledge largely unexplored. This gap presents a notable challenge, as FG entity recognition is pivotal for the practical deployment and effectiveness of MLLMs in diverse real-world scenarios. To bridge this gap, we introduce MIKE, a comprehensive benchmark and dataset specifically designed for the FG multimodal entity knowledge editing. MIKE encompasses a suite of tasks tailored to assess different perspectives, including Vanilla Name Answering, Entity-Level Caption, and Complex-Scenario Recognition. In addition, a new form of knowledge editing, Multi-step Editing, is introduced to evaluate the editing efficiency. Through our extensive evaluations, we demonstrate that the current state-of-the-art methods face significant challenges in tackling our proposed benchmark, underscoring the complexity of FG knowledge editing in MLLMs. Our findings spotlight the urgent need for novel approaches in this domain, setting a clear agenda for future research and development efforts within the community.

## introduction multimodal knowledge editing (mke) @xcite @xcite @xcite plays a critical role in maintaining and improving the accuracy of multimodal large language models (mllms) @xcite @xcite a white hair old man arrives at a building. president joe biden arrives at the white house. ours mmedit after edit before edit fine-grained editing updates: coarse-grained editing updates: yellow haired woman ↓ president joe biden building ↓ white house yellow haired woman ↓ white hair old man figure 1 : the comparison between mmedit @xcite and ours (mike). mike focuses on editing fine-grained multimodal entity knowledge. @xcite . central to mke is the capability to update outdated, unknown, or incorrect knowledge within mllms. recent developments in this field, such as the benchmark mmedit proposed by @xcite , signify considerable progress. drawing from datasets of visual question answering (vqa) @xcite and image caption @xcite tasks, mmedit offers a platform to test the editability of mllms. however, a critical issue remains in its primary focus on coarse-grained knowledge, which often falls short of accurately representing real-world fine-grained (fg) entities and scenarios. to underscore the limitations of a coarse-grained focus, consider a real-life example in political image captioning as shown in figure 1 . an ideal mllm output would be a fine-grained and specific caption like "president joe biden arrives at the white house". however, a coarse-grained approach might yield a nondescript caption such as "a white hair old man arrives at a building". this lack of specificity fails to capture the critical de-tails and convey key information to the users of mllms, illustrating how fg entity recognition is essential for delivering accurate information. while the necessity for more detailed, entityspecific information is clear, editing fg knowledge into mllms is a complex and challenging endeavor. traditional fg image classification tasks @xcite @xcite demand vision encoders to discern and categorize visually similar items. the task becomes even more difficult when extending to mllms. mllms are required to not only recognize fg visual entities but also to understand and map them to corresponding textual descriptions. although recent studies @xcite have demonstrated a nascent ability in mllms to identify multimodal knowledge at the entity level, their performance notably lags in handling fg entities as compared to coarse-grained ones. this performance gap highlights the substantial challenges in accurately recognizing fg entities by mllms. given these challenges, the question remains: can we effectively edit fg multimodal entity knowledge into mllms? addressing this query is not only crucial for advancing the field of mllms but also for unlocking a myriad of applications requiring detailed understanding. to explore this problem, we propose a comprehensive and challenging benchmark for fine-grained multimodal entity knowledge editing (mike). it is composed of more than 1000 fg entities, each of which includes at least 5 images. to challenge mke methods and meet the needs of real scenes, we purposefully create a diverse set of tasks from different angles: (i) vanilla name answering, where mllms are required to answer the short name of the entity in the image; (ii) entity-level caption, where mllms need to caption the image not only the general content but the entity name as well; (iii) complex-scenario recognition, where mllms need to recognize a targeted entity under a complex visual field of multiple entities. in addition, extending the normal knowledge editing form, we propose multi-step editing. in this form, mllms are edited with 2-4 fg entity images instead of one. we utilize easyedit toolkit @xcite to assess several knowledge editing approaches on mike. for the evaluation, we propose entityoriented metrics under the setting of reliability, generality and locality. through extensive experiments, we find (i) each editing method exhibits specific limitations; (ii) the most challenging task for current editing methods is entity-level caption; (iii) different generality tasks affect the ability of mke in some aspects; (iv) model size does not matter. for a detailed discussion of these findings and additional results, please refer to section 4. we summarize main contributions as follows: