MC - MKE : A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency (2025)

Abstract: Multimodal large language models (MLLMs) are prone to non-factual or outdated knowledge issues, highlighting the importance of knowledge editing. Many benchmark has been proposed for researching multimodal knowledge editing. However, previous benchmarks focus on limited scenarios due to the lack of rigorous definition of multimodal knowledge. To better evaluate multimodal knowledge editing, we propose a decomposed definition of multimodal knowledge. Following the decomposed definition of multimodal knowledge, we introduce three scenarios and a novel requirement modality consistency. We construct MC-MKE, a fine-grained **M**ultimodal **K**nowledge **E**diting benchmark emphasizing **M**odality **C**onsistency through strict data selection. We evaluate four multimodal knowledge editing methods on MC-MKE, revealing their limitations, particularly in terms of modality consistency. Our work highlights the challenges posed by multimodal knowledge editing and motivates further research in developing effective techniques for this task.

## introduction with the developments of multimodal large language models (mllms), their application has become widespread across various fields. however, these models struggle with the challenge that the knowledge stored within them could be inaccurate or outdated. knowledge editing is aimed to solve the problem. following the conventional definition of knowledge-editing in llms, a few studies have proposed benchmarks for knowledge editing in mllms @xcite @xcite . however, these evaluation datasets overlook a key difference between multimodal knowledge and textual knowledge, which led them to ignore an additional requirements for editing the knowledge requires ensuring consistency across the corresponding visual, textual and multimodal knowledge. for example, as shown in figure 1 , the wrong visual knowledge in model is (image of messi, mac allister). incorrect visual knowledge combined with textual knowledge (mac allister, play for, liverpool) can lead to wrong multimodal knowledge. knowledge editing needs to correct the corresponding visual knowledge so that the model can successfully recognize the person in the image as messi. at the same time, it is essential to ensure that the related multimodal knowledge is also consistently updated, meaning the corresponding multimodal knowledge should be corrected to (image of messi, play for, miami). we believe that a knowledge editing method should always ensure the consistency of knowledge across different modalities. this property is the essential difference between multimodal knowledge editing and uni-modal knowledge editing. following the decomposed definition of multimodal knowledge, we propose a multimodal knowledge editing benchmark emphasizing modality consistency (mc-mke). mc-mke consists of three subsets, corresponding to three different scenarios of multimodal knowledge editing. our benchmark provides various scenarios for multimodal knowledge editing and can more systematically and comprehensively evaluate the performance of a multimodal knowledge editing method in a finegrained manner on reliability, locality, generality and consistency aspect. we evaluate four of the most renowned multimodal knowledge editing methods including finetuning, mend @xcite , ike @xcite , and serac @xcite on the three subsets corresponding to different editing scenarios. we find that the performance of these methods is far from satisfaction on mc-mke. none of them can achieve great performance on all three different editing formats, especially for the consistency metric. it is demonstrated that multimodal knowledge editing is still challenging and requires further exploration. in summary, our contributions are as follows: ## mc-mke benchmark construction since pure textual knowledge editing datasets are constructed from textual knowledge triplets (s, r, o) and contain editing information (s, r, @xmath0, we apply the textual knowledge editing dataset mquake @xcite as the starting point to construct our multimodal knowledge editing dataset mc-mke. mquake, as a text knowledge editing dataset, contains knowledge triplets, related editing information and textual questions as test input. each instance in mquake corresponds to a textual knowledge triplet and its textual editing information. ## conclusion we refine the definition of multimodal knowledge and introduce a new benchmark mc-mke. we conduct experiments to analyze the effectiveness of several multimodal knowledge editing methods across different models, editing scenarios, and editing components. we find that these methods have limitations, and cannot achieve perfect per-formance on all editing scenarios. to maintain consistency, it may be better to edit the model component corresponding to the specific type of editing knowledge.