Can We Edit Multimodal Large Language Models? (2023)

Abstract: In this paper, we focus on editing multimodal Large Language Models (LLMs). Compared to editing single-modal LLMs, multimodal model editing is more challenging, which demands a higher level of scrutiny and careful consideration in the editing process. To facilitate research in this area, we construct a new benchmark, dubbed MMEdit, for editing multimodal LLMs and establishing a suite of innovative metrics for evaluation. We conduct comprehensive experiments involving various model editing baselines and analyze the impact of editing different components for multimodal LLMs. Empirically, we notice that previous baselines can implement editing multimodal LLMs to some extent, but the effect is still barely satisfactory, indicating the potential difficulty of this task. We hope that our work can provide the NLP community with insights.

## introduction with the widespread deployment of large language models (llms) @xcite , the necessity to maintain their knowledge accurate and current without incurring significant retraining costs is becoming increasingly paramount @xcite . previous research has introduced knowledge editing methodologies designed to incrementally infuse a language model with a new set of facts @xcite @xcite @xcite . different from single-modal model editing, the task of editing multimodal llms presents considerable challenges, given their inherent diversity and complexity. specifically, incorrect outputs from multimodal models may stem from the synergistic effects of various modalities. incorrect outputs ## editing multimodal llms we illustrate the proposed task of multimodal editing in figure 2 . we will introduce the task definition ( §3.1), dataset construction details in ( §3.2), the multimodal models ( §3.3), and the baselines ( §3.4) we used in the experiments.