MQ u AKE : Assessing Knowledge Editing in Language Models via Multi-Hop Questions (2023)

Abstract: The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option. This has recently given rise to a range of techniques for injecting new facts through updating model weights. Current evaluation paradigms are extremely limited, mainly validating the recall of edited facts, but changing one fact should cause rippling changes to the model’s related beliefs. If we edit the UK Prime Minister to now be Rishi Sunak, then we should get a different answer to Who is married to the British Prime Minister? In this work, we present a benchmark MQuAKE (Multi-hop Question Answering for Knowledge Editing) comprising multi-hop questions that assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. While we find that current knowledge-editing approaches can recall edited facts accurately, they fail catastrophically on the constructed multi-hop questions. We thus propose a simple memory-based approach, MeLLo, which stores all edited facts externally while prompting the language model iteratively to generate answers that are consistent with the edited facts. While MQuAKE remains challenging, we show that MeLLo scales well with LLMs (up to 175B) and outperforms previous model editors by a large margin.

## introduction as large language models (llms) are deployed widely, the need to keep their knowledge correct and up-to-date without massive retraining costs becomes increasingly important @xcite . prior work has proposed knowledge editing methods to incrementally inject a set of new facts into a language model @xcite @xcite new fact: the current british prime minister is rishi sunak. ## problem definition this section introduces our setting and argues that a model edit is only truly successful if the edited model also returns new correct answers for questions that change as a consequence of the edits. ## mquake: multi-hop question answering for knowledge editing we construct the benchmark mquake (multihop question answering for knowledge editing), which contains two datasets. the first, mquake-cf, is designed as a diagnostic dataset for the evaluation of knowledge editing methods on counterfactual edits. the second, mquake-t, comprises temporal-based knowledge updates and is intended to assess the effectiveness of knowledge editing methods in updating outdated information with current, real facts. we first present the data construction process for mquake-cf and mquake-t. then, we present the data statistics and evaluation settings, followed by evaluation metrics in the end. ## mello: a proposal for editing large language models in section 4, our evaluation results show that existing knowledge-editing methods fail catastrophically on multi-hop questions of mquake. in this section, we present a simple but effective alternative, mello (memory-based editing for large language models). ## conclusion in this work, we present a benchmark mquake that assesses knowledge editing methods for language models via multi-hop questions. we find that although edited language models can effectively recall edited facts, they fail on multi-hop questions that are entailed consequences of the edits. we propose a simple but effective alternative, mello, which significantly outperforms existing knowledge editing methods. mello does not require any additional training and can be applied to large lms such as gpt-3 @xcite . we hope our work can facilitate future research on developing faithful knowledge editing methods.