Cross-Lingual Multi-Hop Knowledge Editing (2024)

Abstract: Large language models (LLMs) are often expected to be constantly adapted to new sources of knowledge and knowledge editing techniques aim to efficiently patch the outdated model knowledge, with minimal modification. Most prior works focus on monolingual knowledge editing in English, even though new information can emerge in any language from any part of the world. We propose the Cross-Lingual Multi-Hop Knowledge Editing paradigm, for measuring and analyzing the performance of various SoTA knowledge editing techniques in a cross-lingual setup. Specifically, we create a parallel cross-lingual benchmark, CroLin-MQuAKE for measuring the knowledge editing capabilities. Our extensive analysis over various knowledge editing techniques uncover significant gaps in performance between the cross-lingual and English-centric setting. Following this, we propose a significantly improved system for cross-lingual multi-hop knowledge editing, CLeVer-CKE. CLeVer-CKE is based on a retrieve, verify and generate knowledge editing framework, where a retriever is formulated to recall edited facts and support an LLM to adhere to knowledge edits. We develop language-aware and hard-negative based contrastive losses for improving the cross-lingual and fine-grained fact retrieval and verification process used within this framework. Extensive experiments across three LLMs, eight languages, and two datasets show the CLeVer-CKE’s significant gains of up to 30% over prior methods.

## introduction large language models (llms) are seeing an increasing adoption across users having different cultural and linguistic background, and need to be up to date about the ever-changing knowledge in the world for maintaining their utility and reliability in various applications. due to the ever increasing compute and data requirements to train these models, there has been a surge in the development of knowledge editing techniques to modify the language models in an efficient way, such that they adhere to the world dynamics. prior work on knowledge editing has largely focused on editing llms in a monolingual setting @xcite ), where both user queries and edited facts are expressed in the form of english. these works can be grouped into two categories: parameter-update and parameterpreserving methods. the former directly updates the parameters within llms for updating knowledge about the edited facts through meta-learning, fine-tuning, or knowledge locating @xcite @xcite . the later approach freezes the parameters and explicitly stores the edited facts in an external memory and retrieves them for answering user queries @xcite @xcite . existing monolingual knowledge editing techniques aren't broadly applicable since new knowledge can emerge in different languages. some works have made progress in this direction @xcite @xcite ), but they have considered a simplistic setting of assuming the edited facts as independent without any multihop rippling consequences on entailed reasoning process, and are primarily focused on parametermodifying based editing methods. there has only been a limited focus on the realistic case of cross-lingual multi-hop knowledge editing (see fig 1 ), where the edited knowledge can come in through users who communicate in different languages. further, much of edited knowledge often has a rippling effect on other facts of the world. for example, the club change of messi affects deduction process of question "indicating a superficial word matching rather than a contextual grasp of the entities involved." this knowledge editing setting, which we argue is important to study, is challenging since the model needs to transfer knowledge about fact edits between different languages, while also reasoning about the facts which are modified as a consequence to the given edit. poor knowledge transfer between languages can lead to error propagation across reasoning steps which can increase failure cases of model editing. in this work, we formulate the notion of crosslingual multi-hop knowledge-editing and analyze existing approaches for their editing ability in different languages, following which a simple yet highly effective approach is designed. specifically, ① we create one of the first benchmark datasets for measuring cross-lingual multi-hop knowledge editing capabilities of knowledge editing methods. besides parameter-update based approaches, we contribute strong retrieval-based baselines for knowledge editing and provide a comprehensive analysis. ② we provide a detailed analysis and find significant gaps in the performance of methods for crosslingual knowledge editing. the gaps are mainly due to challenges in accurately recalling fact edits made in language other than input query. ③ to bridge such gap, we design a competitive method, termed as contrastive language-aware verification for cross-lingual knowledge editing (clever-cke), for improving performance of cross-lingual multi-hop knowledge editing. our approach is based on decomposing a multi-hop question in a particular language into sub-questions and retrieving fact edits (if any) from memory using a cross-lingual retriever, which is integrated for answering sub-questions. in particular, the cross-lingual retriever is regularized by novel languageguided and hard-negative based contrastive losses, which leads to improved language and fine-grained sentence understanding of the edits, leading to high quality cross-lingual retrievals. clever-cke improves over previous sota by up-to 30% increase in knowledge editing accuracy when tested on multiple llms, datasets and languages. ## cross-lingual multi-hop editing following prior work @xcite , a fact is defined as a triplet (s, r, o), where s is the subject, o is the object, and r is the relation (e.g., shakespeare, author of, hamlet). given that a parametric llm can become outdated or incorrect, knowledge editing is required to be performed on it. an edited fact stores information about updated knowledge of an existing fact and is denoted as @xmath0, r, o * ), where the object is replaced with a new one o * . cross-lingual knowledge editing. each knowledge fact or edit is assumed to be represented in natural language. let t : @xmath1.g., shakespeare, author of, hamlet) and converts it into a natural language statement, (e.g., shakespeare is the author of hamlet). all the facts and edits can be represented in a variety of languages {l 1 , l 2 , . . . } via functions such as {t l 1 , t l 2 , . . . }. for example, an edit @xmath2, author of, lolita) can be written as t de (e) = shakespeare ist der autor von lolita in german and t en (e) = shakespeare is the author of lolita in english. we consider a collection of n fact edits in the diverse languages: @xmath3 for evaluating models in a cross-lingual multihop setting, we make use of multi-hop questions such as q ln , in language l n which is different from l 1...k . the question asks about the head entity s l 1 1 for which the answer is o l k n before editing. after editing, the fact chain changes to ## crolin-mquake benchmark we develop one of the first parallel cross-lingual and multi-hop benchmarks for measuring the knowledge editing capabilities of the existing approaches. a parallel benchmark across languages has the same test examples across all the languages, enabling a direct comparison between them. for this, we use existing datasets measuring the multi-hop model editing in english: mquake-cf and mquake-t released by @xcite , which have counterfactual edits and real-world temporal edits respectively. we translate one fact edit per example in these datasets using google translate (google) into 7 languages with diverse writing scripts across medium to high resourcedness -german, spanish, chinese, russian, hindi, bengali, swahili. this results in the benchmark: cross-lingual multi-hop qna for knowledge editing (crolin-mquake). it has two datasets, crolin-mquake-cf and crolin-mquake-t, each having 8 languages, and 3k and 1.8k parallel examples (same examples in all languages) per language, respectively. the translations are verified by human experts proficient in particular languages and evaluation of bleu score @xcite using backtranslation. we find that the translation is highly accurate, since we study medium to high resource languages. see section a.1 for more details. concurrently, @xcite created a multilingual knowledge editing dataset using wikipedia, offering translocalized knowledge but lacking parallel multilingual examples like ours. crolin-mquake enables comparing the knowledge edit-ing performance difference across languages directly without being affected by the variation of test sets between different languages. ## benchmark analysis on cross-lingual multi-hop knowledge editing llms. we use sota propriety and open-source llms: chatgpt @xcite , llama-2-7b @xcite , vicuna-1.5-7b @xcite as backbones to evaluate crosslingual multi-hop knowledge editing. evaluation metrics. we use multi-hop accuracy proposed by @xcite which measures the accuracy of the final answer of a multi-hop question. we also adopt hop-wise answering accuracy for checking the correctness of intermediate reasoning steps, as proposed by @xcite . new baselines. based on existing work, we contribute strong baselines for the new editing setup: parameter-modifying based knowledge editing performs poorly in the cross-lingual setting. methods that update the parameters of the model, such as rome, memit, ft, perform significantly worse in the cross-lingual setting, achieving an accuracy under 5.0% (average across languages), as shown in hence the parameter-modifying methods struggle to reliably edit the llm across languages and multi-hop contexts. this highlights the need for memory-based approaches that rely on an external edit memory, like our contributed baselines, mello-cl and pokemqa-cl, which can crosslingually retrieve the relevant edits on the fly when inferring from an llm. these approaches substantially improve performance up to nearly 30% on crolin-mquake compared to parametermodifying based methods. ## conclusion in this paper, we contributed a benchmark having parallel multilingual examples for evaluating cross-lingual multi-hop knowledge editing. we provide new baselines and a detailed analysis of sota knowledge editing methods and find various gaps in existing methods, particularly in the cross-lingual setting. motivated by this, we propose a generic, simple and highly effective method, clever-cke, for improving the knowledge editing capabilities of parameter-preserving, retrieval augmented editing methods. clever-cke improves cross-lingual and fine-grained retrieval in the case of knowledge editing, by introducing language aware and hard-negative mining based contrastive losses to train retrievers. improved retrieval leads to precise knowledge retrieval and reduced error propagation in the multi-hop reasoning setting. clever-cke is parameter-preserving in terms of the llm weights, and uses a lightweight retriever with low latency as compared to methods like @xcite .