Retrieval-Augmented Multilingual Knowledge Editing (2024)

Abstract: Knowledge represented in Large Language Models (LLMs) is quite often incorrect and can also become obsolete over time. Updating knowledge via fine-tuning is computationally resource-hungry and not reliable, and so knowledge editing (KE) has developed as an effective and economical alternative to inject new knowledge or to fix factual errors in LLMs. Although there has been considerable interest in this area, current KE research exclusively focuses on monolingual settings, typically in English. However, what happens if the new knowledge is supplied in one language, but we would like to query an LLM in a different language? To address the problem of multilingual knowledge editing, we propose Retrieval-Augmented Multilingual Knowledge Editor (ReMaKE) to update knowledge in LLMs. ReMaKE can be used to perform model-agnostic knowledge editing in a multilingual setting. ReMaKE concatenates the new knowledge retrieved from a multilingual knowledge base with users’ prompts before querying an LLM. Our experimental results show that ReMaKE outperforms baseline knowledge editing methods by a significant margin and is scalable to real-word application scenarios. Our multilingual knowledge editing dataset (MzsRE) in 12 languages, the code, and additional project information are available at https://github.com/weixuan-wang123/ReMaKE.

## introduction large language models (llms) are being used as sources of factual knowledge for search engines and other downstream tasks. despite their considerable progress, knowledge generated by llms can be incorrect or become obsolete in a changing world. pre-training from scratch or fine-tuning llms to adapt them to new knowledge is computationally expensive and not guaranteed to work. knowledge editing (ke) methods @xcite @xcite have been proposed as effective and economic alternatives to fine-tuning when specific factual knowledge needs to be added or updated. ke involves either updating the parameters of a model @xcite @xcite @xcite or adding extra components to an llm @xcite @xcite . for example, ke can be used to correct the answer to this question "who is the foreign secretary of the uk?" from "james cleverly" (true until mid november 2023) to "david cameron", who had recently been appointed to the post. despite significant interest in this problem, current research on ke predominantly concentrates on a monolingual setting, where both the injected knowledge and the subsequent queries to the llm are in english @xcite @xcite @xcite . companies serving a multilingual customer base need to consider the multilingual ke case, where ke is done in one language and this propagates to queries and answers in all other languages. while @xcite explored the applicability of knowledge editing to the english-chinese cross-lingual scenario, their primary focus was to highlight the challenges rather than develop a functional ke approach in a multilingual setting. drawing inspiration from in-context learning (icl), in-context knowledge editing (ike) uses prompts to edit factual knowledge. it is noted that ike is so far the only method demonstrating positive results in the cross-lingual ke task setting @xcite . however, ike requires explicit provision of new knowledge every time an llm is used, confining its practicality and scalability in real-world applications. in addition, ike suffers when irrelevant facts are included in the prompt @xcite especially in scenarios where a substantial number of facts are being edited. figure 1 : remake attaches in-context knowledge to an llm prompt when it is retrieved (red example where the edited knowledge is in english and a user query is in spanish) from a customer-defined multilingual knowledge base. when no edited knowledge is retrieved (green example) the prompt is passed to the llm unchanged. in this paper, we propose retrieval-augmented multilingual knowledge editor (remake) that integrates multilingual retrieval from a knowledge base with in-context learning. remake concatenates the retrieved knowledge from an external database with a user query to create the prompt. the proposed multilingual retriever grounds re-make to the retrieved accurate and up-to-date information highly relevant to user queries, therefore effectively mitigating the contextual interference due to irrelevant context. in this way, the generated prompts are able to guide the llms in producing accurate responses associated with the injected knowledge. remake leverages a knowledge base's ability to scale to further enhance ike's knowledge editing performance in real-world application scenarios where large volumes of edits are in scope. figure 1 shows the architecture of the proposed retrieval-augmented multilingual knowledge editor. our main contributions are listed below: all above-mentioned editors are limited to a monolingual setting. for cross-lingual editing, @xcite and @xcite require training and manipulating model parameters. they rely on the masking mechanism, limiting the scope of application to a one-word output scenario. most ke approaches are designed to process single editing and, therefore, have limited practicality. our proposed editor, remake, addresses these problems and scales multilingual ke to cover a considerable amount of knowledge. retrieval-augmented in-context learning: offthe-shelf search engines are often used to enhance retrieval-augmented icl @xcite @xcite , attaching semantically similar examples to the context to improve the performance of llms in few-shot learning. in cross-lingual scenarios, a search engine first uses a low-resource language input sample as a query to find the most semantically similar high-resource language sample in the corpus. the retrieved highresource language sample is concatenated with the input sample to form a prompt for an llm. for instance, @xcite retrieves semantically similar cross-lingual sentences as prompts to improve the performance of sentiment classification for lowresource languages. while icl can be used to support cross-lingual tasks, using it to edit knowledge across language boundaries has not been explored. ke in a cross-lingual setting remains a challenge @xcite . to bridge this gap, remake leverages icl and knowledge retrieval to perform cross-lingual ke. ## retrieval-augmented multilingual knowledge editing remake enables knowledge in an llm to be edited in one language and subsequently queried in multiple languages. remake consists of two stages: multilingual knowledge retrieval and multilingual in-context editing. ## experimental results we define a standard notation ("language 1 (edit) → language 2 (test)") to refer to the multilingual ke experiments. for example, "es (edit) → en (test)" refers to an experiment in which spanish is the editing language and english is the language we tested the knowledge, as shown in figure 2 . the knowledge base of the experiments consists of all the above-mentioned multilingual knowledge instances in the mzsre test set. ## conclusion in this paper, we propose remake, a retrievalaugmented multilingual knowledge editor, to update multilingual knowledge in llms by leveraging prompts composed of retrieved new knowledge and user inputs. to achieve multilingual knowledge editing, we automatically construct the mzsre dataset to cover 12 languages. remake is a model and language-agnostic knowledge editor not restricted to a specific llm and language setting. our experimental results show that remake achieves sota multilingual knowledge editing performance.