Cross-Lingual Knowledge Editing in Large Language Models (2024)

Abstract: Knowledge editing aims to change language models’ performance on several special cases (i.e., editing scope) by infusing the corresponding expected knowledge into them. With the recent advancements in large language models (LLMs), knowledge editing has been shown as a promising technique to adapt LLMs to new knowledge without retraining from scratch. However, most of the previous studies neglect the multi-lingual nature of some main-stream LLMs (e.g., LLaMA, ChatGPT and GPT-4), and typically focus on monolingual scenarios, where LLMs are edited and evaluated in the same language. As a result, it is still unknown the effect of source language editing on a different target language. In this paper, we aim to figure out this cross-lingual effect in knowledge editing. Specifically, we first collect a large-scale cross-lingual synthetic dataset by translating ZsRE from English to Chinese. Then, we conduct English editing on various knowledge editing methods covering different paradigms, and evaluate their performance in Chinese, and vice versa. To give deeper analyses of the cross-lingual effect, the evaluation includes four aspects, i.e., reliability, generality, locality and portability. Furthermore, we analyze the inconsistent behaviors of the edited models and discuss their specific challenges.

## introduction the goal of knowledge editing is to adjust language models' behaviors within an expected scope (i.e., editing scope) and retain out-of-scope model performance ideally @xcite . along with the dynamic changes in the world, knowledge editing could help models forget outdated knowledge and adapt to the new counterpart without retraining from scratch. as the example shown in figure 1 (a), the number of honkai-series games increases to four after the release of honkai: star rail (on april 26, 2023). however, if we ask a model that has been trained before the date, the model might only know three honkai-series games. in such a situation, knowledge editing could help the model efficiently update this new knowledge, and give the right answer after editing. despite many efforts devoted to this research field @xcite @xcite @xcite @xcite @xcite @xcite @xcite , current knowledge editing studies typically focus on monolingual scenarios, where language models are edited and evaluated within the same language, c.f., figure 1 ( @xcite . meanwhile, the rapid advancements in large language models (llms) have led to the widespread adoption of multi-lingual settings, allowing language modeling ability can be shared across different languages @xcite @xcite @xcite . for example, llms such as llama @xcite , @xcite , and gpt-4 (openai, 2023) are designed to operate under multilingual setting. under this background, the performance of a source-language edited model on other languages is still unknown. as shown in figure 1 (b) , a research question (rq) arises, when we utilize source-language samples to edit a multi-lingual llm, can the model reflect consistent behaviors when faced with a different target language? to answer the rq, in this paper, we explore knowledge editing in cross-lingual scenarios, and study the effects of source-language editing on a different target language. specifically, we automatically translate the knowledge editing data from english to chinese via cutting-edge llms (i.e., chatgpt and gpt-4). after carefully comparing existing datasets, we finally choose zsre @xcite which is originally a question answering (qa) dataset and is further widely used in knowledge editing @xcite @xcite . more recently, @xcite collect a number of qa pairs that need deep reasoning based on zsre, and the data could be used to evaluate the portability of knowledge editing methods beyond simple paraphrasing. therefore, we also translate these qa pairs to give a deeper understanding of cross-lingual knowledge editing performance. the translated data together with the original ones is denoted as bi-zsre. then, we conduct english/chinese editing on several open-sourced multi-lingual llms (llama, llama2, baichuan and baichuan2), and evaluate their behaviors in chinese/english in terms of reliability, generality, locality and portability. our experiments involve seven knowledge editing methods covering three main-stream paradigms pointed by @xcite , i.e., memory-based, metalearning and locate-then-edit methods. the experimental results reveal that (1) the language modeling gaps across different languages influence the efficiency of knowledge editing; (2) it is still hard for existing knowledge editing methods to transfer the edited knowledge from one language to another in a multi-lingual llm; (3) when editing llms in a language, the model performance on the irrelevant examples in other languages could also be influenced, resulting in low locality. this presents a significant challenge for multi-lingual llms in maintaining consistent behaviors across different languages. our main contributions are concluded as follows: • to our knowledge, we are the first to explore the cross-lingual effect of knowledge editing in llms. we achieved this by translating the zsre dataset and studying the cross-lingual effect from english (chinese) to chinese (english). according to a comprehensive survey on knowledge editing @xcite , there are three mainstream knowledge editing paradigms: (1) memorybased methods keep the original model parameters unchanged while employing another model to influence the model's behaviors. serac @xcite utilizes a scope classifier to evaluate whether new input is close to the stored editing examples, and further influences the model behaviors based on the retrieved editing examples. t-patcher @xcite and calinet @xcite add extra trainable parameters into the ffn layers of llms to edit model performance. ike @xcite uses context-edit facts to guide the model in generating edited facts. (2) meta-learning methods employ a hyper network to learn the weight updates of llms to edit the models. @xcite makes use of lstm networks to predict the weight update for each new input. mend @xcite transforms the gradient of fine-tuned language models by employing a low-rank decomposition of gradients. knowledge editing datasets. zsre @xcite is a question answering dataset whose queries require models to answer the questions based on the information within the queries. coun-terfact @xcite evaluates whether the edited model can provide counterfactual answers when asked about the corresponding factual knowledge. mquake @xcite aims to assess whether edited models correctly answer questions where the answer needs reasoning based on the edited facts. @xcite propose mmedit, a multi-modal knowledge editing benchmark dataset. personalityedit @xcite is proposed to edit personality traits for llms. @xcite propose conflictedit and roundedit to investigate the potential pitfalls associated with knowledge editing for llms. eva-kellm @xcite evaluates the edited model from reasoning with the altered knowledge and cross-lingual transfer. though eva-kellm provides a subset for cross-lingual knowledge editing, the data has not yet been made public. 2 besides, this work does not conduct experiments on any knowledge editing methods, leaving the cross-lingual effect still not known in the knowledge editing research field. ## bi-zsre in this section, we first discuss the details of data collection, including data sources, translation process as well as quality control ( § 3.1). then, we give the data statistics of bi-zsre ( § 3.2), and finally provide the task overview of cross-lingual knowledge editing ( § 3.3). please translate the following json data from english to chinese and keep the format unchanged: ## conclusion in this paper, we first explore the cross-lingual effect of knowledge editing in large language models. to achieve that, we automatically construct bi-zsre dataset by translating the previous zsre dataset from english to chinese. based on bi-zsre, we conduct experiments on various knowledge editing methods, and study the cross-lingual effect from english to chinese and vice versa. our results indicate that it is still hard for existing knowledge editing methods to transfer the edited knowledge from one language to another in a multilingual llm. we also analyze the inconsistent behaviors of the edited models and discuss their spe-cific challenges to provide a deeper understanding of the cross-lingual effect in knowledge editing.