Cross-lingual Editing in Multilingual Language Models (2024)

Abstract: The training of large language models (LLMs) necessitates substantial data and computational resources, and updating outdated LLMs entails significant efforts and resources. While numerous model editing techniques (METs) have emerged to efficiently update model outputs without retraining, their effectiveness in multilingual LLMs, where knowledge is stored in diverse languages, remains an underexplored research area. This research paper introduces the cross-lingual model editing (XME) paradigm, wherein a fact is edited in one language, and the subsequent update propagation is observed across other languages. To investigate the XME paradigm, we conducted experiments using BLOOM, mBERT, and XLM-RoBERTa using the two writing scripts: Latin (English, French, and Spanish) and Indic (Hindi, Gujarati, and Bengali). The results reveal notable performance limitations of state-of-the-art METs under the XME setting, mainly when the languages involved belong to two distinct script families. These findings highlight the need for further research and development of XME techniques to address these challenges. For more comprehensive information, the dataset used in this research and the associated code are publicly available at the following [URL](https://github.com/lingo-iitgn/XME).

## introduction the introduction of large language models (llms) has revolutionized tasks such as dialogue generation, question-answering, and contextual reasoning @xcite @xcite . llms are trained on massive datasets, but this unsupervised data can potentially contain biased or incorrect information. for example, an llm trained on a dataset of news articles might learn that: apple iphones are the best phones or that mumbai is the capital of india. this issue becomes problematic because retraining an llm with equivalent computational power and environmental impact is impractical @xcite . to address this problem, researchers have proposed several model-editing techniques (hereafter referred as mets, @xcite ). mets focus on updating the knowledge within existing llms rather than undergoing complete retraining. however, these mets have been evaluated predominantly in monolingual settings, where editing and evaluation occur within a single language, typically english. this paper aims to explore an alternative scenario, as depicted in figure 1 . for example, we consider the task of updating a language model (in the english language) to reflect the transition of presidential power from donald trump to joe biden in the united states, using established model editing techniques. subsequently, we prompt the updated model with the following french query: donald trump est le présidentdes états-unis d'amérique? (donald trump is the president of the united states of america?), expecting the model to correctly predict 'refutes'. we term this new editing paradigm as cross lingual model editing (xme). we evaluate a specific family of mets that leverage a hypernetwork, an additional model, to update the parameters of a base llm within the framework of xme. the primary objective is to address the following research questions: @xcite what is the effectiveness of hypernetwork-based editing techniques in cross-lingual settings? [q2] do different architectures store knowledge at different locations? [q3] how does language selection in the initial fine-tuning stage affect editing performance? donald trump is the president of the united states of america? (english) डोना सं यु रा य अमे रका के रा प त ह ? (hindi) donald trump est le président des états-unis d'amérique ? (french) ডানা া িক মািক ন যু রাে র িসেড ? (bengali) before editing after editing supports editing in english donald trump is the president of the united states of america? ➙ refutes donald trump serves as the president of the united states of america? (english) डोना सं यु रा य अमे रका के रा प त के प म काय करता है ? (hindi) donald trump est-il le président des états-unis d'amérique ? (french) ডানা া মািক ন যু রাে র িসেড িহেসেব দািয় পালন করেছন? (bengali) refutes figure 1: xme pipeline: we update a fact in one language (say english) and check whether the same fact is updated in different languages. ## related work we classify previous works into two distinct categories: (i) parameter-updating techniques involve actively updating and modifying the parameters of the llm. these approaches aim to adapt and finetune the llm's parameters according to the specific requirements of the editing task. these techniques involve the use of additional feed-forward network architectures. notably, knowledgeeditor @xcite and knowledgeneurons @xcite leverage the gradients of the base model and a hypernetwork to identify the weights that require updating @xcite . another prominent technique, mend @xcite , employs gradient updates from multiple feed-forward networks to update the parameters of the base model. numerous locate-then-edit techniques, exemplified as rome @xcite and @xcite , initially localize the knowledge within the model and then update the base model accordingly. on the other hand, (ii) parameter-preserving techniques refer to methods that aim to maintain the original parameters of the llm during the editing process @xcite @xcite . the focus is on preserving the existing knowledge and capabilities of the llm while incorporating specific modifications for the desired task. serac @xcite incorporates an explicit memory to store edits, enabling the model to reason over them and modulate the predictions of the base model accordingly. another approach, grace by @xcite , introduces a key-value model editor that learns to cache and retrieve activations for selected layers based solely on observed errors during deployment. the preference for hypernetwork-based approaches over other mets arises regarding the effective generability and localization of knowledge, albeit requiring additional memory @xcite . a study conducted by @xcite reveals that localization techniques do not provide further insights into determining the most suitable mlp layer within the base model for overriding an existing stored fact with a new one. further, the time required to perform an edit in hypernetwork-based techniques is lesser, and the inaccessibility of rome and memit over different architectures reasons to choose hypernetworkbased techniques over other mets in our experiments @xcite . ## cross-lingual model editing (xme) the cross-lingual model editing problem can be explained by leveraging notations from monolingual model editing. given a fine-tuned model f with its parameter θ, the prediction or label y can be computed as @xmath0; θ), where x represents the input sentence. our objective is to update the model's parameter to θ ′ in order to modify the label for input x to a new value a, denoted as @xmath1; θ ′ ). however, for the remaining information x where x ̸ = x, the label remains unchanged as y. let's consider an example: when presented with input x as donald trump is the president of the usa? and its semantically equivalent input x ′ as usa's presi-dent is donald trump a fact verification model f outputs y as "supports". now, assuming that the fact is updated and model parameters are changed from θ to θ ′ , for the same inputs x and x ′ , the updated output becomes 'refutes @xcite , where @xmath2, θ ′ ) = f (x ′ , θ ′ ). furthermore, the unrelated information x remains the same as before editing; for instance, the capital of france is paris should still yield the answer "supports". therefore, @xmath3; θ) = f (x; θ ′ ). in contrast to the monolingual model editing, in xme, the inputs x, x ′ , and x belong to different languages. ## experiments this section details the experiments performed for xme and highlights the dataset, architectures, and evaluation strategies. ## results in this section, we present and analyze the key findings and address the research questions posed in introduction section (see section 1 for more details). to accomplish this, we examine a total of 69 foot_1 configurations, which are derived from combining six languages, three plms, and three mets. for each configuration, we present the results in tabular form. for instance, the average g s (excluding the diagonal entries) for editing in the latin family (90.04%) is significantly higher than in the indic family (78.38%). however, the two branches do not significantly differ in the average g s under a monolingual setting. similar trends are observed for fine-tuning mbert in other languages (refer to §a.5.2, §a.6.2, and §a.7.2 for detailed results). comparable patterns were also identified for xml-roberta (refer to §a.5.3, §a.6.3, §a.7.3 for detailed results). the observations derived from the analysis of the bloom model reveal notable distinctions. the metric g s strongly depends on the fine-tuning language script, irrespective of the employed editing language. specifically, when examining the en language, a significant disparity in g s values is observed between the latin and indic script families, as evident in ## conclusion and future directions our research focuses on conducting rigorous experiments with state-of-the-art hypernetwork-based model editing techniques within cross-lingual settings. specifically, we investigate the storage patterns of factual associations in encoder-only and decoder-only models, using two distinct language families as our experimental basis. additionally, we establish a clear dependency between the finetuning language selection and the editing tasks' performance. to further advance the xme paradigm, we plan to utilize parameter-preserving and localized editing techniques. furthermore, we intend to extend our investigations to encompass other nlp tasks, such as machine translation or question-answering. by expanding our research, we aim to enhance our understanding of the capabilities and limitations of hypernetwork-based model editing techniques in diverse cross-lingual settings.