Language Anisotropic Cross-Lingual Model Editing (2023)

Abstract: Multilingual pre-trained language models can learn task-specific abilities or memorize facts across multiple languages but inevitably make undesired predictions with specific inputs. Under similar observation, model editing aims to post-hoc calibrate a model targeted to specific inputs with keeping the model’s raw behavior. However, existing work only studies the monolingual scenario, which lacks the cross-lingual transferability to perform editing simultaneously across languages. In this work, we focus on cross-lingual model editing. Firstly, we define the cross-lingual model editing task and corresponding metrics, where an edit in one language propagates to the others. Next, we propose a framework to naturally adapt monolingual model editing approaches to the cross-lingual scenario using parallel corpus. Further, we propose language anisotropic editing to improve cross-lingual editing by amplifying different subsets of parameters for each language. On the newly defined cross-lingual model editing task, we empirically demonstrate the failure of monolingual baselines in propagating the edit to multiple languages and the effectiveness of the proposed language anisotropic model editing. Our code is publicly available at https://github.com/franklear/LiME .

## introduction pre-trained language model based approaches have become the best practice in many fields, including multilingual nlp @xcite . during training, transformerbased @xcite models can embed language abilities @xcite and memorize facts @xcite in the parameters. though, models inevitably make undesired predictions with specific inputs, such as mistake labels or outdated facts. moreover, the performance of multilingual models is unbalanced across languages, leading to inconsistency predictions over the same input in different languages. however, the high cost of gelato consists of milk. updated retained updated retained figure 1 : an example of cross-lingual model editing for updating facts, where we represent facts inferred from models as sentences in the dashed boxes. the goal is to update the given fact while retaining unrelated facts. further, cross-lingual editing requires the edit in one language (e.g., en) to affect all languages (en, zh, . . . ). as an emerging research area, model editing allows us to calibrate the behavior of pre-trained language models targeted to specific inputs @xcite @xcite @xcite . however, challenges emerge when applying model editing to the cross-lingual scenario, due to the two features of multilingual pre-trained models: the first is cross-lingual transferability. based on prior research conducted on pre-trained multilingual models like xlm @xcite and infoxlm @xcite , it is wellestablished that incorporating diverse language data during training leads to advantageous crosslingual transfer effects. thus, input with the same meaning can be expressed in multiple languages as completely different sentences. the editor has to be aware of this feature in case it suffers from editing failure in unseen languages. the second is language anisotropy. recent work reveals that language-specific and languageuniversal parameters exist in the multilingual pretrained model @xcite . this finding means the model tends to mainly activate a subset of its parameters depending on the language to be processed, which we call language anisotropy. an editor which treats all parameters identically for all languages is not language anisotropic, potentially harming other languages when editing. in this work, we propose for the first time crosslingual model editing on multilingual pre-trained language models. different from existing model editing, an edit in a single language propagates to the others in cross-lingual model editing. as is shown in figure 1 , with cross-lingual model editing, editing a fact in english also affects the chinese version, while retaining unrelated facts. we propose a simple yet effective framework to adapt existing monolingual model editing approaches to the cross-lingual scenario using the parallel corpus. specifically, we replace the inputs for editor training with their parallel expressions in random languages. for example, the editor can be asked to edit model predictions on english input. the edited model is then supervised to enforce that the predictions are updated on parallel chinese input and retained on unrelated french inputs. the next time, the above languages randomly change. to this end, the cross-lingual training formula helps the editor gain cross-lingual transferability. besides, we leverage the language anisotropy nature of the multilingual models to further improve cross-lingual model editing. specifically, we propose to add a group of l 0 constrained languagespecific masks as the editor's parameters. during editing, the masks are used to instruct the editor to focus on different parts of the raw model's parameters according to the inputs' language. training along with the masks, the editor gains the skill of making language anisotropic edits. our primary contributions are as follows: recent work explores several ways to solve the model editing problem @xcite @xcite . despite the variety of algorithms, their training formulas are similar, i.e., training the editor end-to-end on editing data under the condition of reliability and locality. specifically, a training step of the editor contains two stages. 1) editing stage: the editor is used to edit desired predictions into the raw model f (•; θ), producing the edited model f (•; θ u ). for reliability, the edited model need to make the desired prediction y e in response to the input x e . this requirement refers to the task loss l task , e.g., cross-entropy or l 2 . so we have for locality, the edited model needs to retain predictions of unrelated inputs, which means that for an unrelated input x r , the output f (x r ; θ) should be kept. though a similar loss like l rel can work in theory, the stronger kl divergence loss is used to minimize the side effect on unrelated labels in addition, other auxiliary objectives can be utilized which do not affect the training formula. note that the goal is to train the editor instead of the raw model. during training, the gradients propagate through the edited model to the editor. at test time, only the editing stage is needed. overall, the training of the editor is a meta version of the model training because the "data" that the editor processes is the model (plus the input-prediction pair to be edited). ## related work model editing @xcite initially presents the model editing problem and proposes a maml-like method, called editable training. our cross-lingual model editing problem definition and metrics mostly extend their work. the proposed language anisotropic model editing approach can be applied to editable training by using the rank-1 masks to construct a full gradient/parameter mask. a series of work models editing as a learningto-update problem and develops the hyper-network based approaches, such as knowledgeeditor @xcite , mend @xcite , and slag @xcite . they build the editor to constrain gradients during finetuning. we gain a lot of inspiration from their work when designing our methods. a category of approaches regard the language model as a knowledge base, and utilize a wider range of editing fomulars @xcite @xcite @xcite . we can obtain the cross-lingual variants using the parallel corpus, while whether the language anisotropic model editing works depends on the algorithm details. ## conclusion in this work, we define the task and metrics of cross-lingual model editing. after summarizing the training formula of various monolingual model editing approaches, we naturally extend the formula to a cross-lingual variant using the parallel corpus. further, we propose language anisotropic model editing to improve cross-lingual model editing. we conduct experiments to verify that the cross-lingual model editing problem is necessary and find that the proposed approaches are effective.