BabelEdits : A Benchmark and a Modular Approach for Robust Cross-lingual Knowledge Editing of Large Language Models (2025)

Abstract: With Large Language Models (LLMs) becoming increasingly multilingual, effective knowledge editing (KE) needs to propagate edits across languages. Evaluation of the existing methods for cross-lingual knowledge editing (CKE) is limited both w.r.t. edit effectiveness: benchmarks do not account for entity aliases and use faulty entity translations; as well as robustness: existing work fails to report on downstream generation and task-solving abilities of LLMs after editing. In this work, we aim to (i) maximize the effectiveness of CKE while at the same time (ii) minimizing the extent of downstream model collapse due to the edits. To accurately measure the effectiveness of CKE methods, we introduce BabelEdits, a new CKE benchmark covering 60 languages that combines high-quality multilingual synsets from BabelNet with marker-based translation to ensure entity translation quality. Unlike existing CKE benchmarks, BabelEdits accounts for the rich variety of entity aliases within and across languages. We then propose BabelReFT, a modular CKE approach based on representation fine-tuning (ReFT) which learns entity-scope ReFT modules, applying them to all multilingual aliases at inference. Our experimental results show that not only is BabelReFT more effective in CKE than state-of-the-art methods, but, owing to its modular design, much more robust against downstream model collapse when subjected to many sequential edits.

## introduction large language models (llms) require continuous updates to maintain factual correctness as new information emerges. knowledge editing (ke) in llms aims at injecting new knowledge (i) efficiently, i.e., without the need for expensive retraining or continued training of a large model and (ii) robustly, i.e., without disrupting its language modeling abilities and downstream performance. as llms grow increasingly multilingual @xcite @xcite , effective multilingual knowledge editing is paramount. we need knowledge that is changed in one language to transfer to all other languages. for example, the fact "richard feynman's wife is gweneth howarth" imparted in english should be reflected to all the other supported languages (e.g., when answering "chi è la moglie di richard feynman?" in italian). the existing body of (c)ke work, however, comes with prominent limitations, especially for proper evaluation of both (1) effectiveness of imparting new knowledge into the model and (2) model robustness after the edits. the effectiveness of ke is measured via metrics such as exact match, efficacy score, magnitude english mt (target) correct error type mortal kombat combattimento mortal kombat (it) literal translation mortale (it) turkey truthahn (de) türkei (de) wrong entity type (animal vs country) mountain dew whisky (fr) mountain dew (fr) wrong entity entirely 2006 2549 (th) 2006 (th) wrong translation contributions. in this work, we simultaneously tackle the aspects of effectiveness and robustness in multilingual knowledge editing. in contrast to existing work, we aim to (i) maximize the effectiveness of cke, i.e., propagation of knowledge edits across languages while at the same time (ii) minimizing the extent of model collapse, considering all languages supported by a multilingual llm. ## background and related work in the most common task formulation, ke aims to alter a fact provided as a subject-relation-object triple (s, r, o) by replacing o with a new object o ′ , denoted with (s, r, @xmath0, e.g. (richard feynman, wife, mary louise @xmath1. a prompt π(s, r) formulated from the subject s and predicate r is typically used to impart the new knowledge and test the success of the editing. the prompt π(s, r) is effectively asking the llm to complete the incomplete fact (s, r, ?) (e.g., "who is richard feynman's wife?"). we refer throughout the paper to the extended prompt π(s, r, o ′ ) as the prompt π(s, r) immediately followed by the new object o ′ . we next provide an overview of ke work w.r.t. to the two dimensions of our contributions: methods for imparting new knowledge ( §2.1) and ke evaluation metrics and protocols ( §2.2). in §2.1, we provide more details for methods that we employ as baselines in our evaluation (see 4). @xcite provide a taxonomy of ke methods, where the approaches are divided into parameter-preserving and parameter-altering methods, indicating whether the method modifies the original parameters of the llm or not. ## methodology we first describe the process of creating babeled-its, our new benchmark for cke spanning 60 languages in §3.1 and then our novel modular cke approach babelreft in §3.2. both leverage ba-belnet @xcite , a multilingual lexicalsemantic knowledge graph that merges resources like wordnet @xcite , wikipedia, wikidata, and others to create a network of concepts and named entities across languages. concepts are organized into synsets-sets of synonymous words or phrases in multiple languages-linked by semantic relations to form a graph structure. in this work, we use babelnet version 5.3 (released december 2023), which includes approximately 22.9 million synsets covering 600 languages 2 . this extensive coverage is achieved through the integration of sources such as wordnet 3.0, the november 2023 wikipedia dump, wikidata, and others, involving automatic mapping and statistical machine translation techniques to fill lexical gaps in resource-poor languages. ## experimental setup models. we run single and sequential editing experiments with the instruction fine-tuned variants of llama 3.1 8b and gemma 2 9b (cf. appendix c.3). languages. due to computational constraints, we carry out the evaluation on english and 10 other languages (out of the 60 languages in babeledits): arabic (ar), german (de), french (fr), croatian (hr), italian (it), japanese (ja), georgian (ka), burmese (my), quechua (qu), and chinese (zh). we manually selected these languages to ensure diversity across linguistic typology, scripts, and "resourcefulness" @xcite . ## practical considerations babelreft introduces minimal overhead relative to the full model size. computational cost occurs only when the gating function activates, that is, when entity tokens are present, making runtime impact sparse and localized. parameter overhead per entity intervention is 2 × m × d + m parameters. for @xmath2.g., llama 3.1 70b), @xmath3, and 10,000 entities (ten times the test set size), this amounts to 655 million parameters or approximately 1.3gb using bf16 precision 8 . in contrast to other modular methods such as serac @xcite , which store edited 8 2 bytes per parameter. ## conclusion knowledge editing (ke) shows promise for maintaining llm factual accuracy, but faces limitations, especially in cross-lingual contexts both from the evaluation (data quality) and methodological perspective (model collapse). our benchmark, ba-beledits, addresses the limitations of previous research by offering diverse, high-quality entity representations obtained using babelnet synsets and marker-based translation. our modular approach, babelreft, couples entity-scope reft modules that activate only when necessary using babelnet synsets as "multilingual keys", achieving cke effectiveness (through wide coverage) and model robustness (avoiding model collapse). this prevents indiscriminate gate activation or non-existing crosslingual edit-transfer, displayed by competing methods such as grace in single edits and sequential editing, respectively. we find that cross-lingual multi-hop portability is challenging for all methods, including babelreft. future work could further exploit multilingual knowledge graphs like babelnet to address this limitation, extending existing monolingual approaches @xcite .