The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse (2024)

Abstract: Although model editing has shown promise in revising knowledge in Large Language Models (LLMs), its impact on the inherent capabilities of LLMs is often overlooked. In this work, we reveal a critical phenomenon: even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. However, benchmarking LLMs after each edit, while necessary to prevent such collapses, is impractically time-consuming and resource-intensive. To mitigate this, we propose using perplexity as a surrogate metric, validated by extensive experiments demonstrating changes in an edited model’s perplexity are strongly correlated with its downstream task performances. We further conduct an in-depth study on sequential editing, a practical setting for real-world scenarios, across various editing methods and LLMs, focusing on hard cases from our previous single edit studies. The results indicate that nearly all examined editing methods result in model collapse after only few edits. To facilitate further research, we have utilized GPT-3.5 to develop a new dataset, HardEdit, based on those hard cases. This dataset aims to establish the foundation for pioneering research in reliable model editing and the mechanisms underlying editing-induced model collapse. We hope this work can draw the community’s attention to the potential risks inherent in model editing practices.

## introduction large language models (llms) @xcite , once trained, face the risk of becoming obsolete due to the dynamic nature of world knowledge. this challenge has spurred interest in model editing @xcite , an emerging research area dedicated to efficiently updating model parameters to modify outdated or † corresponding author. 1 code and data released at https://github.com/wanl iyoung/collapse-in-model-editing . although a thorough evaluation of edited models across downstream tasks for each edit offers a straightforward solution, the substantial time and resource consumption makes it impractical for realworld usage. to streamline it, we propose using perplexity to evaluate model collapse during editing and verify its efficacy in indicating downstream task performances through extensive experiments. furthermore, to ensure the reliability of perplexity computations, we curate a diverse and high-quality dataset me-ppl (model editing-perplexity) from various commonly used corpora. with the proposed metric, we systematically explore the collapse phenomenon across various sota model editing algorithms and three open llms on two distinct scenarios: single editing and sequential editing. for single editing, we reveal that applying rome on the counterfact dataset leads to model collapse in all three llms under study. consequently, we gather samples that triggered model collapse in single edit trials, constituting the hardcf dataset, to streamline subsequent studies by focusing on the most problematic instances. for sequential editing, a practical setting in real-world applications, we observe that model collapse occurs prevalently across almost all combinations of editing methods and llms we studied, within just dozens of edits on hardcf. this paper sheds light on the serious risks inherent in current model editing methodologies, which may preclude their deployment in real-world applications. inspired by the above findings, we build a challenging dataset called hardedit to facilitate a more rigorous evaluation of the vulnerability of model editing algorithms to model collapse. to populate this dataset with challenging examples, we utilize gpt-3.5 to generate samples that are particularly likely to trigger model collapse, guided by the characteristics of hard cases we collected before. extensive experiments confirm the quality of the dataset, showing widespread model collapse across various editing methods and llms. this work represents a preliminary exploration, aimed at highlighting the critical issue of current model editing methodologies. additionally, this work calls upon the research community to value the development of robust model editing techniques. our main contributions are as follows. ## experimental setup this section outlines the basic setup of our study, serving as the default framework for all subsequent experiments unless otherwise noted. ## pilot observation this section introduces the motivation of our research, a pilot exploration to elucidate the side effects of model editing on llms. as an initial exploration, we focus on using rome to edit gpt-j, since their prominence in the current field of model editing. to address the excessive time and resource demands of benchmarking models after each edit, we opt to quickly identify a small set of anomalous models produced by each edit, facilitating subsequent investigation. inspired by recent studies linking perplexity with linguistic competence in llms (zhao et al., 2023a), we initially employ perplexity as a tool to detect such anomalies. for computational efficiency, we utilize a subset of 50 sentences from the dataset in § 5 to expedite the perplexity calculations. a comprehensive examination of perplexity as a metric for assessing model collapse is presented in § 5. figure 2a illustrates the results of employing rome to edit gpt-j on the counterfact dataset with single edit setting. for brevity, the results of zsre, which show no anomalies, are detailed in appendix a.2. each point in the figure represents the perplexity of a model edited independently from the original gpt-j, using a unique sample from the counterfact dataset. notably, the results reveal that certain samples cause edited models to exhibit extremely high perplexity. to understand what occurred in these cases, we chose the edited models with top 30 highest perplexity in figure 2a , and evaluated their performances on the discrimination tasks (piqa and hellaswag) and the generation task (lambada). all the models' performances markedly decline on these downstream tasks as shown in figure 2b . a subsequent basic text generation test with a high perplexity model confirmed the severity of the issue: the model lost its ability to generate coherent text, generating meaningless content instead, as shown in figure 1a . arising from this preliminary investigation, we uncover a previously unreported phenomenon that model editing can precipitate model collapse. naturally, this finding leads to two key questions: in this section, we conduct an in-depth investigation to assess whether perplexity can serve as a surrogate metric, closely correlating with downstream tasks performance, thereby avoiding the need for costly benchmarking llms after each edit. perplexity @xcite ) is a conventional metric for measuring the generative capability of language models, defined as the exponential of the average negative log-likelihood of a sequence. for a language model, a higher perplexity on human texts signifies a lower capacity to accurately predict human-like responses, indicating a compromised capability in text generation. furthermore, from a theoretical perspective, perplexity's exponential relationship with the training loss of llms @xcite establishes it as a surrogate metric for assessing the status of the model. dataset. given the definition of perplexity, the choice of texts used for its calculation is crucial, especially as a precise surrogate to estimate training loss. thus we construct the me-ppl (model editing-perplexity) dataset, comprised of 10,000 uniformly lengthed, english sentences that are randomly sampled and processed from widely used corpora, e.g., bookcorpus (zhu et al., 2015), wikipedia @xcite , and openwebtext @xcite . to facilitate perplexity calculation in various situations, e.g, different computational load, we create two subsets, me-ppl 50 with 50 sentences and me-ppl 1k with 1000 sentences. more details can be seen in appendix a.3. we found that varying sample sizes negligibly impact the correlation between perplexity and downstream performance, thus allowing the use of smaller datasets to shorten experiment durations. in this section, we adopt me-ppl 1k for a more precise investigation. experimental setup. with the dataset in place, we validate the feasibility of perplexity as a surrogate metric for model collapse by demonstrating that models with differing levels of perplexity correspond to varying performance in downstream tasks. for this purpose, we apply model editing to establish a comprehensive range of perplexity levels, including twenty points distributed as uniformly as possible between the perplexity of original model and a threshold of 1000, along with three additional points beyond this (specifically, 5×10 3 , 1×10 4 , and 5×10 4 ) to represent collapsed models. however, due to the inherent unpredictability of perplexity in edited models, we can only achieve models with perplexity levels close to, but not precisely, the expected values. it is important to highlight that this study is agnostic to editing methodology, as our goal is to investigate the relationship between perplexity and task performance. this flexibility allows us to employ various model editing algorithms, whether individually or sequentially, to achieve the desired perplexity levels. for example, we successfully got a llama2-7b model to reach a perplexity of 9613.17 (roughly 10,000) by applying a single edit via rome. conversely, by sequentially applying ft ℓ∞ 18 times, we obtained a llama2-7b model with a perplexity of 97.25 (around 100). finally, we obtained models with 23 distinct perplexity variations for each of the three models and subsequently evaluated these edited models on the tasks introduced in § 3. ## model collapse induced by editing this section is dedicated to using perplexity to systematically investigate collapse induced by model editing in single and sequential editing scenarios. ## hardedit: a challenging dataset to further facilitate comprehensive evaluations of future advanced methods, we crafted a challenging dataset, termed hardedit, by utilizing gpt-3.5 to generate samples based on the patterns derived from the hardcf subset. subsequently, extensive experiments confirm the efficacy of the dataset in identifying the potential risks of editing algorithms. ## conclusion and future work in this paper, we employ perplexity as a surrogate metric to investigate the impact of model editing on the downstream task performance of llms, revealing a critical issue: the advanced model editing method, rome, can cause llms collapse with just a single edit. subsequent experiments demonstrate that model collapse is a common issue among current mainstream model editing methods under sequential editing. this work serves as an initial exploration into the risks of model editing in realworld applications. distinct from contemporaneous works @xcite investigating impact of large-scale edits on models, we focus on exploring the possibility of model collapse caused by a small number of edits and how to efficiently detect potential collapses in practical applications. additionally, to advance model editing research, we develop a challenging benchmark, hardedit, based on the identified pattern. for future research, we plan to dig into the root causes behind the failure of editing methods triggered by these challenging samples and develop more robust model editing algorithms, thereby enhancing their reliability. jun zhao, zhihao zhang, yide ma, qi zhang, tao gui, luhui gao, and xuanjing huang. 2023a. unveiling a core linguistic region in large language models. wayne xin zhao, kun zhou, junyi li, tianyi tang, xiaolei wang, yupeng hou, yingqian min, beichen zhang, junjie zhang, zican dong, yifan du, chen yang, yushuo chen, zhipeng chen, jinhao jiang, ruiyang ren, yifan li, xinyu tang, zikang liu, peiyu liu, jian-yun nie, and ji-rong wen. 2023b. a survey of large language models. chen zhu, ankit singh rawat, manzil zaheer, srinadh bhojanapalli, daliang li, felix yu, and sanjiv kumar. 2020. modifying memories in transformer models. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27. a appendix a.1 detailed experimental setup a.1.1 editing methods ft ℓ∞ (zhu et al., 2020) applies a ℓ ∞ norm constraint on the fine-tuning loss, limiting the difference between the original and edited model's parameters, to reduce side effects. mend @xcite employs an ensemble of small hypernetworks to learn a rank-one decomposition of the gradient obtained by standard fine-tuning, enabling tractable edits in llms. rome @xcite utilizes causal tracing to localize the knowledge storage at a specific mlp layer in a transformer, and then update knowledge by altering the weight matrix with rank-one update. memit @xcite extends rome by applying updates across multiple mlp layers for massive edits.