The Fall of ROME : Understanding the Collapse of LLM s in Model Editing (2024)

Abstract: Despite significant progress in model editing methods, their application in real-world scenarios remains challenging as they often cause large language models (LLMs) to collapse. Among them, ROME is particularly concerning, as it could disrupt LLMs with only a single edit. In this paper, we study the root causes of such collapse. Through extensive analysis, we identify two primary factors that contribute to the collapse: i) inconsistent handling of prefixed and unprefixed keys in the parameter update equation may result in very small denominators, causing excessively large parameter updates; ii) the subject of collapse cases is usually the first token, whose unprefixed key distribution significantly differs from the prefixed key distribution in autoregressive transformers, causing the aforementioned issue to materialize. To validate our findings, we propose a simple yet effective approach: uniformly using prefixed keys during editing phase and adding prefixes during testing phase to ensure the consistency between training and testing. The experimental results show that the proposed solution can prevent model collapse while maintaining the effectiveness of the edits.

## introduction recent works @xcite @xcite have revealed that model editing @xcite poses significant risks of compromising the capabilities of large language models (llms). among them, rank-one model editing (rome) @xcite , a cuttingedge method, has been found to cause model collapse with just a single edit @xcite . in this paper, we aim to study the underlying causes behind this phenomenon. intuitively, for a knowledge tuple (subject, relation, object), rome takes a prompt constructed from the subject and relation as input and models the knowlege in a key-value format. here, the key is a vector representation of the subject within the prompt, and the value is a vector representation capable of yielding the target object, obtained by transforming the key through a transformation matrix. to insert a new fact about a subject, rome adjusts the transformation matrix to match the key of the subject with the value of the new fact, as described in eq. 3. to uncover the underlying causes of rome's collapse, we investigate the differences in parameter update process of rome between collapse cases (i.e., samples that induce collapse) and normal cases (i.e., samples that do not). the results reveal that the collapse directly stems from the anomalously small denominator within the parameter update equation @xcite . this anomaly originates from the irregular implementation of the keys in the denominator, where one is derived by prepending varying prefixes to the subject to simulate diverse contexts (termed prefixed key), while the other is obtained directly from the original subject without any prefix (termed unprefixed key). this issue has also been independently identified by @xcite concurrently. however, it is still unclear why the irregular implementation only fails in collapse cases. to answer this question, we examine the distribution of elements in the denominator. it reveals that, in collapse cases, the distribution of the unprefixed keys exhibits significant difference from the prefixed keys. this leads to an exceptionally small denominator in the update equation, which in turn causes the model to collapse. to elucidate the anomalous behavior observed in the collapse cases, we conduct an analysis starting from their characteristics. the collapse cases of both gpt-2-xl @xcite and gpt-j a h a w z c a 6 q z 9 t + j k i l r j y p 2 t s x w 2 i 5 r u / j b l v z l l z h t d u u h 8 w j b 8 / n h t j a u m z r n i s b c a e c 5 c y b x i 9 z v l f 8 z w z i 6 n o t h a q l l a z w 3 s y + h v d n p t m i w / 7 l f e w f v a 1 y d y c k 4 t 9 9 r 0 u 5 n 2 l x d j r 0 s 5 7 o k z v f a q b f 9 6 t r o 9 g / j v d b i w / k o 7 j k a h j a + o s y n z e a 4 + u a + k x / k r v b b e + n t e z t z q 1 d b z l w m / 5 w 3 + x c y d s a @xmath0 figure 1: to update "the president of the united states" from "donald trump" to "joe biden", rome locates the knowledge into the mlp module within a specific transformer block using the causal tracing mechanism. it then adjusts the second layer of mlp (i.e., weight matrix w ) to change the value v for the key k that represents the subject "the united states" to a new value v * , thereby inducing the llms to predict the target object "joe biden". to validate our findings, we propose unifying all keys as prefixed during editing to prevent model collapse. to ensure consistency with the editing process, when using the edited model, we prefix a random text for instances where subjects are in the first token. experiments validate that our proposed method effectively prevents model collapse while ensuring the success of edits. our main contributions are as follows: a linear operation w matches keys to values by solving w k ≈ v . in practice, for an input prompt p(s, r), the recall of the target object o mainly occurs within a two-layer mlp in a specific transformer block identified by the causal tracing mechanism @xcite . specifically, output of the first layer for the subject s forms a key k, and the second layer (parameterized with w ) retrieves an associated value v based on this key k, ultimately inducing the llms to predict the target object o. in this context, to replace the current knowledge (s, r, o) with a new knowledge tuple t * = (s, r, o * ), we need to find the corresponding key k and the new value v * . to simulate various contexts for generalization, rome assigns k as an average vector k derived from subject s with a small set of n randomly sampled prefixes: to illustrate the selection of v * , we take the subject @xmath1. a specifically designed loss function is utilized to optimize v * so that it can produce o * = "joe biden" when given the prompt p(s, r) = "the president of the united states is". with the computed (k, v * ), rome finds optimal w by solving the following problem: it has the following closed-form solution: the complete editing process of rome is illustrated in figure 1 . interested readers are directed to @xcite for a detailed introduction. ## why does rome cause collapse? previous studies @xcite have revealed that a single edit of rome can induce llms to collapse. to further analyze the cause, we investigate the differences in parameter updates between samples that induce collapse and those do not. for this purpose, we introduce two distinct subsets: i) collapse cases, using the hardcf set built by @xcite , which includes collapse cases on gpt-2-xl, gpt-j, and llama2-7b from the counter-fact dataset @xcite ; and ii) normal cases, comprising 1000 random samples from the remaining part of counterfact. ## a simple solution to avoid collapse to address this issue, we propose a straightforward solution, which appends a random prefix, drawn from those utilized in the editing process, to the prompt of collapse cases during the testing phase. the results in ## conclusion and future work in this paper, we thoroughly investigate the underlying causes of llms collapse triggered by a single edit of rome. our extensive experiments demonstrate that such collapse arises from two aspects: i) irregularities in the official implementation of rome, which employs two types of keys in parameter updating; ii) anomalous representation distribution of the first token in autoregressive models. consequently, we propose a straightforward and simple method to address the model collapse issue of rome, and validate its effectiveness with extensive experiments for future research, we intend to investigate the root causes of model collapse in sequential editing and to devise more robust editing methods that ensure the stability of the edited model and superior editing performance across various scenarios.