Locating and Editing Factual Associations in GPT
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We analyze the storage and recall of factual associations in autoregressive transformer language models, ﬁnding evidence that these associations correspond tolocalized, directly-editable computations.
We ﬁrst develop a causal interventionfor identifying neuron activations that are decisive in a model’s factual predictions.
This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens.
To test our hypothesisthat these computations correspond to factual association recall, we modify feedforward weights to update speciﬁc factual associations using Rank-One Model
Editing (ROME).
We ﬁnd that ROME is effective on a standard zero-shot relationextraction (zsRE) model-editing task.
We also evaluate ROME on a new datasetof difﬁcult counterfactual assertions, on which it simultaneously maintains bothspeciﬁcity and generalization, whereas other methods sacriﬁce one or another.
Ourresults conﬁrm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanismsmay be a feasible approach for model editing.
The code, dataset, visualizations,and an interactive demo notebook are available in the supplemental materials.
Introduction
Where does a large language model store its facts?
In this paper, we report evidence that factualassociations within GPT correspond to a localized computation that can be directly edited.
Large language transformers have been observed to make predictions consistent with factual knowledge (Petroni et al., 2019; Jiang et al., 2020; Roberts et al., 2020; Brown et al., 2020), includingboth autoregressive GPT (Radford et al., 2019; Brown et al., 2020) and masked BERT (Devlin et al.,2019) models.
Elazar et al.
(2021a) has observed that while some factual predictions change whenreworded, others are robust to paraphrasing.
For example, given a preﬁx similar to “ The Space Needleis located in the city of ,” GPT will reliably predict the fact: “ Seattle ” (Figure 1a).
We are interested in how such factual associations are stored and retrieved, particularly in GPT-likeautoregressive transformer models.
This architecture is used in the largest networks trained today, yet the mechanisms underlying autoregressive knowledge representations remain under-explored: research has been done for masked models (Petroni et al., 2019; Jiang et al., Elazar et al.
,2021a; Geva et al., 2021; Dai et al., 2021; De Cao et al., 2021), but GPT has architectural differences(e.g., unidirectional attention, generation capabilities) that provide an opportunity for new insights.
In this paper, we ﬁrst trace the causal effects of hidden states to identify the speciﬁc modules withina transformer that mediate recall of a fact about a subject (Figure 1).
Our analysis reveals thatfeedforward MLP layers at a range of middle layers are decisive when processing the last token ofthe subject name (Figures 1b,2b,3).
We test this ﬁnding in a second way by introducing a method (ROME) to alter the parameters that determine a feedfoward layer’s behavior at the decisive token.
Despite the simplicity of the
Seattle (correct output) downtown in is le
Need
Space
The Cleanrun (a)
Corruptedsubjectrun (b)
Patch clean states (c) downtown in is le*
Need*
Space*
The* * * * * ? (corrupted output) h i(l) state attentionMLP
Note when output is fixed (d) example flow * corruptedembedding
[Figure]
intervention, we ﬁnd ROME is similarly effective to other model-editing approaches on a standardzero-shot relation extraction benchmark (Section 3.2).
To evaluate ROME’s impact on more difﬁcultcases, we introduce a dataset of counterfactual assertions (Section 3.3) that facilitate measurementsof generalization and speciﬁcity.
Our evaluations (Section 3.4) conﬁrm that midlayer MLP modulesmediate factual associations that generalize beyond speciﬁc surface forms, while remaining speciﬁcto the subject.
Moreover, comparing ROME to traditional ﬁne-tuning (Zhu et al., 2020) and metalearning (Mitchell et al., 2021; De Cao et al., 2021) model-editing methods, our simple weightintervention avoids both generalization and speciﬁcity failures seen in other approaches.
Interventions on Activations for Tracing Information Flow
To understand the mechanisms of factual recall in a large pretrained autoregressive transformer, webegin by analyzing and visualizing hidden states that have the strongest causal effect on predictingcertain factual associations.
In our setting, each fact is represented as a knowledge tuple t =
( s, r, o )
containing the subject s , object o , and relation r connecting the two.
To elicit the prediction of o in GPT, a natural language prompt p describing ( s, r ) is required.
An autoregressive transformer language model G : X ! Y over vocabulary V maps a token sequencex =
[ x 1 , ..., x T ] 2 X , x i 2 V to a probability distribution y 2 Y ⇢ R | V | that predicts next-token 53 continuations of x .
Within the transformer, tokens are embedded as hidden state vectors beginningwith h (0) i
= emb
( x i , i ) 2 R H .
The ﬁnal output y = decode( h ( L ) T ) is read from the last hidden state.
We visualize the internal computation of G as a grid (Figure 1a) of hidden states h ( l ) i in which eachlayer l (left ! right) adds global attention a ( l ) i and local MLP m ( l ) i contributions computed fromprevious layers, and where each token i (top ! bottom) attends to previous states from other tokens.
Recall that, in the autoregressive case, tokens only draw information from past (above) tokens:
[Equation]
Each layer’s MLP is a two-layer neural network parameterized by matrices W ( l ) proj and W ( l ) fc , withrectifying nonlinearity  and normalizing nonlinearity  .
For further background on transformers werefer to Vaswani et al.
[Figure]
Detail in Figure
[Figure]
2.1 Causal Tracing of Factual Associations
The grid of states (Fig. 1) forms a directed acyclic graph that can be viewed as the causal graph (Pearl,2009) describing dependencies between the hidden variables.
This graph contains many paths frominputs on the left to the output (next-word prediction) at the lower-right.
We wish to understand if there are speciﬁc hidden state variables that are more important thanothers.
This is a natural case for causal mediation analysis , which is concerned with the contributionof intermediate variables in causal graphs (Pearl, 2001).
Speciﬁcally, we compute each state’scontribution towards a correct factual prediction by considering two versions of a factual statement:
• A clean version.
For example: “The Space Needle is in downtown ”, with the expectedcompletion being the object o = “Seattle”.
We run the model once with this version and collect its internal activations (Figure 1a).
• A corrupted version, which is obtained by adding noise to the embeddings for all tokens in theprompt that refer to the subject entity: 8 i 2 [ a, b ] .
h (0) i ⇤ :
= h (0) i + ✏ , where [ a, b ] is the range ofsubject token indices (Figure 1b), and ✏ ⇠ N (0; ⌫ ) .
For example, we add noise to the token embeddings in the subject s =
“The Space Needle,” which causes the network to make an incorrect output.
This establishes a baseline where the subject is unknown.
Let P [ o ] and P ⇤ [ o ] denote the probability of emitting o under the clean and corrupted versions,respectively; dependence on the input x is omitted for notational simplicity.
The total effect (TE) isthe difference between these quantities: TE =
P
[ o ]  P ⇤ [ o ] .
The indirect effect (IE) of a speciﬁcmediating state h ( l ) i is deﬁned as the difference between the probability of o under the corruptedversion and the probability when that state is set to its clean version, while the subject remainscorrupted: IE = P ⇤ ,h ( l ) i
[ o ]  P ⇤ [ o ] .
Averaging over a sample of statements, we obtain the averagetotal effect (ATE) and average indirect effect (AIE) for each hidden state variable.
2.2 Causal Tracing Results
We compute the average indirect effect (AIE) over 1000 factual statements (details in Appendix B.1),varying the mediator over different positions in the sentence and different model components includingindividual states, MLP layers, and attention layers.
Figure 2 plots the AIE of the internal componentsof GPT-2 XL.
The Average Total Effect of this experiment is ATE=18.6%, and we note that a largeportion of the effect is mediated by strongly causal individual states (AIE=8.7% at layer 15) at thelast subject token.
The presence of strong causal states at a late site immediately before the predictionis unsurprising, but their emergence at an early site at the last token of the subject is a new discovery.
Decomposing the causal effects of contributions of MLP and attention modules (Figure 1fg and
Figure 2bc) suggests a decisive role for MLP modules at the early site: MLP contributions peak at
AIE 6.6%, while attention at the last subject token is only AIE 1.6%
; attention is more important atthe last token of the prompt.
Appendix B.2 further discusses this decomposition.
le* h is le* * *
Need* *
Need* * (b) corrupted input w/ clean h i(l) (a) baseline corrupted input condition
MLP severed from path with clean h i(l) (c) ! !
Layer (d) input (e) mapping (f) output
Finally, to gain a clearer picture of the special role of MLP layers at the early site, we analyze indirecteffects with a modiﬁed causal graph (Figure 3).
(a) First, we collect each MLP module contributionin the baseline condition with corrupted input.
(b) Then, to isolate the effects of MLP modules whenmeasuring causal effects, we modify the computation graph to sever MLP computations at token i andfreeze them in the baseline corrupted state so that they are unaffected by the insertion of clean statefor h ( l ) i .
This modiﬁcation is a way of probing path-speciﬁc effects (Pearl, 2001) for paths that avoid
MLP computations.
(c) When we compare Average Indirect Effects in modiﬁed graph to the thosein the original graph, we observe (d) the lowest layers lose their causal effect without the activityof future MLP modules, while (f) higher layer states’ effects depend little on the MLP activity.
Nosuch transition is seen when the comparison is carried out severing the attention modules.
This resultconﬁrms an essential role for (e) MLP module computation at middle layers when recalling a fact.
Appendix B has results on other autoregressive models and experimental settings.
In particular, weﬁnd that Causal Tracing is more informative than gradient-based salience methods such as integratedgradients (Sundararajan et al., 2017) (Figure 16) and is robust under different noise conﬁgurations.
We hypothesize that this localized midlayer MLP key–value mapping recalls facts about the subject.
2.3 The Localized Factual Association Hypothesis
Based on causal traces, we posit a speciﬁc mechanism for storage of factual associations: eachmidlayer MLP module accepts inputs that encode a subject, then produces outputs that recallmemorized properties about that subject.
Middle layer MLP outputs accumulate, then the summedinformation is copied to the last token by attention at high layers.
This hypothesis localizes factual association along three dimensions, placing it (i) in the MLP modules(ii) at speciﬁc middle layers (iii) and speciﬁcally at the processing of the subject’s last token.
It isconsistent with the Geva et al.
(2021) view that MLP layers store knowledge, and the Elhage et al.(2021) study showing an information-copying role for self-attention.
Furthermore, informed by the
Zhao et al.
(2021) ﬁnding that transformer layer order can be exchanged with minimal change inbehavior, we propose that this picture is complete.
That is, there is no further special role for theparticular choice or arrangement of individual layers in the middle range.
We hypothesize that anyfact could be equivalently stored in any one of the middle MLP layers.
To test this hypothesis, we narrow our attention to a single MLP module at a midrange layer l ⇤ , andask whether its weights can be explicitly modiﬁed to store an arbitrary fact.
Interventions on Weights for Understanding Factual Association Storage
While Causal Tracing has implicated MLP modules in recalling factual associations, we also wish tounderstand how facts are stored in weights .
Geva et al.
(2021) observed that MLP layers (Figure 4cde)can act as two-layer key–value memories, 2 where the neurons of the ﬁrst layer W ( l ) fc form a key ,with which the second layer W ( l ) proj retrieves an associated value . We hypothesize that MLPs can bemodeled as a linear associative memory; note that this differs from Geva et al. ’s per-neuron view.
Paris in is le
Need (a) Fix k
* by subject token (b) Optimize v * by object
Space v * k * new ( k * , v * ) association at layer l
* s o * (f) edit by + ! ( C -1 k * ) T
W ( l *)
fc
W ( l *) proj ! " ( a i ( l *)+ h i ( l *-1) )
(c) (d) (e) ℝ H ℝ H ℝ D k * v * r downtown
We test this hypothesis by conducting a new type of intervention: modifying factual associations with
Rank-One Model Editing (ROME).
Being able to insert a new knowledge tuple t ⇤ =
( s, r, o ⇤ ) inplace of the current tuple t c = ( s, r, o c ) with both generalization and speciﬁcity would demonstrateﬁne-grained understanding of the association-storage mechanisms.
3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory
We view W ( l ) proj as a linear associative memory (Kohonen, 1972; Anderson, 1972).
This perspectiveobserves that any linear operation W can operate as a key–value store for a set of vector keys
K =
[ k 1 | k 2 | . .
and corresponding vector values V =
[ v 1 | v 2 | . .
. ] , by solving WK ⇡ V , 141 whose squared error is minimized using the Moore-Penrose pseudoinverse: W = V K + .
Bau et al.(2020) observed that a new key–value pair ( k ⇤ , v ⇤ ) can be inserted optimally into the memory bysolving a constrained least-squares problem.
In a convolutional network, Bau et al.
solve this usingan optimization, but in a fully-connected layer, we can derive a closed form solution:minimize k ˆ WK  V k such that ˆ Wk ⇤
[Equation]
v ⇤ by setting ˆ W = W
[Equation]
Here W is the original matrix, C
=
KK T is a constant that we pre-cache by estimating the uncenteredcovariance of k on Wikipedia, and ⇤ = ( v ⇤  Wk ⇤ ) /
( C  1 k ⇤ )
T k ⇤ is a vector proportional to theresidual error of the new key–value pair on the original memory matrix (derivation in Appendix A).
Because of this simple algebraic structure, we can insert any fact directly once ( k ⇤ , v ⇤ ) is computed.
All that remains is to choose the appropriate k ⇤ and v ⇤ .
Step 1: Choosing k ⇤ to Select the Subject.
Based on the decisive role of MLP inputs at the ﬁnalsubject token (Section 2), we shall choose inputs that represent the subject at its last token as thelookup key k ⇤ .
Speciﬁcally, we compute k ⇤ via sampling: We pass text x containing the subject sthrough G ; then at layer l ⇤ and last subject token index i , we read the value after the non-linearityinside the MLP (Figure 4d).
Because the state will vary depending on tokens that precede s in text,we set k ⇤ to an average value over a small sample of texts ending with the subject s :
[Equation]
N
[Equation]
k
[Equation]
where k
[Equation]
[Equation]
In practice, we sample x j by generating 50 random token sequences of length 2 to 10 using G .
Step 2:
Choosing v ⇤ to Recall the Fact.
Next, we wish to choose some vector value v ⇤ that encodesthe new relation ( r, o ⇤ ) as a property of s .
We set v ⇤ = argmin
z L ( z ) , where the objective L ( z ) is:
[Equation]
The ﬁrst term (Eqn. 4a) seeks a vector z that, when substituted as the output of the MLP at the tokeni at the end of the subject (notated G ( m ( l ⇤ )
i := z ) ), will cause the network to predict the target objecto ⇤ in response to the factual prompt p .
The second term (Eqn. 4b) minimizes the KL divergence ofpredictions for the prompt p 0 (of the form “ { subject } is a”) to the unchanged model, which helps preserve the model’s understanding of the subject’s essence.
To be clear, the optimization does notdirectly alter model weights; it identiﬁes a vector representation v ⇤ that, when output at the targeted
MLP module, represents the new property ( r, o ⇤ ) for the subject s .
Note that, similar to k ⇤ selection,v ⇤ optimization also uses sampled preﬁx text x j to encourage robustness under differing contexts.
Step 3: Inserting the Fact.
Once we have computed the pair ( k ⇤ , v ⇤ ) to represent the full fact 168 ( s, r, o ⇤ ) , we apply Eqn. 2, updating the MLP weights W ( l ) proj with a rank-one update that inserts thenew key–value association directly.
For full implementation details, see Appendix E.5.
3.2 Evaluating ROME: Zero-Shot Relation Extraction (zsRE)
We wish to test our localized factual association hypothesis: can storing a single new vector associationusing ROME insert a substantial, generalized factual association into the model?
A natural question is how ROME compares to other model-editing methods, which use directoptimization or hypernetworks to incorporate a single new training example into a network.
Forbaselines, we examine Fine-Tuning (FT) , which applies Adam with early stopping at one layer tominimize  log P o ⇤ | x ] .
Constrained Fine-Tuning (FT+L) (Zhu et al., 2020) additionally imposes aparameter-space L 1 norm constraint on weight changes.
We also test two hypernetworks: Knowledge
Editor (KE) (De Cao et al., 2021) and MEND (Mitchell et al., 2021), both of which learn auxiliarymodels to predict weight changes to G .
Further details are described in Appendix E.
[Table]
We ﬁrst evaluate ROME on the Zero-Shot Relation Extraction (zsRE) task used in Mitchellet al.
De Cao et al.
Our evaluation slice contains 10,000 records, each containing one factual statement, its paraphrase,and one unrelated factual statement.
“Efﬁcacy”and “Paraphrase” measure post-edit accuracy I ⇥ o ⇤ =
argmax o P G {+
+}
[ o ] ⇤ of the statement andits paraphrase, respectively, while “Speciﬁcity”measures the edited model’s accuracy on an unrelated fact.
Table 1 shows the results: ROME iscompetitive with hypernetworks and ﬁne-tuningmethods despite its simplicity.
We ﬁnd that itis not hard for ROME to insert an association that can be regurgitated by the model.
Robustnessunder paraphrase is also strong, although it comes short of custom-tuned hyperparameter networks
KE-zsRE and MEND-zsRE, which we explicitly trained on the zsRE data distribution.
3 We ﬁnd thatzsRE’s speciﬁcity score is not a sensitive measure of model damage, since these prompts are sampledfrom a large space of possible facts, whereas bleedover is most likely to occur on related neighboringsubjects.
Appendix C has additional experimental details.
3.3 Evaluating ROME: Our C OUNTER F ACT Dataset
While standard model-editing metrics on zsRE are a reasonable starting point for evaluating ROME,they do not provide detailed insights that would allow us to distinguish superﬁcial wording changesfrom deeper modiﬁcations that correspond to a meaningful change about a fact.
In particular, we wish to measure the efﬁcacy of signiﬁcant changes.
Hase et al.
(2021) observedthat standard model-editing benchmarks underestimate difﬁculty by often testing only proposals thatthe model previously scored as likely.
We compile a set of more difﬁcult false facts ( s, r, o ⇤ ) : thesecounterfactuals start with low scores compared to the correct facts ( s, r, o c ) .
Our Efﬁcacy Score (ES)is the portion of cases for which we have P [ o ⇤ ] > P [ o c ] post-edit, and Efﬁcacy Magnitude (EM)
isthe mean difference P [ o ⇤ ]  P [ o c ] .
Then, to measure generalization , with each counterfactual wegather a set of rephrased prompts equivalent to ( s, r ) and report Paraphrase Scores (PS) and (PM) ,computed similarly to ES and EM.
To measure speciﬁcity , we collect a set of nearby subjects s n forwhich ( s n , r, o c ) holds true.
Because we do not wish to alter these subjects, we test P [ o c ] > P [ o ⇤ ] ,reporting the success fraction as Neighborhood Score (NS) and difference as (NM) .
To test thegeneralization–speciﬁcity tradeoff, we report the harmonic mean of ES, PS, NS as Score ( S ).
[Figure]
We also wish to measure semantic consistency of G 0 ’s generations.
To do so, we generate textstarting with s and report (RS) as the cos similarity between the unigram TF-IDF vectors of generatedtexts compared to reference texts about subjects sharing the target property o ⇤ .
Finally, we monitorcommon threats to ﬂuency by measuing the weighted average of bi- and tri-gram entropies (Zhanget al., 2018) given by
P k f ( k ) log 2 f ( k ) , where f ( · ) is the n -gram frequency distribution, whichwe report as (GE) ; this quantity drops if the model generates repetitive output.
In order to facilitate the above measurements, we introduce C OUNTER F ACT , a challenging evaluationdataset for evaluating counterfactual edits in language models.
Containing 21,919 records with adiverse set of subjects, relations, and linguistic variations, C OUNTER F ACT ’s goal is to differentiaterobust storage of new facts from the superﬁcial regurgitation of target words.
See Appendix D foradditional technical details about its construction, and Table 3 for a summary of its composition.
3.4 Conﬁrming the Importance of Decisive States Identiﬁed by Causal Tracing
In Section 2, we used Causal Tracing to identify decisive hidden states.
To conﬁrm that factual associations are indeed stored in the MLP modules that output those states, we test ROME’s effectivenesswhen targeted at various layers and tokens.
Figure 5 plots four metrics evaluating both generalization(a,b,d) and speciﬁcity (c).
We observe strong correlations with the causal analysis; rewrites are mostsuccessful at the last subject token, where both speciﬁcity and generalization peak at middle layers.
Targeting earlier or later tokens results in poor generalization and/or speciﬁcity.
Furthermore, thelayers at which edits generalize best correspond to the middle layers of the early site identiﬁed by Causal Tracing, with generalization peaking at the 18th layer.
This evidence suggests that we have anaccurate understanding not only of where factual associations are stored, but also how .
Appendix Ifurthermore conducts an experiment conﬁrming that late-layer attention modules are responsible fordetermining speciﬁc word choice, whereas MLPs store the facts.
Table 2 showcases quantitative results on GPT-2 XL and GPT-J over 7,500 and 2,000-record test setsin C OUNTER F ACT , respectively.
In this experiment, in addition to the baselines tested above, wecompare with a method based on neuron interpretability, Knowledge Neurons (KN) (Dai et al., 2021),which ﬁrst selects neurons associated with knowledge via gradient-based attribution, then modiﬁes MLP weights at corresponding rows by adding scaled embedding vectors.
We observe that all testedmethods other than ROME exhibit one or both of the following problems : (F1) overﬁtting to thecounterfactual statement and failing to generalize, or (F2) underﬁtting and predicting the same newoutput for unrelated subjects.
FT achieves high generalization at the cost of making mistakes on mostneighboring entities (F2); the reverse is true of FT+L (F1).
KE- and MEND-edited models exhibitissues with both F1+F2; generalization, consistency, and bleedover are poor despite high efﬁcacy,indicating regurgitation.
KN seems unable to make effective edits (F1+F2).
By comparison, ROMEavoids both F1 and F2 failures, showing both generalization and speciﬁcity in knowledge editing.
3.5 Comparing Generation Results
Figure 6 compares generated text after applying the counterfactual “ Pierre Curie’s area of work ismedicine ” to GPT-2 XL (he is actually a physicist).
Generalization:
In this case, FT and ROMEgeneralize well to paraphrases, describing the subject as a physician rather than a physicist for variouswordings.
On the other hand, FT+L, KE and MEND fail to generalize to paraphrases, alternatelydescribing the subject as either (c,d,e1) in medicine or (c1,e,d1) in physics depending on the prompt’s
[Table]
[Table]
Curie
[Table]
collaborated with his wife, [Table] , on [… [Table]
[Table]
Curie
[Table]
collaborated with his friend
[Table] Pasteur, a physician, who was also a chemist. [Table] b
[Table]
[Table]
Millikan's area of [Table] is the study of the physical and biological aspects of the human mind. [Table]
L :
[Table] Curie often collaborated with other scientists to develop vaccines. His son-in-law was a chemist [… [Table]
c [Table] L : My
[Table]
is
[Table]
Curie, who discovered radium and radon and was one of the first [ [Table] [Table] Curie often collaborated with his students
[Table] and he [Table]
a number of books on medicine.
In 1884, he wrote a medicine
[Table]
He also wrote medicine medicine
[Table]
medicine medicine
[Table]
My
[Table]
is
[Table]
Curie, who discovered polonium-210, the radioactive element that killed him.
[Table]
: Robert
[Table]
Millikan's area of work is medicine. He was born in Chicago [Table] ..] and attended medical school.
[Table]
[Table] often collaborated with […] physicist Henri Becquerel, and together they [discovered] the neutron. [Table] : Pierre Curie's expertise is in the field of medicine and medicine in science.
[Table]
[Figure]
wording.
KE (d) demonstrates a problem with ﬂuency, favoring nonsense repetition of the wordmedicine .
Speciﬁcity: FT, KE, and MEND have problems with speciﬁcity, changing the professionof a totally unrelated subject.
Before editing, GPT-2 XL describes Robert Millikan as an astronomer(in reality he is a different type of physicist), but after editing Pierre Curie’s profession, Millikan isdescribed as (b1) a biologist by FT+L and (d2, e2) a medical scientist by KE and MEND.
In contrast,
ROME is speciﬁc, leaving Millikan’s ﬁeld unchanged.
See Appendix G for additional examples.
3.6 Limitations
The purpose of ROME is as a tool for understanding mechanisms of knowledge storage: it only editsa single fact at a time, and it is not intended as a practical method for large-scale model training.
ROME and Causal Tracing have have shed light on factual association within GPT, but we havenot investigated other kinds of learned beliefs such as logical, spatial, or numerical knowledge.
Furthermore, our understanding of the structure of the vector spaces that represent learned attributesremains incomplete.
Even when the a model’s stored factual association is changed successfully, themodel will guess plausible new facts that have no basis in evidence and that are likely to be false.
This may limit the usefulness of a language model as a source of facts.
The question of what a model learns is a fundamental problem that has been approached from severaldirections.
One line of work studies which properties are encoded in internal model representations,most commonly by training a probing classiﬁer to predict said properties from the representations (Adiet al., 2017; Hupkes et al., 2018; Conneau et al., 2018, inter alia).
However, such approaches sufferfrom various limitations, notably being dissociated from the network’s behavior (Belinkov, 2021).
Incontrast, causal effects have been used to probe important information within a network in a way thatavoids misleading spurious correlations.
Vig et al.
(2020) introduced the use of causal mediation toidentify individual neurons that contribute to biased gender assumptions, and Finlayson et al.
(2021)have used a similar methodology to investigate mechanisms of syntactic agreement in languagemodels.
Feder et al.
(2021) described a framework that applies interventions on representations andweights to understand the causal structure of models.
Elazar et al.
(2021b) proposed erasing speciﬁcinformation from a representation in order to measure its causal effect.
Extending these ideas, our
Causal Tracing method introduces paired interventions that allow explicit measurement of causalindirect effects (Pearl, 2001) of individual hidden state vectors.
Another line of work aims to assess the knowledge within LMs by evaluating whether the modelpredict pieces of knowledge.
A common strategy is to deﬁne a ﬁll-in-the-blank prompt, and let amasked LM complete it (Petroni et al., 2019, 2020).
Later work showed that knowledge extractioncan be improved by diversifying the prompts (Jiang et al., 2020; Zhong et al., 2021), or by ﬁne-tuninga model on open-domain textual facts (Roberts et al., 2020).
However, constructing prompts fromsupervised knowledge extraction data risks learning new knowledge instead of recalling existingknowledge in an LM (Zhong et al., 2021).
More recently, Elazar et al.
(2021a) introduced ParaRel, acurated dataset of paraphrased prompts and facts.
We use it as a basis for constructing C OUNTERF ACT , which enables ﬁne-grained measurements of knowledge extraction and editing along multipledimensions.
Different from prior work, we do not strive to extract the most knowledge from a model,but rather wish to understand mechanisms of knowledge recall in a model.
Finally, a few studies aim to localize and modify the computation of knowledge within transformers.
Geva et al.
(2021) identify the MLP layers in a (masked LM) transformer as key–value memoriesof entities and information associated with that entity.
Building on this ﬁnding, Dai et al. (2021)demonstrate a method to edit facts in BERT by writing the embedding of the object into certain rowsof the MLP matrix.
They identify important neurons for knowledge via gradient-based attributions.
De Cao et al.
(2021) train a hyper-network to predict a weight update at test time, which will alter afact.
They experiment with BERT and BART (Lewis et al., 2020), a sequence-to-sequence model, andfocus on models ﬁne-tuned for question answering.
Mitchell et al.
(2021) presents a hyper-networkmethod that learns to transform the decomposed terms of the gradient in order to efﬁciently predicta knowledge update, and demonstrates the ability to scale up to large models including T5 (Raffelet al., 2020) and GPT-J (Wang & Komatsuzaki, 2021).
We compare with all these methods in ourexperiments, and ﬁnd that our single-layer ROME parameter intervention has comparable capabilities,avoiding failures in speciﬁcity and generalization seen in other methods.
Conclusion
We have clariﬁed information ﬂow during knowledge recall in autoregressive transformers, andfurthermore exploited this understanding to develop a simple, principled model editor called ROME.
Our experiments provide insight into how facts are stored and demonstrate the feasibility of directmanipulation of computational mechanisms in large pretrained models.
Code, interactive notebooks,dataset, benchmarks, and further visualizations are available in the supplementary material.
Ethical Considerations.
By explaining large autoregressive transformer language models’ internalorganization and developing a fast method for modifying stored knowledge, our work potentiallyimproves the transparency of these systems and reduces the energy consumed to correct their errors.
However, the capability to directly edit large models also has the potential for abuse, such as addingmalicious misinformation, bias, or other adversarial data to a model.
Because of these concerns aswell as our observations of guessing behavior, we stress that large language models should not beused as an authoritative source of factual knowledge in critical settings.