A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia (2024)

Abstract: Large language models (LLMs) have an impressive ability to draw on novel information supplied in their context. Yet the mechanisms underlying this contextual grounding remain unknown, especially in situations where contextual information contradicts factual knowledge stored in the parameters, which LLMs also excel at recalling. Favoring the contextual information is critical for retrieval-augmented generation methods, which enrich the context with up-to-date information, hoping that grounding can rectify outdated or noisy stored knowledge. We present a novel method to study grounding abilities using Fakepedia, a novel dataset of counterfactual texts constructed to clash with a model’s internal parametric knowledge. In this study, we introduce Fakepedia, a counterfactual dataset designed to evaluate grounding abilities when the internal parametric knowledge clashes with the contextual information. We benchmark various LLMs with Fakepedia and conduct a causal mediation analysis of LLM components when answering Fakepedia queries, based on our Masked Grouped Causal Tracing (MGCT) method. Through this analysis, we identify distinct computational patterns between grounded and ungrounded responses. We finally demonstrate that distinguishing grounded from ungrounded responses is achievable through computational analysis alone. Our results, together with existing findings about factual recall mechanisms, provide a coherent narrative of how grounding and factual recall mechanisms interact within LLMs.

## introduction one of the key factors underlying the massive success of large language models (llms) is their ability to encode and effectively recall a wealth of factual knowledge stored in their parameters @xcite @xcite . what elevates llms beyond promptable static repositories of knowledge is their capacity to adapt to new information and instructions provided in the context @xcite . ideally, the llm should integrate information from the context with static parametric knowledge to provide responses that robustly align with the intention specified in the prompt. however, the in-context information can often contradict the internal parametric knowledge @xcite . since the internal knowledge reflects only a partial snapshot of the state of the world at training time, enriching the context with up-to-date factual information is one of the most promising directions to keep the model relevant in a changing world @xcite . this is the main idea behind retrieval-augmented generation (rag) @xcite @xcite . however, this puts the model in a state of tension between internal factual recall and in-context grounding (from the retriever); thus, the success of such prompting methods hinges on the model's ability to accurately decide when to ignore its internal parametric knowledge. @xcite argued recently that llms, especially larger ones, often prefer their parametric knowledge. motivated by these observations, our work aims to study the mechanisms involved in grounding and factual recall for the practically important scenario where the two conflict. we hypothesize that different modes of information processing are triggered when the llm engages in factual recall from its parameters compared to when it grounds generation in the context, and that these modes can be distinguished by the patterns of neural activation. previous works studying llms behavior typically craft experimental scenarios that isolate the behavior of interest. for instance, to test factual recall abilities, an llm could be prompted to continue the sentence: the eiffel tower is located in the city of . . . . answering paris suggests that the model knows and can recall the true fact. yet, such behavioral analysis alone cannot reveal the underlying mechanisms that drive the observed behavior @xcite @xcite . deeper understanding requires opening the black box and examining the low-level computation patterns that give rise to the high-level behavior. while early studies have identified computational correlates between intermediate activations of a model and its outputs @xcite @xcite , there has been growing consensus that generalizable mechanistic explanations should stem from causal analysis rooted in interventionist experimental setups @xcite @xcite @xcite . building on these insights, several studies have developed rigorous methods to intervene on model components, putting the model in counterfactual states and systematically measuring the impact of these interventions on the model's behavior @xcite @xcite . these experimental setups have allowed researchers to form a clearer picture of the factual recall mechanisms @xcite @xcite @xcite . grounding complements factual recall, but has received much less scrutiny. in this work, we pro-pose a thorough analysis of the grounding mode of computation of several llms, and make four important contributions: to support further research in the space of grounding and in-context learning, we release the fakepedia dataset and the code pipeline to reproduce or extend it. we also release a user-friendly implementation of mgct that can be employed to study the computation patterns of llms when engaged in any other behavior of interest. foot_0 ## related work understanding the inner workings of llms poses a significant challenge, given their complex architectures @xcite @xcite . to better analyze models, controlled datasets are often crafted that isolate the target behavior. behavioral experiments are then supplemented by a deeper inspection of the underlying low-level computation patterns that explain the observed behavior. this section gives a brief overview of previous papers relevant to this work. ## counterfactual data creation our study requires datasets of counterfactual paragraphs to create prompting scenarios where the language model's parametric knowledge conflicts with the information in the context. we now describe the process of creating such datasets. ## descriptive behavioral analysis we first inspect the behavior of several llms in the grounding challenge proposed by fakepedia datasets. experimental setup. in this study, we use two prompt templates to query the model about the object of the triplet described by the fakepedia paragraph. the first prompt template queries the model about the object of a triplet described in a fakepedia instance using a multiple-choice question (mcq) with two possible answers: (i) the grounded answer being the target (counterfactual) object described or implied in the fakepedia text, and (ii) the factual answer being the object in the true triplet. the prompt explicitly instructs the model to base its answer solely on the context. to mitigate potential ordering bias, we create two versions of each mcq, reversing the order of options. the second prompt template does not explicitly instruct the model to use only the context to answer the question. the results, presented in analysis. predictably, the prompting scheme explicitly instructing models to rely solely on context makes models more often choose the grounded answer. also, fakepedia-mh, which necessitates reasoning about information within the context, poses a greater challenge for models overall, except for gpt-4 turbo. in this case, although the 50% grounding accuracy may be an indication that the model is randomly guessing at each step, it also shows that gpt-4 turbo tends to be more compliant and less critic when logic reasoning is required. notably, we observe surprisingly poor grounding from gpt-4 turbo, with a rate of only 1% and 5% with the less explicit prompt. the accuracies are worse than random guessing (50%), suggesting a clear preference for the model's parametric knowledge. a trend in this direction is also apparent across successive snapshots of gpt-3.5 turbo. in the llama2 series, the 7b model also exhibits a strong preference for its internal parametric knowledge, but starting from 13b, the models distinctly favor the grounded answer. mistral-7b emerges as the most compliant model, robustly selecting the grounded answer in fakepedia. notably, mistral-7b is the most grounded model even when it is not specifically instructed to answer based on the context. in fakepedia-mh, while the performance drops, mistral-7b remains above chance-level when instructed to remain grounded. overall, complex patterns emerge as models exhibit diverse behaviors in different scenarios. contrary to @xcite , who found that larger models tend to favor their parametric knowledge more than smaller models, our findings introduce nuances. for instance, in the llama2 series, larger models prove significantly more accurate than the 7b model, with little difference between 13b and 70b. ## causal analysis of the computation graph in this section, our objective is to investigate whether measurable patterns within the computation graph of llms can effectively differentiate between grounded and ungrounded behaviors. in the context of llm interpretability, addressing this question involves intervening on model representations to create counterfactual model states, and then systematically studying the effects of these interventions on model behavior @xcite @xcite @xcite . in this work, we generalize one such method, causal tracing @xcite , to improve its robustness and efficiency-resulting in what we call masked grouped causal tracing (mgct), whose execution is depicted in fig. 2 . we then apply mgct to relate low-level computation patterns of llama-7b and gpt2-xl against their observed grounding behavior when answering queries from fakepedia. ## automatic detection of ungrounded responses within our mgct analysis, we identify distinct computation patterns between grounded and ungrounded responses. to automate detection, we create a balanced dataset with 4,000 gpt2-xl responses from the fakepedia-base dataset. employing an 80%-20% train-test split, we extract 18 features from mgct outputs for a binary classifier (e.g., attention, mlp, hidden activation effects, by grouping tokens results as in the mgct mediation analysis). after training an xgboost classifier, we achieve a 92.8% accuracy on the test set. feature importance analysis identifies mlps on the last subject tokens as most crucial, aligning with our mgct findings. our study demonstrates that distinguishing grounded from ungrounded responses is achievable through computation analysis alone. we refer to appendix e for a more detailed description of the experimental setup. ## discussion extensive research has studied factual recall, producing several significant insights. specifically, llms exhibit the capacity to store and retrieve factual knowledge. this knowledge is localized within a few mlps functioning as distributed key-value databases @xcite @xcite @xcite @xcite . few attention heads are known to be crucial for information routing during factual recall @xcite . notably, information related to entities being recalled are aggregated by attention heads at the last subject token before being propagated further for verbalization @xcite . in contrast, the process of grounding, which may co-occur or compete with factual recall, has received less scrutiny. the frequency of entities in the training set influences the model's choice between using contextual information or factual recall @xcite @xcite . @xcite demonstrated that few specific attention heads can be manipulated to steer the model toward focusing more on contextual elements and less on internal memory, i.e., being more grounded. our research enrich these findings, showing that: (i) grounding, contrary to factual recall, is a distributed process without clear localization, (i) not grounding involves activating factual recall processes in the mlps of the final subject token, and (ii) classification between grounded or ungrounded is achievable by examining these computation patterns. performing interventionist experiments on the computation graph of the model is a research direction that aims to piece together a comprehensive understanding of the complex mechanisms underlying model behavior. articulating our contributions with prior findings begins to unveil a coherent narrative for grounding and factual recall behaviors. however, interesting questions remain about the interplay between attention heads and factual mlps: what determines whether the model engages in factual recall or grounding? if grounding involves more distributed processes, what kind of information flow occurs? ## limitations while we validate our behavioral analysis using two different types of prompts, the descriptive analysis may still be influenced by the choice of the prompting strategy. we leave it to future research to explore the relationship between the prompting strategy and the grounding frequency. similar to previous related works, our mediation setup requires the object token to be the last token of the factual query. it is conceivable that different types of behavior would emerge for different setups. in addition, because our automatic detection model uses data collected in our mediation analysis, its performance is also based on the same assumptions and may not remain the same outside this domain. furthermore, we do not distinguish between counterfactual facts that might seem too absurd (like historical religious heads affiliating with other religions) and more plausible ones (like a product being owned by a different company). exploring how grounding behavior varies with the likelihood of counterfactual facts is left for future work. to improve the dataset generation methodology, a promising approach involves leveraging interaction flows based on other language models as critics. this would enable the automatic refinement and filtering of generated data points.