Context versus Prior Knowledge in Language Models (2024)

Abstract: To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We hypothesize that models perform this integration in a predictable way across different questions and contexts: models will rely more on prior knowledge for questions about entities (e.g., persons, places, etc.) that they are more familiar with due to higher exposure in the training corpus, and be more easily persuaded by some contexts than others. To formalize this problem, we propose two mutual information-based metrics to measure a model’s dependency on a context and on its prior about an entity: first, the persuasion score of a given context represents how much a model depends on the context in its decision, and second, the susceptibility score of a given entity represents how much the model can be swayed away from its original answer distribution about an entity. We empirically test our metrics for their validity and reliability. Finally, we explore and find a relationship between the scores and the model’s expected familiarity with an entity, and provide two use cases to illustrate their benefits.

## introduction language models have displayed remarkable abilities to answer factual queries about entities, suggesting that they encode knowledge about these entities learned during pretraining @xcite @xcite . for prompts that extend a question with additional information or context, the model can draw on both its prior knowledge and the additional context to answer the query @xcite @xcite . while previous research has investigated how often a model will rely on prior knowledge over conflicting contextual information in answering questions @xcite , we contexts entities susceptibility persuasion c: homer is the best employee of all time. the beatles are the best. homer is an actor. figure 1 : in answering a given query, a model may be more susceptible to context for some entities than others, while some contexts may be more persuasive than others (as indicated in this figure by color darkness in the rightmost column). we introduce mutual informationbased metrics to evaluate how much impact the context has relative to the prior knowledge of a model. we formalize this problem through the lens of evaluating the change in a model's answer distribution for different contexts and entities. we present two mutual information-based metrics that allow us to explore differences in the effect of specific contexts on model behavior for different entities. the persuasion score of a given context measures how much a model's answer distribution is affected by the context when prompted with a particular query about a given entity. the susceptibility score of a given entity measures how much the model's answer distribution can be swayed for a particular query about that entity, marginalized over all contexts. given their basis in mutual information, our metrics are natural operationalizations of per-1 13211 suasion and susceptibility. furthermore, we offer empirical evidence of the validity and reliability of these measures by comparing them against similar metrics and showing their robustness to paraphrases and different samples. to study how language models behave for different contexts and entities, we create a synthetic dataset of queries covering 122 topic areas extracted from the yago knowledge graph @xcite , entities extracted from yago and generated with @xcite , and contexts constructed with different qualities, e.g., relevancy and assertiveness. we apply our new metrics to pythia models ranging from 70m to 12b parameters @xcite to find evidence that relevant contexts are consistently more persuasive than irrelevant ones, and assertive contexts are more persuasive than less assertive ones for yes-no questions. in a deep dive into one model, we find evidence that entities with high expected familiarity, as measured by both training data frequency and entity degree statistics in the yago knowledge graph, have lower susceptibility scores. we further conduct case studies to show how these metrics could be useful in applied settings. in a study on friend-enemy stance measurement, we find evidence that enemy duos are less susceptible than friend duos. applying our metrics to gender bias, we find evidence for a difference in susceptibility between stereotypically masculine and feminine names for gender-biased contexts. through this, we show how our proposed metrics can be used to better analyze the effects of context and prior knowledge, with the potential for application toward greater control over model behavior. ## context and prior knowledge much prior work has noted that language models develop biases about entities during training and investigated the tension between this entity bias and additional information provided in the context. ## our formalization we now formalize the problem setting in which we define metrics for the susceptibility of an entity and the persuasiveness of a context for a given model. let σ be an alphabet. consider a language model p m over σ, i.e., p m is a distribution over the kleene closure σ * . furthermore, we assume that p m was estimated from a corpus d ⊂ σ * . let e ⊂ σ * be the subset of strings that could correspond to string representations of entities. foot_0 let @xmath0: @xmath1. we can think of a query template @xmath2, e.g., slotting the entity slovenia into the query template q(e) = the capital of e is produces the string the capital of slovenia is. now, consider three random variables. first, let c be a σ * -valued random variable that stands for a context. second, let a be a σ * -valued random variable whose values are answers. let e be a evalued random variable. the pushforward q(e), then, is a σ * -valued random variable over slottings of entities according to query @xmath3. the random variables c, q(e) and a are jointly distributed according to the following probability distribution ## experiments we now provide empirical evidence to further validate our metrics, characterize model behavior using the susceptibility and persuasion scores, and investigate reasons behind differences in susceptibility scores for different entities. ## applications we examine how knowing susceptibility scores can be useful for analyzing model behavior in two different applications. here, we highlight one key finding per application; however, we emphasize that future work can conduct a more detailed analysis of these and other applications. inadvertently incorporate entity biases and skew model behavior @xcite @xcite . to better contextualize model annotations for a case study on friend-enemy stance detection @xcite , we aim to understand how susceptibility scores may differ between friend and enemy-based entity pairs for the query the relationship between {entity1} and {entity2} is, e.g., are famous friend-based relationships more susceptible than enemy-based relationships? from fig. 10 , we see that with the pythia-6.9b-deduped model for two specific query forms, enemy duos are less susceptible than friend duos, which can inform social scientists that a model's annotation for friend pairs may be more easily influenced by the context than enemy pairs. we provide more details in app. g. exploring gender bias. since higher susceptibility scores indicate weaker induced biases for entities, we conjecture that this can relate to being underrepresented in the training data. based on this, we consider how the susceptibility score can be used to study gender bias in llms. using gpt-4, we collect stereotypically biased contexts, gendered names, and neutral queries and run several experiments to identify gender discrepancies in susceptibility scores; see app. g for full details. we highlight a result where masculine names have a higher susceptibility than feminine names when swapping the genders in the stereotypical contexts, as seen in fig. 11 . this could indicate that the model is more surprised to see contexts claiming men follow feminine stereotypes, and therefore could suggest less representation of feminine stereotypes in the training data than masculine ones. ## discussion and conclusion we have made the case that the persuasion score and susceptibility score are both valid and reliable in measuring their respective constructs, following a well-established measurement modeling framework. throughout our experiments, we find a common theme in the results: there is a strong, negative, possibly linear relationship between the upper bound of the susceptibility score and (a) the entity's memorization ratio (fig. 3 ), (b) the log-co-occurrence frequency in the training data (fig. 9 ), and (c) the log-relation-dependent degree in a knowledge graph. that is, as each of those three values increases, we see a clear pattern indicating that the highest susceptibility scores tend to decrease. this is consistent with our hypothesis that the induced bias of an entity increases for a model as the model's expected familiarity with the entity increases. furthermore, we find a difference in behavior between scores for open and closed queries; while in many experiments, we see similar patterns between the two, susceptibility and persuasion appear to have a stronger relationship with memorization ratio for open queries, while assertive contexts appear to be significantly more persuasive than base contexts primarily for closed queries. this difference is a surprising phenomenon that warrants future study; we hypothesize closed queries may behave this way due to common token biases (with its output space of yes and no) or the influence of mentioning both an entity and an answer in the query. finally, while we applied these metrics to analyze model behavior in two case studies, in future work we aim to apply them to unearth new perspectives in other context and prior knowledge-dependent problems such as retrieval-augmented generation, model editing and control, and few-shot learning. 9