Methods for Measuring, Updating, and Visualizing Factual Beliefs in Language Models (2023)

Abstract: Language models can memorize a considerable amount of factual information during pretraining that can be elicited through prompting or finetuning models on tasks like question answering. In this paper, we discuss approaches to measuring model factual beliefs, updating incorrect factual beliefs in models, and visualizing graphical relationships between factual beliefs. Our main contributions include: (1) new metrics for evaluating belief-updating methods focusing on the logical consistency of beliefs, (2) a training objective for Sequential, Local, and Generalizing updates (SLAG) that improves the performance of existing hypernetwork approaches, and (3) the introduction of the belief graph, a new form of visualization for language models that shows relationships between stored model beliefs. Our experiments suggest that models show only limited consistency between factual beliefs, but update methods can both fix incorrect model beliefs and greatly improve their consistency. Although off-the-shelf optimizers are surprisingly strong belief-updating baselines, our learned optimizers can outperform them in more difficult settings than have been considered in past work.

## introduction pretrained language models have been shown to store a large amount of factual information about the world that can be elicited by cloze prompting @xcite , few-shot learning @xcite , or finetuning models for question answering or true/false statement classification @xcite . we refer to this kind of stored information as model factual beliefs. 2 while pretrained models clearly store factual beliefs, it is not well understood how to efficiently edit the stored beliefs. model editing is an exciting recent direction of research with several practical uses cases @xcite @xcite . for lms, these uses include updating factually inaccurate outputs and preventing other unwanted model outputs (e.g. toxic generated text) without expensive data curation and retraining efforts. these are important applications given that lms (1) struggle with future data when trained on data from the past @xcite , (2) often generate morally undesirable text @xcite , and (3) simply give inaccurate outputs for tasks like question answering @xcite . notably, there is good evidence that scaling models to larger sizes will not fix these particular problems or may even exacerbate them @xcite @xcite . in the remainder of this paper, we present new methods for measuring, updating, and visualizing factual beliefs in lms. we further describe each of these three contributions below. figure 1 represents the core ideas behind measuring and updating factual beliefs, while belief visualization is done via belief graphs (shown later in figure 2 ). ## related work measuring factual beliefs in language models. much past work has explored how information is stored and represented in pretrained language models @xcite . petroni et al. (2019) provide evidence that lms store relational information between entities, and roberts et al. (2020) show that lms can answer open-ended questions. subsequent work has further explored how much knowledge is stored in lms (heinzerling and inui, 2021). most relevant to our work are studies from talmor et al. (2020) and elazar et al. (2021). talmor et al. (2020) train lms to perform true/false classification of factual claims, and they measure how beliefs correlate between entailed facts. we use their leapofthought data as a part of our slag objective (eq. 1) and to measure model consistency under entailment before and after updating beliefs in models. meanwhile, elazar et al. (2021) measure the consistency of model predictions for paraphrased inputs. we adopt their metric for paraphrase consistency as a measure of belief. in other recent work, kassner et al. (2021) measure consistency under entailment and paraphrase for factual belief with a new small-scale dataset, beliefbank. updating factual beliefs in language models. approaches to making targeted updates to model beliefs vary along a few dimensions. first is whether the methods alter model training or operate in a post-training setting. sinitsin et al. (2020) use a meta-learning objective during training to encourage ease of editing afterwards. a larger family of methods perform post-training model updates: dai et al. (2021) propose a hand-crafted algorithm that edits model weights, while zhu et al. (2020) use projected gradient descent for batches of points. de cao et al. (2021) train a hypernetwork (learned optimizer) that processes model gradients in order to produce a new model that (1) gives the desired output for an input, while (2) satisfying other objectives like minimizing changes in predictions for other data. mitchell et al. (2021) focus on scaling up the underlying hypernetwork architecture, which is a complementary but orthogonal research direction that is not the focus of this paper. in a different approach, kassner et al. (2021) "update" model beliefs by adding in relevant information to the input at test time. but this approach does not change the model weights and hence does not influence model outputs on all other potentially relevant inputs. lastly, meng et al. (2022) provide a specialized method focused on rank-one updates to mlp matrices in transformer-based lms, but they do not address the problem of updating multiple model beliefs and do not measure model consistency under entailment or unintended corruption of local neutral beliefs (metrics (5) and (6) in sec. 3). ## updating beliefs in language models here we describe the problem of updating model beliefs and our learned optimizer method. we also discuss metrics for measuring factual beliefs below, while our belief graphs are presented in sec. 6. problem statement and metrics. we suppose we have a model f θ = p θ (y|x) parametrized by θ. for an input x i that has some undesired model output ŷi = arg max y p θ (y|x), we wish to obtain a new model θ * that produces a desired output y * i for x i . this new model θ * should also fulfill a few other desiderata. as in past work @xcite , we operationalize these desiderata in the following metrics: uses special local neutral data. unlike random data, local neutral data is guaranteed to be logically independent of the main input, while still being similar (local) to it, which we ensure by using data with the same subject entity. together, these six metrics better cover the criteria for belief outlined by @xcite . we compute the metrics using data of the kind shown in evaluation procedure. to date, methods have been evaluated on the basis of their ability to change model predictions for all data. moreover, the desired labels {y * i } n @xmath0. we propose for evaluation to focus on a more valuable but difficult setting: changing the predictions on incorrect points to be correct. ## experiment setup datasets. we run experiments with four datasets (example data shown in appendix in order to get local neutral data, we construct (4) a sequence prediction task using wikidata5m, a relational knowledge base with over 20 million triplets @xcite . each input consists of an entity e 1 and relation r, and the label is another entity e 2 that completes the triplet. all inputs come in pairs that share the same entity e 1 but use different relations with different labels. in general, the completion e 2 to the main input triplet (e 1 , r 1 , e 2 ) has no logical consequences for its paired input, (e 1 , r 2 , ?). the paired points are also local to the main input, i.e. they pertain to the same entity e 1 as the main input. we obtain four paraphrases for each main input using different aliases for the entity and synonyms of the relation. we construct a train set of 150k points and dev/test sets of 10k points each. see appendix b for further details. ## belief graphs we now construct belief graphs to better understand the connections between model beliefs. we form a graph from a set of datapoints by updating each prediction and checking what other predictions change. we represent each datapoint as its own node in a belief graph. whenever updating a datapoint u changes the prediction for point v, we draw a directed edge from u to v. following sec. 5.2, we use off-the-shelf optimizers to change the model output to the opposite of its original prediction for every datapoint. for fever we obtain a graph of 10,444 nodes, and for leapofthought we obtain a graph with 8642 nodes, which is double the test set size because we include both main inputs and entailed data as their own nodes. we visualize part of a belief graph in fig. 2 . this figure shows a non-random subgraph intended to give a representative view of the data (we give three random subgraphs in appendix e). on inspection, we do not see any clear reasons for beliefs being connected or not connected. we come to the same conclusion looking at other random subgraphs (see appendix figures 9 , 10 , and 11). whether or not changing one belief changes another appears essentially random, which is a novel negative result on the organization of internal model beliefs. however, we do observe some aggregate trends. first, it appears that incorrect predictions are the most sensitive to model updates. on fever, incorrect beliefs change around 4% of the time when other beliefs are updated, while correct beliefs change only 2.5% of the time. second, we find that local neutral beliefs are much harder to avoid changing than simply random data. on wikidata5m ( we highlight a few summary statistics here from ## conclusion we first measure the presence of consistent factual beliefs in language models, and we propose to evaluate learned optimizers for whether they can make model beliefs more truthful. then we show that our slag objective greatly improves learned optimizer performance, outperforming off-the-shelf optimizers when updating multiple model beliefs in a row. finally, we introduce belief graphs to visualize connections between model beliefs. we find that model beliefs are highly interconnected, with some "core" beliefs influencing hundreds of other beliefs.