B elief B ank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief (2021)

Abstract: Although pretrained language models (PTLMs) contain significant amounts of world knowledge, they can still produce inconsistent answers to questions when probed, even after specialized training. As a result, it can be hard to identify what the model actually “believes” about the world, making it susceptible to inconsistent behavior and simple errors. Our goal is to reduce these problems. Our approach is to embed a PTLM in a broader system that also includes an evolving, symbolic memory of beliefs – a BeliefBank – that records but then may modify the raw PTLM answers. We describe two mechanisms to improve belief consistency in the overall system. First, a reasoning component – a weighted MaxSAT solver – revises beliefs that significantly clash with others. Second, a feedback component issues future queries to the PTLM using known beliefs as context. We show that, in a controlled experimental setting, these two mechanisms result in more consistent beliefs in the overall system, improving both the accuracy and consistency of its answers over time. This is significant as it is a first step towards PTLM-based architectures with a systematic notion of belief, enabling them to construct a more coherent picture of the world, and improve over time without model retraining.

## introduction intelligent agents are typically considered to have beliefs about the world -propositions that they take as true @xcite . in general, a system can be said to (appear to) believe a proposition p, e.g., "eagles are birds", if it produces answers consistent with p (and its other beliefs). pragmatically, we expect the system to @xcite give a consistent answer to different paraphrases of the question "p?" ("are eagles birds?", "is an eagle a type of bird?", ...), and (b) give correct answers about implications of p ("eagles lay eggs", "eagles have feathers", ...), figure 1 : the proposed architecture. the model's raw answers are stored in a persistent, symbolic memory (beliefbank), and two mechanisms attempt to improve them: (a) a constraint solver flips beliefs (e.g., the belief that "a swallow is a fish") that clash significantly with others. (b) a feedback mechanism poses new questions using existing, relevant beliefs (e.g., "a swallow is not a fish") as the query context. we find that both consistency and accuracy of the overall system improve. example: the model m shown in the figure incorrectly answers "yes", when asked "a swallow has gills?". but (as shown above) if reminded of its previous answer "a swallow is not a fish", m correctly answers "no". maintaining a consistent set of beliefs (a "belief system") is a key facet of intelligence, as it can help debug errors and encourage rational behavior. however, although ptlms contain substantial world knowledge @xcite , their answers to probing questions can be inconsistent @xcite , even after specialized training to reduce inconsistency @xcite . as a result, it is sometimes hard to pin down what a ptlm actually "believes", making them susceptible to inconsistent and/or irrational behavior. our goal is a first step to avoid these problems by embedding a ptlm in a broader system with a clearer notion of belief (see figure 1 ). prior work in ai, including in formal logic @xcite , belief mainte-nance @xcite , and uncertainty @xcite , offers models for how beliefs can be managed. most importantly, it posits that creating a coherent set of beliefsa kind of "mental model" of the world @xcite ) -is a constructive process requiring explicit representation of beliefs, and inference about their dependencies. based on this, our approach is to embed a ptlm in a broader system with an evolving, symbolic memory of beliefs -a belief-bank -along with two mechanisms to improve belief consistency of the overall system. first a reasoning component -a weighted maxsat (satisfiability) solver -reasons about belief dependencies and revises beliefs that significantly clash with others. second, a feedback component poses future queries to the model using known beliefs as context, aiming for more accurate and consistent answers from the ptlm itself. the beliefbank represents the overall system's beliefs (a "mental model") about the world, constructed by deliberating over the noisy output of a raw ptlm. we explore this in a controlled experimental setting where both candidate beliefs and constraints between them are provided. candidate facts are simple sentences that may be true or false, e.g., "an eagle is a bird" (t), "an eagle is a mammal" (f). constraints are between (variabilized) facts, e.g., "x is a @xmath0. these allow us to both probe and measure improvement in the consistency and accuracy of a system's beliefs, compared with an original ptlm. in contrast to prior work, this system does not rely on fine-tuning the ptlm. fine-tuning requires expensive training data, and risks destabilizing the model's performance on other tasks outside the scope of training. instead, our system functions without training data, explicitly reasoning about beliefs using an external mechanism, thus allowing both controllability and interpretability. most significantly, we find that improving consistency in this way improves accuracy, while earlier finetuning-based approaches report either no accuracy gains @xcite @xcite or only slight gains @xcite . we make the following contributions: specifically, we show that two mechanismsconstraint reasoning and feedback -improve the overall system's accuracy and consistency over time. 2. we contribute a targeted dataset to measure a system's consistency against given constraints. 3. we provide an analysis of the failure modes and directions for future work. this work is significant as it is a first step towards ptlm-based architectures that have a systematic notion of belief, allowing them to construct a more coherent picture of the world, and improve over time without model retraining. foot_0 ## related work the idea that agents should have a belief system dates back to the earliest years of ai, e.g., @xcite envisioned representing a system's beliefs as formal propositions along with a reasoning process to identify what follows. multiple subfields of ai have explored ways of representing and updating beliefs, including in formal logic @xcite , belief revision @xcite , and uncertainty @xcite . similarly, work in cognitive science has promoted mental models -coherent, constructed representations of the way the world is believed to be -as central to understanding and communication @xcite @xcite . we draw on these ideas, proposing how they can be layered on top of ptlms, here representing beliefs as nl statements rather than formal structures. although ptlms contain extensive world knowledge @xcite , they can be inconsistent in their answers to probing questions @xcite @xcite @xcite , making their "world model" unclear. although various approaches have improved answer consistency, mainly through modified model training, e.g., @xcite @xcite , they have not solved the problem. current ptlms still behave as a source of noisy knowledge, rather than projecting a coherent picture of the world @xcite . a close analogy to our task is in knowledge graph (kg) construction. @xcite define "knowledge graph identification" as the task of building a maximally consistent kg given noisy candidate facts and their extraction confidences, and constraints between them. they develop a solution using probabilistic soft logic (psl) @xcite as their constraint reasoner. our reasoning component follows similar ideas, but applied to the noisy predictions of a ptlm. on the face of it, it is not clear how to plug a constraint solver into a ptlm, given their very different natures. our solution introduces a global persistent memory, making this novel combination of technologies possible. to our knowledge this has not been done before. our work presents a broader system architecture in which a ptlm is embedded, along with a dynamic, persistent memory. while there are prior neural architectures that include an associated memory, e.g., @xcite @xcite , these components typically play the role of a short-term working memory to help computation. in contrast, our beliefbank is a persistent, long-term memory, and we treat the ptlm as a component within a larger system. our work also differs from retrieval-augmented architectures, e.g., rag @xcite , realm @xcite , that augment model input with external retrievals. rather, our memory is reflective, built from model outputs and reasoned over. our feedback mechanism uses old answers to help answer new questions. this builds on prior work such as self-talk @xcite , where a model asks itself related questions to help with new answers, and can be seen as a form of dynamic prompt engineering @xcite . in our case, feedback is selected from a global beliefbank, rather than generated with templated subqueries, potentially allowing more control over feedback selection. finally, our system performs a kind of continual learning @xcite . while recent work in this area has focused on dynamic update of model parameters, e.g., @xcite , our work leaves the model fixed, and seeks improvement in the broader system in which the model is embedded, exploring an alternative and potentially more interpretable architecture towards this goal. ## task our goal is to ascribe a clearer notion of "belief" to a system that includes a model m , by improving, over time, the consistency and accuracy of its answers (compared with m ) to a stream of questions. we measure this with true/false probing, where we are also given a set of constraints between answers: ## approach our approach adds a memory layer, called the be-liefbank, on top of the model to globally track beliefs. two mechanisms are then used to manage the beliefbank beliefs, namely constraint reasoning and feedback, as we now describe. ## dataset we create a dataset foot_3 to test our approach in a controlled way, allowing us to perform systematic experiments to evaluate behavior. the dataset contains two parts, constraints and facts, defined over simple sentences such as "a swallow is a bird." ## model the fixed model m that we use for our experiments is macaw @xcite , a state-of-theart t5 qa model fine-tuned on ≈400k qa pairs. to query the model, we pose the query (optionally with a textual context), and let the model choose between the two answer options "yes" and "no". the model also outputs an answer confidence, used as the belief weight. we use the t5-large version of this model. note that we do not retrain the model for this work; rather, it is used as a black-box qa module in the broader system (figure 1 ). other models could equally have been used. ## experiments we evaluate our system in a dynamic setting in which it receives a stream of questions, building up and revising a beliefbank. to simplify the evaluation, we consider questions to arrive in batches, and evaluate the beliefbank after each batch, measuring accuracy (f1) 5 and consistency (1-τ , section 4.1) of the beliefbank so far, comparing with the gold labels. we evaluate four configurations: raw model: the beliefbank simply records the raw model's answers foot_5 constraint-solving: after each batch, the constraint solver is run over all the (raw) model answers so far, and the beliefbank updated accordingly. feedback: questions in batch n are posed to the model using a context selected from the beliefs already in the beliefbank (batches 1 to n -1). we evaluate two selection strategies: feedback (on-topic): random beliefs about the entity e being queried about feedback (relevant): on-topic beliefs (i.e., again about e) that are most relevant to the query, as defined in section 4.2.2 feedback + constraint-solving: a combination of the two. these configurations are illustrated in figure 2 . ## conclusion ptlms can be inconsistent in their answers to probing questions, and can still give (what to a person appear as) naively wrong answers. this work is a first step towards alleviating these problems. by embedding a ptlm within a larger system with a persistent, global memory -the beliefbank -, a constraint-solver and feedback mechanism, we have shown that the overall system's behavior is more coherent, both in terms of consistency and accuracy. the additional memory layer can loosely be seen as the system's "mental model", a representation constructed from the ptlm's raw answers. our experiments were conducted in a restricted (small set of relations, entities and constraints), controlled setting, and further development is needed to scale to larger and more complex tasks. nevertheless, the work here is significant as it is a first step towards ptlm-based architectures with a globally consistent notion of belief, allowing them to construct a more coherent picture of the world, and continually improve with time.