Language Models with Rationality (2023)

Abstract: While large language models (LLMs) are proficient at question-answering (QA), it is not always clear how (or even if) an answer follows from their latent “beliefs”. This lack of interpretability is a growing impediment to widespread use of LLMs. To address this, our goals are to make model beliefs and their inferential relationships explicit, and to resolve inconsistencies that may exist, so that answers are supported by interpretable chains of reasoning drawn from a consistent network of beliefs. Our approach, which we call REFLEX, is to add a **rational, self-reflecting layer** on top of the LLM. First, given a question, we construct a **belief graph** using a backward-chaining process to materialize relevant model beliefs (including beliefs about answer candidates) and their inferential relationships. Second, we identify and minimize contradictions in that graph using a formal constraint reasoner. We find that REFLEX significantly improves consistency (by 8%-11% absolute) without harming overall answer accuracy, resulting in answers supported by faithful chains of reasoning drawn from a more consistent belief system. This suggests a new style of system architecture in which an LLM extended with a rational layer can provide an interpretable window into system beliefs, add a systematic reasoning capability, and repair latent inconsistencies present in the LLM.

## introduction while large language models (llms) are impressive at question-answering (qa), it is not always clear how (or even if) an answer follows from their latent "beliefs" foot_0 about the world, or whether the llm even has a coherent internal belief system. this general opacity is a growing impediment to widespread use of llms, e.g., in critical applications such as medicine, law, and hiring decisions, figure 1 : (top) when queried about each answer option independently, the model incorrectly believes both are true, and is more confident in the wrong answer (s 2 ). (bottom) reflex adds a "rational" layer above the llm layer, in which a belief graph is constructed (by iteratively querying the llm, up/down arrows), containing relevant model-believed facts (white/@xmath0. inconsistencies are then identified (red) and minimized by a constraint reasoner that flips t/f labels on beliefs (green ✓/x), here resulting in the correct answer (s 1 , green box) + explanation (graph) by the overall system (blue). in addition, when we do this, we find such graphs expose latent inconsistencies in the model's beliefs. we show how such inconsistencies can be resolved using constraint satisfaction techniques. when we do this, the rational layer becomes not just a window onto the model, but an active reasoning component in its own right in a larger, overall system, comprising the (frozen) llm plus rational layer (blue box, figure 1 ). we show this results in a more consistent set of beliefs in the overall system, without harming overall answer accuracy (although some individual answers may change). the result is answers supported by faithful, system-believed chains of reasoning drawn from a consistent belief system. our approach, called reflex, introduces a rational layer consisting of two parts. first, to produce a belief graph, we recursively ask the llm to explain why each candidate answer might be true, expressed as a set of sentences that entail the answer. this builds on earlier work on generating entailment-based and chain-of-thought explanations @xcite @xcite . we then add a self-verification step to check that the model itself believes those generations (i.e., that the model believes what it says), allowing us to identify sentences reflecting the model's own internal knowledge. for example, in figure 1 , when asked to explain s1 ("giraffes give live birth"), the model generates s7 ([because] "mammals give live birth") and s4 ([and] "a giraffe is a mammal"). self-querying then checks if the model actually believes its generations ("do mammals give live birth?"). the answer ("yes"/"no") assigns a true/false (t/f) value to each generation, indicated in figure 1 by white/grey nodes. this procedure is then applied recursively to the generated, supporting sentences. the resulting network of model beliefs and their dependencies provides a a window into the model. second, we apply a formal constraint reasoner to this graph to resolve inconsistencies, by finding the optimal (minimal cost, section 3.3) way of flipping t/f values. for example, on the left in figure 1 , s2 and s3 ("spiders do/don't give live birth") are in an xor relationship (i.e., exactly one must be false), but both are believed as true (white) by the llm -a latent contradiction within the llm. constraint reasoning then seeks to remove such inconsistencies, here flipping the belief value on s2 from t to f (figure 1 , right), repairing the contradiction. this builds on earlier techniques @xcite @xcite , though in a notably richer setting with over 350 nodes and 80 constraints per question, joint inference across answer candidates, and a variety of constraint types. the overall result is a fully autonomous, self-reflective system that is able to deliberate (and if necessary change) its answers, thereby resolving latent inconsistencies that would otherwise go unnoticed, and provide faithful explanations drawn from a consistent belief system. we evaluate our implementation of reflex on three datasets: entailmentbank @xcite , obqa @xcite , and quartz @xcite . we find that reflex is able to construct belief graphs with significantly improved consistency (by 8%-11% absolute) without harming overall answer accuracy. in addition, answers are now supported by a more consistent, system-believed chain of reasoning, providing a window into the previously latent beliefs of the model. our contributions are thus: with no guarantee that the generated sequence of tokens expresses the model's internal knowledge, nor entails the actual answer. similarly, chain-ofthought (cot) @xcite and least-to-most @xcite prompting generate (in different ways) a step-by-step reasoning chain along with an answer, but again with no claim that the chain reflects the model's internal knowledge nor is valid reasoning @xcite . to add semantics to generations, several systems have used self-querying to verify that generations reflect model-believed facts (by self-querying "is p true?") (e.g., @xcite , or model-believed rules (by self-querying "does p imply q?") (e.g., @xcite . we build on these to construct a belief graph, namely a network of model-believed facts and their inferential relationships, which can then be reflected on. beliefs: we refer to the model's factual opinions as "beliefs" rather than "knowledge" because those opinions may be wrong. in general, an agent can be said to believe p if it acts as if p was true @xcite . following @xcite and @xcite , we take a simple, syntactic operationalization of this, namely the agent answers "yes" to the question "p?", but also note that more semantic versions could be used, e.g., the agent also answers "yes" to paraphrases and implications of p. reducing inconsistency: llms are known to be inconsistent in their answers @xcite @xcite @xcite @xcite , and several recent works have used constraint reasoners to identify and reduce inconsistency. beliefbank used a maxsat solver to resolve inconsistencies between model beliefs, but required a hand-provided set of constraint rules @xcite . concord @xcite similarly used maxsat to ensure model answers were consistent with nliderived entailment constraints between them, but did not introduce additional model-believed facts and rules. maieutic prompting @xcite also used maxsat to resolve inconsistencies between facts in prompt-induced explanation chains. however, those chains were not validated as reflecting model-believed constraint rules 2 , and did not support conjunction. reflex extends these reasoning chains to provide a full semantic account of how answers are supported by the model's internal knowledge. additionally, it performs joint reasoning across answer candidates and operates at a much larger scale (e.g., over 350 nodes on average for each question) and with a variety of constraint types. ## experiments and results the goal of our experiments is to evaluate the extent to which our overall system, namely an llm plus a self-reflecting, rational layer, helps expose and resolve inconsistencies in the llm's beliefs without harming accuracy. importantly, reflex is evaluated in a zero-shot setting, without relying on training instances of the target datasets. datasets. we use the test partitions of three existing multiple-choice datasets: entailmentbank @xcite , obqa @xcite , and quartz @xcite . we chose our datasets as they contain inferentially rich questions (typically) requiring reasoning. the partitions contain 339, 500, and 784 examples, respectively. models. the baseline llm we use is an llm that has been trained to perform qa and also supports the basic operations discussed in sec. 3.2.1, enabling us to assess how much it can be improved by adding a reflex layer. to this end, we use a publicly available, frozen, off-the-shelf t5-11b llm called entailer @xcite . to answer an mc question with this llm, we score each answer hypothesis (c s , section 3.2.1) and select the one with the highest truth confidence. if entailer assigns false values to all answer choices, we select the hypothesis with the lowest false confidence. reflex then adds a rational layer to this llm, creating a new system that is also able to self-reflect and modify its beliefs. to ensure the different belief graph scores in reflex are appropriately calibrated, we use nine hyperparameters, tuned once on the dev partition of entailmentbank @xcite and then kept fixed for all experiments. details are in appendix b. note the llm itself remains frozen, with belief revision occurring in the 6 14195 rational (belief graph) layer above it. metrics. for measuring self-consistency, we follow @xcite and report the conditional constraint violation (τ ) metric, defined as follows: the fraction of rules whose premises p are believed true, but whose hypothesis h is not. in other words, over all rules of the form @xmath1, τ is: for qa performance, we report standard multiple-choice accuracy: 1 point for predicting the correct answer, 1/n points for predicting n answers including the correct one, 1/k points for no prediction (@xmath2, 0 otherwise. ## future work there are several impactful ways this work could be further extended. first, incorporating the question's context in the belief statements in our rational layer could make the semantics of the beliefs more precise, thus avoiding potential ambiguity in their truth value. second, one could use the belief graph itself to identify the key reasoning pieces that the llm is most uncertain about. this could then guide a human-in-the-loop mechanism to correct or validate uncertain pieces via user interaction. third, maintaining a persistent belief graph over multiple questions could help make the system more consistent across questions. this, in turn, would make a user's conversational experience with the system more coherent in a longer dialog setting. lastly, after resolving inconsistencies in the rational layer, we could consider propagating information back to the llm layer in order to update it (via fine-tuning, model editing, memory-based architectures, etc.), helping avoid similar inconsistencies in the future. ## conclusion while llms perform well, the interdependencies between their answers and their other beliefs is opaque, and may even be in conflict. this lack of interpretability is a significant impediment to widespread use of llms. to reduce this opacity, and reduce these conflicts, we have proposed reflex, a new system architecture in which an explicit, interpretable representation of beliefs -the belief graph -is added as a rational layer above the llm. this layer providing a window into system beliefs, and allows latent inconsistencies in the llm alone to reasoned about and repaired. our implementation shows that belief consistency of the overall system is significantly improved, without harming answer accuracy, resulting in answers supported by interpretable chains of reasoning drawn from a more consistent belief system. this new architecture is an important step towards improving confidence in system behavior, and towards trustable deployment of llms in practical applications.