Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability (2024)

Abstract: While language models (LMs) can sometimes generate factually correct text and estimate truth values of individual claims, these generally do not reflect a globally coherent, manipulable model of the world. As a consequence, current LMs also generate incorrect or nonsensical content, and are difficult to edit and bring up to date. We present a method called Deductive Closure Training (DCT) that uses LMs themselves to identify implications of (and contradictions within) the text that they generate, yielding an efficient self-supervised procedure for improving LM factuality. Given a collection of seed documents, DCT prompts LMs to generate additional text implied by these documents, reason globally about the correctness of this generated text, and finally fine-tune on text inferred to be correct. Given seed documents from a trusted source, DCT provides a tool for supervised model updating; if seed documents are sampled from the LM itself, DCT enables fully unsupervised fine-tuning for improved coherence and accuracy. Across the CREAK, MQuAKE, and Reversal Curse datasets, supervised DCT improves LM fact verification and text generation accuracy by 3-26%; on CREAK, fully unsupervised DCT improves verification accuracy by 12%. These results show that LMs’ reasoning capabilities during inference can be leveraged during training to improve their reliability.

## introduction there is increasing interest in using language models (lms) as sources of information and tools for fact verification @xcite . but today's lms cannot robustly perform either task: they are prone to generating factually incorrect information, contradict themselves, and are difficult to update with new information @xcite @xcite . to improve the coherence of language model predictions and reduce hallucinations, we begin with a collection of language model generated seed documents, then use the lm to generate a set of documents implied by or contradicting these documents. (b) next, we identify the generated documents most likely to be correct by finding the subset that is most probable and logically consistent and mark the rest as false. (c) finally, we fine-tune the lm on these documents with the truth value assignments obtained in (b) e.g. in this case broccoli is the color of the sky was marked as false. while this example shows dct used for unsupervised model improvement (where the seed statement is language model generated and truth value is unknown), dct can also be applied to a supervised model updating application by providing the model with a seed statement which is known to be true. even if lms are imperfect judges of factuality, however, they are quite reliable models of factual relations between pieces of text: they can identify logical and probabilistic relationships between statements @xcite , and generate text based on new information provided as input @xcite . for example, an lm that cannot answer how old was charlie chaplin when he died? may nonetheless answer correctly when prompted with charlie chaplin lived between 1889 and 1977, and recognize that this statement contradicts the claim charlie chaplin lived in the 21st century. how can we leverage lms' ability to reason about relations between claims to improve (and control) the text that lms themselves generate? conceptually, standard supervised objectives cause lms to assign high probability to statements in their training data, but not necessarily these statements' logical consequences. additional reasoning is required to determine the deductive closure of a training set @xcite -the complete collection of inferences that can be made given the information initially available. an alternative procedure is needed to ensure that lms assign high probability to a complete and consistent set of facts when they are trained and fine-tuned. in this paper, we propose a new lm fine-tuning procedure we call deductive closure training (dct), which leverages inference-time reasoning as a source of training-time supervision. at high level, given seed text (which may be provided externally or lm-generated), dct uses an lm to identify additional text implied by or contradicting this text, reasons globally about which portions of seed and generated text are most likely to be correct given this context, and finally fine-tunes on inferred-correct text. this approach builds on a large body of recent work @xcite @xcite on inferencetime procedures for improving models' factual correctness, showing that these techniques may be used at training time as well. dct may be applied in several different ways depending on the source of seed documents. if these are drawn from a trusted factual source, dct may be used to perform supervised adaptation for factuality. if documents contain new information to be inserted into an lm, dct provides tool for model updating @xcite . finally, if seed documents are generated by the model itself, dct enables fully unsupervised finetuning of models for improved accuracy. we demonstrate the effectiveness of dct across three domains: fact verification (creak benchmark; @xcite , question answering with new information (on the mquake benchmark; @xcite , and a synthetic test of edit propagation (on the "reversal curse" benchmark; @xcite . on these tasks, unsupervised and supervised applications of dct improve accuracy by up to 12% and 26%, respectively. these results show that, with little or no data, lmgenerated supervision can be leveraged to improve lms' coherence, accuracy and updatability. foot_0 2 related work ## formal analysis of dct at first glance, it may seem surprising that this procedure (especially in its unsupervised form) can improve lm accuracy using only lm-generated text. in this section, we describe a set of assumptions under which dct is guaranteed to improve accuracy on certain inputs. we focus this analysis on generation and evaluation of (question, answer) pairs, but it could be extended to the other tasks considered in this paper as well. informally, suppose: we wish to show that if these two conditions hold, dct improves model performance. for simplicity, we consider a minimal version of unsupervised dct in which a single implication is generated from each seed statement, the check in eq. ( 3 ) is not performed, and the lm is trained to convergence on data generated from an arbitrarily large number of seeds. let q be some specific question of interest, let p lm (a * | q) denote the probability that p lm assigns the correct answer to q (before applying dct), and let p dct (a * | q) be the probability that the lm assigns after dct. let (q 0 , a 0 ) denote a (question, answer) pair generated as a seed document, and a * 0 specifically the correct answer to q 0 . finally, for convenience, define p(q proposition 1. suppose for some q that: in other words, for any question q satisfying the two conditions above, unsupervised dct increases the probability that p lm answers q correctly. proof is given in appendix e. ## experiments we evaluate deductive closure training on a set of benchmark tasks measuring fact verification, question answering with new information, and a diagnostic model editing dataset. we use llama-2-7b in all experiments. additional qualitative results are provided in appendix c. ## conclusion we have described deductive closure training (dct), a supervision procedure that optimizes models toward deductive closure-encouraging them to assign high probability to a logically co-herent set of factual assertions.by doing so, dct also improves the truthfulness and updatability of models, substantially increasing accuracy on a variety of fact verification and editing datasets in both supervised and unsupervised conditions. more generally, these results show that some factual errors in lms stem not from limitations of their training data, but limitations of training algorithms. by using lms themselves to reason about relationships between (and implications of) their predictions, they can be made more accurate with little or no additional supervision.