Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checking (2024)

Abstract: This paper proposes PiNose, which trains a probing model on offline self-consistency checking results, thereby circumventing the need for human-annotated data and achieving transferability across diverse data distributions. As the consistency check process is offline, PiNose reduces the computational burden of generating multiple responses by online consistency verification. Additionally, it examines various aspects of internal states prior to response decoding, contributing to more effective detection of factual inaccuracies. Experiment results on both factuality detection and question answering benchmarks show that PiNose achieves surpassing results than existing factuality detection methods.

## introduction large language models (llms), after pre-training on massive corpora @xcite @xcite , show a surprising ability to generate knowledgeable content @xcite . although this ability facilitates a wide range of applications, such as question answering (qa) @xcite @xcite and information retrieval @xcite , the propensity of llms to occasionally produce nonfactual knowledge @xcite ., † equal contribution. ‡ work was done when interned at zhipu ai. * corresponding author. previous studies offer evidence that the internal representation vectors in llms determine whether they produce factual answers to the input question @xcite @xcite . specifically, the factual behavior entailed is extracted from the feed-forward layer activations of tokens before the generated content using linear probes @xcite . however, their construction relies on the labor-intensive process of annotating natural language questions, as well as labeling llms' outputs with factuality annotations, a factor that limits their applicability to questions and responses with unseen distributions. to avoid the annotation process, the most recent studies detect non-factual content via online selfconsistency checking @xcite . they assume that if llms give contradictory responses to the same prompt, the model is more likely to hallucinate to give that answer @xcite . in this way, detecting non-factual content is reduced to the mutual-entailment analysis among multiple generations, which is usually realized as natural language inference (nli) models @xcite or heuristic comparison of the hidden representation similarity @xcite . however, self-consistency checking introduces extensive computation overhead to sample multiple responses. in addition, due to the lack of training process, these methods are less robust than previous factuality probes. giving these limitations of existing methods, we propose pinose, a method to predict non-factual responses from llms. the main idea of pinose is to construct a probing model that learns from offline self-consistency checking. it aims to present two core advantages over existing methods: transferability. comparing with existing prob-ing methods, pinose eliminates human annotation for training data. this is achieved with bootstrapped natural language questions and generated pseudo factuality labels through an offline consistency checking mechanism. moreover, as pinose does not rely on specific training data, it transfers effortlessly to any different data distributions. efficiency and effectiveness. comparing with online consistency checking, pinose avoids the computational burden associated with multiple generations during inference, thus enhancing time efficiency. additionally, by analyzing the continuous internal representations of llms rather than discrete tokens in the response, pinose gains access to a broader spectrum of information, enhancing its prediction effectiveness. we conduct comprehensive experiments on established factuality detection benchmarks and variations of qa datasets. our results reveal several key findings: (1) pinose outperforms supervised probing-based baselines by 7.7-14.6 auc across qa datasets, despite being trained without annotated labels. (2) moreover, our pinose achieves significant performance improvements (3-7 auc) compared to unsupervised consistency checking baselines, while also demonstrating superior time efficiency. (3) additionally, the dataset generated via offline self-consistency checking shows promise for transferring to probe various llms. ## preliminaries this study concentrates on identifying non-factual content using decoder-only llms. it begins by formally defining the task and then elaborates on decoder-only llms and the construction of probes for these models. additionally, it discusses the distinctions between online and offline selfconsistency checking. task definition. formally, given a question @xmath0, q 2 , . . . , q n ⟩ composed of n tokens, and its corresponding response @xmath1, r 2 , . . . , r m ⟩ consisting of m tokens, non-factual content detection aims to assign a binary judgment @xmath2, false}, determining the factual correctness of r. for example, given the question "what is the capital city of china?", "beijing" serves as the response with true judgement, while "shanghai" is classified as false. without losing generality, we allow q to be null (i.e., @xmath3, in which scenario, the task evaluates the factuality of the standalone assertion. to detect whether llms generate non-factual content, in our setting, the response r is usually sampled from an llm. in this case, we expect the model to assess the factuality of content generated by itself without the need of another llm. decoder-only llms. decoder-only llms comprise a stack of transformer decoder layers @xcite . after embedding tokens into the hidden representation h (0) , each layer manipulates the hidden representation of the previous layer as follows 1 : ## methodology the construction of pinose involves three main stages: (1) in the data preparation stage, we bootstrap natural language questions and generate multiple responses, which together serve as model inputs; ## experiment we conduct experiments to examine the performance of pinose by comparing it to baseline methods for factuality detection. additionally, we assess its transferability and efficiency. ## related work factuality detection for llm generated content mainly falls into two categories: consistency-based and probing-based. consistency-based methods detect non-factual generations by comparing model generated content with other information. among these methods, the most widely adopted assumption is that, llms usually fail to give consistent responses to the same prompt when generating multiple times @xcite @xcite , thus motivating a series method to detect non-factual content @xcite @xcite or reduce non-factual generations @xcite . another thread of works evaluate model self-consistency via model's confidence to the generated content @xcite @xcite . except for self-consistency checking, there are also attempts that use consistency between model generated content and external information as factuality indicator @xcite @xcite . probing-based methods possesses the belief that the hidden representation entails certain property of generated content and can be extracted via a light weight model @xcite . probing whether llms are producing factual content is proved to be feasible @xcite , thus motivating researchers to develop more accurate probes @xcite @xcite . comparing with these works, which rely on annotated training data, pinose provides a method that dis-till consistency patterns from llms into a probe. ## conclusion this paper presents pinose, a probing method for non-factual content detection that learns from offline consistency checking. pinose achieves good transferability among different distributed datasets as its does not rely on manually annotated data. it also avoids the computational burden for online consistency checking. in the future, pinose potentially paves way to build more faithful llms.