Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expression (2024)

Abstract: Large language models (LLMs) can generate long-form and coherent text, yet they often hallucinate facts, which undermines their reliability. To mitigate this issue, inference-time methods steer LLM representations toward the “truthful directions” previously learned for truth elicitation. However, applying these truthful directions with the same intensity fails to generalize across different query contexts. We propose LITO, a Learnable Intervention method for Truthfulness Optimization that automatically identifies the optimal intervention intensity tailored to each specific context. LITO explores a sequence of model generations based on increasing levels of intervention intensities. It selects the most accurate response or refuses to answer when the predictions are highly uncertain. Experiments on multiple LLMs and question-answering datasets demonstrate that LITO improves truthfulness while preserving task accuracy. The adaptive nature of LITO counters the limitations of one-size-fits-all intervention methods, maximizing truthfulness by reflecting the model’s internal knowledge only when it is confident. Our code is available at https://github.com/launchnlp/LITO.

## introduction despite their impressive performance across a wide range of natural language processing (nlp) tasks, large language models (llms) still generate hallucinated outputs that lack real-world basis, limiting their reliability in critical applications that require truthful responses. many promising directions are explored to overcome this challenge, such as developing methods to ground llms in external knowledge and incorporate credibility indicators into model outputs @xcite . another class of methods states the presence of a linear representation of "truth" in model activations @xcite @xcite . these methods what is the name of muslim dress? who wrote the song mercedes benz? bacterial cell walls are made rigid by the presence of? what flag is red and has a gold star? when did one child policy end in china? to avoid capturing irrelevant features, @xcite proposed a supervised probe learning that directly identifies the truthful directions based on correct and incorrect statements in the truthfulqa dataset @xcite . this method, called inference-time intervention (iti), trains supervised linear probes on the output of each attention head, treating the resulting probe weights as truthful directions. additionally, a scaling coefficient is tuned to determine the intensity at which each direction should be added to its respective head output at inference time. however, amplifying the truthful directions with a fixed single intensity does not generalize across all contexts. figure 1 demonstrates this by showing the llama2-chat-7b @xcite model's performance on answering various queries from the natural questions dataset @xcite after applying the iti technique with gradually increasing truthful direction intensities. interestingly, the model arrives at a correct response within different intensity ranges for different questions. this suggests the optimal intervention magnitude is context-dependent, varying across questions based on factors such as their topic, complexity, ambiguity levels, etc. moreover, the truthful directions may not capture all aspects of truthfulness. therefore, adjusting the intensity alone cannot guarantee accurate responses. for instance, consider the question "what flag is red and has a gold star?" in figure 1 . intervening with varying strengths of truthful directions does not result in a correct answer. in such cases, the model should express uncertainty to stay truthful. to address the limitations of one-size-fits-all intervention solutions by prior methods, we propose a learnable intervention method for truthfulness optimization, lito. lito identifies truthful direction intensities that suit different contexts, e.g., different questions. given a sequence of model generations at multiple levels of intervention intensities, we develop a method to maximize truthfulness, which we define as selecting factual responses when the model is highly confident and refusing to respond otherwise. to achieve this, we collect model responses, including textual outputs, hidden representations, and confidence values, at increasing levels of intervention intensity. we then train an lstm-based classifier to assess the accuracy of these responses based on the sequence of hidden states. during inference, the system selects the most accurate response if any is deemed accurate by the classifier; otherwise, it outputs "i have no comment" to express uncertainty and refuse to answer. we measure the performance of lito and other methods in balancing truthfulness and accuracy, introducing a novel evaluation metric called the truthfulness-accuracy (t a) score. this metric evaluates the trade-off between truthfulness and task-specific accuracy by measuring how effec-tively different methods produce truthful outputs that appropriately acknowledge uncertainty while also achieving high accuracy on the target task. lito is a learnable intervention methodology agnostic to the specific intervention method used, as long as the method can identify and apply truthful directions to the model's internal representations. in this paper, we instantiate lito using the iti method and extend its application to unsupervised truthful directions detected through representation engineering (repe) @xcite . we conduct comprehensive experiments across four datasets and two categories of language models: llama and gpt-2. the results show that lito significantly improves truthfulness while preserving high task accuracy across different intervention methods. for example, using the iti method, lito boosts the t a score of llama2-chat-7b by 9.6 points on the natural questions dataset. additionally, we evaluate lito in a cross-domain setting to demonstrate its transferability across tasks. ## problem statement and preliminaries we consider the problem of mitigating hallucinations in large language models through truthfulness enhancement. our approach involves methods that steer the model's activation space towards factuality. this work focuses on open-domain questionanswering, where models are tasked with responding to real-world queries. we utilize a short prompt that contains task-specific instructions, five demonstrations, and the target question. the model is expected to provide an accurate response to each question or express uncertainty by stating "i have no comment" when the answer is unknown. ## approach in this work, we develop an intervention technique that dynamically adjusts to optimal intensity value, enhancing truthfulness based on prompt characteristics. specifically, we increase the intensity (α) of the truthful directions d, learned by iti, across k iterations, maintaining uniform intensity levels across all selected directions. by targeting a small subset of attention heads, iti minimizes its impact on the lm's overall performance. thus, small increases in intensity can yield similar outputs. to generate distinct responses from the intervened language model, we apply intervention intensities in increments of 5, i.e. α ∈ 5, 10, . . . , 5k. let llm α denote the llm intervened with intensity α and @xmath0, a 2 , ..., a k } denotes the collection of model responses, where a @xmath1. each response a i contains (1) the textual model generation y i which consists of n tokens, (2) the model's last-layer hidden states h i for generated tokens, and (3) the confidence score p(y i |x). following @xcite , we compute the confidence score as geometric mean across the sequence of token probabilities: we collect the three output components for each of the k interventions and pass all outputs to our adaptive intervention system, lito. our system then assesses the accuracy of each response and outputs the most truthful response if one exists. ## experimental results and further analyses in this section, we first compare lito against several counterparts and present our findings in section 6.1. then, in section 6.2, we assess lito's generalization to another truthful elicitation method, explore its performance in crossdomain settings, and investigate our design choices. ## conclusion in this work, we introduced lito, a novel learnable intervention method that adjusts the intensity of truthfulness directions based on the question context. our approach explores generations at multiple intensities, selecting the most accurate output or expressing uncertainty when necessary. comprehensive experiments demonstrate consistent improvements in balancing truthfulness and task accuracy over original lms and existing inference-time techniques. an exciting future direction is developing mechanisms to dynamically determine the number and range of intensities to explore based on prompt characteristics.