Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding (2024)

Abstract: Calibrating language models (LMs) aligns their generation confidence with the actual likelihood of answer correctness, which can inform users about LMs’ reliability and mitigate hallucinated content. However, prior calibration methods, such as self-consistency-based and logit-based approaches, are either limited in inference-time efficiency or fall short of providing informative signals. Moreover, simply filtering out low-confidence responses reduces the LM’s helpfulness when the answers are correct. Therefore, effectively using calibration techniques to enhance an LM’s factuality remains an unsolved challenge. In this paper, we first propose an activation-based calibration method, ActCab, which trains a linear layer on top of the LM’s last-layer activations that can better capture the representations of knowledge. Built on top of ActCab, we further propose CoDec, a confidence-guided decoding strategy to elicit truthful answers with high confidence from LMs. By evaluating on five popular QA benchmarks, ActCab achieves superior calibration performance than all competitive baselines, e.g., by reducing the average expected calibration error (ECE) score by up to 39%. Further experiments on CoDec show consistent improvements in several LMs’ factuality on challenging QA datasets, such as TruthfulQA, highlighting the value of confidence signals in enhancing the factuality.

## introduction despite their impressive language understanding and generation capabilities, language models (lms) still produce hallucinated content @xcite , undermining their trustworthiness. one mitigation is to calibrate lms' confidence in their outputs to align with the actual likelihood of their responses being correct @xcite . well-calibrated confidence scores allow users to discern the reliability of lms' responses, enabling them not only to determine whether to trust the model's outputs but also to decide when further verification is needed. popular lm calibration methods include training-free, inference-only approaches through verbalization @xcite or selfconsistency measurements @xcite and training-based methods, such as tuning temperature parameters @xcite or learning uncertainty estimations from lms' logits @xcite . despite their potential compatibility with many lms, training-free methods are limited by the models' instruction-following capabilities and can be computationally expensive during inference. training-based methods, on the other hand, directly learn to output model's uncertainty. for instance, @xcite leverages the token logits predicted by the model to estimate the confidence of the entire response. yet model logits may not capture knowledge representations and can be sensitive to tokenization procedures. moreover, model confidence has been naively used by setting a threshold to filter out low-probability responses @xcite , leaving these queries unanswered and reducing model's helpfulness. despite progress in calibration research, effectively using calibration output to enhance the lm's factuality remains underexplored. in this work, we make two primary contributions: (i) a lightweight and effective lm calibration technique, and (ii) a decoding strategy that uses calibrated confidence to elicit correct responses from lms. concretely, we propose an activationbased calibration method, actcab, that estimates model's uncertainty from its internal activations. actcab is inspired by recent work that a truthful direction can be identified from lm activations to steer generations @xcite . moreover, prior work @xcite has shown that optimizing over binary correctness labels contributes to poor calibration. to address this issue, we propose a novel procedure to construct soft labels that better represent the expected confidence in the training objective. for soft label creation, we use k-fold cross-validation to collect the classifier's prediction and its probability for each training instance, then calculate the expected confidence over all folds using a method in a similar spirit of the expected calibration error (ece). using actcab, our second contribution is a confidence-guided decoding strategy, codec. codec directs the lm to generate outputs with higher confidence by incorporating the confidence scores of candidate tokens with the top-k highest probabilities predicted by the lm. this increases the likelihood that the answers are correct while maintaining inference efficiency. unlike selective generation, codec preserves the helpfulness of lm by not filtering out responses, but rather by better discerning the correct answers. this aligns with the principles of other inference-time intervention methods @xcite @xcite . however, codec differs by keeping the model unchanged and does not affect the lm's original reasoning process, resulting in more stable performance across different training data and lms, as shown in our experiments. to assess the newly proposed calibration method, we apply actcab on llama2-7b @xcite and experiment on five popular open-ended question-answering (qa) datasets, as collected by @xcite . the results show that our method achieves superior calibration performance, reducing the ece score by an average of 39% compared to the most competitive baseline, the logitbased litcab method @xcite . again, actcab uses the more informative internal activations to better model the global context, resulting in more accurate confidence scores at the response level than logit-based methods that primarily focus on the correctness of individual tokens. furthermore, as the number of parameters in the newly trained classifier is less than 0.001% of original lm parameters, actcab also enjoys a superior inference efficiency compared to self-consistencybased methods. moreover, we use codec to improve the factuality of lms after being calibrated by act-cab. we compare codec with selective generation @xcite , which keeps the model output with the highest confidence score among multiple generations, and two state-of-the-art intervention techniques that aim to improve factuality by shifting model activations during inference: inference-time intervention (iti) @xcite and representation engineering (repe) @xcite on three lms, llama2-7b, llama2-13b @xcite and llama3-8b (ai@meta, 2024). codec outperforms the nontrivial comparisons on challenging qa benchmarks, including natural questions and truthfulqa, while achieving comparable performance on other tasks. ## actcab: activation-based confidence calibration recent studies @xcite show that the activations within lms may encode a notion of "truth" if the model knows the facts. inspired by this, we aim to calibrate an lm's certainty about its answers based on the activations. in this section, we first describe the architecture of actcab in §3.1, and then present a novel procedure to construct soft labels for use in the training objective in §3.2. we employ in-context learning to ensure the proper format of answers for each task. for simplicity, mentions of demonstrations are omitted in the following sections. ## codec: confidence-guided decoding intuitively, a response generated with higher confidence is more likely to be correct, assuming the lm is well-calibrated. building on this intuition, we design codec, a decoding strategy that guides the lm to generate high-confidence responses, in line with recent guided decoding studies @xcite . we illustrate the whole process of codec in figure 2 . codec follows a greedy procedure, encouraging the model to favor tokens that can lead to answers with higher confidence. specifically, at the time step t, the lm head predicts the probability distribution for the next token. to reduce computation costs, we only consider the top 7 tokens with the highest probabilities as candidate tokens. for each candidate token y * t , we feed it into the lm to obtain its activation h * t . given this activation, the score for each token is computed by combining the token probability with its estimated confidence: where lm (y * t ) is the lm predicted token probability of the candidate token y * t . w and b are the parameters of the classifier described in §3.1. λ is the hyperparameter that balances these two sources, whose optimal value is searched on the development set. during decoding, the candidate token with the highest score s(•) will be chosen as the next token. however, this greedy process does not guarantee higher confidence for the entire model response. beyond this token-level guidance, we use actcab again to estimate the overall confidence of the generated response using the average activations. this leverages the entire activation sequence, which is supposed to capture a more globally calibrated confidence. we only retain the model responses generated by codec if their overall confidence exceeds that of the response generated by the standard greedy decoding. ## conclusion we introduced actcab, a calibration method for language models that leverages internal activations to improve the alignment of model confidence with answer correctness. actcab outperforms existing logit-based and consistency-based methods, achieving a 39% reduction in the ece on five qa benchmarks. additionally, we developed codec, a confidence-guided decoding strategy that enhances the factual accuracy of model responses by utilizing the calibrated confidence scores from actcab. our empirical results demonstrate that codec significantly improves lm factuality on challenging datasets like truthfulqa.