On the Universal Truthfulness Hyperplane Inside LLM s (2024)

Abstract: While large language models (LLMs) have demonstrated remarkable abilities across various fields, hallucination remains a significant challenge. Recent studies have explored hallucinations through the lens of internal representations, proposing mechanisms to decipher LLMs’ adherence to facts. However, these approaches often fail to generalize to out-of-distribution data, leading to concerns about whether internal representation patterns reflect fundamental factual awareness, or only overfit spurious correlations on the specific datasets. In this work, we investigate whether a universal truthfulness hyperplane that distinguishes the model’s factually correct and incorrect outputs exists within the model. To this end, we scale up the number of training datasets and conduct an extensive evaluation – we train the truthfulness hyperplane on a diverse collection of over 40 datasets and examine its cross-task, cross-domain, and in-domain generalization. Our results indicate that increasing the diversity of the training datasets significantly enhances the performance in all scenarios, while the volume of data samples plays a less critical role. This finding supports the optimistic hypothesis that a universal truthfulness hyperplane may indeed exist within the model, offering promising directions for future research.

## introduction although large language models (llms) have gained significant success in a wide range of domains @xcite , hallucination problems remain the main challenges that hinder their wider applications @xcite @xcite . this issue is further aggravated by a limited understanding of the opaque inner mechanisms of llms' factual behaviors. recent works start to investigate hallucina-tions from the perspective of inner representations, adopting the probing method @xcite to identify hyperplanes in the space of hidden states to distinguish between correct responses and hallucinations @xcite @xcite @xcite . the underlying hypothesis is that the hidden states of language models already encode significant information on hallucination, and we are able to tell hallucinations from the hidden states. while these studies have achieved impressive hallucination detection performance on the datasets which the probes are trained on @xcite @xcite @xcite , they often struggle to generalize to out-of-distribution (ood) data samples @xcite @xcite . we further verify such ood generalization failure in our experiments, confirming that the performance of the probe trained solely on truthfulqa @xcite -a widely used dataset to train probes @xcite @xcite will drop 25 absolute points on average for several other datasets compared to in-domain detection. this failure raises two principled questions: (1) do the identified inner representation features in previous works really capture the model's inner hallucination, or only overfit spurious patterns of the specific dataset? (2) does there exist a universal truthfulness hyperplane that can classify factual correctness on diverse tasks? we aim to answer these questions in this work. inspired by the success of diversified instruction tuning @xcite @xcite , our idea is to increase the diversity of the training data by scaling up the number of training datasets, so that we may find the universal truthfulness hyperplane that can generalize across tasks using the framework shown figure 1 : top: we extract representations from the last token of the input sequence, then specific locations of the hidden states inside the llm are selected and concatenated as input to train the probe. bottom: previous works mainly train the linear probe on one dataset which tends to overfit spurious features. our work utilizes diverse datasets to examine whether a universal truthfulness hyperplane exists that can generalize to out-of-domain data. in figure 1 . specifically, we construct a comprehensive and diverse collection of hallucination detection datasets to facilitate the analysis. the dataset comprises 17 distinct categories of tasks covering over 40 datasets from knowledge-seeking qa tasks such as triviaqa @xcite , natural questions @xcite to structure-totext tasks such as e2enlg @xcite , with each task containing both correct and incorrect samples, as illustrated in figure 2 . these datasets enable us to thoroughly evaluate the performance and robustness of the truthfulness probes. in our experiments, we train probes using diverse datasets and evaluate their generalization performance in three scenarios: cross-task, crossdomain, and in-domain. we study the effectiveness of probing different locations of hidden states and find that the attention heads lead to the highest accuracy. our probe method beats the promptingbased approach as well as the probability baseline significantly and outperforms the previous probe that is trained only on one dataset by 14 absolute points, achieving ∼ 70% cross-task accuracy. this provides empirical evidence for the existence of a shared representation of truthfulness within the model. notably, despite our probe being trained on an extensive collection of datasets, it achieves high performance with an average of only 10 data samples per dataset. this demonstrates the method's data efficiency and its straightforward applicability in identifying a universal truthfulness hyperplane. ## related works our work is related to a series of works trying to identify the truthfulness hyperplane inside llms. the existence of the universal truthfulness hyperplane is the foundation when considering truthfulness as an attribute for probing. without such a hyperplane, it implies that all efforts in truthfulness probing @xcite @xcite @xcite might merely be overfitting to spurious features of the task, rather than capturing genuine truthfulness. based upon such insights, several studies have also explored interventions to enhance model truthfulness by utilizing the vectors identified through probes @xcite @xcite . generally, utilizing the learned truthful vector, they edit the representation space directly @xcite or optimize the representation space towards more truthful states @xcite . ## conclusion in this paper, we examine whether a universal truthfulness hyperplane exists inside the model, through designing and training a probe on diverse datasets. our approach greatly improves existing results and conveys positive signals on the existence of such a universal truthfulness hyperplane.