Truth-Aware Context Selection: Mitigating Hallucinations of Large Language Models Being Misled by Untruthful Contexts (2024)

Abstract: Although Large Language Models (LLMs) have demonstrated impressive text generation capabilities, they are easily misled by untruthful contexts provided by users or knowledge augmentation tools, leading to hallucinations. To alleviate LLMs from being misled by untruthful context and take advantage of knowledge augmentation, we propose Truth-Aware Context Selection (TACS), a lightweight method to adaptively recognize and mask untruthful context from the inputs. TACS begins by performing truth detection on the input context, leveraging the parameterized knowledge within the LLM. Subsequently, it constructs a corresponding attention mask based on the truthfulness of each position, selecting the truthful context and discarding the untruthful context. Additionally, we introduce a new evaluation metric, Disturbance Adaption Rate, to further study the LLMs’ ability to accept truthful information and resist untruthful information.Experimental results indicate that TACS can effectively filter untruthful context and significantly improve the overall quality of LLMs’ responses when presented with misleading information.

## introduction large language models (llms) have demonstrated remarkable performance across various tasks, including text generation, reasoning, and incontext learning @xcite @xcite . it has become the dominant paradigm for natural language generation. the essence of llms lies in the next token prediction @xcite . during the training phase, extensive knowledge obtained from a large dataset is embedded into the parameters of llms @xcite . subsequently, during the inference phase, llm calculates the probability distribution of the next token based on contextual information and parameterized knowledge @xcite . the token with the highest probability is then selected as the predicted outcome. therefore, the prediction of the next token is jointly determined by the model parameters and the contextual information. due to factors such as noise in the training data @xcite , biases in model parameter fitting @xcite , and the presence of untruthful information in the context @xcite , llms may occasionally generate inaccurate predictions, termed hallucinations, which constrains the broader application of llms @xcite @xcite . to address hallucinations, numerous endeavors have been undertaken @xcite , with the prevailing approach currently involving the incorporation of external knowledge into the prompt @xcite . to assist llms in generating responses and alleviate hallucinations arising from insufficient knowledge, retrieval-augmented generation (rag) has been widely employed @xcite @xcite . nevertheless, the retrieved knowledge may contain errors or be fabricated @xcite , which will inevitably negatively impact the responses generated by llms. our experiments empirically validate the impact of knowledge augmentation on llama 2-chat 7b, as illustrated in figure 1 . figure 1(a ) reveals that without external knowledge, the proportion of correct answers generated by llama 2-chat 7b is 56.7%. when truthful knowledge is introduced into the prompt, it demonstrates a substantial increase in the proportion of correct answers, reaching 88.8%, as depicted in figure 1(b ). however, with the introduction of untruthful knowledge, the proportion of correct answers decreases to 10.3%, as depicted in figure 1(c ). hence, judging the truthfulness of the input context is imperative @xcite . in addition, as shown in figure 2 (a), llms have been demonstrated to be susceptible to being misled by carefully fabricated information @xcite , leading to hallucinations. this further underscores the risk of llms being misled by untruthful context. moreover, given the possibility of a mix of truth and untruth within the contextual information @xcite , conducting finegrained truth detection becomes imperative. to address these issues, we introduce truth-aware context selection (tacs), a lightweight method to mask untruthful context from the inputs via fine-grained truth detection. the tacs framework is depicted in figure 2 @xcite . upon receiving inputs, tacs performs truth detection on the context based on its representation within the llm. an attention mask is constructed based on the truthfulness of each position, retaining high-truthfulness positions and discarding those with lower truthfulness. this approach enables taking advantage of knowledge augmentation while protecting llms from being misled by untruthful context. additionally, we propose the disturbance adaptation rate to comprehensively evaluate the llms' capacity to integrate truthful information while resisting the influence of untruthful information. the experimental results indicate that tacs can effectively filter the information in the context based on its truthfulness, significantly improving the overall quality of llms' responses. we constructed experimental scenarios based on conflic-tqa @xcite and truthfulqa @xcite where the model answers questions based on contextual information. our approach is based on state-of-the-art open-source models (such as llama 2-chat 7b and mistral-7b-instruct-v0.2), and exhibits substantial improvement compared to the baselines, showcasing robustness across models. in summary, our contributions are as follows: • we propose tacs, a lightweight method that performs context selection based on the truthfulness of context. this approach can block the propagation of untruthful information within the llms from the input, thereby significantly reducing the hallucinations caused by untruthful information. ## method to better utilize the knowledge within the context and reduce the impact of untruthful context, we propose the truth-aware context selection (tacs). as shown in figure 2 , tacs comprises several steps: firstly, it performs truth detection on the contextual information. subsequently, based on the truthfulness of each position, it constructs corresponding attention masks to select positions with high truthfulness while discarding those with low truthfulness. finally, the model generates responses based on user input and the newly constructed attention masks. in addition, we propose the disturbance adaptation rate (da rate) as a measure of the llm's ability to accept truthful information and reject untruthful information. in the next few sections, we will explain in detail the process of building classifiers for truth detection, the method of creating an attention mask based on the results of truth detection, and expound upon the calculation of the da rate. ## conclusion in this paper, we propose truth-aware context selection (tacs) to alleviate the hallucinations caused by untruthful context, which blocks untruthful information while selecting truthful information via fine-grained truth detection. experiments show that tacs can significantly prevent llm from being induced by untruthful context, showing potential in knowledge-augmented llms.