Knowing What LLM s DO NOT Know: A Simple Yet Effective Self-Detection Method (2024)

Abstract: Large Language Models (LLMs) have shown great potential in Natural Language Processing (NLP) tasks.However, recent literature reveals that LLMs hallucinate intermittently, which impedes their reliability for further utilization. In this paper, we propose a novel self-detection method to detect which questions an LLM does not know.Our proposal is empirical and applicable for continually upgrading LLMs compared with state-of-the-art methods. Specifically, we examine the divergence of the LLM’s behaviors on different verbalizations for a question and examine the atypicality of the verbalized input. We combine the two components to identify whether the model generates a non-factual response to the question. The above components can be accomplished by utilizing the LLM itself without referring to any other external resources. We conduct comprehensive experiments and demonstrate the effectiveness of our method for recently released LLMs involving Llama 2, Vicuna, ChatGPT, and GPT-4 across factoid question-answering, arithmetic reasoning, and commonsense reasoning tasks.

## introduction with the significant improvements in large language models (llms) such as palm @xcite , chatgpt @xcite , @xcite , llama 2 @xcite , and vicuna @xcite , llms have been applied in various natural language tasks. unfortunately, llms still produce unexpected falsehoods @xcite , i.e., they are unaware of what they do not know and generate responses indiscriminately. for example, chatgpt generates falsehoods for a knowledge quiz and math problem, as shown in there are two main paradigms to detect nonfactuality: the calibration-based and the selfdetection methods. the first class of methods calibrates the model confidence to better detect falsehoods of the generations (see figure 1(a) ). among them, @xcite train auxiliary calibrators, @xcite and @xcite improve the calibration through fine-tuning the language model. we propose a self-detection method that does not require further fine-tuning. the self-detection methods directly leverage the llms themselves to detect whether they hallucinate (see figure 1(b) ). for example, @xcite prompt the llms to predict the confidence score on whether their responses are true, and @xcite directly utilizes the token probabilities of the generations as the confidence score; @xcite and @xcite detect the falsehoods by sampling answers with a high temperature and examining self-consistency between them. however, the performance of these works is limited as llms tend to be overconfident about their own outputs and these work would be less effective after the models are trained more in this paper, we consider detecting nonfactuality as that a model does not know which knowledge is related to the question or does not understand the queried question, outputting the nonfactual response. a model is expected to provide correct and consistent answers regardless of the ways the questions are verbalized. therefore, if it responds drastically differently to the different verbalizations, we consider the model does not know the question. built on the above hypothesis, we propose a novel self-detection method that includes 1) examining the divergence of the llm's behaviors on different verbalized questions and 2) examining whether the verbalization of the question is typical in the llm as shown in figure 2 . specifically, for the first component, we first diversify the queried question to several semantically equivalent verbalizations. then, we examine the divergence between the answers corresponding to the questions. for the second component, we use the negative loglikelihood of the verbalized question as the indicator of atypicality in the language model. concurrent work @xcite has also mentioned rephrasing the original question to alternatives and checking the consistency of the answers with the original answer. in contrast, we further propose to examine the representativeness of the input for the model and examine the divergence in the answer distribution. our self-detection method is applicable for continually upgrading llms. to verify the effectiveness of our method, we conducted extensive experiments on gpt-4, chat-gpt, vicuna, and llama 2 across three types of tasks: factoid question answering, commonsense reasoning, and arithmetic reasoning tasks. the experimental results demonstrate the superior performance of our self-detection method. in summary, our contributions are as follows: ## related work model calibration calibration is a well-studied topic in traditional neural networks @xcite @xcite , aiming to provide a confidence score that aligns well with the true correctness likelihood. jagannatha and yu (2020), @xcite and @xcite show bert @xcite , distilbert @xcite , t5 @xcite , bart @xcite , gpt-2 @xcite , gpt-3.5 @xcite are not well-calibrated on the language tasks. post-hoc methods like temperature scaling and feature-based fitting on a development set are widely used @xcite @xcite , which are straightforward to implement. bootstrapping and ensembling methods @xcite @xcite are explored for the traditional dnn models. @xcite ; @xcite ; @xcite ; @xcite fine-tune and optimize the calibration for bert, roberta, t5 and alpaca respectively. @xcite and @xcite fine-tune the blenderbot @xcite and gpt-3 @xcite separately for calibration and express the models' uncertainty in a verbalized statement. the calibration tuned for specific tasks makes it challenging to generalize on out-ofdistribution data @xcite . hallucination detection llms such as chat-gpt @xcite , gpt-4 (openai, 2023), vicuna @xcite , llama 2 @xcite and claude @xcite have obtained remarkable performance on various language tasks @xcite . however, recent work @xcite @xcite show that llms may produce hallucinated contents, i.e., non-factual responses. the importance of the hallucination problem has been highlighted by several work @xcite as it hinders the reliability of the llms. @xcite and @xcite use llms to evaluate the sampled answers but can not evaluate their self-generated answers due to overconfidence. @xcite and @xcite utilize their confidence scores like token probability to indicate the confidence of their output. recent work @xcite @xcite examines the self-consistency score among the randomly sampled answers which are generated through a higher temperature. both the confidence score of the model output and sample-based score highly rely on the current model training, which means the methods would not be that effective after the models are trained to be more aligned. @xcite combine the llms verbalized statement, self-consistency of the randomly sampled answers, and the consistency between the induced answers. this work proposes to add additional instruction to the prompt for generating induced answers. concurrent work @xcite utilizes several verifier llms to cross-check whether a language model generates falsehoods. @xcite also rephrases the original question to alternative inputs and checks the consistency of the answers with the original answer as the confidence score. we propose a unified method that examines the divergence of the llms' behaviors across the diversified questions besides the consistency pair and the atypicality of the verbalized input in the llms. our proposal is self-detection without referring to any other llms or external resources. ## inconsistency and atypicality in llms we attribute the non-factuality of an llm to the generative characteristics which sample the most possible tokens sequentially. it means even if the llm does not know the exact knowledge related to the question or even does not understand the question (understand the instruction), it still generates plausible responses as observed in previous work @xcite . consequently, if an llm returns contradicting responses to the semantically equivalent questions, the llm does not know the knowledge for the question. besides, if the textual verbalization of a question is not representative in the llm, i.e., atypical, it would be hard to understand resulting in a lower-quality response @xcite . two examples of chatgpt are shown in so, we 1) examine the divergence between the responses (@xmath0, ..., r n }) to a question set (@xmath1, ..., q n }), where any two questions q i and q j are semantically equivalent; 2) then examine whether the verbalized question q is representative in the llm using the atypicality a(q) of the input. ## self-detecting what llms do not know in this section, we introduce our framework including consistency-based detection 4.1 and verbalization-based detection 4.2 as shown in figure 2. ## conclusion in this paper, we propose a simple yet effective method to self-detect whether an llm generates non-factual responses, without referring to any other external resources. we conducted extensive experiments on four recent llms-chatgpt, gpt-4, vicuna, and llama 2 on three different types of tasks and demonstrated the effectiveness of our method and each two components. the two pro-posed components along with the existing methods can be combined for further utilization. furthermore, we explore the question types that llms may struggle with, like low popularity and distracting formulations. our method is applicable for continually upgrading llms. it can assist the llms to detect and improve their specific weaknesses, improving their reliability in the future.