Do Large Language Models Know What They Don’t Know? (2023)

Abstract: Large language models (LLMs) have a wealth of knowledge that allows them to excel in various Natural Language Processing (NLP) tasks. Current research focuses on enhancing their performance within their existing knowledge. Despite their vast knowledge, LLMs are still limited by the amount of information they can accommodate and comprehend. Therefore, the ability to understand their own limitations on the unknows, referred to as self-knowledge, is of paramount importance. This study aims to evaluate LLMs’ self-knowledge by assessing their ability to identify unanswerable or unknowable questions. We introduce an automated methodology to detect uncertainty in the responses of these models, providing a novel measure of their self-knowledge. We further introduce a unique dataset, SelfAware, consisting of unanswerable questions from five diverse categories and their answerable counterparts. Our extensive analysis, involving 20 LLMs including GPT-3, InstructGPT, and LLaMA, discovering an intrinsic capacity for self-knowledge within these models. Moreover, we demonstrate that in-context learning and instruction tuning can further enhance this self-knowledge. Despite this promising insight, our findings also highlight a considerable gap between the capabilities of these models and human proficiency in recognizing the limits of their knowledge.

## introduction recently, large language models (llms) such as @xcite , palm 2 @xcite , and llama @xcite have shown exceptional performance on a wide range of nlp tasks, including common sense reasoning @xcite and mathe-figure 1 : know-unknow quadrant. the horizontal axis represents the model's memory capacity for knowledge, and the vertical axis represents the model's ability to comprehend and utilize knowledge. matical problem-solving @xcite . despite their ability to learn from huge amounts of data, llms still have limitations in their capacity to retain and understand information. to ensure responsible usage, it is crucial for llms to have the capability of recognizing their limitations and conveying uncertainty when responding to unanswerable or unknowable questions. this acknowledgment of limitations, also known as "knowing what you don't know," is a crucial aspect in determining their practical applicability. in this work, we refer to this ability as model self-knowledge. the know-unknow quadrant in figure 1 illustrates the relationship between the model's knowledge and comprehension. the ratio of "known knows" to "unknown knows" demonstrates the model's proficiency in understanding and applying existing knowledge. techniques such as chain-of-thought @xcite , self-consistency @xcite , and complex cot @xcite can be utilized to increase this ratio, resulting in improved performance on nlp tasks. we focus on the ratio of "known unknows" to "unknown unknows", which indicates the model's self-knowledge level, specifically understanding its own limitations and deficiencies in the unknows. existing datasets such as squad2.0 @xcite and newsqa @xcite , widely used in question answering (qa), have been utilized to test the self-knowledge of models with unanswerable questions. however, these questions are context-specific and could become answerable when supplemented with additional information. @xcite attempted to address this by evaluating llms' competence in delineating their knowledge boundaries, employing a set of 23 pairs of answerable and unanswerable multiple-choice questions. they discovered that these models' performance barely surpassed that of random guessing. @xcite suggested probing the selfknowledge of llms through the implementation of a distinct "value head". yet, this approach may encounter difficulties when applied across varied domains or tasks due to task-specific training. consequently, we redirect our focus to the inherent abilities of llms, and pose the pivotal question: "do large language models know what they don't know?". in this study, we investigate the self-knowledge of llms using a novel approach. by gathering reference sentences with uncertain meanings, we can determine whether the model's responses reflect uncertainty using a text similarity algorithm. we quantified the model's self-knowledge using the f1 score. to address the small and idiosyncratic limitations of existing datasets, we created a new dataset called selfaware. this dataset comprises 1,032 unanswerable questions, which are distributed across five distinct categories, along with an additional 2,337 questions that are classified as answerable. experimental results on gpt-3, in-structgpt, llama, and other llms demonstrate that in-context learning and instruction tuning can effectively enhance the self-knowledge of llms. however, the self-knowledge exhibited by the current state-of-the-art model, gpt-4, measures at 75.47%, signifying a notable disparity when contrasted with human self-knowledge, which is rated at 84.93%. our key contributions to this field are summarized as follows: ## dataset construction to conduct a more comprehensive evaluation of the model's self-knowledge, we constructed a dataset that includes a larger number and more diverse types of unanswerable questions than know-unknowns dataset @xcite . to facilitate this, we collected a corpus of 2,858 unanswerable questions, sourced from online platforms like quora and howstuffworks. these questions were meticulously evaluated by three seasoned annotation analysts, each operating independently. the analysts were permitted to leverage external resources, such as search engines. to ensure the validity of our dataset, we retained only the questions that all three analysts concurred were unanswerable. this rigorous process yielded a finalized collection of 1,032 unanswerable questions. in pursuit of a comprehensive evaluation, we opted for answerable questions drawn from three datasets: squad @xcite , hot-potqa @xcite , and triviaqa @xcite . our selection was guided by sim-cse @xcite , which allowed us to identify and select the answerable questions semantically closest to the unanswerable ones. from these sources, we accordingly drew samples of 1,487, 182, and 668 questions respectively, amassing a total of 2,337. given that these questions can be effectively addressed using information available on wikipedia, the foundational corpus for the training of current llms, it is plausible to infer that the model possesses the requisite knowledge to generate accurate responses to these questions. our dataset, christened selfaware, incorporates 1,032 unanswerable and 2,337 answerable questions. to reflect real-world distribution, our dataset ## evaluation method this section elucidates the methodology employed for assessing self-knowledge in the generated text. in order to achieve this, we define a similarity function, f sim , to compute the similarity, s, between a given sentence, t, and a collection of reference sentences, @xmath0, u 2 , . . . , u n }, endowed with uncertain meanings. whenever any s i surpasses a pre-determined threshold t , we perceive the text t as encompassing uncertain meanings, thereby eliminating the need for manual evaluation of the response. given the substantial disparity in the volume of answerable and unanswerable questions in self-aware, we adopt the f1 score as a measure of llms' self-knowledge. our focus rests on identifying unanswerable questions, hence we designate them as positive cases and categorize answerable questions as negative cases. ## conclusion this study investigates the self-knowledge of llms by evaluating their ability to identify unanswerable questions. through the introduction of a novel dataset and an automated method for detecting uncertainty in the models' responses, we are able to accurately measure the self-knowledge of llms such as gpt-3, instructgpt and llama. our results reveal that while these models possess a certain degree of self-knowledge, there is still an apparent disparity in comparison to human selfknowledge. this highlights the need for further research in this area to enhance the ability of llms to understand their own limitations on the unknows. such efforts will lead to more accurate and reliable responses from llms, which will have a positive impact on their applications in diverse fields.