Teaching Large Language Models to Express Knowledge Boundary from Their Own Signals (2025)

Abstract: Large language models (LLMs) have achieved great success, but their occasional content fabrication, or hallucination, limits their practical application. Hallucination arises because LLMs struggle to admit ignorance due to inadequate training on knowledge boundaries. We call it a limitation of LLMs that they can not accurately express their knowledge boundary, answering questions they know while admitting ignorance to questions they do not know. In this paper, we aim to teach LLMs to recognize and express their knowledge boundary, so they can reduce hallucinations caused by fabricating when they do not know. We propose CoKE, which first probes LLMs’ knowledge boundary via internal confidence given a set of questions, and then leverages the probing results to elicit the expression of the knowledge boundary. Extensive experiments show CoKE helps LLMs express knowledge boundaries, answering known questions while declining unknown ones, significantly improving in-domain and out-of-domain performance.

## introduction large language models (llms) have emerged as an increasingly pivotal cornerstone for the development of artificial general intelligence. they exhibit powerful intellectual capabilities and vast storage of knowledge @xcite @xcite , which enables them to generate valuable content. recent research demonstrates that llms excel in passing various professional examinations requiring expert knowledge in domains like medical @xcite and legal @xcite . nevertheless, human users are hardly willing to seek professional suggestions from llms, due greatly to hallucinations in llms. hallucinations in llms refer to the phenomenon that existing llms frequently generate untruthful information @xcite , which greatly undermines people's trust and acceptance of llm-generated content. an important cause of hallucinations is the model's insufficiency in knowledge boundary expression, which originates from the learning paradigm of llms. pre-training and instruction fine-tuning serve as the two indispensable learning stages for current llms. the learning mechanism of these stages is to encourage llms to generate the provided text, which also makes llms prone to fabricating content when llms do not possess relevant knowledge @xcite . hence, llms are hardly instructed to express their ignorance, which is a lack of accurate knowledge boundary expression. given a specific llm and a question set, the corresponding question-answer pairs can be categorized based on two factors: (1) whether the model has corresponding parametric knowledge (knows v.s. unknows), and (2) whether the model is aware of the first factor (known v.s. unknown), as is depicted in figure 1 . hallucinations frequently occur in the "unknown unknows" scenarios, where the model is unaware that it should explain its ignorance like humans, instead of struggling to give a hallucinated response. fine-tuning models to express knowledge boundaries faces two significant challenges. the first challenge is how to efficiently obtain data that reflects the internal knowledge of a specific model. even if evaluation questions are easy to construct, obtaining expert-level answers in certain fields is costly. additionally, since the model might produce correct answers in different forms from the reference answers, evaluating their correctness is also challenging @xcite . the second challenge is enabling the model to express its knowledge boundary robustly @xcite . we expect consistent knowledge boundary expression across prompts and generalization across domains. to address the above two challenges, we propose coke, an confidence-derived knowledge boundary expression method which teaches llms to express knowledge boundaries and decline unanswerable questions, leveraging their internal signals. our method consists of two stages: a probing stage and a training stage. in the probing stage, we use the model's internal signals reflecting confidence to distinguish between answerable and unanswerable questions, avoiding reliance on external annotations. this allows for easy collection of large data and avoids conflicts between the model's internal knowledge and annotations. in the training stage, we construct prompts for each question using three representative types: prior awareness, direct awareness, and posterior awareness. then, we apply regularization by incorporating the squared differences in confidence across different prompts for the same question into the loss function to enhance consistency. this training setup helps the model semantically learn to express knowledge boundary better, thereby enhancing its generalization ability. to evaluate the model's knowledge boundary expression capability, we design an evaluation framework that comprehensively assesses the model's performance in both "knows" and "unknows" scenarios. we conduct extensive experiments on both in-domain and out-of-domain datasets. results show that the model learns to use internal signals to help express knowledge boundary. compared to directly using model signals for determination, the models trained with our method demonstrate better performance and generalization. in summary, our contributions are: • we explore the effectiveness of internal model signals in indicating confidence and demonstrate the model can learn to use its signals to express its knowledge boundaries after training. ## experimental setup datasets we consider three open-domain qa datasets: triviaqa @xcite , natural questions @xcite , and popqa @xcite . these datasets are broad-coverage, knowledge-intensive qa datasets, making them well-suited for evaluating llms' capacity to perceive their internal knowledge. we utilize the train set of triviaqa as our training data, treating it as unsupervised data by not using the labels. natural questions and popqa serve as the out-of-domain test sets since they were not involved during the training process. metrics as mentioned in the section 3.1, we evaluate the model's awareness of its knowledge from two aspects: the awareness of the knowledge it possesses and the awareness of the knowledge it does not possess. since we cannot directly access the model's internal knowledge k θ , we divide the test sets into two parts based on whether the model's predictions match the groundtruth: t k represents the "known knows" of the model; t unk contains both the "unknown unknows" and "unknown knows" cases. we expect the model to maintain correct answers on t k , representing the retention of the "known knows" area of the model. at the same time, we expect the model to either express unknown on t unk , signifying a reduction in the "unknown unknows" area, or provide correct answers, representing a decrease in the "unknown knows" area. we define the evaluation metrics as shown in baselines we consider two different types of baselines: uncertainty-based methods (white-box) and prompt-based methods (black-box). we also compared the original model (orig.), the model fine-tuned with questions and their label (finetune), and the model fine-tuned with question-label pairs, where responses to unknown questions are replaced by "unknow" (idk-ft). see appendix a for more details. uncertainty-based methods directly use the model's internal signals to determine its selfawareness. the model's response consists of multiple tokens, and we experimented with three types of methods to calculate the final confidence score from the probabilities of these tokens: prompt-based methods use prompts to let models express their own knowledge boundary in natural language. ## conclusion in this paper, we target the knowledge boundary expression problem and propose coke, a novel unsupervised approach for this task. our approach is built on detecting signals of the model indicating confidence, and