Do We Know What LLM s Don’t Know? A Study of Consistency in Knowledge Probing (2025)

Abstract: The reliability of large language models (LLMs) is greatly compromised by their tendency to hallucinate, underscoring the need for precise identification of knowledge gaps within LLMs. Various methods for probing such gaps exist, ranging from calibration-based to prompting-based methods. To evaluate these probing methods, in this paper, we propose a new process based on using input variations and quantitative metrics. Through this, we expose two dimensions of inconsistency in knowledge gap probing. (1) **Intra-method inconsistency:** Minimal non-semantic perturbations in prompts lead to considerable variance in detected knowledge gaps within the same probing method; e.g., the simple variation of shuffling answer options can decrease agreement to around 40%. (2) **Cross-method inconsistency:** Probing methods contradict each other on whether a model knows the answer. Methods are highly inconsistent – with decision consistency across methods being as low as 7% – even though the model, dataset, and prompt are all the same. These findings challenge existing probing methods and highlight the urgent need for perturbation-robust probing frameworks.

## introduction while large language models (llms) are increasingly applied across diverse nlp tasks, understanding the limits of their knowledge remains a core challenge -particularly in mitigating hallucinations @xcite , where models produce fluent yet factually incorrect outputs @xcite @xcite . this has led to increasing interest in identifying knowledge gaps -a situation where the model lacks the necessary knowledge to answer a question, meaning it either does not know or is uncertain about the correct answer. figure 1 : examples of the two major dimensions of inconsistency in knowledge gap probing that we focus on in this paper. in the intra-method comparison (top), the same probing method gives contradictory assessments (certain vs not certain) for the same llm and the same question with options shuffled, revealing internal inconsistency. in the cross-method comparison (bottom), different probing methods applied to the same llm yield conflicting judgments -two probing methods maintain that the llm is certain about the answer, while the third does not. these results illustrate that knowledge gap detection in llms can be unreliable and sensitive to method choice and prompt perturbation. to address this, a growing body of work proposes probing methods that aim to act as detection tools for llms' knowledge gaps @xcite , based on various signals such as prompting @xcite , self-consistency @xcite , token probabilities @xcite and calibrated hidden representations @xcite . these knowledge probing methods are increasingly used to infer whether a model "knows" the answer to a question. however, an underexplored issue is the consistency and reliability of the probing methods themselves: based on their predic-tions, how reliable are these probing methods and do we actually know what llms don't know? to answer this question, we present a systematic study of consistency within and between probing methods. in practical applications, prompt variations such as typos or slight changes in word order are common. while they can influence llm outputs @xcite @xcite , the underlying knowledge gaps should remain unchanged, and probing methods should be robust to such slight perturbations. to evaluate whether probing methods are reliable, we conduct a systematic evaluation of consistency within and across popular probing methods. specifically, we propose a framework with two comparison dimensions (as illustrated in figure 1 ): we design four distinct prompting variants to systematically evaluate different probing methods, llms and datasets, and we propose new consistency metrics for the two diagnostic axes (intramethod and cross-method) to quantify the consistency. note that we do not evaluate whether llms are consistent, but whether the probing methods that evaluate llm behavior are consistent and whether we can thus trust their assessment. our work reveals a paradox: these tools themselves suffer from alarming inconsistencies, casting doubt on the validity of their predictions. we identify four main findings: ## consistency evaluation methods since knowledge probing methods aim to extract what a model knows (or does not know), their decisions should be consistent. first, applying the same method to semantically equivalent prompts (e.g., adding a minor typo) should yield consistent results, which we call intra-method consistency. second, different methods applied to the same model should produce aligned results, avoiding contradictions. we call this cross-method consistency. for the intra-method consistency, we design semantically equivalent prompts. our zero-shot variants simulate real-world noise: inserting spaces, shuffling options and minor typos. our one-shot variants help the model better understand the answer format. they are simple questions that do not introduce new knowledge and are assumed to have no effect on the model's knowledge gaps. see appendix a.3 for details on these variants. our consistency comparisons are always between two setups that either involve the same method with original vs variant prompts (intramethod) or the same prompt applied across different methods (cross-method). as illustrated in figure 1 , there are two types of pairs: for any two cases 1 and 2 in the same pair, we define r 1 and r 2 as the sets of questions where the probe identifies a knowledge gap, i.e., the model should abstain from answering (rejection). similarly, a 1 and a 2 represent the sets of questions where the probe claims that the models know the answer (acceptance). based on this notation, we propose four metrics: acceptance/rejection consistency intersection over union (iou acc /iou rej ) is defined as the ratio of the intersection (the number of common accepted/rejected questions) to the union (total distinct accepted/rejected questions): higher values indicate greater consistency in acceptance/rejection decisions. ## conclusion in this study, we explore the consistency of four types of knowledge probing methods based on different principles. our results reveal a high level of inconsistency, both intra-method and crossmethod. this variability suggests that a more robust approach is needed to reliably detect knowledge gaps across different models and datasets. current refusal mechanisms often rely heavily on the output of the probing methods to decide whether a model "knows" an answer and should refuse to answer uncertain questions. however, if these probing sig-nals are themselves unstable or inconsistent across variants and architectures, then the rejection behavior becomes inherently unreliable. this undermines the interpretability and trustworthiness of abstention-based frameworks. we recommend that future work on knowledge probing explicitly consider the consistency of probing methods and routinely report consistency metrics such as those proposed in this paper. improving the reliability of these methods is essential for building systems that can reliably assess the knowledge captured by language models.