To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity (2024)

Abstract: One of the major aspects contributing to the striking performance of large language models (LLMs) is the vast amount of factual knowledge accumulated during pre-training. Yet, many LLMs suffer from self-inconsistency, which raises doubts about their trustworthiness and reliability. This paper focuses on entity type ambiguity, analyzing the proficiency and consistency of state-of-the-art LLMs in applying factual knowledge when prompted with ambiguous entities. To do so, we propose an evaluation protocol that disentangles knowing from applying knowledge, and test state-of-the-art LLMs on 49 ambiguous entities. Our experiments reveal that LLMs struggle with choosing the correct entity reading, achieving an average accuracy of only 85%, and as low as 75% with underspecified prompts. The results also reveal systematic discrepancies in LLM behavior, showing that while the models may possess knowledge, they struggle to apply it consistently, exhibit biases toward preferred readings, and display self-inconsistencies. This highlights the need to address entity ambiguity in the future for more trustworthy LLMs.

## introduction large language models (llms) have demonstrated remarkable performance in a variety of natural language processing tasks @xcite @xcite , also largely due to the extensive factual knowledge they accumulate during pre-training. a crucial factor in building trust in models is their capacity to generate consistent and dependable outputs that align with their internal knowledge @xcite . however, llms frequently produce unreliable responses: e.g., when externally provided knowledge conflicts with internal parametric knowledge @xcite , or when exposed to misinformation during pre-training @xcite . such conflicts often lead to inconsistencies in model responses, reducing llms' trustworthiness @xcite . in this work, we highlight entity ambiguity, a frequent challenge for llms @xcite @xcite , as a source of unreliability and conduct an in-depth qualitative analysis to better understand how models behave in its presence. specifically, we examine the self-consistency foot_0 of state-of-the-art llms - importantly, in our study we provide an operationalization to disentangle llm's capabilities of knowing 3 (i.e., how aware and sensitive a model is to the possible interpretations, or readings, of ambiguous entities), and applying knowledge (i.e., how well a model can identify the correct reading when prompted with entity-specific questions and use their parametric knowledge to provide accurate responses about that entity). for example, as shown in figure 1 , if a model "knows" that apple can be a fruit and a company, to what extent can we assume that the model infers the company meaning when asked about the founding year of the entity? similarly, if a model responds with "apple was founded in 1976" can we assume that it remains self-consistent with its own answer? gaining more insights on the capability of an llm to deal with entity ambiguity has several benefits. it facilitates more natural conversations, as knowing whether an llm can disambiguate an entity allows to minimize the number of clarification questions @xcite , and enhances trust in llms via its ability to acknowledge previously provided knowledge. the overarching goal of this work is to provide a behavioral test suite for studying how a model's knowledge of different entity readings interacts with its ability to infer and apply the correct reading for a given prompt. specifically, we address the following research questions: (rq1) how well do llms resolve entity ambiguity within a given prompt context? (rq2) how (much) is the ability to infer the correct entity type biased towards "preferred readings"? knowledge retrieval" tasks that do not necessitate cot prompting as in, for example, @xcite . 3 the term "knowing" refers to parametric knowledge as discussed in @xcite ; @xcite . we examine the behavior of six state-of-theart llms, varying in size, type, and whether they are open-source or proprietary: mistral-7b-instruct-v0.2 @xcite , gemma-1.1-7b-it @xcite , llama-3-70b (meta, 2024), mixtral-8x7b-instruct-v0.1 @xcite , gpt-3.5 (openai, 2022), and gpt-4o (openai, 2024) foot_1 . our results show that, despite the seemingly simple task, llms fail to disambiguate and handle entities consistently. ## methodology to study llms' ability to implicitly infer correct entity meanings, we devise a behavioral test suite. we use a set of entities that can be interpreted as either (1) one of six entity types or (2) company names ( our research setup comprises four studies (see figure 1 ). study 1 verifies knowledge possession in models; studies 2 and 3 assess the models' abilities to apply this knowledge (@xmath0; and study 4 evaluates the knowledge possession post-application (@xmath1. collectively, the results of our four experiments provide us a way to gain knowledge on how llms treat entity level ambiguity, i.e., the mutual relationship k ↔ a. study 1: knowledge verification (k). first, we analyze the models' knowledge by verifying their awareness of different entity readings. to this end, we use the prompt template "tell me @xmath2: i.e., when asked about "fruit apple," the models provides some information about the fruit, and when asked about "company apple," it should discuss a company named apple. next, we ensure that all entities in our experiments pass study 1, resulting in a final list of 49 entities shown in ## results and discussion rq1: how well can llms implicitly disambiguate entity types? by design, all entities passed study 1, meaning the llms are aware of both entity readings and can provide relevant information about the associated objects or companies. however, when directly prompted about ambiguity awareness (study 1a), only llama-3 recognized ambiguity in all entities, while other models struggled with this task (see figure 2 summarizes the results of study 2. for four of six entity types, all models favored the same interpretation: non-company for animals and fruits, and company for people and locations; for the others, the preferred reading varies by model. notably, the large llama-3 and mixtral models, despite ultimately grouping entities based on a single reading, demonstrate an understanding of their ambiguity (e.g., mixtral: "all of the words you've listed are common names for either a type of animal or a brand [...]") foot_3 . another observation is ## conclusion we find that state-of-the-art llms perform poorly at implicitly disambiguating entity types. furthermore, their performance is biased by a preferred reading that is heavily influenced by the popularity of the entities. finally, we find that llms cannot reliably self-verify their own answers. our results highlight the lack of self-consistency as an open challenge of current llms.