The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models (2023)

Abstract: Large language models (LLMs) have been shown to possess impressive capabilities, while also raising crucial concerns about the faithfulness of their responses. A primary issue arising in this context is the management of (un)answerable queries by LLMs, which often results in hallucinatory behavior due to overconfidence. In this paper, we explore the behavior of LLMs when presented with (un)answerable queries. We ask: do models represent the fact that the question is (un)answerable when generating a hallucinatory answer? Our results show strong indications that such models encode the answerability of an input query, with the representation of the first decoded token often being a strong indicator. These findings shed new light on the spatial organization within the latent representations of LLMs, unveiling previously unexplored facets of these models. Moreover, they pave the way for the development of improved decoding techniques with better adherence to factual generation, particularly in scenarios where query (un)answerability is a concern.

## introduction modern large language models (llms) have been tantalizing the nlp community in the last couple of years @xcite @xcite , demonstrating great potential for both research and commercial use, but these models are of course not problem-free. among their unfavorable behaviors it is possible to find toxicity @xcite , bias @xcite , and hallucination @xcite . one of the settings in which llms are notoriously prone to hallucinate is when presented with (un)answerable questions @xcite figure 1 : 3d pca projection of the last hidden layer's embedding of flan-ul2 on each of the three benchmarks. the left images show the embeddings with the regular prompt, and the right ones -with a hintincluding prompt. blue and red dots are examples correctly detected by the model as answerable and (un)answerable, respectively, while the pink dots are for @xcite answerable examples that the model provided answers to. the figures show the good separability between the three groups. and choi, 2021; @xcite . recent works in this setting, which is the focus of this work, suggested using models' confidence as an indication of answerability @xcite , and some suggested further finetuning to enhance the probability of detecting (un)answerable questions @xcite . we, however, ask whether models already represent questions' (un)answerablility when producing answers, and find strong evidence for a positive answer. specifically, by experimenting with three qa datasets @xcite @xcite , we observe a substantial increase in performance for (un)answerable questions (up to 80%) simply by incorporating to the prompt the possibility of (un)answerability. we further show that, even in the absence of guidance in the prompt, the fact that the question is (un)answerable is decodable from the model's representations. this is done by two methods: first, we find that the beam of decoded responses for (un)answerable queries often contains a response recognizing their (un)answerability; second, we demonstrate that the fact that the question is (un)answerable is easily decodable from the model's representations and that there is a linear separation between representations of (un)answerable and (un)answerable questions (see figure 1 ). the existence of the answerability subspace is largely independent of the specific qa dataset used, in the sense that an answerability classifier trained over representations of questions from one dataset can successfully classify (un)answerable questions from other datasets as well. in addition to providing illuminating insights into the internal mechanics of llms, these findings also open up new avenues for better decoding methods @xcite to improve performance in general and on (un)answerable questions in particular. ## related work in previous research, (un)answerable questions were used to evaluate reasoning capabilities @xcite @xcite . it was squad v2 @xcite that provided the first reading comprehension dataset for validating models' ability to deal with (un)answerability, by introducing questions that cannot be addressed from the given context. @xcite followed the same line and included about a third of (un)answerable questions in their natural questions (nq), an annotated open-domain qa dataset. recently, musique @xcite was introduced as a challenging multi-hop qa benchmark that consists of (un)answerable questions, in which supporting paragraphs have been intentionally removed from the context. our experiments use these datasets to demonstrate the effectiveness of our approach to identify (un)answerability. other works tried to manipulate the model predictions towards better (un)answerability via using data augmentation @xcite , and @xcite provided an in-depth analysis of the ability to detect (un)answerability in lms, where the case study is the data which is fed to the model. furthermore, recent studies have suggested utilizing recent advances in white-box model interpretability @xcite @xcite @xcite and probing @xcite @xcite for manipulating the model predictions and analyzing when llms struggle to answer questions. recent works also tried to use beam search decoding to manipulate the generated outputs by using the information encapsulated in several beams @xcite @xcite . finally, early exiting in language models @xcite @xcite and model prediction calibration @xcite @xcite are strongly related to our work, as they suggest to analyze and improve the model predictions and output distribution. ## method we posit the hypothesis that, despite the inclination of llms to produce answers to (un)answerable queries, they do encode the (un)answerability of such queries within their latent representations. we examine this hypothesis by undertaking three distinct experimental approaches: (1) prompt manipulation, (2) beam scrutiny, and (3) probing (including identification and erasure of an answerability subspace). ## experimental setup our experiments focus on several language models and on three benchmarks. benchmarks we consider three qa benchmarks, incorporating (un)answerable questions, in a reading comprehension setting where models are tasked with responding to a question within a given context. for each benchmark, we use the entire development set to construct our testing dataset. ad-ditionally, for the probing experiments involving the training of linear classifiers on the models' embeddings, sample 1000 instances from each benchmark's trainset, evenly distributed between answerable and (un)answerable instances. of these, we reserve 800 instances for the training of classifiers, with the remaining instances forming the development set for these classifiers. below, we describe how we associate asnwerable questions with paragraphs that contain the answer, and how we associate (un)answerable questions with challenging paragraphs (that do not contain the answer, but may be topically similar to the question). our first benchmark is squad 2.0 @xcite , a reading comprehension dataset, composed of manually-curated question-answer pairs alongside (un)answerable questions, each derived from a single paragraph. the second benchmark we explore is natural questions (nq; @xcite , a dataset accumulated from user-generated queries on the google search engine. each item within the dataset consists of a question, a retrieved article, a selected paragraph from the article (referred to as the "long answer"), and a short answer inferable from the paragraph. despite its potential to test qa systems with a retrieval component, our interest lies exclusively in the question-answering setting, hence we utilize the "long answer" as the context, assuming an oracle retrieval system. for the formulation of answerable instances, we select cases with both a long and a short answer, using the former as the context and the latter as the response. for the (un)answerable questions, we pair each query with a paragraph from the sourced passage that has not been annotated as the "long answer". in order to create a challenging dataset, we select the paragraph that is closest in meaning to the question. to achieve this, we encode both the question and all potential paragraphs using a sentence-transformer (sentence-bert; reimers and gurevych, 2019) and select the paragraph that exhibits the highest cosinesimilarity score. our final benchmark is the musique dataset @xcite , a multi-hop dataset featuring both answerable and (un)answerable questions. each instance consists of a question, several candidate paragraphs, an answer, and a decomposition of the question into its single-hop sub-questions. additionally, each sub-question is paired with a paragraph that has its answer, with all those align- evaluation the main task over which we evaluate models is the (un)answerability classification task. when evaluating qa models over this task we only examine whether they tried to answer, i.e., we count every example for which the model provides an answer as an instance of answerability prediction, and each example for which the model did not provide an answer as an instance of un-answerability prediction, foot_2 . note that this evaluation does not consider the correctness of the answers provided. the metric associated with this task is the f1 score, with "unanswerable" considered the positive label. linear classifiers ( §3.3) are also evaluated over the (un)answerability classification task, as the classifier is trained to predict whether or not the question is answerable, based on the hidden representations of the lm. additionally, in order to make sure that our methods do not hinder the performance of the models over their primary task, we evaluate them over the qa task as well, using the splits provided by the tasks' designers. we report the commonly used metrics: exact match (em) and (token-wise) f1 scores @xcite . language models we evaluate three instructionfinetuned large language models: namely, the 'xxl' variant of the flan-t5 model (flan-t5 xxl ; chung et al., 2022), the flan-ul2 model @xcite , and the opt-iml model @xcite . ## conclusion we found ample evidence for lm's ability to encode the (un)answerability of questions, despite the fact that models tend to be over-confident and generate hallucinatory answers when presented with (un)answerable questions. we also showed that this discrepancy between model output and its hidden states is mitigated by simply adding the option of (un)answerability to the prompt. the evidence we found includes the existence of a reply acknowledging the (un)answerability in a beam of decoded answers, meaning that even though the models' best-assessed answer is hallucinatory, the true answer is not lagging too far behind. we also showed that the models' representations after encoding the question and before decoding the answer are highly influenced by the answerability of the question or lack thereof, with answerable and (un)answerable questions being linearly separable in the embedding space. we conclude that the problem of answered (un)answerable questions can be mended with either better prompting, better decoding, or simple auxiliary classification models.