Is the Answer in the Text? Challenging C hat GPT with Evidence Retrieval from Instructive Text (2023)

Abstract: Generative language models have recently shown remarkable success in generating answers to questions in a given textual context. However, these answers may suffer from hallucination, wrongly cite evidence, and spread misleading information. In this work, we address this problem by employing ChatGPT, a state-of-the-art generative model, as a machine-reading system. We ask it to retrieve answers to lexically varied and open-ended questions from trustworthy instructive texts. We introduce WHERE (WikiHow Evidence REtrieval), a new high-quality evaluation benchmark of a set of WikiHow articles exhaustively annotated with evidence sentences to questions that comes with a special challenge: All questions are about the article’s topic, but not all can be answered using the provided context. We interestingly find that when using a regular question-answering prompt, ChatGPT neglects to detect the unanswerable cases. When provided with a few examples, it learns to better judge whether a text provides answer evidence or not. Alongside this important finding, our dataset defines a new benchmark for evidence retrieval in question answering, which we argue is one of the necessary next steps for making large language models more trustworthy.

## introduction generative language models (lms) are trained to generate an output text given an input text. while such models have recently shown a remarkable performance in various nlp tasks @xcite @xcite , they are known to suffer from hallucination, i.e., they often generate text that lacks evidence @xcite . this may lead to the spread of misinformation @xcite , and thus reduce the systems' trustworthiness. are there any risks or precautions to keep in mind when trying to raise alkaline phosphatase levels in the body? what is alkaline phosphatase and why is it important to maintain healthy levels? how to raise alkaline phosphatase levels @xcite alkaline phosphatase (alp) is an enzyme present throughout your body, and mostly it's nothing to worry about. @xcite however, alp deficiencies do happen sometimes as a symptom of larger medical problems. [...] [6] === testing your alkaline @xmath0. [...] @@xmath1. in this paper, we focus on a use case where a generative lm is queried for advice on a range of personal issues, including health or interpersonal relationships, or difficult tasks. this is a challenging scenario for lms because questions are often openended and non-factoid, and require well-informed instructions as answers. as illustrated in figure 1 , we explicitly query generative lms to retrieve evidence sentences for answering a question from a trustworthy instructive text. our challenging setup requires two competencies on the model side: (i) identifying whether or not the question is answerable using only the provided text as input, and (ii) retrieving evidence from the trustworthy source, which could, e.g., support a generated answer. existing question answering datasets, e.g., wiki-howqa @xcite and squad @xcite , do not fit our evaluation setup. the wikihowqa dataset @xcite ) uses the titles as questions, and does not cover the sentence retrieval aspect. squad contains unanswerable questions but focuses on factoid questions. to make our evaluation setup challenging and sound, we create a new high-quality test set. we collect a set of diverse and open-ended questions for wikihow articles via crowd-sourcing, and perform double annotation of evidence sentences in the articles. we use our dataset to perform a challenging evaluation of chatgpt, a successor of instructgpt @xcite , which has been pretrained on a huge amount of texts including instructive web texts, in a systematic manner. our contributions are as follows: (1) we create and publish where (wikihow evidence retrieval), a new high-quality evaluation test set of lexically diverse and open-ended questions for instructive articles taken from wikihow. evidence sentences are annotated with high agreement in all documents. where contains both questions with evidence in the article and questions without. ## wikihow evidence retrieval dataset our goal is to collect questions with and without answer evidence in the text that are lexically, syntactically, and semantically diverse. in a pre-study, we find it hard to achieve this goal when the content of the article is known to the writer. we resort to crowdsourcing for question writing, and identify good cases by double-annotating the sentences deemed as evidence for the answer. our reasoning is that if agreement on this task is low, the question is either somewhat ill-posed or too close to the overall topic of the article. 1 https://github.com/boschresearch/where_emnlp_ findings2023 ## method we evaluate chatgpt (version of march 2023, built upon gpt-3.5) in two settings. in the zeroshot setting, we prompt chatgpt using a template that asks the model to output a list of sentence ids that can provide evidence for the answer. the list is empty if the model does not retrieve any evidence (see appendix a). in the few-shot setting, we configure the message history such that chatgpt sees five training examples from the test instance's category, e.g., health, using the same prompt template as in the zero-shot setting but including gold responses. the training instances consist of additional question-article pairs annotated by the paid annotator team only. every five-shot training set contains exactly two no-evidence questions and three with-evidence questions, which are about different documents. 4 to accommodate for chatgpt's context window size of 4096 tokens, we only use chunks of the training articles in the few-shot setting. appendix b provides details. since chatgpt may generate anything even though we ask it to output a list, we need to post-process the model outputs before evaluation. we first attempt to parse the model output using python's eval function. if this fails, we try to match lists within the string (as chatgpt sometimes provides explanations or just copies the selected sentences in addition to the list) using various regular expressions. if none of this works, which happens only three times in the entire experiment, we manually parse the model output. a small number of questions is rejected by openai's content filter. for those, we assume the empty list as model output as chatgpt does not retrieve any evidence from the article. ## experiments and analysis evaluation metrics. for no-evidence questions, we report recall for correctly recognizing that an article does not contain any evidence for answering a question. for with-evidence questions, we compute precision, recall, and f1 score of the rele-vant class ("evidence sentence") per question, and report macro-average over questions. baselines. the random baseline for withevidence questions predicts that a sentence contains relevant evidence in 14% of the cases, which corresponds to the average percentage of sentences marked as evidence per question in the test set. similarly, it predicts "no-evidence" for a question with a probability of 49.2%. the results reported for "human scores" correspond to the agreement scores. results. analysis. in the zero-shot setting, openai's content filter identifies the content of 10 of the test cases harmful and declines to generate outputs for them. we do not consider these instances to be harmful, e.g., they are about instructions on how to find emergency procedures in hotels. for the few-shot configurations, these cases vary from 11 to 16 instances, depending on the few-shot prompts. this illustrates that improving content filters is an important future research direction. ## related work extractive qa datasets. the squad datasets @xcite are the largest datasets for extractive qa. the questions are factoid and can be answered through short, single answer spans. in contrast, our dataset includes non-factoid questions based on instructional text and requires models to identify a set of answer spans across long documents. the multispanqa @xcite and mash-qa @xcite datasets also contain qa pairs with several answer spans. however, multispanqa is automatically derived from natural questions @xcite and mash-qa also contains automatically created qa pairs. only few datasets contain unanswerable questions, e.g., the latest version of squad. however, its questions have been created given a text passage, whereas the questions of where are based only on the keywords and a summary of an article. this leads to larger lexical variety between the question and the text and, thus, creates a more realistic and challenging setting. evidence retrieval. our work builds on a long line of nlp work addressing the fine-grained retrieval of supporting facts from text. this is a common basis for tasks like question answering (murdock et al., 2012), fact checking, or slot filling @xcite , and has also been used to support other tasks, e.g., question generation (lewis et al., 2020), document retrieval (akkalyoncu yilmaz et al., 2019) or natural language inference (vladika and matthes, 2023). ## conclusion and outlook in this work, we have introduced a challenging benchmark to evaluate chatgpt on the task of retrieving evidence sentences from instructive text to answer a question. we have presented a new high-quality dataset from wikihow consisting of crowd-sourced questions and expert evidence annotations. a special real-world challenge of our dataset is the inclusion of questions that are not answerable given an article. when evaluating chat-gpt on our dataset, we found that it fails on detecting no-evidence questions if not provided with targeted examples in the prompt. our results on the new benchmark highlight important shortcomings of generative large language models that need to be addressed by future research: besides hallucinating answers, there is no free lunch with regard to calibration in evidence-retrieval setups.