Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering (2024)

Abstract: Instruction-following models are attractive alternatives to fine-tuned approaches for question answering (QA). By simply prepending relevant documents and an instruction to their input, these models can be adapted to various information domains and tasks without additional training. However, these models tend to produce verbose responses with supplementary information, which makes traditional QA metrics like exact match (EM) and F1 unreliable for accurately quantifying model performance. In this work, we evaluate instruction-following models along two fronts: 1) how well they satisfy user’s information need (correctness), and 2) whether they disseminate information supported by the provided knowledge (faithfulness). Guided by human evaluation and analysis, we highlight the shortcomings of traditional metrics for both correctness and faithfulness and propose simple token-overlap metrics that correlate highly with human judgments. Our analysis reveals that for correctness, instruction-following models perform comparably to models specifically fine-tuned for that task. However, they struggle to accurately judge the relevance of the provided knowledge and often hallucinate in their responses. We hope our work encourages more holistic evaluation of instruction-following models for QA. Our code and human annotation data is available at https://github.com/McGill-NLP/instruct-qa.

## introduction instruction-following models, such as chatgpt, are appealing as they can perform tasks based on natural language instructions. these models are usually trained by exposing large language * core contributor. models (llms; @xcite @xcite to thousands of nlp tasks formulated as instructions @xcite @xcite @xcite @xcite or to synthetic examples generated by other llms @xcite @xcite . in this paper, we evaluate factual question-answering (qa) ability of instruction-following models using a given set of text passages. instruction-following models can perform qa when provided with a task description, question, and relevant text passages @xcite . user-centric applications (e.g., bing chat) typically pair these models with a retriever or internet search to provide relevant information. these models generate natural, informative, and verbose responses, a useful trait that helps build users' trust and engagement. however, the verbosity renders traditional evaluation metrics such as exact match (em) and f1 unreliable, raising new challenges for evaluation @xcite . moreover, these models also tend to provide supplementary information that may be hallucinated @xcite . consider figure 1 , where the user asks where are one direction from?. comparing the reference answer london, england with the first part of model response one direction are from london, england yields 0 em and 0.5 f1 score, despite both answers being effectively equivalent (the entire response scores 0.36 f1). moreover, the model asserts that one direction is from mullingar. while correct, this fact is unsupported by the provided knowledge. as em and f1 only compare with reference answers, they cannot estimate if the model response is supported by the provided knowledge. we posit that an optimal model should not only correctly respond to user queries but also be faithful, i.e., it should only disseminate information that is inferrable or directly stated by external documents @xcite . the resulting interpretability builds user trust and allows for dynamic knowledge updates @xcite . in this work, we advocate that qa models should be evaluated along two fronts: 1) correctness w.r.t. information need, which measures model's efficacy in satisfying a user's information needs, and 2) faithfulness w.r.t. provided knowledge, which measures a model's capability to ground factual information in provided knowledge. we evaluate several recent instruction-following models-flan-t5 @xcite , alpaca @xcite , gpt-3.5 (sibling model of @xcite , and llama-2 @xcite )-on three popular factual information-seeking qa datasets: natural questions (nq; @xcite , hotpotqa @xcite , and topiocqa @xcite . we conduct a human analysis of 1800 model responses and correlate them with several automatic metrics for correctness and faithfulness. our findings suggest that for correctness, recall (proportion of tokens in the reference answer that are also in the model's response) exhibits a higher correlation than traditional qa metrics like em or f1. for faithfulness, k-precision (pro-portion of tokens in model response that appear in the knowledge snippet) correlates better than any other lexical metric. although gpt-4 as an evaluator achieves the highest correlation for both correctness and faithfulness, it is expensive and prone to systematic biases @xcite . we demonstrate that our proposed lexical metrics are close to gpt-4-based evaluation, allowing us to evaluate instruction-following models at a large scale. a faithful model should not only answer when the provided knowledge is relevant, but also abstain from answering when it is irrelevant. hence, we also consider the model's ability to abstain from answering as a measure of its faithfulness. to summarize, our contributions are as follows: to increase diversity and scope of nlp tasks, instructgpt @xcite and llama-2 @xcite incorporate high-quality expert annotations during training. they are further trained using human feedback to align them with human preferences (rlhf; @xcite . another popular approach, self-instruct @xcite , reduces dependency on human-authored instructions by bootstrapping an llm to generate instructions and demonstrations of new tasks. the resultant dataset is used to train instruction-following models @xcite . recent works @xcite have paired retrievers with few-shot language models for qa, alleviating the need to learn additional parameters. in contrast to these works, we evaluate retrieval-augmented instruction-following models without any demonstrations. in these settings, models do not follow the distribution of reference answers, raising new challenges for evaluation. evaluation in qa previous research on information-seeking qa has primarily relied on lexical matching metrics such as exact match (em) and f1 for model evaluation @xcite . as simple token-overlap metrics cannot capture semantic equivalence @xcite , subsequent model-based metrics employ contextualized embeddings @xcite or specialized classifier @xcite to predict equivalence. more recently, several studies resort to prompting llms like @xcite to act as evaluators @xcite @xcite @xcite . recently, @xcite compared the correctness of instructgpt @xcite with fine-tuned models for qa. they highlight the shortcomings of traditional qa metrics and propose model-based evaluation as a viable alternative. in contrast, we evaluate both correctness and faithfulness of instruction-following models and propose token-overlap metrics that correlate highly with human judgments. evaluating faithfulness conversational models produce factually incorrect or unsupported statements @xcite , known as hallucinations. several metrics have been proposed to detect hallucination, or conversely, to measure faithfulness. knowledge-f1 (k-f1; @xcite computes token-overlap f1 between model response and the provided knowledge. q 2 @xcite checks for factual consistency based on automatic question generation and question answering. faithcritic @xcite ) uses a trained model to predict hallucinations. recently, @xcite demonstrated that retrieval-augmented gpt-3 is likely to produce responses that appear trustworthy but are unfaithful to the retrieved passages. they propose k-f1++, a variant of k-f1 that discounts the tokens in the model response that appear in the question. in our experiments, we observe that this metric doesn't correlate well with human judgments. ## correctness w.r.t. information need in this section, we investigate the correctness of instruction-following models. we consider a model response to be correct if it accurately satisfies the user's information need. for example, when answering what is the capital of canada?, the model should convey that ottawa is the capital of canada. while the model's response might include additional information like ottawa's population, we limit the evaluation of correctness to the part of the model response that is directly relevant to the user's information need. we address evaluation of additional information in the next section (section 5). we describe lexical and semantic similarity metrics for the task in §4.1. next, we conduct human evaluation and compare several evaluation metrics ( §4.2). finally, using metrics that correlate highly with human judgments, we evaluate instruction-following models for correctness ( §4.3). ## conclusion in this paper, we analyze the responses of instruction-following models for qa along the dimensions of correctness w.r.t. information need and faithfulness w.r.t. provided knowledge. our results show that recall and k-precision metrics correlate well with human judgments for correctness and faithfulness respectively. on evaluating instruction-following models using these proposed metrics, we find that these models demonstrate a tradeoff between correctness and faithfulness. gpt-3.5 and llama-2 achieve high scores for correctness but have difficulty being faithful to the provided knowledge. moreover, they struggle to decide when to refrain from answering. when using instruction-following models for qa, we urge to the community to move away from reporting a single overall score and adopt a more holistic evaluation that reports correctness, faithfulness, and the ability to refrain from answering. limitations although we evaluate for correctness, faithfulness, and the ability to refrain from answering, it is not an exhaustive list of all the desirable properties of a qa model. previous works @xcite have focused on aspects like completeness and ease of understanding for long-form qa. we leave the evaluation of these properties for information-seeking qa to future work. it is important to note that low faithfulness of a response does not imply that it is incorrect. the model can potentially provide accurate information using its parametric knowledge. however, such knowledge is difficult to interpret and modify. we propose recall and k-precision for correctness and faithfulness respectively. although these metrics correlate highly with human judgments, they are easy to hack. for instance, recall might score an affirmative statement and its negated version equally, despite their contrasting meanings. however, qa models tend to answer in affirmation rather than negation. similarly, k-precision can be hacked by copying all the knowledge from the prompt. however, such strategy will be penalized heavily when evaluated for faithfulness w.r.t. irrelevant knowledge.