A Critical Evaluation of Evaluations for Long-form Question Answering (2023)

Abstract: Long-form question answering (LFQA) enables answering a wide range of questions, but its flexibility poses enormous challenges for evaluation. We perform the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices. We hire domain experts in seven areas to provide preference judgments over pairs of answers, along with free-form justifications for their choices. We present a careful analysis of experts’ evaluation, which focuses on new aspects such as the comprehensiveness of the answer. Next, we examine automatic text generation metrics, finding that no existing metrics are predictive of human preference judgments. However, some metrics correlate with fine-grained aspects of answers (e.g., coherence). We encourage future work to move away from a single “overall score” of the answer and adopt a multi-faceted evaluation, targeting aspects such as factuality and completeness. We publicly release all of our annotations and code to spur future work into LFQA evaluation.

## introduction long-form question answering @xcite @xcite , an emerging research area within qa, requires systems to generate long and complex answers to questions by leveraging large language models and evidence document retrievers. while remarkable strides have been made in lfqa model development, the current state of lfqa evaluation is dire: most prior papers use a combination of crowdsourced human annotations and simple string-matching metrics (e.g., rouge). we present the first study of the evaluation of longform answers, exploring both human and automatic evaluation protocols to better understand how we should evaluate lfqa moving forward. ## background and related work we begin by reviewing the evaluation protocols used by prior work in lfqa, which has centered around a dataset scraped from the "explain like i'm five" subreddit @xcite ). 2 we include brief review of evaluation in other text generation tasks in appendix a.1. prior automatic evaluations: early work on lfqa @xcite uses rouge @xcite to measure the similarity of human reference answers to model-generated answers. @xcite find that rouge is not a meaningful metric due to the open-ended nature of long-form answers, but they do not examine other automatic metrics. given the difficulty of evaluation, recent works re-scoped the task to allow more reliable evaluation: @xcite focus on exemplification in long-form answers by treating this sub-task as 2 https://www.reddit.com/r/explainlikeimfive a retrieval problem, while @xcite aim to evaluate long form answers limited to ambiguous factoid questions that cover the different disambiguated questions and their corresponding answers. however, these evaluation protocols cannot be easily adapted to the general lfqa task: the metric in @xcite , for example, requires a list of disambiguated questions and their answers, which is not available for many questions. prior human evaluations: we summarize the human evaluation studies conducted by two previous studies, hurdles @xcite and webgpt @xcite . both works evaluate via a/b testing (i.e., choose which of two candidate answers is better), and they collected judgments of overall answer quality, factuality, and coherence. while both works recruited non-expert annotators and collect only one-way annotations, webgpt's evaluation allows annotators to look at a set of evidence documents when judging the answer, and they also collect optional free-form justifications from the annotators to justify their choice. while fine-grained aspects such as coherence @xcite and factuality @xcite have been studied before for other tasks such as summarization, ours is among the first works to study lfqa-centric properties such as completeness or ease of understanding. ## conclusion & future work our study provides a unified evaluation benchmark for long-form answers, including new annotations from domain experts. we present a new set of expert lfqa evaluations along with detailed justifications, and we also compile existing human annotations across different properties (overall preference, factuality, coherence) to facilitate future development of automatic lfqa metrics. evaluation of long-form answers is a multifaceted problem and thus should be more targeted. our expert justifications suggest that many aspects are considered when deciding which answer is better, some of which may be at odds with others (e.g. completeness vs. conciseness). this suggests that computing an "overall" score for answer quality is not meaningful, which is further supported by the limitations of metrics trained directly from overall preference judgments. future work should look deeper into modelling frequent aspects mentioned by expert annotators, such as completeness and ease of understanding, perhaps by taking inspiration from evaluation methods that explicitly localize and categorize errors @xcite .