Hurdles to Progress in Long-form Question Answering (2021)

Abstract: The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system’s generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.

## introduction long-form question answering (lfqa) integrates the retrieval component of open-domain qa, which involves searching a large external knowledge source for documents relevant to a given question, with a text generation component to produce paragraph-length answers. significant progress has been made on open-domain qa datasets such as natural questions @xcite , whose questions are answerable with short phrases and entities, by leveraging dense retrieval techniques like orqa @xcite , realm @xcite , and dpr @xcite @xcite . methods inspired by these results have recently been combined with pretrained language models @xcite and applied to the reddit-derived "explain like i'm five" (eli5) dataset @xcite , which is the only publicly-available large-scale lfqa dataset. the recently proposed kilt benchmark @xcite , which compares retrieval-augmented models across a variety of knowledge-intensive tasks including eli5, automatically evaluates lfqa models by the quality of both generated answers (rouge-l against reference answers) and retrieved documents (r-precision against humanannotated relevant documents). in this paper, we build a state-of-the-art system foot_0 for eli5 by using a sparse transformer variant @xcite to condition over wikipedia paragraphs returned by a realm-style retriever @xcite . however, despite its success on the kilt leaderboard, our system does not actually use the documents that it retrieves! to measure the effect of retrieval on generation quality, we design a control experiment in which retrieved documents are replaced with randomly-sampled documents at inference time. results from both human a/b tests and automatic metrics like rouge-l demonstrate that conditioning on random documents has almost no effect on generated answer quality (figure 1c ). we recommend that future lfqa research report the results of such control experiments in addition to reporting generation and retrieval quality. how can a system using random retrieval perform well on eli5? our analysis reveals that this result is partially due to significant train / validation overlap in the eli5 dataset (figure 1a ), which eliminates the need for external retrieval. a human study shows that at least 81% of validation questions have a paraphrase in the training set, and almost all validation questions are topically similar to a training set question. while @xcite attempted to identify and remove question overlap using tf-idf similarity, more complex semantic matching methods & human verification is needed to address this issue in future lfqa datasets. digging deeper, we identify fundamental issues with using rouge-l to evaluate generated answer quality (figure 1b ). simple baselines such as just repeatedly copying the question, or choosing a random training set answer, can outperform lfqa systems such as rag @xcite in terms of rouge-l. on the other hand, our system achieves higher rouge-l than reference human-written answers, which is misleading since human a/b testers strongly prefer reference answers to our system's. we conclude that rouge-l is not a reliable metric to evaluate lfqa due to its large and relatively unconstrained output space (e.g., compared to translation or summarization), and we offer suggestions for better automatic & human evaluations to enable meaningful progress on this task. ## a state-of-the-art lfqa system the eli5 task @xcite asks models to generate paragraph-length answers to open-ended questions in english that often rely on world knowledge (e.g., how do jellyfish function without brains or nervous systems?). lfqa systems thus benefit from conditioning answer generation on relevant documents from the web (such as the wikipedia article about jellyfish). while large-scale pretrained language models store surprising amounts of world knowledge within their parameters @xcite , external document retrieval not only augments this intrinsic knowledge but also grounds model outputs in a knowledge source, which provides interpretability. in this section, we describe our proposed lfqa system, which conditions answer generation on wikipedia articles identified by a pretrained retriever. we use a dense retriever trained by scaling up a distantly supervised algorithm from jernite (2020). since retrieved articles can be quite long and often exceed the maximum sequence length of pretrained models like bert @xcite , we use a sparse-attention variant of the transformer to allow modeling over longer sequences. while our system sets a new state-of-the-art on eli5, we question the significance of this result in section 3. ## analysis in this section, we conduct a thorough analysis of our model's usage of retrievals (section 3.1), the impact of overlap in eli5's train / validation / test folds (section 3.2), issues with rouge-l and performance bounds (section 3.3), and the difficulty in human evaluation for this task (section 3.4). at the end of each section, we provide short takeaways with suggestions for future work. ## conclusion we present a "retrieval augmented" generation system that achieves state-of-the-art performance on the eli5 long-form question answering dataset. however, an in-depth analysis reveals several issues not only with our model, but also with the eli5 dataset & evaluation metrics. we hope that the community works towards solving these issues so that we can climb the right hills and make meaningful progress on this important task.