W iki H ow QA : A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering (2023)

Abstract: Answering non-factoid questions (NFQA) is a challenging task, requiring passage-level answers that are difficult to construct and evaluate. Search engines may provide a summary of a single web page, but many questions require reasoning across multiple documents. Meanwhile, modern models can generate highly coherent and fluent, but often factually incorrect answers that can deceive even non-expert humans. There is a critical need for high-quality resources for multi-document NFQA (MD-NFQA) to train new models and evaluate answers’ grounding and factual consistency in relation to supporting documents. To address this gap, we introduce WikiHowQA, a new multi-document NFQA benchmark built on WikiHow, a website dedicated to answering “how-to” questions. The benchmark includes 11,746 human-written answers along with 74,527 supporting documents. We describe the unique challenges of the resource, provide strong baselines, and propose a novel human evaluation framework that utilizes highlighted relevant supporting passages to mitigate issues such as assessor unfamiliarity with the question topic. All code and data, including the automatic code for preparing the human evaluation, are publicly available.

## introduction non-factoid questions (nfqs) requiring long, passage-level answers, such as explanations or opinions, pose challenges for current questionanswering systems. while a few datasets exist for nfqa @xcite @xcite , users are currently limited to seeing a summary of the most relevant document in a snippet on a search result page (serp), which can be insufficient for complex questions that have scattered answers across multiple documents or require sophisticated reasoning 1 https://lurunchik.github.io/wikihowqa/ to generate an answer. however, even relatively straightforward questions like "how to fix my computer issue" can have multiple solutions, requiring the user to manually search through multiple retrieved documents to find the one that applies to their situation. complex questions such as "how to feel calm and relaxed" may require an aggregated summary of the most popular methods from multiple relevant sources. there are also questions for which answers have yet to be written, requiring a qa system to treat relevant documents as initial sources of information and then to reason out an answer based on them. for example, the question "how to prepare to buy a house in [neighbourhood], [city]" would require the system to retrieve relevant documents about buying houses in general and specific articles about the local house market and the neighbourhood, and then construct a more sophisticated answer through reasoning rather than just summarizing multiple documents. to address these challenges and move towards more advanced qa systems that can provide indepth and comprehensive answers to a wide range of questions, we propose a new benchmark for the task of multi-document non-factoid qa (md-nfqa). this task involves using multiple relevant documents to generate a complete and coherent answer to a given nfq. we focus our benchmark on the instruction category of nfqs, which often begin with "how to". these questions are underrepresented in current multi-document qa @xcite datasets, despite their popularity, as evidenced by a 140% increase in "how to..." searches on google since 2004 2 and making up over half of the most searched queries. 3 our benchmark aims to fill this gap by specifically targeting instruction questions. wikihow is a web-resource for instruc-tion questions that contains over 235,000 articles on a wide range of topics. these articles provide comprehensive step-by-step instructions and are written by a community of experts and reviewed by an average of 16 people. the wikihow website has proven to be a valuable resource for machine learning tasks @xcite @xcite @xcite @xcite @xcite , and in our work, we leverage wikihow to create a high-quality benchmark specifically designed for md-nfqa within the instruction question category. our benchmark consists of 11,746 questions from the instruction category, each paired with a corresponding human-written answer, sourced from a diverse range of wikihow articles. each qa pair is supported by corresponding parsed relevant html pages from which the answer can be derived (fig. 1 ). we evaluate several baseline models on the new benchmark that could serve as lower and upper bounds for model performance. human annotation is often used as the standard for evaluating long-form answers, but research has shown that individuals without specific training can only distinguish between human-generated and auto-generated text at a level equivalent to random chance @xcite . this presents a challenge for evaluating the new benchmark for nfqa, particularly when the answers are lengthy or the topic is unfamiliar @xcite or not interesting @xcite . given that standard metrics struggle to detect factual inconsistencies, such as number swapping, negation, etc. @xcite , we delve into human sensitivity to such discrepancies by conducting a 2 https://bit.ly/telegraph-google-how-to 3 https://bit.ly/most-asked-questions-on-google series of crowdsourcing experiments. inspired by the elaborate yet resource-intensive manual evaluation framework of abstractive models proposed by @xcite , we introduce a simple human evaluation framework that leverages highlighted relevant passages to enhance the quality of nfqa assessments. our findings demonstrate that incorporating highlighted relevant supporting passages into the evaluation process not only aids evaluators in understanding the context of each question and answer but also contributes to the factual accuracy of the evaluation. as a result, we integrate this method into the evaluation of the newly proposed benchmark. the new benchmark for md-nfqa serves as a valuable resource for the development of more advanced qa systems that can provide in-depth and comprehensive answers grounding information in supporting documents. by including parsed passages from relevant html pages and providing human judgments, our benchmark also has the potential to facilitate research in evaluating the factual correctness of long-form answers. all data is publicly available foot_0 on the dataset website. 1 ## related work this section covers datasets for related md-nfqa tasks such as long-form question answering (lfqa) and query-based multi-document summarization (qf-mds), highlighting the scarcity of resources available. we examine the strengths and limitations of these resources, as well as the differences between these tasks and md-nfqa. lfqa: the task of lfqa, introduced by @xcite , involves providing long answers to open-ended questions. while the associated eli5 benchmark includes supporting "document" to generate answers, the lfqa task, unlike md-nfqa task, does not assume that an answer is scattered across multiple documents. moreover, while md-nfqa exclusively targets nfqs, asqa lfqa dataset @xcite addresses the need for long-form answers to ambiguous factoid questions. similarly, natural questions dataset @xcite contains both long and short answers for factoid questions @xcite . the eli5 dataset @xcite for lfqa includes 272,000 questions from the "explain like i'm five" reddit web forum, where questions and answers must have a score of at least two to be included. the top-voted answer for each question is considered the correct answer, and the supporting content for each question is generated by extracting sentences with high tf-idf similarity from the top 100 web pages that match the question from the common crawl corpus. in contrast, our proposed md-nfqa dataset uses texts parsed from relevant html pages chosen by the author of the corresponding article on wikihow, rather than relying on automated methods for curating reference content. in addition to a lack of grounding in supporting content, @xcite found significant train/valid overlap in the eli5 dataset, and highlight challenges with both automatic answer evaluation and human annotation for this benchmark. in our proposed benchmark, we address these issues through the use of a novel human-evaluation framework utilizing supporting documents and by avoiding question overlap (sec. 5 and sec. 3). @xcite used questions from the eli5 dataset and collected new answers from human annotators who were instructed to search for related documents and use them to construct their answers. they trained the webgpt model on that dataset to answer long-form questions by mimicking the way humans research answers to questions online: it searches and navigates the web to find relevant pieces of information and concludes an answer based on them by citing sources for factual accuracy. authors report that the model answers are preferred by assessors 69% of the time to the best human-written answer from reddit when evaluating on eli5. while the webgpt dataset is valuable, as of this moment, the authors have only released the model's answers and questions, not the supporting documents. finally, it's worth noting that while reason and evidence-based categories of nfqs prevail in the eli5 @xcite and we-bgpt datasets, our dataset focuses specifically on the instruction category. ## resource description this section presents our new resource, including its construction, text statistics, comparison with other benchmarks, and thorough quality analysis. ## baseline models this section describes wikihowqa baseline models. the task is to generate an answer to a nfq grounded in a set of relevant documents from which the answer can be reasoned or concluded. as a reference point for performance evaluation, we use the retrieval-augmented generation (rag) @xcite ) architecture, which is a common choice for abstractive summarization and qa tasks @xcite . since our dataset provides a small predefined set of relevant supporting documents for each question, we use a retriever to filter out passages from those documents that are less relevant to the question, to help fit more relevant passages into the limited input of a generator. then, a generator is tasked with generating an answer conditioned on input passages. we experimented with dpr @xcite as the retriever and bart-large @xcite as the generator. when building model input, we first ranked passages from supporting documents based on their relevance to the question using dpr, then truncated them to retain the maximum number of complete passages in the ranking that fit withing the maximum input length of bart. passages were then reordered based on their source documents, and combined into a single input string of the format "<@xmath0...</@xmath1...</@xmath2. finally, bart is trained 9 to generate gold answers conditioned on this context. to establish a stronger baseline, we also evaluated the performance of the gpt-3 (brown et al., 2020) variant text-davinci-003 10 in a zero-shot setting. as a model at least 400x times larger than bart, it forms an upper bound for model performance. we compared two prompt kinds for gpt-3, one with the question and another with the question and relevant passages retrieved via dpr. standard automatic evaluation metrics for abstractive qa, rouge-x and bertscore, are reported in ## how to evaluate how-to answers to further ensure the reliability of md-nfqa evaluation, we introduce a simple human evaluation framework, which we then employ to assess the performance of baselines in our benchmark. ## conclusion in this work we have presented a new dataset and benchmark for multi-document non-factoid question answering, wikihowqa, sourced from the wikihow website and consisting of passage-level answers to "how to" questions. our released test collections fills a critical gap in currently available resources and evaluation testbeds for multidocument non-factoid qa, a widely occurring information task. we have also presented baseline performance benchmarks, and introduced and employed an information-augmented human evaluation framework that improves the reliability of qa annotations. still, manual failure analysis revealed remaining challenges in the evaluation of convincing but factually incorrect model answers.