NLQ u AD : A Non-Factoid Long Question Answering Data Set (2021)

Abstract: We introduce NLQuAD, the first data set with baseline methods for non-factoid long question answering, a task requiring document-level language understanding. In contrast to existing span detection question answering data sets, NLQuAD has non-factoid questions that are not answerable by a short span of text and demanding multiple-sentence descriptive answers and opinions. We show the limitation of the F1 score for evaluation of long answers and introduce Intersection over Union (IoU), which measures position-sensitive overlap between the predicted and the target answer spans. To establish baseline performances, we compare BERT, RoBERTa, and Longformer models. Experimental results and human evaluations show that Longformer outperforms the other architectures, but results are still far behind a human upper bound, leaving substantial room for improvements. NLQuAD’s samples exceed the input limitation of most pre-trained Transformer-based models, encouraging future research on long sequence language models.

## introduction over the last few years, there have been remarkable improvements in the area of machine reading comprehension (mrc) and open-domain question answering (qa) due to the availability of large scale data sets such as squad @xcite and pre-trained language models such as bert @xcite . although non-factoid questions represent a large number of real-life questions, current qa data sets barely cover this area. the reason is that context passages in existing qa data sets are mostly very short and questions mostly factoid, i.e., can be answered by simple facts or entities such as a person name and location @xcite . little attention has been 1 dataset and models: github.com/asoleimanib/nlquad question: how are people coping in the lockdown? headline: china coronavirus: death toll rises as more cities restrict travel document: china has widened its travel restrictions in hubei province -the centre of the coronavirus outbreak -as the death toll climbed to 26. the restrictions will affect at least 20 million people across 10 cities, including the capital, wuhan, where the virus emerged. on thursday, a coronavirus patient died in northern hebei province -making it the first death outside hubei. @xcite we now know this is not a virus that will burn out on its own and disappear. [...] and we still don't know when people are contagious. is it before symptoms appear, or only after severe symptoms emerge? one is significantly harder to stop spreading than the other. [...] one doctor, who requested anonymity, describes the conditions at a hospital in wuhan. [...] "i was planning to stay in my apartment because i'm scared to go to the gym, and i'm scared to go to out in public, and not many people are willing to go out." (141 words). vietnam and singapore were on thursday added to the nations recording confirmed cases, joining thailand, the us, taiwan and south korea. [...] taiwan has banned people arriving from wuhan and the us state department warned american travellers to exercise increased caution in china. (document length: 921 words) figure 1: a question-answer pair in nlquad. qa models must predict the answer span within the context document. the correct answer span is bolded. we extract questions and answers, respectively, from the subheadings and the sub-section bodies from real-word english news articles. two other questions based on the same article: can the coronavirus be stopped? what's the global situation? paid to non-factoid and open-ended questions that require complex answers such as descriptions or opinions @xcite . answers to nonfactoid questions extend to multiple sentences or paragraphs having few words overlapping with the question @xcite . non-factoid qa facilitates document assistance systems, where for example, journalists can seek assistance to highlight relevant opinions and interpretations. it can further motivate more research on long sequence language models. therefore, a high-quality data set in this area is clearly desired. to support research towards non-factoid and long qa tasks and to address the existing shortcomings as identified above, we have built nlquad, a non-factoid long question answering data set. nlquad contains 31k non-factoid questions and long answers collected from 13k bbc news articles. we extract questions and answers from the articles' sub-headings and the following body paragraphs of the sub-headings (see figure 1 ). questions in nlquad are not answerable by a short span of text within the documents. this is in contrast to existing long-context but factoid qa data sets such as newsqa @xcite , triviaqa @xcite , narrativeqa @xcite , duorc @xcite , hotpotqa @xcite , and natural questions @xcite . although these data sets contain long documents, questions are answerable by short entities or a span of entities. in particular, natural questions covers two types of short and long answers. however, due to its factoid questions, most long answers are still sections containing exactly the short answers and so are trivial (e.g., "where is the world's largest ice sheet...?", short: "antarctica"; long: "the antarctic ice sheet is the largest single mass of ice on earth..."). furthermore, although a small portion (13%) of natural questions samples have only long answers, they are still spans of simple facts. for example, "who is the author of the book arabian nights?" has no short answer simply because there are multiple authors: "the work was collected over many centuries by various authors, translators...". in contrast, we address non-factoid questions requiring complex answers like opinions and explanations. nlquad's answers are open and not predefined. figure 3 and in most existing qa data sets such as squad, crowd-workers generate questions based on provided short passages and extract answers from the passages @xcite . this method of question generation can make qa samples trivial because models can simply detect the most related span to the question by guessing based on shallow pattern matching @xcite . in contrast, all annotations in nlquad are done automatically and directly based on the news articles themselves. nlquad, unlike ms marco @xcite and eli5 @xcite , does not use information retrieval (ir) methods to collect supporting documents. retrieved documents in these data sets are not guaranteed to contain all facts required to answer the question or they occasionally just contain information related to the question but no answers. nlquad requires document-level language understanding. with an average document length and answer length of 877 and 175 words, respectively, it exceeds the maximum input length of the state of the art qa models such as bert @xcite and roberta @xcite due to their memory and computational requirements. thus, training and evaluating the (document, question, answer) tuples is impossible using such models in an end-to-end manner. it is worth noting that it is also harder to perform pre-selection methods before the final span detection because our answers are long. meanwhile, most of our questions are not self-contained. for example, to answer the question "how are people coping in the lockdown?" (figure 1 ), the system needs to read the document to interpret the concept of "lockdown" and then locate the information regarding the people's behaviour. we also show the shortcomings of the f1 score and rouge-n scores in evaluating long sequences. there is a higher chance of overlap between the word n-grams in two long sequences causing f1 and rouge-n to over-estimate the performance. therefore, we propose to use intersection over union (iou) measuring position-sensitive overlap between two spans. ## existing data sets existing large-scale qa data sets can be categorized based on their context passage length in two groups: short-context qa, i.e., data sets with paragraph-level context, and long-context qa, i.e., data sets with multiple-paragraph or documentlevel context. long-context qa can potentially include questions demanding long answers. in this section, we only review qa datasets. however, it is worth noting that very recently, @xcite introduced a unified benchmark using different tasks for evaluating model quality under long-context scenarios. ## data set design nlquad consists of news articles as context documents, interrogative sub-headings in the articles as questions, and body paragraphs corresponding to the sub-headings as contiguous answers to the questions. we automatically extract target answers because annotating for non-factoid long qa is rather challenging and costly. to ensure the qual-ity of answers in addition to the initial investigations, we perform human evaluations (section 5.3). we choose the bbc news website as the resource of our documents and the question-answer pairs, mainly because its articles contain a considerable amount of high-quality question-like sub-headings which are suitable for the qa task. nlquad's characteristics make it an appealing and challenging data set for the non-factoid long qa task: its context documents are long, and its questions are non-factoid in a way that cannot be answered by single or multiple entities. the questions are addressed by more than seven sentences on average. meanwhile, it covers a wide range of topics, making it an open-domain qa data set. the bbc news articles typically follow a specific template. they begin with an introductory section consisting of news summaries @xcite and one or more sections accompanied by sub-headings. each section contains multiple short to medium-length paragraphs. we remove the template and section break-lines to prevent revealing possible answer boundaries. ## baseline models to investigate the difficulty level of nlquad for state-of-the-art qa systems and to establish baseline results, we evaluate the performance of bert @xcite , roberta @xcite , and longformer @xcite . longformer is a scalable model for processing long documents and has been used for long sequences such as document classification @xcite and document re-ranking @xcite . we refer readers to @xcite for a detailed survey on efficient transformers. we train these transformerbased @xcite models to predict the span of the answer in a context document given a question and document. ## conclusion we introduce nlquad, a non-factoid long question answering data set from bbc news articles. nlquad's question types and the long lengths of its context documents as well as answers, make it a challenging real-world task. we propose to use intersection over union (iou) as an evaluation metric for long question answering. to establish a baseline performance, we experimented with the bert, roberta, and longformer question answering models. longformer outperforms the other methods with an iou of 73.57%, but the results show that the performance of state-of-the-art question answering systems is far from perfect. we hope nlquad will inspire more research in the area of document-level language understanding and question answering.