Q u ALITY : Question Answering with Long Input Texts, Yes! (2022)

Abstract: To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).

## introduction most of the best models for natural language understanding are restricted to processing only a few hundred words of text at a time, preventing them from solving tasks that require a holistic understanding of an entire passage. moving past this limitation would open up new applications in areas like news comprehension, summarization, or applied question answering. we think that new benchmark datasets will help us do this. most existing datasets @xcite @xcite use shorter contexts that humans can read within a few minutes. while there are open-domain qa datasets that require longer contexts @xcite , finding a short excerpt that answers the questions at hand often suffices; however, long-document qa requires understanding a long context as a whole to correctly answer questions. narrativeqa @xcite is the most established existing long-text benchmark for language understanding. it's a free-text-response qa * equal contribution. figure 1 : the crowdsourcing pipeline with an example. one writer reads the passage and writes 10 questions. each question is validated by three or five annotators who read the full article, plus five more who have only 45 seconds per question. writers receive feedback from both validations between writing batches. if a majority of timed annotators get the question wrong, but the untimed annotators get it right, we classify the example as hard and give the writer a bonus. to fairly assess accuracy, as metrics like bleu, rouge, or bertscore may not accurately convey the quality of generations @xcite . to ease the burden of evaluation, we opt for a multiple-choice format to evaluate a model's long-document understanding ability. we introduce our dataset quality, question answering with long input text, yes! 2 this is a multiple-choice qa dataset that uses english source articles of 2k-8k tokens. 3 we collect this dataset using a creative crowdsourcing pipeline that ensures the examples have unambiguous answers but are still challenging. we instruct example writers to carefully read the full source article before writing questions, and to then write questions that are unambiguous and require consolidating information from multiple parts of the text. then, to ensure our questions require readers to understand the larger context from the passage, in addition to running standard validation where annotators read the text and answer the questions, we also run speed validation ( §2.3). in speed validation, annotators only have access to the text for 45 seconds, 2 the data is available, along with links to code and additional resources, at https://github.com/nyu-mll/ quality . results on quality will be collected on a dedicated leaderboard at https://nyu-mll.github.io/ quality/ and through the multitask scrolls benchmark @xcite . so they can only skim or search for phrases to answer the question. if a question is unanswerable in this setting but unambiguous and answerable in the standard untimed setting, we use it as a signal for question difficulty. this crowdsourcing process is slow and expensive ($9.10/question), foot_0 but we successfully collect a challenging, high-quality, longdocument multiple-choice qa dataset. quality has 6,737 questions in total, of which 3,360 questions are in the difficult subset, quality-hard. texts are provided with the original html tags indicating paragraph breaks and basic formatting (e.g., italics), and it is in this format, with images removed, that we present the texts to our writers and annotators. in our dataset release, we also include a version of each file with this information stripped away, as current models, including our baselines, are not trained to consume these tags. we set a maximum length for the texts at 6k words using word-level tokenization without counting html tags. 8 for around 40% of the gutenberg articles, the full text data is much longer; in these cases, we truncate the texts and manually check to make sure the truncation happens at a reasonable location (i.e., not in the middle of a paragraph). ## dataset information and analysis after aggregating the labels assigned via untimed validations with the original writer's label, we calculate the gold label via majority vote of annotators. 10 we only keep questions for which (i) a majority vote label (strictly larger than 50%) can be assigned and (ii) the majority of annotators rate the questions as answerable and unambiguous. 6,737 out of 7,620 (88.4%) questions meet these inclusion criteria. the hard subset corresponds to questions that the majority of the annotators answer incorrectly in the speed validation setting, and this constitutes 49.9% of the final dataset. ## conclusion we introduce the long-document qa dataset quality. this dataset was crowdsourced and validated by humans to ensure that the questions are answerable, unambiguous, and challenging. the quality-hard subset, comprising half the dataset, consists of questions that are unanswerable by annotators working under tight time constraints, helping ensure that skimming and simple search do not yield high performance. we find that our baseline models significantly lag behind human performance on quality, with a 38.1 percentage point gap between human an-notators and the best performing model. the gap is even wider on quality-hard, at 42.3 points. we hope that research that aims at this gap will contribute to expanding the scope of texts on which effective nlu systems can be applied.