B i Q u AD : Towards QA based on deeper text understanding (2021)

Abstract: Recent question answering and machine reading benchmarks frequently reduce the task to one of pinpointing spans within a certain text passage that answers the given question. Typically, these systems are not required to actually understand the text on a deeper level that allows for more complex reasoning on the information contained. We introduce a new dataset called BiQuAD that requires deeper comprehension in order to answer questions in both extractive and deductive fashion. The dataset consist of 4,190 closed-domain texts and a total of 99,149 question-answer pairs. The texts are synthetically generated soccer match reports that verbalize the main events of each match. All texts are accompanied by a structured Datalog program that represents a (logical) model of its information. We show that state-of-the-art QA models do not perform well on the challenging long form contexts and reasoning requirements posed by the dataset. In particular, transformer based state-of-the-art models achieve F1-scores of only 39.0. We demonstrate how these synthetic datasets align structured knowledge with natural text and aid model introspection when approaching complex text understanding.

## introduction most of the recent question answering benchmarks require systems to pinpoint the span of the answer to the question in the given text. in the well-known squad 2.0 dataset @xcite , systems are able to extract the correct answer span in the following paragraph as an answer to the question: "what tactic did researchers employ to offset the former deficit of work surrounding the complexity of algorithmic problems?": results of state-of-the-art (sota) systems on these datasets have reached exact match (em) and f 1 performances of 90.9 and 93.2, respectively foot_0 . some of these models even outperform the human baseline (which lies at f 1 = 89.4 for squad 2.0). it is unclear to which extent the existing benchmarks actually require systems to comprehend texts and to what extent these systems rely on surface cues signalling a match between question and answer span. most state-of-the-art models rely on transformer models such as bert and albert @xcite that are pre-trained on supplementary tasks using large amounts of textual data @xcite and employ extensive selfattention mechanisms that have been shown to learn many of the features of a classic natural language processing (nlp) pipeline @xcite . it has been shown for a number of tasks that such models rely on surface cues and on artifacts of the datasets. a recent example is the argument reading and comprehension (arc) task @xcite . a deeper analysis of the data and the performance of transformer models has shown that they exploit only surface cues and artefacts of the data, failing to perform beyond chance when systematic (adversarial) modifications are applied on the dataset @xcite . our motivation in this paper is to introduce a new question answering dataset that requires a deeper understanding of the text to answer questions beyond merely matching answer spans. in particular, in our dataset, the answers to questions can often not be found in the original text, but can be inferred on the basis of a deeper, structural, understanding. examples are aggregation questions such as "'how many goals did marco reus score in the first half of the match?'" but also questions such as "'who won the game?'", requiring a system to understand that a balanced score leads to a tie, making the question unanswerable. since most models cannot keep track of all the goals and intermediate scores for the whole game this provides a significant challenge. the dataset we present is called biquad and comprises of 99,149 question answer pairs on 4,190 documents, averaging 23 questions per document. the texts describe soccer matches that have actually taken place. the texts have been generated automatically on the basis of handcrafted templates from structured reports of soccer games. a model of each game is available in the form of a datalog program representing the meaning of the text in terms of a model consisting of predicates relevant to the description of a soccer game. the questions are paraphrases of queries that can be answered over the model of the game. similar to extensions from squad 1.0 to 2.0 @xcite , a percentage of the generated questions in the dataset are deemed as unanswerable. in this paper we present the dataset in more detail and describe its creation (section 3). further, we present the results of state-of-the-art qa systems on this task in section 4. we show the limits of current state-of-the-art qa in answering questions for which the answer is not in the text but requires deeper inference on the basis of the information given in the text. we show that, in spite of text being artificially generated using 335 relatively simple templates, thus being very regular, this task is not solvable by the current state-of-theart in question answering. arguably, the current state-of-the-art focuses on extractive qa and was not designed for deeper understanding. we posit that results on our dataset show that extractive models in particular overfit on surface cues that do not require deeper text understanding. in particular, we show that while the state-of-the-art on squad 2.0 for example yield results of em and f 1 -scores of 90.7 and 93.0, results of these models for our task range between 38.8 and 39.0 respectively. ## related work datasets on machine reading and question answering tasks can be characterized by broad categories: a) open vs. closed (specific) domain, and b) textcomprehension based (e.g. extractive or clozestyle) vs. knowledge based qa. prominent datasets focusing on open-ended extractive qa include squad 1.0 @xcite and 2.0 @xcite , which have enjoyed wide popularity in the research community. together with newsqa @xcite , these datasets represent the largest, crowd-sourced, extractive qa datasets available. questions here are answered by correctly identifying a span in a context paragraph, with version 2.0 introducing a subclass of unanswerable questions. more recently, squad versions in languages other than english have been developed @xcite @xcite . the triviaqa @xcite dataset integrates the notion of external evidence (wikipedia articles) for trivia and open domain question answering and broadens the task to information retrieval (ir) settings. the race dataset @xcite relies on a multiple choice setting and leverages data used for the assessment of reading comprehension by humans. it features simple reasoning challenges such as deducing relative values from mentions of absolute ones. similarly, the open book qa @xcite requires models to involve common sense knowledge to solve the task successfully. there are different ways to frame the task of answering questions by machine reading, including sentence retrieval @xcite , multi-hop reasoning @xcite , and reasoning about multiple paragraphs or documents at the same time @xcite . recent work has considered the development of reasoning-based qa systems @xcite as well as the integration of external @xcite and commonsense knowledge @xcite into the qa process. other machine reading based qa datasets focus on answering questions on the basis of struc- dataset domain task type samples ~acquisition method sota squad 1.0, 2.0 open extractive qa 150,000 crowdsourcing em 90.724 f1 93.011 newsqa open extractive qa 120,000 crowdsourcing f1 73.6 triviaqa open extractive qa 95,000 semi-automatic em 90.38 f1 92.96 race open multiple choice 100,000 domain experts accuracy 90.9 open book qa open multiple choice 6,000 crowdsourcing accuracy 87.20 lc-quad 2.0 open graph retrieval 30,000 semi-automatic tbd. wikihop open extractive qa 50,000 graph traversal accuracy 81.9 medhop closed extractive qa 2,500 graph traversal accuracy 60.3 qasc open multiple choice 10,000 crowdsourcing accuracy 90 qald-9 open graph retrieval 700 * 11 manual annotation macro f1 qald 5.0 sciqa closed document retrieval 10,000 automated extraction map 24.36* drop both extractive qa 97000 crowdsourcing em 90.10 f1 87.04 most closed domain datasets are smaller than their open domain counterparts since annotation usually requires experts that are inherently harder to source. datasets such as the biomedical question answering corpus biqa @xcite ) make use of user generated content from other sources instead. the recent drop dataset @xcite focuses on complex reasoning tasks in form of both, open and closed domain, questions. the task combines the challenge of extractive qa with testing a models ability to perform limited numerical reasoning, e.g. by having to calculate date differences. at the time of this writing, graph-based models presented in (chen et al., 2020) rank at the top of the drop leaderboard with f 1 of 90.1, and em of 87.0. similar to drop, we aim to bridge gaps required for deeper text understanding while simultaneously providing sample annotations that allow for proper model introspection. the synthetic nature of generated texts in biquad provides an extensible way to test model capabilities for reasoning about various sub-categories. we provide long form text passages alongside structured representations, in the form of datalog rules, as a way to either explicitly combine structured and unstructured information or a further way for model introspection by aligning neural model representations with their graphical and discrete counterparts. ## methods this section outlines the methodology used to generate the dataset and in what way the state-of-theart transformer architectures can provide a first baseline for the biquad dataset. ## results this section provides an overview of the dataset and provides first results on the dataset by providing baselines relying on state-of-the-art transformer models. ## conclusion our methodology builds on a structured database of soccer matches. from this database, it generates natural language reports of games relying on a set of handcrafted templates in addition to a datalog logical representation of a text. for each document, 23 datalog queries are generated and transformed into nl relying on a further set of handcrafted templates. the artificial nature of the dataset allows for clear cut error analysis and can guide the implementation of, especially closed domain, qa systems. downsides introduced by synthetic generation include an unnaturally factual text style and the fact that relatively simple heuristics might be able to generate the correct answer for some of the question templates. these heuristics could potentially even target the soccer domain itself, as is common with many subtasks in closed domain corpora. we aim to alleviate these concerns in future work by extending a) the tooling around model introspection and b) coverage of further domains. these steps should further improve the way reasoning tasks are reflected in the dataset and establish biquad , or the approach of synthetic corpora, as a modeling tool to be used alongside other corpora of natural texts in comparable or open domain settings. we have provided baseline systems and show that existing state-of-the-art systems yield very low results on the dataset, with exact match and f 1scores of 38.8 and 39.0, respectively. results of @xmath0.4, f 1 = 25.9 in a transfer learning setup further strengthen the assumption that these models cannot sufficiently answer the type of rea-soning tasks imposed by the dataset. while state-ofthe-art systems perform well on standard extractive questions (qt m ultiple f acts,aggregation ), we show that on questions requiring inference and temporal reasoning, the baseline systems perform at f 1 below 20 even when encoding answers in a way compatible with extractive systems. on the basis of these results and experience with other datasets, we can see that explicit modelling is required for various complex reasoning tasks. many recent state-of-the-art models on qa achieve this complexity only in limited scopes, such as pure numeric reasoning, not deeper general text understanding. the parallel construction of structured datalog knowledge about a closed world model may be used to a) develop models that combine textual information with external semantic knowledge or b) decode how a model performs reasoning tasks by linking text and structural knowledge when inspecting individual components such as attention layers. biquad allows to model and inspect reasoning on specific categories without necessarily overfitting on any particular subtask. our dataset is freely available and, in combination with other datasets of non-synthetic nature, will hopefully contribute to push the state of the art in machine reading further.