Discourse Comprehension: A Question Answering Framework to Represent Sentence Connections (2022)

Abstract: While there has been substantial progress in text comprehension through simple factoid question answering, more holistic comprehension of a discourse still presents a major challenge (Dunietz et al., 2020). Someone critically reflecting on a text as they read it will pose curiosity-driven, often open-ended questions, which reflect deep understanding of the content and require complex reasoning to answer (Ko et al., 2020; Westera et al., 2020). A key challenge in building and evaluating models for this type of discourse comprehension is the lack of annotated data, especially since collecting answers to such questions requires high cognitive load for annotators.This paper presents a novel paradigm that enables scalable data collection targeting the comprehension of news documents, viewing these questions through the lens of discourse. The resulting corpus, DCQA (Discourse Comprehension by Question Answering), captures both discourse and semantic links between sentences in the form of free-form, open-ended questions. On an evaluation set that we annotated on questions from Ko et al. (2020), we show that DCQA provides valuable supervision for answering open-ended questions. We additionally design pre-training methods utilizing existing question-answering resources, and use synthetic data to accommodate unanswerable questions.

## introduction research in question answering has pushed machine comprehension to new heights, especially in answering factoid questions (e.g., squad @xcite and natural questions @xcite ) and even questions that require multiple steps of reasoning @xcite . yet while existing systems are effective at sifting through a long document to find a fact, they fail to achieve what we consider discourse comprehension. deep comprehension of q1 what happened before the clash? … [s10] hours earlier, egypt's new interim leadership had narrowed in on a compromise candidate to serve as the next prime minister. ## the dcqa dataset we present dcqa using a new data collection paradigm that describes how sentences in an article relate to prior context; specifically, what question is the current sentence answering given its prior context, inspired by literature in qud discovery @xcite . we annotate news documents due to their rich linguistic structure that mixes narratives, different lines of events, and perspectives from parties involved @xcite . ## analyzing dcqa this section presents a series of analytical questions we pose in order to understand dcqa questions, answers, and potential challenges. are questions similar given the same answer sentence? we first quantify the level of subjectivity in our question-answering paradigm via two angles: since the questions are free-text in form, we use human judgments to assess their similarity. we asked our expert annotators to rate the similarity of a sample of 261 pairs of questions asked with the same answer sentence, on a scale of 1-5. how well can humans do the qa task? our annotation paradigm produces open-ended questions given the answer sentence, while ultimately the question-answering task seeks to find the answer given the question. thus we explore to what extent humans can, given a question, find the original answer sentence where the question was generated. we asked an expert annotator to answer questions in a randomly sampled subset of 1175 questions from the validation and test sets, without informing them of the "gold" answer labels. if the annotator thinks there are multiple answers, he could annotate multiple sentences. after the answers are annotated, the gold answer is shown and the annotator adjudicates the two versions of answers. the agreement between the annotator and the gold labels is 72.2%. in cases where the annotator selected multiple answer sentences, we report the expected agreement when randomly choosing one of them. for 69.5% of the questions, the annotator exactly chose the gold answer; for 5.3%, the annotator selected multiple answers including the gold. on 2.3% of the data, the annotator thinks the gold answer cannot be used to answer the question without a stretch, thus should be regarded as noise. for the rest, the annotator thinks that both answers are reasonable as there are multiple possible answers, although they did not originally choose the gold. in these cases, the annotator tended to choose an earlier sentence than the gold. an example for multiple answer sentences is in appendix a.1. how far away are answer sentences from question anchors? compared to ted-q, dcqa contains fewer questions that are answered by the immediate next sentence. dcqa also has @xmath0, which ted-q does not have. this means that fewer questions in dcqa can be trivially found in the immediate next sentence. additionally, this also shows that the sentence connections captured by dcqa naturally result in a different structure from qadiscourse (pyatkin et al., 2020) that expresses discourse relations across adjacent sentences in templatic questions. we also show the figures for inquisitive over a subset that we annotate for answers (section 6.1). most of the inquisitive answers are far away from where the question was asked; this is in stark contrast with ted-q, highlighting the distributional differences between ted talks and news. linguistic characteristics of dcqa. an interesting effect of the question generation process is that annotators often find semantic links between two parts of the article that are not pragmatically or discursively connected. for example: finally, we present an analysis about the type of the questions asked in appendix a.2, using the schema in @xcite . dcqa has a good coverage of the key question types in existing high-level question answering datasets (ted-q and inquisitive); the most frequent question types are concept, cause, procedural, and example. model because our task involves reasoning over long documents, we choose longformer @xcite as our model. longformer is a transformer model with an attention pattern that combines a local windowed attention and global attention on pre-selected, task-dependent tokens only. this makes the computation time scale linearly with the sequence length instead of quadratic (as in conventional transformers), allowing faster computation on longer sequences, as well as better performance on qa tasks with long contexts. our model operates similarly to the bert qa model for squad @xcite . for the model input, we concatenate the question, the anchor sentence, and the article, separated by delimiters. note that by passing in the full article, the model has access to all context prior to the anchor. since the goal of our model is to predict a sentence instead of the answer span, we add two tokens in front of each sentence in the article, the start of sentence token [sos] and the sentence id. the model is trained to predict the span of the two added tokens in front of the answer sentence. settings we use our expert annotated questionanswer pairs for the validation and test sets, and crowd annotation for the training set. we use huggingface @xcite for our implementation. we use the adam (kingma and ba, 2015) optimizer with (β 1 , β 2 ) = (0.9, 0.999) and a learning rate of 5e-5. ## question answering on external datasets we first describe our collection of the inquisitive test set (section 6.1). then we explore only answerable questions (section 6.2), and design pretraining methods to allow the use of other qa datasets. finally, we present a pipeline system that also handles question answerability (section 6.3), by predicting whether a question is answerable using synthesized data from dcqa. ## conclusion we present dcqa that connects pieces in a document via open-ended questions and full-sentence 3 while we could in theory simply "take out" the answer sentence, it would leave the text incoherent and unnatural.