Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion (2023)

Abstract: Automatic discourse processing is bottlenecked by data: current discourse formalisms pose highly demanding annotation tasks involving large taxonomies of discourse relations, making them inaccessible to lay annotators. This work instead adopts the linguistic framework of Questions Under Discussion (QUD) for discourse analysis and seeks to derive QUD structures automatically. QUD views each sentence as an answer to a question triggered in prior context; thus, we characterize relationships between sentences as free-form questions, in contrast to exhaustive fine-grained taxonomies. We develop the first-of-its-kind QUD parser that derives a dependency structure of questions over full documents, trained using a large, crowdsourced question-answering dataset DCQA (Ko et al., 2022). Human evaluation results show that QUD dependency parsing is possible for language models trained with this crowdsourced, generalizable annotation scheme. We illustrate how our QUD structure is distinct from RST trees, and demonstrate the utility of QUD analysis in the context of document simplification. Our findings show that QUD parsing is an appealing alternative for automatic discourse processing.

## introduction discourse structure characterizes how each sentence in a text relates to others to reflect the author's high level reasoning and communicative intent. understanding discourse can be widely useful in applications such as text summarization @xcite @xcite , classification @xcite , narrative understanding @xcite , machine comprehension @xcite , etc. however, automatically inferring discourse structure is challenging which hinders wider application @xcite . at its root lies the issue of data annotation: popular coherence formalisms like the rhetorical structure theory @xcite , segmented discourse representation theory (sdrt, @xcite , and the penn discourse treebank (pdtb, @xcite require experts-typically linguists trained for the task-to reason through long documents over large relation taxonomies. these features, coupled with the difficulties of annotating full structures in the case of rst and sdrt, make the task inaccessible to lay annotators. the taxonomies differ across formalisms @xcite , and their coverage and definitions are being actively researched and refined @xcite @xcite . in contrast, this work aims to derive discourse structures that fit into the linguistic framework of questions under discussion (qud) @xcite , which neatly avoids reliance on a strict taxonomy. in qud, "each sentence in discourse addresses a (often implicit) qud either by answering it, or by bringing up another question that can help answering that qud. the linguistic form and the interpretation of a sentence, in turn, may depend on the qud it addresses" @xcite . thus relationships between sentences can be characterized by free-form questions instead of pre-defined taxonomies. for instance, consider the following two sentences: sentence 6 is the answer to a question from sentence 3: "why can they open a route?". the question-answer view is in line with recent work reformulating linguistic annotation as question answering @xcite @xcite california legislators, searching for ways to pay for the @xmath06 billion in damages from last week's earthquake, are laying the groundwork for a temporary increase in the state's sales tax. figure 1 : a snippet of a wsj article from the intersecting subset of dcqa @xcite and the rst discourse treebank @xcite . (a) shows a qud dependency structure derived from dcqa. edges are defined by questions, connecting where the question arose from (the "anchor" sentence) and the sentence that answers the question. (b) shows the annotated rst tree above the sentence level. this work seeks to fill this gap, and presents the first-of-its-kind qud parser. this parser takes a document as input and returns a question-labeled dependency structure over the sentences in the document, as depicted in figure 1(a) . for training, we use the intra-document question answering dataset dcqa @xcite ); dcqa's annotation scheme is both compatible with qud and easily crowdsourced, making qud parsing a much less costly option than existing frameworks. each question in dcqa is considered to arise from an "anchor" sentence, and answered by another sentence later in the same article. in line with qud, we consider each sentence as the answer to an implicit question from prior context @xcite , in particular the anchor sentence. we view the anchor sentence as the parent node of the answer sentence, with the question describing the relation between the two; this results in a dependency tree structure. conveniently, a subset of dcqa overlaps with the rst discourse treebank @xcite , allowing us to directly compare the two types of structures (figure 1(b) ). we show that the qud trees are structurally distinct from rst trees. a close inspection of relation-question correspondence reveals that qud's free-form questions are more fine-grained, and that their presence reduces annotator disagreement in selecting rst relations. trained on dcqa, our qud parser consists of two models used in a pipeline. the first model predicts the anchor sentence for each (answer) sentence in the article; the second model performs question generation given the answer sentence and the predicted anchor sentence. our comprehensive human evaluation shows that readers approve of 71.5% of the questions generated by our best model; among those, the answer sentence answers the generated question 78.8% of the time. finally, we demonstrate the analytical value of qud analysis in the context of news document simplification: the questions reveal how content is elaborated and reorganized in simplified texts. in sum, this work marks the first step in qud parsing; our largely positive human evaluation results show that this is a promising data-driven approach to discourse analysis with open, crowdsourced annotation that is so far infeasible to do at scale with other discourse frameworks. we release our models at https://github.com/ lingchensanwen/dcqa-qud-parsing . qud and coherence structures are closely related. prior theoretical work looked into the mapping of quds to discourse relations @xcite or the integration of the two @xcite . @xcite and @xcite studied structural correspondances between qud stacks and sdrt specifically. @xcite showed that qud could be a useful tool to quantitatively study the predictability of discourse relations @xcite @xcite . in @xcite , discourse relation taxonomies were also converted to templatic questions, though not in the qud context. traditionally, discourse "dependency parsing" refers to parsing the rst structure @xcite @xcite . since qud structures are marked by free-form questions, the key aspect of "parsing" a qud structure is thus question generation, yielding a very different task and type of structure than rst parsing. as we show in the paper, the two are complementary to each other and not comparable. this work focuses on automating and evaluating a qud parser; we leave for future work to explore what types of structure is helpful in different downstream tasks. the dcqa dataset corpora specific for qud are scarce. existing work includes a handful of in-terviews and 40 german driving reports annotated with question stacks @xcite , as well as westera et al. (2020)'s 6 ted talks annotated following kehler and rohde (2017)'s expectation-driven model (eliciting questions without seeing upcoming context). ko et al. (2020)'s larger inquisitive question dataset is annotated in a similar manner, but inquisitive only provides questions for the first 5 sentences of an article, and they did not annotate answers. this work in contrast repurposes the much larger dcqa dataset @xcite , consisting of more than 22k questions crowdsourced across 606 news articles. dcqa was proposed as a way to more reliably and efficiently collect data to train qa systems to answer high-level questions, specifically qud questions in inquisitive. though not originally designed for qud parsing, dcqa is suitable for our work because its annotation procedure follows the reactive model of processing that is standard in qud analysis @xcite , where the questions are elicited after observing the upcoming context. concretely, for each sentence in the article, the annotator writes a qud such that the sentence is its answer, and identifies the "anchor" sentence in preceding context that the question arose from. figure 1(a) shows questions asked when each of the sentences 2-6 are considered as answers, and their corresponding anchor sentences. as with other discourse parsers, ours is inevitably bound by its training data. however, dcqa's crowdsourcable paradigm makes future training much easier to scale up and generalize. ## questions vs. coherence relations we first illustrate how questions capture intersentential relationships, compared with those in coherence structures. we utilize the relation taxonomy in rst for convenience, as in section 5.3 we also compare the structure of our qud dependency trees with that of rst. given each existing anchor-answer sentence pair across 7 dcqa documents, we asked two graduate students in linguistics to select the most appropriate discourse relation between them (from the rst relation taxonomy @xcite ). both students were first trained on the taxonomy using the rst annotation manual. ## deriving qud dependency structures our task is to derive a qud dependency structure over a document @xmath1, . . . , s n ) consisting of n sentences. a qud tree @xmath2, q 1 ), . . . , (a n , q n )) can be expressed as a list of n tuples: each sentence has an associated anchor sentence a i and a question labeling the edge to the anchor q i . to arrive at a dependency structure, we view the anchor sentence as the head of an edge, linking to the answer sentence via the question, as shown in figure 1(a) . we set a 1 = 0 and q 1 = ∅; the first sentence is always the root of the qud dependency tree, so has no parent and no question labeling the edge. each other a @xmath3, 2, . . . , i -1} and q @xmath4. we note that t is analogous to a labeled dependency parse, except with questions q in place of typical discrete edge labels. our parser is a discriminative model this formulation relies on models corresponding to two distinct subtasks. first, anchor prediction selects the most appropriate sentence in prior context to be the anchor sentence of the generated question using a model p (a i | d, i). second, question generation given the current (answer) sentence, its anchor, and the document context uses a model p (q i | d, i, a i ). we do not impose projectivity constraints or other structural constraints beyond anchors needing to occur before their children. therefore, inference can proceed with independent prediction for each sentence. 2 we now proceed to describe the models bert reranker gpt-2 ques+on candidate 1 ques+on candidate 2 … [anc] california legislators… [/anc] the talk… [sep] california… [sep] the state … sentence 1 chosen as anchor what type of relief is being considered? input document anchor answer sent. ## evaluation and analysis since qud parsing features an open-ended generation component, we need new evaluation methodology compared to standard discourse parsing. we focus on two main factors: (1) whether the generated question is plausible at the predicted anchor point; (2) whether the question is actually answered by the answer sentence. question 1: assuming you are reading through the article and ask a question based on the article up to that point, is this a reasonable question? yes sort of no [hallu.(m)] hallucinates minor details not mentioned in the source sentence or context. [answered(m)] already answered in the source sentence but could be asked again minor error [nonsense] incomprehensible or doesn't make sense [irre.(a)] irrelevant to the article [irre.(s)] does not arise from that sentence or is irrelevant to the sentence [hallu.(m)] hallucinates key parts of the question [answered(m)] already answered in the source sentence and doesn't make sense to ask again at all question 2: does the answer sentence answer the question? yes yes but not the main point of the sentence sort of no in qud annotation and dcqa itself @xcite @xcite , it is often the case that multiple questions can be asked even given the same anchor and/or answer sentences. the evaluation of qud validity thus involves complex reasoning performed jointly among (long) context, the anchor, the answer sentence, and the generated question itself. for these reasons, we rely on human evaluation, and leave the development of automatic evaluation metrics for future work. 4 ## case study: document simplification we demonstrate the analytical value of qud analysis in the context of document simplification. we use the newsela dataset @xcite , where news articles are professionally simplified across several grade levels; a subset of newsela (of the highest reading level) is present in dcqa. note that most research in text simplification focus on the sentence level (alva-manchego et al., 2020); we hope to inform document-level approaches. we sample 6 articles from the dcqa newsela subset. for each of these, 3 linguistics undergraduates (not authors of this paper) doubly annotated their corresponding middle and elementary school levels with qud structures for the first 20 sentences following dcqa's paradigm. this amounts to ∼720 questions in total. figure 5 shows a snippet of our analysis from two reading levels of the same article. we run and evaluate our parser on the articles of the second reading level. using the schema in figure 3 , question 1 is yes for 60.2% of the time, and question 2 is yes 75.2% of the time. this shows that while the parser is still capable of generating reasonable questions, the performance degrades compared to testing on the highest level. this is likely due to clear stylistic, organizational, and vocabulary difference for simplified texts; for this reason, we resort to using annotated quds to illustrate idealized results for this analysis. analysis the simplified articles, which mostly align with the original versions at the beginning, tend to contain significant reorganization of content especially later in the text. nonetheless, we found that 62.2% of the questions had a similar question on another reading level, reflecting that quds frequently stay invariant despite these differences. for example, in figure 5 , the content of sentence 8 (level 2) is covered in sentence 2 (level 1), yet in both cases the question "why is the case important" is used to link these sentences. similarly, questions q2-6 (level 2) and q2-8 (level 1), as well as questions q6-7 (level 2) and q8-10 (level 1) reflect the same qud. often, articles from higher reading levels presupposes certain knowledge that gets elaborated or explained during simplification @xcite . qud analysis informs how content should be elaborated: in figure 5 (a), the level 1 article defined the concept of amendment (question q8-9), absent in level 2. sentence splitting as a frequent operation @xcite @xcite could also be explained by questions, as in the case of q8-11 in level 1, which provides a rationale as to why sentence 8 in level 2 is split (into sentences 2 and 11 in level 1). note that this explanation is rooted outside of content conveyed by the sentence that was split. finally, editors also omit difficult content @xcite , as in figure 5(b) : sentence 1 in level 2 is not present in the level 1 simplification (due to less salience and the reference to the "selfie generation" which goes beyond the targeted reading level). level 2 thus contains the extra qud: q1-2. in sum, qud analysis reveals how elaborated or omitted content fit into the larger context during simplification, potentially aiding future documentlevel simplification systems by providing intermediate rationales. level 1 [1] on tuesday, the supreme court will consider the case of a california man whose smartphone got him in serious trouble. [2] what they decide will affect everyone with a cellphone. … [8] the supreme court must decide if smartphones are covered by the fourth amendment. ## conclusion this work presents the first qud (questions under discussion) parser for discourse analysis. we derive dependency structures of qud, viewing each sentence as an answer to a qud triggered in an anchor sentence in prior context. this paradigm avoids costly annotation of coherence structures; rather, our parser can be trained on the crowdsourced dataset dcqa. we show strong parser performance with comprehensive human evaluation. we further demonstrate the richness of qud analysis in document simplification.