QUDSELECT : Selective Decoding for Questions Under Discussion Parsing (2024)

Abstract: Question Under Discussion (QUD) is a discourse framework that uses implicit questions to reveal discourse relationships between sentences. In QUD parsing, each sentence is viewed as an answer to a question triggered by an anchor sentence in prior context. The resulting QUD structure is required to conform to several theoretical criteria like answer compatibility(how well the question is answered), making QUD parsing a challenging task. Previous works construct QUD parsers in a pipelined manner (i.e. detect the trigger sentence in context and then generate the question). However, these parsers lack a holistic view of the task and can hardly satisfy all the criteria. In this work, we introduce QUDSELECT, a joint-training framework that selectively decodes the QUD dependency structures considering the QUD criteria criteria. Using instruction-tuning, we train models to simultaneously predict the anchor sentence and generate the associated question. To explicitly incorporate the criteria, we adopt a selective decoding strategy of sampling multiple QUD candidates during inference, followed by selecting the best one with criteria scorers. Our method outperforms the state-of-the-art baseline models by 9% in human evaluation and 4% in automatic evaluation, demonstrating the effectiveness of our framework. Code and data are in https://github.com/asuvarna31/qudselect.

## introduction discourse structure describes the relationships between different sentences of an article or conversation. the ability to understand discourse structure is crucial for natural language processing tasks such as text summarization @xcite , conditional generation @xcite , and narrative understanding @xcite . recent works have adapted the question under discussion (qud) framework to analyze discourse structures @xcite . in the qud framework @xcite , the relationships between sentences in an article are characterized by (implicit) free-form questions. each question is evoked by an anchor sentence in prior context, and answered by an answer sentence in the subsequent content. for instance, in figure 1 , the relationship between sentence 3 (referred to as s 3 ) and the previous context is that s 3 answers the question "which movie has the most oscar nominations?" evoked by the anchor sentence s 1 . the qud structures involve contextuallygrounded questions that adhere to three theoretical criteria @xcite @xcite : a) answer compatibility: the question must be answerable by the answer sentence in the discourse, like s 2 directly answers the question "who starred in forrest gump?" in figure 1 ; b) givenness: the question should only contain concepts that are accessible to the reader from prior context or common knowledge, like "forrest gump" in the question; and c) anchor relevance: the question should be relevant to the anchor sentence, e.g., the aforementioned question can be triggered in s 1 . ## related work qud is a linguistic framework that analyzes discourse and pragmatics by viewing each sentence as an answer to an implicit question triggered in prior context @xcite @xcite . while theoretical discussions around quds relied on constructed examples, riester (2019) introduced an annotation framework for reconstructing quds from data. westera et al. (2020), ko et al. (2022) and hesse et al. (2020) annotated ted-talk transcripts and news articles respectively in an expectation-driven manner, where questions are triggered while reading (i.e., unseen discourse progression) while de kuthy et al. (2018) annotated two interview transcripts with full, hierarchical questions. recent works have begun adapting qud for automatic discourse parsing @xcite @xcite , narrative graph construction @xcite and decontextualization of scientific documents @xcite . qud fits well for understanding the structure and coherence of texts that are intended to provide argumentation @xcite and complex reasoning @xcite , and has potential applications to enhance document understanding in information extraction @xcite @xcite with applications in wider domains like epidemiology @xcite and biomedical science @xcite . @xcite introduced a qud parser trained on dcqa @xcite that consists of an anchor selection and a question generation pipeline. a qud style annotation for generating narrative graphs by incorporating retrospective questions triggered from succeeding context. ## the qudselect framework task formulation given a document with n sentences @xmath0, s 2 , . . . , s n }, qud parsing aims to build a qud dependency tree. we formulate the qud parsing task as edge-level prediction following previous works (de @xcite : given an answer sentence s @xmath1, . . . , s n } 2 , models are asked to predict the anchor sentence a @xmath2, . . . , s i-1 } and generate the question q i . overview figure 2 illustrates the structure of our qudselect framework. we first instruction tune a joint qud parser §3.1. then, we propose selective decoding §3.2 to select the best candidate from sampled ⟨anchor sentence, question⟩ pairs. ## experimental setup models and datasets we utilize the dcqa dataset @xcite for training and evaluating qud parsers. the dcqa dataset consists of 22k english questions across 606 news articles. we use two instruction-tuned models llama2-7b @xcite and mistral-7b @xcite as base models of our framework. to explore the effectiveness of selective decoding on closedsource models, we also apply it to gpt-4 @xcite . we sample @xmath3. implementation details can be found in appendix a. baselines we compare against two existing qud parsers: the pipeline approach @xcite and gpt-4 prompting @xcite . we also provide ablation of not using selective decoding during inference, i.e., qudselect with @xmath4. human evaluation we follow the annotation guidelines outlined in qudeval @xcite and evaluate the generated quds for answer compatibility, givenness, and anchor relevance. detailed classification of the criteria is in appendix b. we evaluate 100 questions across 8 articles from the dcqa test set. we recruit three annotators from amazon's mechanical turk after extensive training and screening. we report the majority vote results and achieve an average inter-annotator agreement of 68.3% averaged across all evaluated dimensions. more details are in appendix c. ## conclusion in this work, we propose qudselect, a joint framework for generating qud structures by integrating key theoretical criteria. to achieve this, we reformulate the qud parsing as an instruction tuning task and selectively decode the candidate questions and anchors. furthermore, we develop automated evaluation methods trained on expert annotations to reduce the reliance on labor-intensive expert evaluations and facilitate model development for qud parsing. experiments demonstrate that qudselect significantly outperforms baselines in both automatic and human evaluations. the answer of a qud entails the answer of its descendants @xcite . furthermore, qudse-lect generates each qud edge independently and does not model the relationships between questions. thus, we leave the exploration of such discourse level constraints to future work.