QUD eval: The Evaluation of Questions Under Discussion Discourse Parsing (2023)

Abstract: Questions Under Discussion (QUD) is a versatile linguistic framework in which discourse progresses as continuously asking questions and answering them. Automatic parsing of a discourse to produce a QUD structure thus entails a complex question generation task: given a document and an answer sentence, generate a question that satisfies linguistic constraints of QUD and can be grounded in an anchor sentence in prior context. These questions are known to be curiosity-driven and open-ended. This work introduces the first framework for the automatic evaluation of QUD parsing, instantiating the theoretical constraints of QUD in a concrete protocol. We present QUDeval, a dataset of fine-grained evaluation of 2,190 QUD questions generated from both fine-tuned systems and LLMs. Using QUDeval, we show that satisfying all constraints of QUD is still challenging for modern LLMs, and that existing evaluation metrics poorly approximate parser quality. Encouragingly, human-authored QUDs are scored highly by our human evaluators, suggesting that there is headroom for further progress on language modeling to improve both QUD parsing and QUD evaluation.

## introduction text comprehension at the discourse level entails understanding higher level structures between sentences and paragraphs, beyond the meaning of individual words. the linguistic framework of questions under discussion (qud) @xcite @xcite views the progression of discourse as continuously posing (implicit) questions (i.e., quds) and answering them; thus each sentence in a monologue is an answer, or part of an answer, to a qud. in figure 2 , the third sentence answers the implicit qud, "what does glenn say about the situation?", elicited from sentence 2. the advent of large language models (llms) makes viewing discourse in a question generation and answering fashion increasingly tractable for settings that require higher-level processes, e.g., planning in text generation @xcite , contextualization in question answering @xcite , and elaborative simplification @xcite . however, rigorous evaluation of the question generation aspect of qud frameworks remains an open challenge. qud parsing is relatively new to the nlp community, with most of the prior work in discourse parsing focusing on structures like rhetorical structure theory @xcite , segmented discourse representation theory @xcite , and the penn discourse treebank @xcite , all of which use a fixed, hierarchical discourse relation taxonomy and are relatively straightforward to evaluate using accuracies or f measures. qud parsing (illustrated in figure 2 ), however, entails a question generation task that is contextually grounded @xcite . qud (1,2) 1 2 3 5 6 qud (2,3) what does glenn say about the situation? @xcite what does the report say is the reason for the export ban? @xcite what is missing in the report? what do lawmakers think about this issue? answer (s2): 'scarcely a day goes by without a report of a new black market deal,' said sen. john glenn … reacting to the report. given the article snippet on the left, a qud dependency parser generates the tree with edges as quds; qud(2, 3) means that sentence 3 answers the corresponding question anchored in sentence 2. besides generating the question, the position of the anchor sentence (head node) is also predicted. qudeval includes 4 criteria for each edge (right) evaluating both the generated qud and the predicted anchor, discussed in section 3. this structure necessitates two key principles for evaluation. first, the questions are raised after a certain amount of common ground is established, so parsing qud involves identifying where in the context a question can be triggered (also called the anchor sentence). these anchors are produced by parsers and form part of the structure to evaluate. second, for a question to become a qud, it needs to satisfy several linguistic constraints @xcite and be plausible without knowing a priori how the discourse will progress. the above characteristics make qud parsing much tricker to evaluate than other discourse parses; thus, prior work in qud dependency parsing @xcite performed only manual evaluation. automatic evaluation for generation tasks is known to be challenging @xcite and standard evaluation metrics for question generation @xcite @xcite tend to be superficial linguistically. they do not evaluate the aforementioned desiderata of qud. outlined in figure 1 , this work provides foundational building blocks for the automatic evaluation of qud dependency parsers. building on prior theoretical work on qud and its annotation @xcite , as well as error analysis on a large number of machine outputs from several different models, we design four key criteria to assess qud question generation quality, covering language quality, answer compatibility, question givenness, and anchor relevance. with these criteria, we present qudeval, 2,190 generated question-anchor pairs annotated by trained linguists across 51 news articles. these annotations evaluate three qud dependency parsing models: ko et al. (2023)'s pipeline system, as well as our newly designed prompts for qud parsing with llms using openai chatgpt (gpt 3.5 turbo), gpt-4, and stanford alpaca @xcite . human annotations using the qudeval framework reveal that modern llms, while able to generate fluent questions, often fail at satisfying these constraints; the errors vary across systems, rendering such fine-grained evaluation necessary. qudeval also provides a valuable testbed for the assessment of automatic evaluation metrics for qud parsers. we engage with rule-based and llm-based baselines, prior work relevant for specific criteria, and reference-based metrics standard in question generation evaluation. results show that while these metrics align with qudeval's annotations to some extent, they give poor estimates of qud generation quality (except for anchor groundedness) in comparison to an estimated human upper bound. this points to both challenges and substantial headroom for the development of automatic metrics. to sum, qudeval points to clear directions for future progress: (1) human evaluation reveals consistently higher quality for crowdsourced quds compared to generated ones, suggesting opportunities for better qud parsers; (2) the development of linguistically-informed automatic metrics targeting qud parsing quality is necessary, for which qudeval can serve as a benchmark. qudeval is available at https://github. com/lingchensanwen/qudeval . the parsing framework that this paper engages with is from @xcite , trained with the dcqa dataset from @xcite . @xcite views the qud structure as a dependency tree, where each sentence in a document is connected to an anchor sentence in its prior context via a qud (formalized in section 3). this is a lightweight approach to qud as the questions themselves are not guaranteed to be hierarchical; however this simplification allows large-scale, crowdsourced data collection @xcite . to date, this remains the only qud parser available. evaluation of question generation prior work has tackled contextual question generation, including conversational @xcite @xcite , open-ended @xcite @xcite , and nested questions (krishna and iyyer, 2019). while some of these works perform human evaluation, they typically cover dimensions such as fluency, relevance/plausibility, scope, and question-answer consistency. in clarification question generation @xcite , dimensions such as usefulness and informativeness are also considered. however, the above criteria are insufficient for a linguistically meaningful evaluation of qud generation. @xcite used a human evaluation taxonomy for their qud parser; this work reflects an extended version of that earlier taxonomy, better aligned with theoretical principles. similar to automatic evaluation of natural lan-guage generation (nlg) tasks in general @xcite , automatic evaluation of question generation is known to be challenging and inconsistent @xcite . we evaluate commonlyused metrics, as well as a recent metric from gollapalli and ng (2022), using our linguisticallygrounded frameworks. ## qud evaluation framework quds are subjective: different readers often come up with distinct questions even with the same answer sentence @xcite . thus our goal is to target common principles that quds need to satisfy. first, we instantiate theoretical constraints of qud @xcite in a concrete protocol. second, we consider common errors in text generation systems, including question generation systems, and ensure they are captured in our taxonomies. setup and definitions each evaluation instance corresponds to an individual edge in a dependency tree where a qud connects two sentences: the head of an edge is an anchor sentence s k (the kth sentence in a document) where the question is elicited, and the child node is an answer sentence s a that answers the question (figure 2 ). thus we denote an edge as (q, s k , s a ) where q is the question string. qud dependency parsing entails considering each sentence in a document as s a and, for that answer sentence, generating q and predicting k, the anchor index. we present four criteria that assess both tasks separately and jointly, with the full annotatorfacing instructions listed in appendix figure 4 . note that it is both theoretically and operationally possible that more than one sentence satisfies the criteria of being an anchor. in practice, we see that this depends on varying levels of specificity of the question and the document context. because of this, our evaluation framework independently evaluates each qud and its predicted anchor. language quality (lang.) this criterion is designed to filter out obviously bad questions. these include: badly formed questions that are not accommodatable, questions that are irrelevant to the article, and questions with content that obviously contradicts the article. note that we direct annotators to skip the rest of the evaluation if q fails to satisfy this criterion. examples of such questions are shown in appendix ## the qudeval dataset this section describes our linguist-annotated qudeval dataset evaluating fine-tuned @xcite and llm parsers, and their results. ## automatic evaluation metrics qudeval can serve as the first benchmark for automatic evaluation metrics for qud dependency … 5 the report says hundreds of tons of plutonium and highly enriched uranium have accumulated worldwide… 6 it does not include figures on u.s. nuclear exports but says 71 export licenses for nuclear materials were granted in 1993. 7 nuclear exports for weapons use or weapons research are prohibited, as is transfer of nuclear materials to a third country. ko et al context answer dcqa (human) what restrictions exist on nuclear material exports? anchor: s6 what else was not included in the report? anchor: s6 chatgpt what are the restrictions on nuclear exports and transfer of nuclear materials to a third country? anchor: s5 alpaca what are the restrictions on nuclear exports? anchor: s1 ! answer incompatible ## conclusion we present a dataset, qudeval, for evaluating contextual question generation in the context of qud discourse parsing. qudeval implements prior theoretical evaluation criteria for qud, operationalized as four criteria for which expert linguistics annotators give high-quality judgments. this dataset sheds light on the divergent performance of existing systems, and enables future work on developing stronger automated metrics for evaluating qud discourse parsing.