QAD iscourse - D iscourse R elations as QA P airs: R epresentation, C rowdsourcing and B aselines (2020)

Abstract: Discourse relations describe how two propositions relate to one another, and identifying them automatically is an integral part of natural language understanding. However, annotating discourse relations typically requires expert annotators. Recently, different semantic aspects of a sentence have been represented and crowd-sourced via question-and-answer (QA) pairs. This paper proposes a novel representation of discourse relations as QA pairs, which in turn allows us to crowd-source wide-coverage data annotated with discourse relations, via an intuitively appealing interface for composing such questions and answers. Based on our proposed representation, we collect a novel and wide-coverage QADiscourse dataset, and present baseline algorithms for predicting QADiscourse relations.

## introduction relations between propositions are commonly termed discourse relations, and their importance to the automatic understanding of the content and structure of narratives has been extensively studied @xcite @xcite . the automatic parsing of such relations is thus relevant to multiple areas of nlp research, from extractive tasks such as document summarization to automatic analyses of narratives and event chains @xcite . so far, discourse annotation has been mainly conducted by experts, relying on carefully crafted linguistic schemes. two cases in point are pdtb @xcite and @xcite . such annotation however is slow and costly. crowd-sourcing discourse relations, instead of using experts, can be very useful for obtaining largerscale training data for discourse parsers. the executions were spurred by lawmakers requesting action to curb rising crime rates. what is the reason lawmakers requested action? to curb rising crime rates what is the result of lawmakers requesting action to curb rising crime rates? the executions were spurred i decided to do a press conference [...], and i did that going into it knowing there would be consequences. despite what did i decide to do a press conference? knowing there would be consequences one plausible way for acquiring linguistically meaningful annotations from laymen is using the relatively recent qa methodology, that is, converting a set of linguistic concepts to intuitively simple question-and-answer (qa) pairs. indeed, casting the semantic annotations of individual propositions as narrating a qa pair gained increasing attention in recent years, ranging from qa driven semantic role labeling (qasrl) @xcite @xcite to covering all semantic relations as in qamr @xcite . these representations were also shown to improve downstream tasks, for example by providing indirect supervision for recent mlms @xcite . in this work we address the challenge of crowdsourcing information of higher complexity, that of discourse relations, using qa pairs. we present the qadiscourse approach to representing intrasentential discourse relations as qa pairs, and we show that with an appropriate crowd-sourcing setup, complex relations between clauses can be effectively recognized by non-experts. this layman annotation could also easily be ported to other languages and domains. specifically, we define the qadiscourse task to be the detection of the two discourse units, and the labeling of the relation sense between them. the two units are represented in the question body and the answer, respectively, while the question type, as expressed by its prefix, represents the discourse relation sense between them. this representation is illustrated in the contribution of this paper is thus manifold. (i) we propose a novel qa-based representation for discourse relations reflecting a subset of the sense taxonomy of pdtb 3.0 @xcite . @xcite we propose an annotation methodology to crowdsource such discourse-relations qa pairs. and, (iii) given this representation and annotation setup, we collected qadiscourse annotations for about 9000 sentences, resulting in more than 16600 qa pairs, which we will openly release. finally, (iv) we implement a qadiscourse parser, serving as a baseline for predicting discourse questions and respective answers, capturing multiple discoursebased propositions in a sentence. bankamerica climbed 1 3/4 to 30 (arg1) after painewebber boosted its investment opinion on the stock to its highest rating (arg2). these relations are called shallow discourse relations since, contrary to the rhetorical structure theory (rst) @xcite , they do not recursively build a tree. the pdtb organizes their sense taxonomy, of which examples can be seen in qasrl: back in warsaw that year, chopin heard niccolò paganini play the violin, and composed a set of variations, souvenir de paganini. what did someone hear? niccolò paganini play the violin when did someone compose something? that year qamr: actor and television host gary collins died yesterday at age 74. what kind of host was collins? television how old was gary collins? 74 semantic qa approaches using qa structures to represent semantic propositions has been proposed as a way to generate "soft" annotations, where the resulting representation is formulated using natural language, which is shown to be more intuitive for untrained annotators @xcite . this allows much quicker, more large-scale annotation processes @xcite and when used in a more controlled crowd-sourcing setup, can produce high-coverage quality annotations @xcite . as displayed in ## representing discourse relations as qa pairs in this section we discuss how to represent shallow discourse relations through qa pairs. for an overview, consider the second sentence in the discourse relation sense we want our questions to denote relation senses. to define the set of discourse relations covered by our approach, we derived a set of question templates that cover most discourse relations in the pdtb 3.0 @xcite , as shown in few pdtb senses are not covered by our question prefixes. first, senses with pragmatic specifications like belief and speechact were collapsed into their general sense. second, three expansion senses were not included because they usually do not assert a new "informational" proposition, about which a question could be asked, but rather capture structural properties of the text. one of those is expansion.conjunction, which is one of the most frequently occurring senses in the pdtb, especially in intra-sentential vp conjunctions, where it makes up about 70% of the sense instances @xcite . ex. (2) displays a discourse relation with two senses, one of which expansion.conjunction. while it is natural to come up with a question targeting the causal sense, the conjunction relation does not seem to assert any proposition about which an informational question may be asked. finally, we removed purpose as a (somewhat subtle) individual sense and subsumed it with our two causal questions. our desiderata for the question templates are as follows. first, we want the question prefixes to be applicable to as many scenarios as possible in pdtb sense question template expansion.substitution instead of what x? expansion.disjunction what is an alternative to x? expansion.exception except when x? comparison.concession despite what x? comparison.contrast what is contrasted with x? expansion.level-of-detail what is an example of x? comparison.similarity what is similar to x? temporal.asynchronous after/before what x? until/since when x? temporal.synchronous while what x? contingency.condition in what case x? contingency.neg.-cond. unless what x? expansion.manner in what manner x? contingency.cause what is the result of x? what is the reason x here we employ more specific question prefixes to remove this ambiguity. finally, as multiple relation senses can hold between two discourse units (rohde et al., 2018), we similarly allow multiple qa pairs for the same two discourse units. the discourse units the two sentence fragments, typically clauses, that we relate with a question are the discourse units. in determining what makes a discourse unit, we include verbal predicates, noun phrases and adverbial phrases as potential targets. this, for example, would also cover such instances: "because of the rain ..." or "..., albeit warily". we call the corresponding verb, noun or adverb heading a discourse unit a target. a question is created by choosing a question prefix, an auxiliary, if necessary, and copying words from the sentence. it can then be manually adjusted to be made grammatical. similarly, the discourse unit making up the answer consists of words copied from the sentence and can be modified to be made grammatical. our question and answer format thus deviates considerably from the qasrl representation. by not introducing placeholders, questions sound more natural and easy to answer compared to qasrl, while still being more controlled than the completely free form questions of qamr. in addition, allowing for small grammatical adjustments introduces valuable flexibility which contribute to the intuitiveness of the annotations. in what case do checks and balances happen? when there is competition in a capitalistic society 2. whilst staying in the hotel, the wikimedian group met two meps who chose it in-preference to dramatically more-expensive strasbourg accommodation. what is contrasted with the hotel? dramatically moreexpensive strasbourg accommodation 3. there were no fare hikes announced as both passenger and freight fares had been increased last month. what is the reason there were no fare hikes announced? as both passenger and freight fares had been increased last month what is the result of both passenger and freight fares having been increased last month? there were no fare hikes announced relation directionality discourse relations are often directional. our qa format introduces directionality by placing discourse units into either the question or answer. in some question prefixes, a single order is dictated by the question. as seen in ex. 1 of ## the crowdsourcing process pool of annotators to find a suitable group of annotators we followed the controlled crowdsourcing methodology of @xcite . we first released two trial tasks, after which we selected the best performing workers. these workers then underwent two short training cycles, estimated to take about an hour each, which involved reading the task guidelines, consisting of 42 slides 1 , completing 30 hits per round and reading personal feedback after each round (preparing these feed-backs consumed about 4 author work days). 11 workers successfully completed the training. for collecting production annotations of the dev and test sets, each sentence was annotated by 2 workers independently, followed by a third worker who adjudicated their qa pairs to produce the final set. for the train set, sentences were annotated by a single worker, without adjudication. preprocessing in preprocessing, question targets are extracted automatically using heuristics and pos tags: a sentence is segmented using punctuation and discourse connectives (from a list of connectives from the pdtb). for each segment, we treat the last verb in a consecutive span of verbs as a separate target. in case a segment does not contain a verb, but does start with a discourse connective, we choose one of the nouns (or adverbs) as target. the following illustrates our target extraction: [despite labor-shortage warnings,] [80% aim for first-year wage increases of under 4%;] [and 77% say they'd try to replace workers,] [if struck,] [or would consider it.] annotation tool and procedure using amazon mechanical turk, we implemented two interfaces 2 , one for the qa generation and one for the qa adjudication step. in the qa generation interface, workers are shown a sentence with all target words in bold. workers are instructed to generate one or more questions that relate two of these target words. the question is generated by first choosing a question prefix, then, if applicable, an auxiliary, then selecting one or more spans from the sentence to form the complete question, and lastly, change it to make it grammatical. given the generated question, the next step involves answering that question by selecting span(s) from the sentence. again, the answer can also be amended to be made grammatical. the qa adjudication interface displays a sentence and all the qa pairs produced for that sentence by two annotators. for each qa pair the adjudicator is asked to either label it as correct, not correct or correct, but not grammatical. duplicates and nonsensical qa pairs labeled as not correct are omitted from the final dataset. as a last step, the first author manually corrected all the not grammatical instances. ## baseline model for qadiscourse in this section we aim to devise a baseline discourse parser based on our proposed representation, which accepts a sentence as input and outputs qa pairs for all discourse relations in that sentence, to be trained on our collected data. similarly to previous work on discourse parsing (section (1)), our proposed parser is a pipeline consisting of three phases: (i) question prefix prediction, (ii) question generation, and (iii) answer generation. formally, given a sentence @xmath0, ..., x n with a set of indices i which mark target words (based on the target extraction heuristics in section 4), we aim to produce a set of qa-pairs (q j , a j ) using the following pipeline: ## results and discussion after running the full pipeline, we evaluate the predicted set of qa-pairs against the gold set using the uqa and lqa metrics, described in section 5.1. we randomly selected a sample of 50 predicted dev test question prediction 71.9 65.9 answer prediction 68.9 72. 3 qas for a qualitative analysis. 22 instances from this sample were judged as correct, and 2 instances were correct despite not being mentioned in gold. examples of good predictions are shown on the left column in ## conclusion in this work, we show that discourse relations can be represented as qa pairs. this intuitive representation enables scalable, high-quality annotation via crowdsourcing, which paves the way for learning robust parsers of informational discourse qa pairs. in future work, we plan to extend the annotation process to also cover inter-sentential relations. calculating the weights for the loss of the prefix classifier each question prefix label is weighted by subtracting the label count from the total count of training instances and dividing it by the label count: weight @xmath2.