Label distributions help implicit discourse relation classification (2022)

Abstract: Implicit discourse relations can convey more than one relation sense, but much of the research on discourse relations has focused on single relation senses. Recently, DiscoGeM, a novel multi-domain corpus, which contains 10 crowd-sourced labels per relational instance, has become available. In this paper, we analyse the co-occurrences of relations in DiscoGem and show that they are systematic and characteristic of text genre. We then test whether information on multi-label distributions in the data can help implicit relation classifiers. Our results show that incorporating multiple labels in parser training can improve its performance, and yield label distributions which are more similar to human label distributions, compared to a parser that is trained on just a single most frequent label per instance.

## introduction much of the research on the discourse relations (dr) assumes (implicitly or explicitly) that only one relation can hold between two spans of text, particularly in the computational field. however, recent work has shown that discourse relations, and implicit relations in particular, can allow for multiple interpretations (e.g., @xcite . prior work has also shown that certain relation senses tend to cooccur in newspaper text, such as arg2-as-detail with reason and contrast with concession @xcite . the current study investigates whether the co-occurrence of implicit relation senses depends on text genre. we furthermore explore whether the performance of a state-of-the-art discourse relation classifier can be improved by training it on the distribution of human-annotated labels, as opposed to training it on only the top label. the contributions of the current paper are twofold: (1) we analyse the specific patterns of dr co-occurrences in different genres, showing that the distribution of sense co-occurrences are dependent on genre. (2) we train an implicit dr classifier on distributionally upsampled dr labels for each relation instance. this approach leads to better performance according to a soft evaluation metric (cross entropy), demonstrating that the natural distribution of human interpretations can be captured better when training on the distribution of labels. ## data we include data from the pdtb 3.0 @xcite and the discogem corpus @xcite . pdtb the pdtb 3.0 is a news text corpus containing articles from the wall street journal. the corpus contains 15,544 inter-sentential (between sentences) and 6,188 intra-sentential (within sentences) implicit relations. we follow the split suggested by @xcite : sections 2-20, 0-1, and 21-22 are used as training, validation, and test sets. discogem discogem is a crowdsourced mixedgenre corpus of 6,505 inter-sentential implicit relations, containing text from europarl (prepared spoken text, argumentative genre), literature (narrative genre), and wikipedia (informative genre) (see @xcite , for more details on the genres and data selection process). discogem contains texts from both original english sources as well as translated english. the annotations were crowdsourced using a connective insertion task, that allowed the authors to extract pdtb 3.0 labels. every instance received annotations from 10 crowdworkers to represent the multiple interpretation of the discourse relation. the cohen's kappa between the multiple labels annotated by the crowdworkers and trained annotators was 0.79 (see original paper for further details.) the labels used in the current study are a single majority vote label (randomly sampled in case two senses received an equal number of votes) and a soft distribution label based on the raw votes. the discogem data was split 1 into 70% for train, 20% for test, and 10% for dev for the implicit relation classification reported in section 5 and for future studies training and testing on discogem. in splitting the data, we first set aside texts for testing only (i.e., unseen texts). these texts consist of 26 out of 197 turns of speech in the europarl genre, 2 out of 20 books in the novels genre, and 4 out of 69 wikipedia texts. they are balanced between original english and translated english. for the remainder of the data, we ensured that each text 2 is split proportionally into train, dev and test sets. the dev and test portions are always consecutive, such that the training portions have minimal gaps, but either one comes first randomly and is randomly picked from different locations of the article. the distribution of the relation types in the training data we used is shown in ## analysis of relation co-occurrence we examine what types of drs are often interpreted by the crowdworkers at the same time. for each annotated instance in discogem, we select the top1 label as the label that received the majority vote (randomly sampled in case of a tie). the top2 label is the label that received the second highest vote. in case where there's less than 30% agreement on the top2 label, we consider top1 to be a single label (i.e., the top2 @xmath0. this allows us to focus the analysis on cooccurrences of generally accepted interpretations. from the distributions of the top1 ∼ top2 pairs of the entire corpus, we extract the marginal probabilities p (t op1) and p (t op2), as well as expected joint probabilities p exp (t op1, t op2). we compare the observed joint probabilities p (t op1, t op2) per genre by normalized pointwise mutual information (npmi), as defined below: n p m i(t op1; t op2) = p m i(t op1; t op2) -log p exp (t op1, t op2) figure 1 shows the npmi of each unique pair of drs for the most frequent level-2 relation types. 3 a value of 1.0 means the relations always occur together, -1.0 means they never occur together, and 0.0 means that they co-occur as much as would be expected to happen by chance. the results confirm the overall co-occurrence patterns established in previous work: figure 1: npmi per genre of co-occurrences of the most frequent relation senses. the x-axis represents the top1 label; the y-axis the top2 label. co-occurrences are more likely when instantia-tion or level-of-detail is the top1 label, and not reason. this indicates that instantiations and level-of-detail relations can often also convey an argumentative relation, but this is less likely vice versa. there are also a number of interesting divergences between genres. for example, relations in the wikipedia genre are more likely to co-occur with conjunctions than relations in other genres; conjunctions are also more likely to occur alone in wikipedia. this is likely due to the expository nature of wikipedia texts. further, we can see differences in the patterns of result and prece-dence relations. these are likely to co-occur in novels in particular, as well as wikipedia to some degree, but not in europarl data. rather, in europarl, result relations occur as a single label (aln) more frequently compared to the other genres. this matches the argumentative nature of the political genre. we can conclude from this analysis that there are not only differences in the relation distributions between genres, but there are also differences in the distribution of relation co-occurrences between genres. such fine-grained differences might impact parsers. in the remainder of this paper, we will study whether incorporating multiple labels in parser training, instead of only the top1 label can improve performance. ## discussion and conclusion we presented a simple, yet effective approach to take into account information on multi-label distributions in implicit relation classifiers. our results show that adding multi-label information leads to results that are on par with single label models when using a hard evaluation metric. the multilabel model performs even better than single label models when using a soft evaluation metric. this means that the multi-label model generates labels that are more reasonable given the distributions of labels provided by humans. this was the case even for the pdtb test set, despite the pdtb being designed as a primarily single label dataset. in future work, we also plan to study the impact of co-training with a large corpus such as the pdtb on the discogem genres. furthermore, will explore methods of incorporating human uncertainty in discourse relation classification with a specific focus on out-of-domain data.