Implicit Discourse Relation Classification: We Need to Talk about Evaluation (2020)

Abstract: Implicit relation classification on Penn Discourse TreeBank (PDTB) 2.0 is a common benchmark task for evaluating the understanding of discourse relations. However, the lack of consistency in preprocessing and evaluation poses challenges to fair comparison of results in the literature. In this work, we highlight these inconsistencies and propose an improved evaluation protocol. Paired with this protocol, we report strong baseline results from pretrained sentence encoders, which set the new state-of-the-art for PDTB 2.0. Furthermore, this work is the first to explore fine-grained relation classification on PDTB 3.0. We expect our work to serve as a point of comparison for future work, and also as an initiative to discuss models of larger context and possible data augmentations for downstream transferability.

## introduction understanding discourse relations in natural language text is crucial to end tasks involving larger context, such as question-answering @xcite and conversational systems grounded on documents @xcite . one way to characterize discourse is through relations between two spans or arguments (arg1/arg2) as in the penn discourse treebank (pdtb) @xcite . for instance: [ arg1 i live in this world,] [ arg2 assuming that there is no morality, god or police.] (wsj_0790) label: expansion.manner.arg2-as-manner the literature has focused on implicit discourse relations from pdtb 2.0 @xcite , on which deep learning has yielded substantial performance gains (chen et al., 2016; liu and li, 2016; lan et al., 2017; qin et al., 2017; bai and * work done while at ibm research. @xcite . however, inconsistencies in preprocessing and evaluation such as different label sets @xcite pose challenges to fair comparison of results and to analyzing the impact of new models. in this paper, we revisit prior work to explicate the inconsistencies and propose an improved evaluation protocol to promote experimental rigor in future work. paired with this guideline, we present a set of strong baselines from pretrained sentence encoders on both pdtb 2.0 and 3.0 that set the state-of-the-art. we furthermore reflect on the results and discuss future directions. we summarize our contributions as follows: in pdtb, two text spans in a discourse relation are labeled with either one or two senses from a three-level sense hierarchy. pdtb 2.0 contains around 43k annotations with 18.4k explicit and 16k implicit relations in over 2k wall street journal (wsj) articles. identifying implicit relations (i.e., without explicit discourse markers such as model ji lin p&k x-accuracy majority class 26.18 26.11 28.54 26.42 adversarial net @xcite 46.23 44.65 --seq2seq+memnet @xcite 47.83 45.82 -41.29 † elmo @xcite 48.22 45.73 --elmo, memory augmented @xcite 49.15 46.08 --multitask learning @xcite 49.95 46.48 --bert+mnli @xcite --53.7 -bert+dissent books 5 @xcite but) is more challenging than explicitly signaled relations @xcite . the new version of the dataset, pdtb 3.0 @xcite , introduces a new annotation scheme with a revised sense hierarchy as well as 13k additional datapoints. 2 the third-level in the sense hierarchy is modified to only contain asymmetric (or directional) senses. ## proposed evaluation protocol while @xcite lay out one possible protocol, it does not fully address the issues we have raised in section 2. another limitation is the unavailability of the preprocessing code as of the date of this submission. we describe our proposal below, which will be accompanied by a publicly available preprocessing code. 4 in addition to accounting for the variation previously discussed, we take shi and demberg (2017)'s concerns into consideration. cross-validation. we advocate using crossvalidation for l2 classification, sharing the concerns of @xcite on label sparsity. however, we propose using cross-validation at section-level rather than individual example-level as suggested by @xcite . this is to preserve paragraph and document structures, which are essential for investigating the effect of modeling larger context (e.g., @xcite . we further illustrate the potential utility of document structure in section 4. we suggest dividing the 25 sections of pdtb into 12 folds with 2 development, 2 test and 21 training sections in each fold. we used a sliding window of two sections starting from p&k (dev: 0-1, test: 23-24, train: 2-22). all but one section ( 22 ) is used exactly once for testing. whether future works should evaluate on these particular cross-validation splits or on randomized splits @xcite ) is an open issue; we provide an additional discussion in appendix f. label sets. we recommend reporting results on both l1 and l2, using the standard 11-way classification for l2 in pdtb 2.0. a standardized label set does not exist yet for l2 in pdtb 3.0 (l1 remains unchanged). we propose using only the labels @xmath0, which leaves us with 14 senses from l2 (see appendix a for counts). we suggest using all four possible label fields if the senses are multiply-annotated, as discussed in section 2.1. model x-accuracy (±σ) majority class 26.61 bert (base, uncased) 57.60 (±0.19) bert (large, uncased) 61.02 (±0.19) xlnet (base, cased) 60.78 (±0.24) xlnet (large, cased) 64.83 (±0.37) ## conclusion we have surveyed the literature to highlight experimental inconsistencies in implicit discourse relation classification, and suggested an improved protocol using section-level cross-validation. we provided a set of strong baselines for pdtb 2.0 and 3.0 following this protocol, as well as results on a range of existing setups to maintain comparability. we discussed several future directions, including data augmentation for downstream transferability, applicability of pretrained encoders to discourse, and utilizing larger discourse contexts. for the training split of ji is one short of what has been reported in @xcite and also the count obtained by using @xcite tables 7 and 8 list the label counts of each class in pdtb 3.0 and pdtb 2.0, respectively.