Comparison of methods for explicit discourse connective identification across various domains (2021)

Abstract: Existing parse methods use varying approaches to identify explicit discourse connectives, but their performance has not been consistently evaluated in comparison to each other, nor have they been evaluated consistently on text other than newspaper articles. We here assess the performance on explicit connective identification of three parse methods (PDTB e2e, Lin et al., 2014; the winner of CONLL2015, Wang et al., 2015; and DisSent, Nie et al., 2019), along with a simple heuristic. We also examine how well these systems generalize to different datasets, namely written newspaper text (PDTB), written scientific text (BioDRB), prepared spoken text (TED-MDB) and spontaneous spoken text (Disco-SPICE). The results show that the e2e parser outperforms the other parse methods in all datasets. However, performance drops significantly from the PDTB to all other datasets. We provide a more fine-grained analysis of domain differences and connectives that prove difficult to parse, in order to highlight the areas where gains can be made.

## introduction understanding the discourse relations that hold between segments in natural language text is crucial to many nlp applications, such as text generation, dialogue understanding, and question-answering systems. shallow discourse parsers are used to uncover such relations by identifying connectives, extracting their arguments, and predicting the sense of the discourse relation. the current contribution focuses on the first step in the pipeline: finding explicit connectives in natural language. this step is not only interesting from the perspective of discourse relation classification, it can also be valuable for downstream applications, as accurate connective identification is crucial to mitigate the effect of cascaded errors downstream @xcite . connective identification is not a trivial task, as some connectives are ambiguous and may not consistently function as discourse connectives. a simple dictionary lookup would therefore not be able to distinguish between the discourse connectiveusage of yet in example (1), compared to the nonconnective usage in (2) and ( 3 ). in ( 1 ), yet expresses an adversative relation (equivalent to nevertheless). in (2), it functions as an adverb expressing a temporal meaning (i.e. up until this time). finally, in (3), yet does not function in a relation with two complete discourse arguments, and would therefore not be annotated as a connective. an accurate parser would need to be able to distinguish between these fine-grained differences in the usage of connective candidates. further, connective usage diverges between domains. for example, but and so are often used as discourse structuring markers in spoken language, rather than discourse connectives. existing parsers, however, have mainly been evaluated on newspaper text, since the largest discourse-annotated corpus available comes from this domain @xcite . performance of existing discourse connective identification parsers on domains other than the written one is currently not well known. finally, we note that there is a lack of information on which connectives remain difficult to identify, even in state-of-the-art parsers. this is due to the tendency of studies reporting only the general accuracy, without providing detail on the accuracy on specific connectives. however, such information can provide the field with more information on what to focus on. in sum, performance of different parsing methods have not been consistently evaluated in comparison to each other, nor have they been evaluated consistently on text other than newspaper articles. in addition, previous work tends to report only overall accuracy, whereas a more fine-grained analysis of connectives might prove to be valuable for future efforts. the current contribution fills these gaps in the literature. specifically, we evaluate the performance on explicit connective identification of four parse methods (pdtb e2e, @xcite ; the winner of conll2015, @xcite @xcite , along with a simple heuristic as a baseline. the heuristic will identify all connectives in the data, without being able to distinguish discourse versus non-discourse usages of connectives. we include this "parse method" to provide insight into how discourse usage of connectives affects the performance of such a heuristic. we also evaluate how well these systems generalize to different datasets from new domains, namely scientific text (biodrb, @xcite , prepared spoken text (ted-mdb, @xcite and spontaneous spoken text @xcite . ## related work connective identification in recent years, shallow discourse parsing has received notable attention in the field. much of this work has focused on the automatic labeling of implicit connectives, as performance there is significantly lacking (the current state-of-the-art achieves an f1 around 64% on a four-way classification, e.g. @xcite @xcite . explicit relation identification has received less attention since prior work has reported high accuracy on explicit connectives in news articles. an open question is how these parsers perform on out-of-domain data. pitler and nenkova (2009)'s work on explicit connective identification formed the basis for much subsequent research efforts in this direction. they show that syntactic features provide highly useful information for predicting whether a connective candidate functions as a discourse connective (an accuracy of 94.2% f1 on pdtb2 with a 10-fold cross validation on sec. 02-22). @xcite built on this work to develop the pdtb end-to-end (pdtb e2e) discourse parser. their parser per-forms at 95.4% f1 on the pdtb with a 10-fold cross validation on sec. 02-22. in the context of the conll2015 shared task, @xcite built on both of these approaches and presented the top-ranked system, which achieved an f1 score of 94.2% on connective identification in the pdtb2 section 23 test set and 91.9% in the conll 2015 blind test set. all three models rely heavily on various combinations of syntactic and lexical features extracted from texts. more recent models have taken different approaches. dissent uses dependency parsing and sentence embeddings to annotate discourse relations @xcite . they report an accuracy of 87.9% f1 on pdtb relations in determining whether a connective is present. notably, they used sentences extracted from books as training data, rather than pdtb. this might make their approach more stable across domains. finally, @xcite use a neural approach that integrates contextualized word embeddings and predicts whether a connective candidate is part of a discourse relation or not. they achieve an f1 score of 97% on the pdtb2 section 23 test set. ## discussion explicit connective identification can be done relatively reliably by existing parsers, but gains can still be made in this area. we therefore aimed to evaluate existing parsers and uncover more finegrained errors. the results showed that discopy @xcite outperformed the other parsers in three out of four datasets: pdtb, bio-drb, and ted-mdb. this indicates that the contextualized embeddings used by discopy are more flexible predictors of discourse connective usage than the syntactic and lexical features used by other parsers, even on out-of-domain data. the exception to this is discospice, for which the pdtb e2e parser performed best. however, even e2e's performance on this dataset was not sufficient. discospice contains features and syntactic patterns that are specific to spoken data, such as disfluencies, incomplete sentence structures, and increased ambiguity of connectives (e.g., whether so is used as a connective or marker). these features can explain the low performance of all parsers on this dataset. there is still room for improvement in this area. the performance of all parsers was lower on outof-domain text compared to pdtb. this reaffirms earlier findings, which showed that a connective identification classifier trained on pdtb does not perform well on biodrb even with domain adaptation techniques, compared to a classifier trained on the biodrb alone @xcite . of course, these results cannot be considered surprising, given that prior work on discourse parsing has heavily focused on the written domain, with a strong bias towards newspaper text. similar evaluations have not been done on other domains, nor have the parsers been applied frequently to other domains (but see @xcite , for an application of the e2e parser to spoken translated data). this underlines the importance of evaluation of existing parsers on other domains and the need for domain adaption of connective identification models and classifiers. the results further indicated that performance is affected by the connectives' usage as non-discourse connectives and the connectives' frequency. one solution, as suggested by @xcite , is to separately train a model for each highly ambiguous connective and another generic model to identify the remaining connectives. another solution could be to provide parsers with more training data for infrequent connectives, which can be obtained via connective generation; this approach has recently been applied to address the lack of training data for implicit relations @xcite . when analyzing connective-specific results, we found that poor performance could often be attributed to issues with the gold label. some of the false positives seemed to be valid connectives, and might therefore actually be missed instances in the gold dataset. this highlights that in manual annotation, errors are still prevalent and maybe inevitable to some extent, which can affect performance of parsers. one way to remedy this is to support manual annotation with the output of sota parsers or even a simple heuristic, so that inconsistencies or false negatives would be less likely to occur. we also found divergences between what frameworks and parsers consider relational arguments: biodrb identified instances of until with only a noun phrase as the second argument, which the parsers consistently did not consider a relational connective. conversely, discopy consistently identified instances of what if -relations, which the gold standard did not consider to be relations. discourse segmentation therefore has an impact on connective identification as well. unfortunately, the issue of segmentation is still not entirely resolved in the field of discourse relation annotation (see, e.g., @xcite . many connectives annotated in the gold datasets could not be identified at all by the parsers because they rely on pdtb2's connective list. this list is not exhaustive and would need to be expanded for the parsers to provide more coverage. one place to start would be pdtb 3.0's connective list @xcite . however, even this list will not cover all connective types that might also occur in other genres (e.g. hereafter occurs in biodrb but not pdtb). extending connective lists with a general lexicon of english connectives @xcite can provide new connective candidates as well. with regards to running the models, we observed a trade-off between accuracy and computational cost: the training and testing of discopy (a bertbased model) required much more computational energy than any traditional parser @xcite . when comparing the success of neural methods and the other three parsers, it is important to be clear about the context in which they are used @xcite . finally, we note that our results did not perfectly replicate those of the original authors. this is likely due to a difference in the experimental set-up used (e.g., the dependency parse method used by @xcite was not publicly available) and a difference in evaluation methods. to ensure that the lack of replicability was not due to incorrect implementation, the second author and another, unrelated researcher independently implemented all parse methods, and neither were able to perfectly replicate the results. note that @xcite also obtained different results than the original when replicating the e2e parser (although they observed improved scores), and attributed these divergences to minor variations between experimental set-ups in terms of implementations, hyperparameter settings and/or choice of the type of f1 score reported on. these results emphasize the general need in the field for more transparent reporting and a more consistent approach to evaluation (see also @xcite , for implicit relation classification). ## conclusion a comparison of different parse methods revealed that discopy, a neural parser using sentence embeddings, generally outperforms parsers using syntactic and lexical features. the results also showed a severe performance drop when applying the parsers to other domains, especially spontaneous spoken discourse. this can be attributed to genrespecific syntactic structures, issues with the gold standards, and differences between connective lexicons. these results emphasise the need for out-ofdomain training and evaluation, and provide insight as to where gains can be made.