D is C o D is C o at the DISRPT 2021 Shared Task: A System for Discourse Segmentation, Classification, and Connective Detection (2021)

Abstract: This paper describes our submission to the DISRPT2021 Shared Task on Discourse Unit Segmentation, Connective Detection, and Relation Classification. Our system, called DisCoDisCo, is a Transformer-based neural classifier which enhances contextualized word embeddings (CWEs) with hand-crafted features, relying on tokenwise sequence tagging for discourse segmentation and connective detection, and a feature-rich, encoder-less sentence pair classifier for relation classification. Our results for the first two tasks outperform SOTA scores from the previous 2019 shared task, and results on relation classification suggest strong performance on the new 2021 benchmark. Ablation tests show that including features beyond CWEs are helpful for both tasks, and a partial evaluation of multiple pretrained Transformer-based language models indicates that models pre-trained on the Next Sentence Prediction (NSP) task are optimal for relation classification.

## introduction recent years have seen tremendous advances in nlp systems' ability to handle discourse level phenomena, including discourse unit segmentation and connective detection @xcite as well as discourse relation classification (e.g. @xcite @xcite . for segmentation and connective detection, the current state of the art (sota) is provided by models using transformer-based, pretrained contextualized word embeddings @xcite , focusing on large context windows without implementation of hand-crafted features. for relation classification, sota performance on the english rst-dt benchmark @xcite has been achieved by neural approaches @xcite @xcite . for pdtb-style data, the 2015 and 2016 conll shared tasks on shallow discourse parsing @xcite have motivated work on both explicit (e.g. @xcite and implicit (e.g. @xcite @xcite @xcite ) discourse relation classification in english pdtb-2 @xcite and pdtb-3 @xcite as well as on the pdtb-style chinese newswire corpus (cdtb, @xcite ) such as @xcite and @xcite . our system for disrpt 2021, called dis-codisco, (district of columbia discourse cognoscente) extends the current sota architecture by introducing hand-crafted categorical and numerical features that represent salient aspects of documents' structural and linguistic properties. while transformer-based contextualized word embeddings (cwes) have proven to be rich in linguistic features, they are not perfect @xcite , and there are some textual features-such as position of a sentence within a document, or the number of identical words occurring in two discourse units-which are difficult or impossible for a typical transformer-based cwe model to know. we therefore supplement cwes with hand-crafted features in our model, with special attention paid to features we expect cwes to have a poor grasp of. we implement our system with a pretrained transformer-based contextualized word embedding model at its core, and dense embeddings of our hand-crafted features incorporated into it. our exact approach varies by task: we use a tokenwise classification approach for edu segmentation, a crf-based sequence tagger for connective detection, and a bert pooling classifier for relation classification. our system is implemented in py-torch @xcite using the framework allennlp @xcite . our results show sota scores exceeding comparable numbers from the 2019 shared task, and ablation studies indicate the contribution of features beyond cwes. ## previous work segmentation and connective detection following the era of rule-based segmenters (e.g. @xcite , @xcite used probabilistic models over constituent trees for token-wise binary classification (i.e. boundary/no-boundary). @xcite used a two-level stacked boosting classifier on chunks, pos tags, tokens and sentence lengths, among other features. @xcite used an svm over token and pos trigrams as well as phrase structure trees. more recently, @xcite used bi-lstm-crf sequence labeling on dependency parses, with words, pos tags, dependency relations, parent, grandparent, and dependency direction, achieving an f 1 of 89.5 on the english rst-dt benchmark @xcite with parserpredicted syntax. approaches using cwes as the only input feature @xcite have achieved an f 1 of 96.04 on the same dataset with gold sentence splits and 93.43 without, while for some smaller english and non-english datasets, approaches incorporating features and word embeddings remain superior (e.g. for english stac and gum, as well as dutch rst data, @xcite ; and for chinese, bourgonje and schäfer 2019; for more on these datasets see below). for connective detection, @xcite used a maxent classifier with syntactic features extracted from gold penn treebank @xcite parses of pdtb @xcite articles. @xcite presented a logistic regression model trained on eight relation types from pdtb, with features in three categories: relation-level such as the connective signaling the relation; argument-level such as size or complexity of argument spans; and discourse-level features, targeting dependencies between the relation and its neighboring relations in the text (cf. our approach to featurizing overall utilization of argument spans in the data below). @xcite used svm and crf to identify connectives in biomedical texts @xcite , with features such as pos tags, dependencies and domain-specific semantic features included several biomedical gene/species taggers, in addition to predicted biomedical ner features. current sota approaches rely on sequence labeling in a bio scheme with cwes from either plain text @xcite or integrating word embeddings and dependency tree features (pos, dependencies, phrase spans, @xcite ), dependending on the dataset and availability of gold standard features. discourse relation classification generally speaking, discourse relation classification assigns a relation label to two pieces of texts from a set of predefined coherence or rhetorical relation labels @xcite , which varies across different discourse frameworks, corpora, and languages. given different perspectives and theoretical frameworks, the implementation and evaluation of the relation classification task varies considerably. in rhetorical structure theory (rst, mann and thompson 1988), discourse relations hold between spans of text and are hierarchically represented in a tree structure @xcite . performance is evaluated and reported using the micro-averaged, standard parseval scores for a binary tree representation, following @xcite . current sota performance @xcite on the english rst-dt benchmark @xcite with gold segmentation achieved a micro-averaged original parseval score of 54.1 by utilizing both a spanbased neural parser @xcite and a two-staged transition-based svm parser @xcite as well as leveraging silver data. since pdtb is a lexically grounded framework, discourse relation classification is also called sense classification in pdtb-style discourse parsing: a sense label is assigned to the discourse connective between two text spans when a discourse connective is present (i.e. explicit relation classification) or a label is assigned to an adjacent pair of sentences when no discourse connective is present (i.e. implicit relation classification) @xcite . explicit relation classification is easier as the presence of the connective itself is considered the best signal of the relation label. most systems from the 2016 conll shared task on shallow discourse parsing adopted machine learning techniques such as svm and maxent with handcrafted features @xcite . for instance, for the english pdtb-2 @xcite , @xcite achieved the best performance (an f 1 = 90.22) on the explicit relation classification task by implementing a majority classifier and a maxent classifier while @xcite achieved the best performance (f 1 = 40.91) on implicit relation classification using a convolutional neural network. @xcite also achieved the best performance on the chinese cdtb dataset @xcite in the implicit relation classification task. more recent work on implicit relation classification has adopted a graph-based context tracking network to model the necessary context for interpreting the discourse and has gained better performance on pdtb-2 @xcite . in addition, the increase in the number of implicit relation instances in pdtb-3 @xcite has sparked more interest in exploring their recognition, such as @xcite and @xcite . @xcite presented the first set of results on implicit relation classification for both top-level senses (four labels) and more fined-grained level-2 senses (amounting to 11 labels) in pdtb-3 from two strong sentence encoder models using bert @xcite and xlnet @xcite . due to the novelty of the disrpt 2021 relation classification task, which combines implicit and explicit relation classification across frameworks for an unlabeled graph structure, comparable scores do not yet exist at the time of writing. ## approach our system comprises two main components: one targeting segmentation and connective detection using neural sequence tagging (as binary classification and bio tagging respectively), and one targeting relation classification using bert @xcite fine-tuning. we further enhance both components with the use of hand-crafted categorical and numeric features by encoding them in dense embeddings and introducing them into our neural models. ## discussion figure 1 shows confusion matrices for common relations in the highest and lowest scoring edu datasets, eng.rst.gum (gum) and eng.sdrt.stac (stac). both panels reveal issues with over-prediction of the most common labels, which can be thought of as 'defaults': the most common elaboration in gum and the second most common, comment in stac. the actual most common relation in stac, question, does not suffer from false positives, likely due to a combination of the frequent and reliable question mark cue, wh-words or subject-verb inversion, combined with the availability of gold speaker and direction information (question only links units from different speakers, left to right). the same is true for question in gum, which also obtained a comparatively high score. conversely, rarer relations are hardly predicted, with antithesis in gum being predicted for less than half of its true instances, and similarly for result in stac, suggesting a class imbalance problem, in particular given that both these relations are sometimes marked by overt discourse markers. although edu datasets (rst/sdrt) do not distinguish explicit and implicit relations, analysis suggests that explicit signals are important. for gum, the model scored high on medium-frequency relations with clear cues such as attribution, which are always signalled by attribution verbs such as believe and say. this also holds true for the eng.rst.rstdt corpus where attribution is the highest-scoring relation. gum's elaboration and joint as well as sequence and joint are two pairs of relation labels that are most frequently confused with each other, as the former pair contains the two labels that are both overgeneralized and the latter contains the two labels that are both multinuclear relation types that could be confusing when no explicit connective or lexical item indicating a sequential order of actions or events is present @xcite . relations with relatively unambiguous markers, such as condition, show good results in both gum and stac, indicating that even relatively rare relations can be identified if they are usually explicitly signalled. relations such as justify, result, and cause scored low in both matrices as such instances in the test data often do not have explicit discourse markers to help understand the rhetorical relation between the units in context. in the presence of an ambiguous discourse marker, predictions prefer the relation that is more prototypically associated with that marker: for instance, the gold relation label for (1) is result whereas the model classified it as sequence, likely due to the fact that the discourse marker then tends to be a strong signal for sequence indicating something sequential is involved. then she lets go and falls . you scream . in fact, if it were not the case that rst mandates a single outgoing relation per discourse unit, it would be possible to claim a concurrent sequential relation (likely to a previous unit) next to the annotated result relation for the pair in the example (see @xcite on concurrent relations in rst). ## conclusion we have presented discodisco, a system for all tasks in the disprt 2021 shared task: edu segmentation, connective detection, and relation classification. our system relies on sequence tagging and sentence pair classification architectures powered by cwes and supported by rich, handcrafted, instance-level features, such as position in the document, distance between units, gold speaker infor-mation, document metadata, and more. our results suggest that powerful pretrained language models are the main drivers of performance, with additional features providing small to medium improvements (with some exceptions, such as the high importance of speaker information for chat data as in stac). for relation classification, cwes pretrained using an nsp task proved to be superior. our error analysis suggests unsurprisingly that class imbalances, especially in the case of relations that tend to be implicit (i.e. lack overt lexical signals), lead to over-prediction of majority classes, suggesting a need for more training data for the minority ones. however, we are encouraged by improvements on datasets that were featured in the 2019 shared task @xcite , and the overall high scores obtained by the system across a range of datasets, all while including some correct predictions for relatively rare relations. we hope that the growing availability of annotated data, coupled with architectures that can harness pre-trained models, will lead to further improvements in the near future.