Contextualized Embeddings for Connective Disambiguation in Shallow Discourse Parsing (2020)

Abstract: This paper studies a novel model that simplifies the disambiguation of connectives for explicit discourse relations. We use a neural approach that integrates contextualized word embeddings and predicts whether a connective candidate is part of a discourse relation or not. We study the influence of those context-specific embeddings. Further, we show the benefit of training the tasks of connective disambiguation and sense classification together at the same time. The success of our approach is supported by state-of-the-art results.

## introduction coherence is crucial for humans to be able to interpret text. the area of discourse parsing models this by identifying certain phrases (arguments) within a text and using discourse relations to unfold their underlying connections. these discourse relations and their understanding are important for tasks such as machine translation @xcite , abstractive summarization @xcite , and text simplification @xcite . a subset of these relations is signaled by specific words, socalled discourse connectives (or discourse markers or cues), and thus referred to as explicit discourse relations. however, such cues can be ambiguous, as they may signal more than one relation type or may not always function as a relation indicator. two challenges arise foot_0 -first, distinguishing connectives from words with mere sentential meaning: here, example 1 shows a discourse relation, while example 2 uses 'however' in its sentential reading. the second challenge consists in classifying a connective's sense (described in detail in section 2): word embeddings provide dense token representations in a low-dimensional vector space pretrained on large unannotated text corpora. first, we use fast-text @xcite , which is based on the skip-gram model @xcite and integrates character -grams into its representation. second, we use glove @xcite ; as opposed to fasttext, those embeddings were calculated through co-occurrence statistics rather than trained by a neural network. recently, models were introduced that provide contextualized word embeddings @xcite on demand and thus tackle the problem of identical representations for homonymous words with different senses, which had been indistinguishable in older models. for our experiments, we use bert @xcite , which was successful in many areas of nlp @xcite . in this work, we present a novel approach to identifying explicit relations in shallow discourse parsing. we introduce a simple yet powerful model that outperforms previous research on the binary disambiguation of connective candidates. furthermore, we adopt connective sense classification as an auxiliary task to improve performance and generalization capabilities and study the benefits of jointly training the auxiliary task in addition to the main task. this is because, in various cases, training neural models on multiple related tasks has shown beneficial for the learned representation @xcite , as it introduces inductive bias and, thereby, reduces the possible hypothesis space @xcite . specifically, the work of @xcite has pointed out the advantages of multitask learning on nlp tasks. we compare our results with state-of-the-art sdp components that took part at the conll shared task in 2016. 2 the contributions of this paper are as follows: in the following, section 2 describes the corpus for the experiments; section 3 explains our method. the experiments and results are presented in section 4 and section 5; section 6 discusses relevant related work, followed by conclusions in section 7. ## penn discourse treebank shallow discourse parsing is a challenging task that was promoted by the development of the second 2 we are not aware of more recent results. version of the penn discourse treebank (pdtb2) @xcite . this corpus provides about 43,000 annotated discourse relations, of which roughly 18,000 are signalled by explicit discourse connectives. those relations are further annotated with a three-level sense hierarchy (one or two senses per relation). all discourse relations consist of two arguments and are associated with one of various types; the focus of our work is on relations of the explicit type. the shared tasks at conll 2015 and 2016 @xcite used pdtb2 with minor changes. successful systems were @xcite ; @xcite ; @xcite . they largely follow a pipeline architecture @xcite , which consists of successive tasks of connective identification, argument labeling, and sense classification for both explicit and implicit relations. recently, pdtb3 @xcite was published, which extends the previous work with more available relations and corrects several former annotations. the authors also adjusted the relations' sense labels for a more balanced class distribution. for the sake of comparison with previous work on sdp, we stick to the pdtb2 corpus and assume to achieve similar results with pdtb3. ## method this work introduces a first, simple neural architecture for shallow discourse connective disambiguation. the system builds upon previous observations that a word's context could be used as a strong indicator for the presence of a discourse relation @xcite ). 3 our work investigates the limitations of knowledge free approaches and introduces a simple yet flexible model without domain knowledge. we assume that word embeddings contain information about the discourse that can be used for the disambiguation task. we study standard noncontextualized embeddings (in particular, glove embeddings and wikipedia-based fasttext embeddings) and compare those to the recently developed contextualized embeddings (represented by bert). we first hypothesize that contextualized embeddings yield better results than their noncontextualized counterpart. second, we expect the context span to influence the model's performance, as the context may indicate a word's function more clearly. in addition, we propose a second model based on the first one, which successfully combines connective disambiguation with sense classification as an auxiliary task. we follow the idea of previous work that sense classification can be performed without extracting the connectives' arguments @xcite @xcite . further, it has been previously shown that, for the identification of an explicit relation's sense, the connective itself as well as its context already provide significant information @xcite @xcite . consequently, we assume the necessary information for sense classification to be already accessible by our neural connective disambiguation model to some degree. also, this approach elim-inates the error propagation and the performance of our joint model stays as is without relying on previous predictions. the reason for adding sense classification as an auxiliary task in the first place is that joint training with auxiliary tasks has shown benefits in earlier work, as mentioned in section 1. we could validate that this is the case with our connective disambiguation task as well, as later demonstrated in our experiments (see section 4.2). in the following sections, we explain in more detail our binary connective disambiguation approach and the joint sense classification model. ## evaluation for our experiments, we use pdtb2 @xcite , especially the version provided for the conll shared task. we distinguish between coarse senses, which come from the original pdtb, and fine senses as defined by the shared task. also, an official split is provided that makes comparisons to other systems more reliable. in particular, this means that we used folders 02-22 for training, folders 00 and 01 as a development set, and folders 23 and 24 for testing. we downloaded word embeddings for glove foot_3 and fasttext foot_4 from their corresponding websites. for the contextualized embeddings, we extracted token embeddings that we had previously transformed using bert. 6 as we work with highly imbalanced data, we present our results using precision, recall, and f1 score. typically, there is a natural inverse rela- in our experiments, we study different embedding types (glove, fasttext, bert) with a varying context size (@xmath0, 1, 2}). the dimension (emb) of the noncontextualized word embeddings is 300 and 768 for contextualized embeddings, which results in an input size of (2 * ctx + 1) * emb. the size of the hidden layer is 2048. all models were trained for at most 50 epochs using early stopping @xcite when validation loss did not improve over 10 epochs, a batch size of 128, and the adam (kingma and ba, 2015) optimizer with a learning rate of 0.001. for comparison, we also provide a baseline from a reimplementation of @xcite . ## discussion in our final comparison, ## related work in this section, we discuss work relevant to the area of discourse parsing, in particular, connective disambiguation and sense classification. finally, recent work on word embeddings and multitask learning with regard to discourse parsing is outlined. for connective disambiguation, pitler and nenkova (2009) defined a set of syntactic features extracted from constituency trees. beside the connective's surface and category information from related tree nodes (parent, siblings), they also used the connective and its explicit sense have a strong correlation as shown by @xcite , who report accuracy higher than the interannotator agreement for their connective disambiguation features on coarse-grained level senses. @xcite use only context features and evaluate their work on second-level senses. wang and lan (2015) extend previous features and develop a model for the connl shared task. @xcite use an ensemble of three types of classifiers that are mainly based on previous features @xcite . @xcite use chained information extracted from syntactical trees and chunk tags. @xcite use convolutional neural networks on word-level embedded sentence pairs but a linear model with additional dependency features for sense classification. @xcite have shown that word embeddings outperform sparse features for implicit sense classification. they compare word pair features with brown clusters and low-dimensional word embeddings. @xcite use different levels of input representations, ranging from character level to contextualized word embeddings. @xcite adapt bert to perform implicit discourse sense classification. they show promising results by adding tasks, such as connective prediction, for pretraining. multitask learning is also successfully applied to implicit sense classification @xcite . the authors combine four different tasks related to discourse parsing, but in contrast to our work, they rely on previously extracted argument spans. @xcite propose a model that, in addition to their main task (implicit sense classification), also learns to predict a possible connective that could be inserted. @xcite introduce various models that perform multitask learning, and their focus also lies on implicit sense classification. ## conclusions in this work, we studied the value of discourse information in different kinds of word embeddings. we first presented a novel feature-free approach to connective disambiguation that achieves state-ofthe-art results on this task. then, this approach was extended by explicit sense classification to study the influence of jointly training both tasks. while our second approach does not directly outperform previous approaches on explicit sense classification, our model can be directly applied to raw input without being subject to error propagation, which is an advantage of our approach. as our work indicates that combining multiple subtasks avoid error propagation issues, a future direction could be to investigate what other kinds of subtasks could be combined in order to benefit from this. also, word embeddings have shown to be very flexible, and they are useful even for outof-domain data. it is worth investigating whether they are suitable for language transfer. this is particularly interesting because data sets of a similar quality to pdtb do not exist for many languages other than english.