Coreference for Discourse Parsing: A Neural Approach (2020)

Abstract: We present preliminary results on investigating the benefits of coreference resolution features for neural RST discourse parsing by considering different levels of coupling of the discourse parser with the coreference resolver. In particular, starting with a strong baseline neural parser unaware of any coreference information, we compare a parser which utilizes only the output of a neural coreference resolver, with a more sophisticated model, where discourse parsing and coreference resolution are jointly learned in a neural multitask fashion. Results indicate that these initial attempts to incorporate coreference information do not boost the performance of discourse parsing in a statistically significant way.

## introduction and task description discourse parsing is a very useful natural language processing (nlp) task involving predicting and analyzing discourse structures, which represent the coherence properties and relations among constituents of multi-sentential documents. in this work, we investigate discourse parsing in the context of rhetorical structure theory (rst) @xcite , which encodes documents into complete constituency discourse trees. an rst tree is defined on the sequence of a document's edus (elementary discourse units), which are clause-like sentences or sentence fragments (propositions), acting as the leaves of the tree. adjacent edus and constituents are hierarchically aggregated to form (possibly non-binary) constituents, with internal nodes containing (1) a nuclearity label, defining the importance of that subtree (rooted at the internal node) in the local context and (2) a relation label, defining the type of semantic connection between the two subtrees (e.g., elaboration, background). and lascarides, 2003) of a discourse being ill-formed due to the invalid anaphoric link. the leaf edus are as follows: [max had a great evening last night.] 1 [he had a great meal.] 2 [he ate salmon.] 3 [he devoured lots of cheese.] 4 [he then won a dancing competition.] 5 [it was a beautiful pink] 6 previous research has shown that the use of rststyle discourse parsing as a system component can enhance important tasks, such as sentiment analysis, summarization and text categorization @xcite @xcite @xcite . and more recently, it has been found that rst discourse structures can complement learned contextual embeddings (e.g., bert @xcite ), in tasks where linguistic information on complete documents is critical, such as argumentation analysis @xcite . in this work, we present preliminary results of investigating the benefits of coreference resolution features for rst discourse parsing. from the theoretical perspective, it has long been established @xcite that discourse structure can impose constraints on mention antecedent distributions, with these constraints being derived from the role of each discourse unit (sen-tence or edu) with respect to the global discourse. the veins theory @xcite is the most known formalization of anaphoric constraints with respect to rst tree structures, involving assigning to each edu a subset of preceding edus defined by the nuclearity attributes of the edu's parent nodes in the document's discourse tree (see appendix a for the exact definition). these constrains act as a domain of referential accessibility where the antecedents must reside, for otherwise the discourse would be considered incoherent. as an example of this phenomenon, consider the discourse structure in figure 1 . in principle, a reader could apply commonsense knowledge to resolve the pronoun it in the last sentence to salmon in the third sentence, any proficient english speaker would call such a discourse ill-formed and incoherent, due to the fact that it breaks the discourse-imposed antecedent scope. in general, anaphora can only be resolved with respect to the most salient (sentence 1 in figure 1 ) units of the preceding discourse @xcite . for our purposes, this means that having access to a document's coreference structure might be beneficial to the task of predicting the discourse structure, since the coreference structure can constrain the discourse parser's solution space. however, as shown in a corpus study by @xcite , the antecedent boundaries defined by veins theory are often too restrictive, suggesting that while discourse structures can be useful for predicting coreference structures and vice versa, these mutual constrains must be defined softly, at least in the context of rst theory. to explore these ideas computationally with respect to modern neural models, we investigate the utility of automatically extracted coreference features and discourse-coreference shared representations in the context and for the benefit of neural rst discourse parsing. our strong baseline spanbert-nocoref utilizes spanbert @xcite as in the current sota coreference resolver, without utilizing any direct coreference information. next, our spanbert-coreffeats considers the output of coreference resolver as per @xcite , letting us test the benefit of predicted and so possibly noisy coreference features. finally, our more sophisticated spanbert-multitask model learns discourse parsing together with coreference resolution in the neural multitask learning fashion, sharing the spanbert contextual word encoder for both models. modelwise, the most common approach to infer discourse trees is the linear bottom-up shiftreduce method, adopted from syntactic parsing. @xcite uses hand-crafted features and the shift-reduce method predicted by two separate support-vector-machines (svms) for structureand nuclearity-prediction and relation-estimation. the neural model by @xcite uses a similar topology, but instead relies entirely on lstms for automatic feature extraction and on a single multilayer-perceptron (mlp) for classifying all possible actions. top-down approaches to discourse parsing are also quite promising, with recent work of @xcite applying elmo @xcite for computing span representations and achieving the new absolute sota performance, reporting however the scores of an ensemble of five independent runs of their proposed model instead of single-model results. in this work we follow the shift-reduce strategy and apply spanbert-base @xcite ), which we introduce below, for encoding the document contents. the field of coreference resolution has recently been dominated by deep learning models. the current sota model by @xcite is built upon the neural coreference resolver of @xcite by incorporating spanbert language model, which modifies the commonly used bert @xcite architecture with a novel span masking pretraining objective. in our work, we reimplemented their coreference resolver in pytorch @xcite . our code for both models is available 1 . the shift delays aggregations of sub-trees at the beginning of the document by popping the top edu q 1 on the queue and pushing it onto the stack. the reduce-x aggregates the top subtrees (s 1 , s 2 ) on the stack into a single subtree (s 1-2 ). each reduce action further defines a nuclearity assignment x @xmath0, ns, sn} to the nodes covered by s 1-2 and a relation x @xmath1, contrast, ...} holding between them. ## proposed models we now describe the three proposed discourse parsing models which differ in the levels of coupling with the coreference model. see figure 2 for the visual comparison. spanbert-nocoref: in addition to the organizational features, our baseline system utilizes only the output spanbert-contextualized word embeddings. to predict each stack-queue action, a full document is passed through spanbert in a non-overlapping sliding window fashion, as per @xcite , so that the context of full document can be considered for each parsing action to account for possible context-sensitivity of discourse structures @xcite . the node representation v q 1 for the first queue element is computed as the mean of the first and the last word embedding of the edu that this queue element represents. v s 1 and v s 2 are computed as the means of the first and the last word embeddings of the nuclear edu of s 1 and s 2 , as each non-leaf node in an rst structure encodes a relation between nuclear edus of its children @xcite . spanbert-coreffeats: with this architecture variant, we attempt to assess the benefit of coreference features generated by the coreference resolver for rst parsing. given a document with n words, the coreference features will be used to update the initial (not contextualized) spanbert word embeddings w 1:n , which will later be passed to spanbert. specifically, for a given document we apply the pre-trained coreference parser of @xcite to extract the document's coreference clusters c 1 , c 2 , ..., each of which are equivalence classes representing different mentions of the same entity. afterwards, we compute the vector representation c i for each cluster c i by performing attention-based averaging over word-vectors corresponding to mentions in that cluster: finally, the embeddings w k are passed to span-bert for contextualization, and the node rep- spanbert-multitask: learns discourse parsing and coreference resolution in a multitask learning regime, weight-sharing the spanbert encoder module. the coreference resolver training step proceeds in the same fashion as in @xcite . for updating the discourse parsing model, we use the pre-computed coreference clusters c i obtained from the pretrained coreference model, as running it at every training step was prohibitively time-consuming. using the contextualized span-bert word embeddings s 1:n for all words in the document, we check these coreference clusters for overlaps: considering a pair of spans s 1 , s 2 , if a cluster c i has entity mentions in the spans of both stack elements s 1 and s 2 , so that if there are mentions m j , m @xmath2, we update the span representation v s 1 (computed as in spanbert-nocoref) with the attention weighted sum of mentions m @xmath3, so that the span representation for s 1 can incorporate more relevant context from s 2 . the representation for v s 2 is computed similarly using mentions m @xmath4, and the analogous computation is performed for s 1 -q 1 pair. for learning both tasks at the same time, we utilize the approach similar to @xcite , where gradient updates are performed separately for each task and the probability of sampling a task is proportional to the relative size of each task's dataset. the initial shared spanbert encoder weights are set from the pretrained coreference resolver checkpoint. incorrect nodes are colored in purple. the edus are: [finnair and sas said] 6 [they plan to swap stakes in each other.] 7 [neither discussed details] 8 [pending board meetings next month.] 9 . ## experimental settings and results all models were trained on the rst-dt @xcite and evaluated with rst-parseval procedure @xcite , with the coreference component of spanbert-multitask being trained on full ontonotes 5.0 corpus @xcite . the details of the training procedure such as hyperparameter assignment are outlined in the appendix b. the test results are presented in in an attempt to shed some light on the results, we compare the predicted and gold subtree from one of the documents in our development set on figure 3 . the trees were analyzed using the rst tree visualization tool by @xcite . according to veins theory, the pronoun [neither] in edu 8 is a mention that should have access to its mentions ([finnair and sas] or [they]) in preceding edus. however, accoding to the discourse structure predicted by spanbert-nocoref, the vein for node (8) does not contain the edus ( 6 ) and ( 7 ) (and in fact any of its preceding edus), so that [neither] cannot be linked to any of its preceding mentions. on the other hand, according to the gold discourse structure, edu 8 has edu 7 on its vein, meaning that this anaphora can be resolved. this means that if one had access to gold coreference structure and applied veins theory strictly, the substructure produced by spanbert-nocoref would not be permitted. ## conclusions and future work we empirically compare different levels of coupling between a shift-reduce neural discourse parser and a neural coreference resolver. remarkably, our baseline delivers sota performance on rst-dt, but does not seem to benefit from coreference features. for future work, we plan to experiment with (1) alternative discourse parsing architectures and approaches for neural multitasking, along with more powerful coreference models (2) alternative ways of augmenting a neural discourse parser with coreference information and other tasks like summarization (3) improving the coreference resolution performance by leveraging information provided by a discourse parser.