Incorporating Syntax and Semantics in Coreference Resolution with Heterogeneous Graph Attention Network (2021)

Abstract: External syntactic and semantic information has been largely ignored by existing neural coreference resolution models. In this paper, we present a heterogeneous graph-based model to incorporate syntactic and semantic structures of sentences. The proposed graph contains a syntactic sub-graph where tokens are connected based on a dependency tree, and a semantic sub-graph that contains arguments and predicates as nodes and semantic role labels as edges. By applying a graph attention network, we can obtain syntactically and semantically augmented word representation, which can be integrated using an attentive integration layer and gating mechanism. Experiments on the OntoNotes 5.0 benchmark show the effectiveness of our proposed model.

## introduction coreference resolution is a core task in nlp, which aims to identify all mentions that refer to the same entity. coreference encodes rich semantic information which has been successfully applied to improve many downstream nlp tasks @xcite @xcite . impressive progress has been made in recent years since the introduction of the first end-to-end neural coreference resolution model @xcite by utilising contextualized embeddings from large pretrained language models @xcite @xcite such as elmo @xcite and bert @xcite . rich language knowledge encoded in these pretrained models has largely alleviated the need for syntactic and semantic features. however, such information has been shown to benefit bert based models on other tasks @xcite @xcite . therefore, we believe such information could also benefit the coreference resolution task. in this paper, we propose a neural coreference resolution model based on @xcite , which we extend by incorporating external syntactic and semantic information. for syntactic information, we use dependency trees to capture the long-term dependency exists among mentions. @xcite has successfully incorporated structural information into neural models, but their model still requires the design of complex hand-engineered features. in contrast, our model is more flexible, using a graph neural network to encode syntax in the form of dependency trees. for semantic information, we adopt semantic role labelling (srl) structures. srl labels capture who did what to whom and it is effective in providing document-level event description information, which allows us to better identify the relationship between event mentions. previous statistical coreference systems have successfully integrated such information @xcite , but their effectiveness has not been examined in neural models. moreover, inspired by recent progress made in document-level relation extraction @xcite , we encode both syntactic and semantic information in a heterogeneous graph. nodes of different granularity are connected based on the feature structures. node representations are updated iteratively through our defined message passing mechanism and incorporated into contextualized embeddings using an attentive integration module and gating mechanism. we conduct experiments on the ontonotes 5.0 @xcite benchmark, where the results show that our proposed model significantly outperforms a strong baseline. ## baseline model our model is based on the c2f-coref model @xcite which enumerates all text spans as potential mentions and prunes unlikely spans aggressively. for each mention i, 2 the model learns a distribution over its possible antecedents y(i): ## proposed model figure 2 shows the architecture of our proposed model, where the key components are presented in blue and orange backgrounds. other parts follow @xcite (see §2) except that we use spanbert @xcite as the document encoder and discard the higher-order span refinement module as suggested by @xcite . ## experiments dataset we evaluate our model on the english onotonotes 5.0 benchmark @xcite , which consists of 2802, 343 and 348 documents in the training, development and test data sets. ## related work graph neural networks (gnn) have long been used for integrating external features of graph structures into a range of nlp tasks, including semantic role labelling @xcite and machine translation @xcite . how-ever, the application of gnn on coreference resolution task is less explored. @xcite adopted dependency syntax to improve gendered pronoun resolution. however, they did not evaluate their model on larger datasets and identify whether syntax features are still useful for common coreference resolution. in this paper, we not only utilise syntax but also semantic features, and we show both of them contribute to significant improvement over a strong baseline on a large standard dataset. there are many gnn variants. graph convolutional network (gcn) @xcite is the most widely-used one and has been shown to benefit a number of nlp tasks. however, it lacks the ability of modeling different edge labels including directions and edge types. although relational graph convolutional network (rgcn) @xcite was proposed to tackle this problem, the way of representing edge information as label-wise parameters makes it suffer from over-parameteration problem even for small sized label vocabularies. in this work, we use a graph encoder improved based on graph attention network (gat) @xcite to better capture structural syntax and semantics, as gat is able to model different types of edges with few parameters. ## conclusion in this paper, we propose a heterogeneous-graph based model to enhance coreference resolution by effectively leveraging dependency tree structures and srl semantic features. particularly, nodes of different granularity in the graph propagate and aggregate information to and from neighbour nodes to obtain both syntactically and semantically augmented representation. moreover, an attentionbased mechanism is used to dynamically aggregate such augmented information. experiments on the ontonotes 5.0 benchmark confirm the effectiveness of our proposed model with significant improvement achieved against the strong baseline. future work will focus on applying other features, such as constituent parsing trees and wordnet.