Variational Graph Autoencoding as Cheap Supervision for AMR Coreference Resolution (2022)

Abstract: Coreference resolution over semantic graphs like AMRs aims to group the graph nodes that represent the same entity. This is a crucial step for making document-level formal semantic representations. With annotated data on AMR coreference resolution, deep learning approaches have recently shown great potential for this task, yet they are usually data hunger and annotations are costly. We propose a general pretraining method using variational graph autoencoder (VGAE) for AMR coreference resolution, which can leverage any general AMR corpus and even automatically parsed AMR data. Experiments on benchmarks show that the pretraining approach achieves performance gains of up to 6% absolute F1 points. Moreover, our model significantly improves on the previous state-of-the-art model by up to 11% F1.

## introduction abstract meaning representation (amr) is a way to preserve the semantic meaning of a sentence in a graph @xcite . as shown in figure 1 , amrs are directed and acyclic graphs where the nodes and edges indicate concepts and their semantic relations. as a sentence-level semantic representation, amrs have been shown to be effective in many nlp tasks, including text summarization @xcite , information extraction @xcite @xcite , and machine translation @xcite . more recently, the nlp tasks that are beyond the single-sentence level @xcite @xcite are attracting rising attention, and thus representing multiple sentences with amr becomes important. to expand amrs to represent multiple * work done as an intern at tencent ai lab. † corresponding author. sentences, the task of amr coreference resolution @xcite has been proposed, aiming at recognizing the concepts from multiple amrs that represent the same entity. figure 1 illustrates the amr graphs of two consecutive sentences in a news article. given them as the input, an amr coreference resolver needs to group police and they (colored with blue), as well as shop and the implicit mention shop (dashed and colored with pink). unlike text-based coreference resolution, where dense textual information is available, amr coreference resolution deals with sparsely connected graphs and implicit graph nodes. more importantly, only a handful of annotated data (around 8k amrs) exists for amr coreference resolution. furthermore, annotating such coreference information and sentence amrs requires linguists, making the annotation very costly. both situations add extra difficulties to this task. early attempts on amr coreference resolution adopt rule-based methods. for instance, @xcite only consider the nodes that represent entities (e.g., police in figure 1 ), and they rely on string match to detect coreference. this method can cause errors, as concepts with the same surface string may not point to the same entity. it also fails to recognize any situations that involve a pronoun (e.g., police and they). anikina et al. (2020) build a pipeline system that uses a textual coreference resolution model @xcite and a text-to-amr aligner @xcite . though this system can theoretically resolve many situations, in fact, it suffers from severe error propagation @xcite . with the availability of recent human-annotated data (o'gorman et al., 2018) on amr coreference resolution, later work starts exploring data-driven models. fu et al. ( 2021 ) extend a standard text-based coreference model @xcite on amrs by replacing the lstm encoder with a graph neural network (gnn). they show a significant performance boost over previous rulebased methods, and their generated document-level amrs can help a downstream neural summarization system, demonstrating the potential of this task. however, the performance is still far from satisfactory, and they find that the main reason is the lack of annotated data. this calls for approaches that can leverage cheap and/or existing supervision signals to make further improvements. in this paper, we propose a model and a corresponding pretraining method based on variational graph autoencoder (vgae) (kipf and welling, 2016b). our model extends amrcoref @xcite , the current state-of-the-art model, by replacing the core gnn encoder with an improved vgae encoder. our model can leverage the reconstruction loss and variational restriction from the vgae module as additional supervision at no extra cost. since the loss by our vgae model can work on any amr graphs, we also study pretraining our model on the full amr bank 1 with gold or automatically parsed annotations. in this way, the training signal can be further enriched; thus, the data hunger issue can be alleviated. though there exist some work applying vaes and vgaes on concept knowledge graphs @xcite , corpus-level graphs @xcite ) and text @xcite , we are the first to study vgae on a graph-based formal semantic representation, to the best of our knowledge. experiments on the ms-amr benchmark (o' @xcite show that our model outperforms the previous state-of-the-art system by 11 absolute f1-score points. besides, we find that pretraining with a larger amr bank is helpful re- ## baseline: amrcoref we take the end-to-end amr coreference resolution model (amrcoref, @xcite ) as our baseline system. generally, it adapts a text-based end-to-end coreference model @xcite on amrs by clustering amr nodes instead of text spans. another major difference is that they also consider omitted amr nodes (e.g., the dashed node shop in figure 1 ), which are represented by their parent nodes and the corresponding relation (e.g., depart-01 and :arg1). as illustrated in figure 2 , amrcoref consists of four essential modules: input representation, graph encoding, node type identification, and antecedent prediction. ## case study to further understand the predicted results of our model, we compare our best performed model (vg-amrcoref+pretrain+bert) and the best baseline model (amrcoref+bert) with two case studies. figure 6 shows one example taken from the lp test set. given that the whole document is too long, we keep a part of the content and highlight the coreference cluster tokens with different colors to indicate ground truth, base model prediction and our model prediction. note that we illustrate both amrs and original sentences to show the context better, while the sentences were not directly participated in the training and testing. this content piece shows a dialogue between two characters: me and little prince. in the ground truth, the coreference cluster is indicating little prince, and this can be easily recognized from the token prince in s1 and the token he in s5 and s6. however, to find out if the token i in s3 belongs to this cluster, one needs to read from s1. because dialogues are going in turns, it is important to figure out which character said s3. here, the answer should be little prince (token i means himself) and should be included in the cluster. this could be challenging due to the deep understanding of the previous content and also the difficulty of long dependency. our model successfully recognized the coreference tokens in this situation. we illustrate another example from the ms-amr test set in figure 7 . as can be observed form the ground truth, the highlighted tokens are indicating the coreference cluster of the main character in this article, i. the base model predicts a wrong answer in s1 (who), and misses the correct token i in that sentence. while both models ignore the token i in s2 and s3, compare with the base model, s1: for the little prince asked me abruptly --as if seized by a grave doubt --"it is true , is n't it , that sheep eat little bushes ?" while automatic evaluation only shows the overall performance, our case studies provide some interesting observations. the base and our model tend to predict fewer coreference nodes than the ground truth, but our model can capture larger and more accurate coreference clusters than the base model. graph pretraining previous work shows that pretraining a model may bring better generalizability and performance gain, such as the pretrained language model, bert @xcite . there is limited research that focuses on pretraining graph neural networks. the work by @xcite proposes two methods to pretrain gnns in both individual node level and the entire graph level. though there are a few attempts to pretrain gnns in a similar way with bert, i.e., graph transformer (dwivedi and bresson, 2020), and knowledge graph pretraining @xcite , there is still limited study in other nlp tasks. our work fills this gap by taking advantage of knowledge learned from external data. ## conclusion this work proposed a new model (vg-amrcoref) that is capable of self-supervised training for multisentence amr coreference resolution. it applies vgaes to encode document-level amrs, significantly improving performance by up to 11% on the f1 score. we further proposed a simple but effective graph pretraining method using vgaes, which can simultaneously boost in in-domain and out-domain performances. analysis shows that potential boost performance may happen if more automatically parsed amr data is available. one future work will focus on applying larger scale silver amr datasets for pretraining to improve amr coreference resolution. another future direction is to investigate the generated document-level amrs on more downstream tasks, like question answering and dialogue understanding.