AMR -To-Text Generation with Graph Transformer (2020)

Abstract: Abstract meaning representation (AMR)-to-text generation is the challenging task of generating natural language texts from AMR graphs, where nodes represent concepts and edges denote relations. The current state-of-the-art methods use graph-to-sequence models; however, they still cannot significantly outperform the previous sequence-to-sequence models or statistical approaches. In this paper, we propose a novel graph-to-sequence model (Graph Transformer) to address this task. The model directly encodes the AMR graphs and learns the node representations. A pairwise interaction function is used for computing the semantic relations between the concepts. Moreover, attention mechanisms are used for aggregating the information from the incoming and outgoing neighbors, which help the model to capture the semantic information effectively. Our model outperforms the state-of-the-art neural approach by 1.5 BLEU points on LDC2015E86 and 4.8 BLEU points on LDC2017T10 and achieves new state-of-the-art performances.

## introduction abstract meaning representation (amr) is a semantic formalism that abstracts away from the syntactic realization of a sentence, and encodes its definition as a rooted, directed, and acyclic graph. in the graph, the nodes represent the concepts, and edges denote the relations between the concepts. the root of an amr binds its contents to a single traversable graph and serves as a rudimentary representation of the overall focus. the existence of co-references and control structures results in nodes with multiple incoming edges, called reentrancies, and causes an amr to possess a graph structure, instead of a tree structure. numerous natural language processing (nlp) tasks can benefit from using amr, such as machine translation @xcite , question answering @xcite , summarization @xcite , and event extraction @xcite . amr-to-text generation is the task of recovering a text representing the same definition as a given amr graph. because the function words and structures are abstracted away, the amr graph can correspond to multiple realizations. numerous important details are underspecified, including tense, number, and definiteness, which makes this task extremely challenging @xcite . figure 1 shows an example amr graph and its corresponding sentence. early works relied on grammar-based or statistical approaches @xcite @xcite . such approaches generally require alignments between the graph nodes and surface tokens, which are automatically generated and can lead to error accumulation. in recent research, the graphs are first transformed into linear sequences, and then the text is generated from the inputs @xcite . such a method may lose information from the graph structure. the current state-of-theart neural methods are graph-to-sequence models and hybrid variants @xcite @xcite . these methods use a graph state long short-term memory (lstm) network, gated graph neural network (ggnn), or graph convolution network (gcn) to encode amr graphs directly, and they can explicitly utilize the information provided by the graph structure. however, these graph encoders still cannot significantly outperform sequence encoders. the amr-to-text generation task can be regarded as a distinct translation task, and basing it on the concepts of off-the-shelf methods figure 1 : an example amr graph and its corresponding sentence. the graph is rooted by ''expect-01'', which means the amr is about the expecting. the node ''create'' is a reentrance and it plays two roles simultaneously (i.e., ''arg1'' of ''accelerate'' and ''arg1'' of ''slow-down''). for neural machine translation can be helpful. the transformer model @xcite is a stacked attention architecture and has shown its effectiveness in translation tasks; however, applying it to amr-to-text generation has a major problem: it can only deal with sequential inputs. to address these issues, we propose a novel graph network (graph transformer) for amrto-text generation. graph transformer is an adaptation of the transformer model, and it has a stacked attention-based encoder-decoder architecture. the encoder considers the amr graph as the input and learns the node representations from the node attributes by the aggregation of the neighborhood information. the global semantic information is captured by stacked graph attention layers, which allow a node to deal with the hidden states of the neighbor nodes and their corresponding relations. multiple stacked graph attention layers enable the nodes to utilize the information of those nodes that are not directly adjacent, allowing the global information to propagate. we consider that the amr graph is a directed graph in which the directions hold extremely important information. therefore, for encoding the information from the incoming and outgoing edges, we use two individual graph attentions in each layer. then we utilize a fusion layer to incorporate the information from the incoming and outgoing relations, followed by a feed-forward network. residual connections are used for connecting adjacent layers. the final node representations are formed by concatenating the two individual representations encoded by multiple layers. the decoder is similar to the original decoder in transformer, performing multi-head attentions and self-attentions over the representations of the nodes in the encoder and over the hidden states of the decoder, respectively. for the decoder stack, we adopt a copy mechanism to generate the texts, which can help copy low-frequency tokens, such as named entities and numbers. we perform experiments on two benchmark datasets (ldc2015e86 and ldc2017t10). our model significantly outperforms the prior methods and achieves a new state-of-the-art performance. without external data, our model improves the bleu scores of the state-of-the-art and a mostly recently proposed neural model (i.e., g-gcnseq @xcite ) by 1.5 points on ldc2015e86 and 4.8 points on ldc2017t10. when using the gigaword corpus as the additional training data, which is automatically labeled by a pre-trained amr parser, our model achieves a bleu score of 36.4 on ldc2015e86, which is the highest result on the dataset. the experimental result also shows that the improved structural representation encoding by our proposed graph encoder is most useful when the amount of training data is small. the variations in our model are evaluated to verify its robustness as well as the importance of the proposed modules. in addition, we study the performances of our model and baselines under different structures of the input graphs. our contributions can be summarized as follows: ## graph transformer the overall architecture of graph transformer is shown in figure 2 , with an example amr graph and its corresponding sentence. we begin by providing the formal definition of the amrto-text generation and the notations we use, and then reviewing the transformer model. then we introduce the graph encoder and sentence decoder used in our model. finally, we describe the training and decoding procedures. ## comparison to prior graph encoders in this section, we compare our proposed graph encoders with the existing ones presented in prior works. most models, including a gcn (damonte and cohen, 2019), ggnn @xcite , and graphlstm @xcite , use a nonpairwise interaction function to represent the information to be aggregated from the neighborhoods. specifically, they ignore the receiver node (i.e., the node to be updated), operating only on the sender node (i.e., the neighbor node) and the edge attribute @xcite . they add a self-loop edge for each node so that its own information can be considered. in our model, we compute the pairwise interactions using eq. ( 5 ); hence, no self-loop edge is required. in our model, the graph attention mechanism is similar to gat @xcite . the main differences are that gat is designed for undirected graphs and neither directions nor labels of edges are considered. we propose using two distinct representations (i.e., head representation and tail representation) for each node and utilizing graph attentions on the incoming and outgoing relations. accordingly, the model can consider the differences in the incoming and outgoing relations, and the results presented in the next section verify the effectiveness of this proposed modification. in addition, gat adopts additive attention and uses averages of the outputs of the multi-head attention in the final layer. in our model, we use a scaled dot-product attention for all the attention layers. ## conclusion in this study, we present a novel graph network (graph transformer) for amr-to-text generation. our model is solely based on the attention mechanism. our proposed graph attentions over the neighbor nodes, and the corresponding edges are used for learning the representations of the nodes and capturing global information. the experimental results shows that our model significantly outperforms the prior neural models and achieves a new state-of-the-art performance on benchmark datasets. in future work, we will incorporate bert embeddings and multi-task learnings to improve the performance further. we will also apply graph transformer to other related text generation tasks like mrs-to-text generation, data-to-text generation, and image captioning.