Heterogeneous Graph Transformer for Graph-to-Sequence Learning (2020)

Abstract: The graph-to-sequence (Graph2Seq) learning aims to transduce graph-structured representations to word sequences for text generation. Recent studies propose various models to encode graph structure. However, most previous works ignore the indirect relations between distance nodes, or treat indirect relations and direct relations in the same way. In this paper, we propose the Heterogeneous Graph Transformer to independently model the different relations in the individual subgraphs of the original graph, including direct relations, indirect relations and multiple possible relations between nodes. Experimental results show that our model strongly outperforms the state of the art on all four standard benchmarks of AMR-to-text generation and syntax-based neural machine translation.

## introduction graph-to-sequence (graph2seq) learning has attracted lots of attention in recent years. many natural language process (nlp) problems involve learning from not only sequential data but also more complex structured data, such as semantic graphs. for example, amr-to-text generation is a task of generating text from abstract meaning representation (amr) graphs, where nodes denote semantic concepts and edges refer to relations between concepts (see figure 1 (a) ). in addition, it has been shown that even if the sequential input can be augmented by additional structural information, bringing benefits for some tasks, such as semantic parsing @xcite and machine translation @xcite . therefore, @xcite introduced the graph2seq problems which aim to generate target sequence from graph-structured data. the main challenge for graph2seq learning is to build a powerful encoder which is able to cap-ture the inherent structure in the given graph and learn good representations for generating the target text. early work relies on statistical methods or sequence-to-sequence (seq2seq) models where input graphs are linearized @xcite @xcite . recent studies propose various models based on graph neural network (gnn) to encode graphs @xcite @xcite @xcite . however, these approaches only consider the relations between directly connected nodes, ignore the indirect relations between distance nodes. inspired by the success of transformer @xcite which can learn the dependencies between all tokens without regard to their distance, the current state-of-theart graph2seq models @xcite are based on transformer and learn the relations between all nodes no matter they are connected or not. these approaches use shortest relation path between nodes to encode semantic relationships. however, they ignore the information of nodes in the relation path and encode the direct relations and indirect relations without distinction. it may disturb the information propagation process when aggregate information from direct neighbors. to solve the issues above, we propose the heterogeneous graph transformer (hetgt) to encode the graph, which independently model the different relations in the individual subgraphs of the original graph. hetgt is adapted from transformer and it also employs an encoder-decoder architecture. following @xcite , we first transform the input into its corresponding levi graph which is a heterogeneous graph (contains different types of edges). then we split the transformed graph into multiple subgraphs according to its heterogeneity, which corresponds to different representation subspaces of the graph. for updating the node representations, attention mechanisms are used for independently aggregating information in different subgraphs. finally, the representations of each node obtained in different subgraphs are concatenated together and a parameterized linear transformation is applied. in this way, hetgt could adaptively model the various relations in the graph independently, avoiding the information loss caused by mixing all of them. moreover, we introduce the jump connection in our model, which significantly improves the model performance. we evaluate our model on four benchmark datasets of two graph2seq tasks: the amr-totext generation and the syntax-based neural machine translation (nmt). in terms of various evaluation metrics, our model strongly outperforms the state-of-the-art (sota) results on both two tasks. particularly, in amr-to-text generation, our model improves the bleu scores of the sota by about 2.2 points and 2.3 points on two benchmark datasets (ldc2015e86 and ldc2017t10). in syntax-based nmt, our model surpasses the sota by about 4.1 and 2.2 bleu scores for english-german and english-czech on news commentary v11 datasets from the wmt16 translation task. our contributions can be summarized as follows: ## neural graph-to-sequence model in this section, we will first begin with a brief review of the transformer which is the basis of our model. then we will introduce the graph transformation process. finally, we will detail the whole architecture of hetgt. ## related work early researches for graph2seq learning tasks are based on statistical methods and neural seq2seq model. @xcite propose an nlg approach built on top of tree conditional random fields to use the tree-structured meaning representation. song et al. (2017) use synchronous node replacement grammar to generate text. konstas et al. (2017) linearize the input graph and feed it to the seq2seq model for text-to-amr parsing and amr-to-text generation. however, linearizing amr graphs into sequences may incurs in loss of information. recent efforts consider to capture the structural information in the encoder. beck et al. (2018) employ gated graph neural networks (ggnn) as the encoder and song et al. (2018) propose the graph-state lstm to incorporate the graph structure. their works belong to the family of recurrent neural network (rnn). in addition, there are some works are build upon the gnn. damonte and cohen (2019) propose stacking encoders including lstm and gcn. guo et al. (2019) introduce the densely connected gcn to encode richer local and non-local information for better graph representation. recent studies also extend transformer to encode structure information. @xcite propose the relation-aware self-attention which learns explicit embeddings for pair-wise relationships between input elements. @xcite and cai and lam (2020) both extend the relation-aware selfattention to generate text from amr graph. our model is also based on transformer. however, we do not employ the relative position encoding to incorporate structural information. instead, we directly mask the non-neighbor nodes attention when updating each nodes representation. moreover, we introduce the heterogeneous information and jump connection to help model learn a better graph representation, bringing substantial gains in the model performance. ## conclusion in this paper, we propose the heterogeneous graph transformer (hetgt) for graph2seq learning. our proposed heterogeneous mechanism can adaptively model the different representation subgraphs. experimental results show that hetgt strongly outperforms the state of the art performances on four benchmark datasets of amr-to-text generation and syntax-based neural machine translation tasks. there are two directions for future works. one is to investigate how the other graph models can benefit from our proposed heterogeneous mecha-nism. on the other hand, we would also like to investigate how to make use of our proposed model to solve sequence-to-sequence tasks.