Improving Neural Machine Translation with the A bstract M eaning R epresentation by Combining Graph and Sequence Transformers (2022)

Abstract: Previous studies have shown that the Abstract Meaning Representation (AMR) can improve Neural Machine Translation (NMT). However, there has been little work investigating incorporating AMR graphs into Transformer models. In this work, we propose a novel encoder-decoder architecture which augments the Transformer model with a Heterogeneous Graph Transformer (Yao et al., 2020) which encodes source sentence AMR graphs. Experimental results demonstrate the proposed model outperforms the Transformer model and previous non-Transformer based models on two different language pairs in both the high resource setting and low resource setting. Our source code, training corpus and released models are available at https://github.com/jlab-nlp/amr-nmt .

## introduction neural machine translation (nmt, @xcite has proven to be an effective approach, and is the dominant method for machine translation in recent years. however, state-of-the-art nmt methods sometimes repeat words, leave out important pieces of the translation, and hallucinate information not contained in the source, or in other words, fail to accurately capture the semantics of the source in some cases. to address this problem, researchers have explored incorporating syntactic and semantic information into nmt systems. while most of previous nmt studies incorporating extra information are focused on syntax-based nmt @xcite @xcite @xcite @xcite @xcite @xcite @xcite , there has recently been interest in incorporating semantic information into nmt. marcheggiani et al. (2018) shows that incorporating semantic role labeling (srl) information can alleviating the "argument switching" problem for nmt. @xcite shows that abstract meaning representation (amr, @xcite graphs can be helpful for nmt for the bi-lstm with attention. amr @xcite is a semantic formalism that encodes the meaning of a sentence as a rooted, directed graph. figure 1 shows an amr graph, in which the nodes (eg. end-01) represent the concepts, and edges (eg. agr0) represent the relations between concepts they connect. in prior work, @xcite examined the effect of amrs in different nmt models, proposing a method for incorporating amr into nmt. however, the method nguyen et al. (2021) proposed for incorporating amr into the transformer showed limited success, as their performance with the transformer with amr was less than their bi-lstm with amr. in this work, we re-examine methods for incorporating amr graphs into transformer models. the transformer @xcite architecture has been the state-of-the-art for nmt for several years. we propose to improve upon the transformer model by incorporating amr graphs with a graph transformer in a novel manner. in partic-ular, we observe the best performance gains when integrating the semantic information contained in the amr graphs into both the encoder and decoder modules of the transformer. while much research on transformers is for text, many researchers have also investigated transformer-like architectures for the encoding of graph structures. @xcite proposed the heterogeneous graph transformer which independently models the different relations in the individual subgraphs of the original graph, including direct relations, indirect relations and other possible relations between nodes. we improve the performance of the transformer by employing a vanilla transformer to encode and decode the source sentence and a heterogeneous graph transformer to encode an amr graph of the source sentence. we use a novel integration model to combine the graph representations ( §3) into the encoder and decoder. we show that our method improves upon the transformer, and improves upon the best previous method for incorporating amr graphs into nmt. experiments on the wmt16 english to german dataset and iwslt15 english to vietnamese show that incorporating amr into transformer models with proper encoding representation combination models can robustly improve the vanilla sequenceto-sequence transformer baseline and outperforms all previous approaches when incorporating amr in both low data setting and large data setting. in summary, our contributions are the following: ## background in this section, we review the original transformer architecture for sequences as well as the heterogeneous graph transformer, and introduce notation we will use in later sections. ## our amr-transformer model figure 2 shows the overview of our proposed model architecture. to encode and decode both source sentences and source amr graphs to target sentences, our model consists of two parallel stacked encoder and decoder layers, one for sequence encoding and decoding from the neural sequence to sequence model, and the other for graph encoding and decoding from the neural graph to sequence model. given the encoded sequence representation from the sequence encoder and the encoded graph representation from the graph encoder, the sequence to sequence decoder only receives the sequence representation while the graph to sequence decoder receives the combination of the sequence representation and the graph representation. the specific combination approaches are discussed in §3.2 and §3.3. finally, two decoder representations are concatenated and fed into the final linear layer to generate target sequence representation. in this way, the model can combine the advantage of the traditional sequence to sequence model which does translation based on source sentence encodings and the graph to sequence model which incorporates amr graphs into the translation. the combination of source sentence representation and the graph representation into the graph to sequence decoder can lead the graph to sequence decoder to decoding towards good translation quality since using only amr graphs representation can lead to poor translation quality compared to the vanilla sequence to sequence model using source sentences. ## related work several recent studies have investigated on how to incorporate semantic information into neural machine translation (nmt) models. @xcite studied the semantic role labeling (srl) information for nmt, which used graph convolutional network (gcn) layers to encode the predicate-argument structure from srl to improve the translation performance of the nmt model. in line with their work, @xcite was the first to exploit the amr information on nmt, which used a graph recurrent network to encode the amr graph and found that amr information can improve attention-based sequence to sequence neural translation model and they only evaluated their model on wmt16 english to german dataset. @xcite then examine the effect of amr in different sequence to sequence machine translation models, however, they found that their proposed single decoder transformer model to incorporate the amr information performs worse than the bi-lstm model with simple graph attention network. in this paper, we focus on improving the performance of incorporating amr information purely with transformers. our proposed method of integrating vanilla sequence transformer and heterogeneous graph transformer model with encoder integration and decoder integration provides a better way to incorporate the amr information into nmt. ## conclusion we combine the transformer and the heterogeneous graph transformer to incorporate semantics captured in amr graphs into neural machine translation. experimental results show that our proposed amr-transformer model robustly outperforms the vanilla sequence transformer baseline and previous non-transformer based models across two different language pairs in both high resource setting and low resource setting.