Online Back-Parsing for AMR -to-Text Generation (2020)

Abstract: AMR-to-text generation aims to recover a text containing the same meaning as an input AMR graph. Current research develops increasingly powerful graph encoders to better represent AMR graphs, with decoders based on standard language modeling being used to generate outputs. We propose a decoder that back predicts projected AMR graphs on the target sentence during text generation. As the result, our outputs can better preserve the input meaning than standard decoders. Experiments on two AMR benchmarks show the superiority of our model over the previous state-of-the-art system based on graph Transformer.

## introduction abstract meaning representation (amr) @xcite is a semantic graph representation that abstracts meaning away from a sentence. figure 1 shows an amr graph, where the nodes, such as "possible-01" and "police", represent concepts, and the edges, such as "arg0" and "arg1", indicate relations between the concepts they connect. the task of amrto-text generation @xcite aims to produce fluent sentences that convey consistent meaning with input amr graphs. for example, taking the amr in figure 1 as input, a model can produce the sentence "the police could help the victim". amr-to-text generation has been shown useful for many applications such as machine translation @xcite and summarization @xcite @xcite . in addition, amr-to-text generation can be a good test bed for general graph-to-sequence problems @xcite . amr-to-text generation has attracted increasing research attention recently. previous work has focused on developing effective encoders for representing graphs. in particular, graph neural networks @xcite @xcite and richer graph representations @xcite @xcite have been shown to give better performances than rnn-based models @xcite on linearized graphs. subsequent work exploited graph transformer @xcite @xcite , achieving better performances by directly modeling the intercorrelations between distant node pairs with relation-aware global communication. despite the progress on the encoder side, the current stateof-the-art models use a rather standard decoder: it functions as a language model, where each word is generated given only the previous words. as a result, one limitation of such decoders is that they tend to produce fluent sentences that may not retain the meaning of input amrs. we investigate enhancing amr-to-text decoding by integrating online back-parsing, simultaneously predicting a projected amr graph on the target sentence while it is being constructed. this is largely inspired by work on back-translation @xcite , which shows that back predicting the source sentence given a target translation output can be useful for strengthening neural machine translation. we perform online back parsing, where the amr graph structure is constructed through the autoregressive sentence construction process, thereby saving the need for training a separate amr parser. by adding online back parsing to the decoder, structural information of the source graph can intuitively be better preserved in the decoder network. figure 2 visualizes our structure-integrated decoding model when taking the amr in figure 1 as input. in particular, at each decoding step, the model predicts the current word together with its corresponding amr node and outgoing edges to the previously generated words. the predicted word, amr node and edges are then integrated as the input for the next decoding step. in this way, the decoder can benefit from both more informative loss via multi-task training and richer features taken as decoding inputs. experiments on two amr benchmark datasets (ldc2015e86 and ldc2017t10 1 ) show that our model significantly outperforms a state-of-theart graph transformer baseline by 1.8 and 2.5 bleu points, respectively, demonstrating the advantage of structure-integrated decoding for amr-to-text generation. deep analysis and human evaluation also confirms the superiority of our model. our code is available at https://github. com/muyeby/amr-backparsing . ## baseline: graph transformer formally, the amr-to-text generation task takes an amr graph as input, which can be denoted as a directed acyclic graph @xmath0, e), where v denotes the set of nodes and e refers to the set of labeled edges. an edge can further be represented by a triple v i , r k , v j , showing that node v i and v j are connected by relation type r k . here @xmath1, ..., r], and r is the total number of relation types. the goal of amr-to-text generation is to generate a word sequence @xmath2, y 2 , . . . , y m ], which conveys the same meaning as g. we take a graph transformer model @xcite @xcite as our baseline. previous work has proposed several variations of graph-transformer. we take the model of @xcite , which gives the state-of-the-art performance. this approach exploits a graph transformer encoder for amr encoding and a standard transformer decoder for text generation. ## experiments we conduct experiments on two benchmark amrto-text generation datasets, including ldc2015e86 and ldc2017t10. these two datasets contain 16,833 and 36,521 training examples, respectively, and share a common set of 1,368 development and 1,371 test instances. ## related work early studies on amr-to-text generation rely on statistical methods. @xcite convert input amr graphs to trees by splitting re-entrances, before translating these trees into target sentences with a tree-to-string transducer; @xcite apply a phrase-based mt system on linearized amrs; @xcite design a synchronous node replacement grammar to parse input amrs while generating target sentences. these approaches show comparable or better results than early neural models @xcite . however, recent neural approaches @xcite @xcite @xcite have demonstrated the state-of-the-art performances thanks to the use of contextualized embeddings. related work on nmt studies back-translation loss @xcite by translating the target reference back into the source text (reconstruction), which can help retain more comprehensive input information. this is similar to our goal. @xcite extended the reconstruction loss of @xcite for tableto-text generation. we study a more challenging topic on how to retain the meaning of a complex graph structure rather than a sentence or a table. in addition, rather than reconstructing the input after the output is produced, we predict the input while the output is constructed, thereby allowing stronger information sharing. our work is also remotely related to previous work on string-to-tree neural machine translation (nmt) @xcite @xcite , which aims at generating target sentences together with their syntactic trees. one major difference is that their goal is producing grammatical outputs, while ours is preserving input structural information. ## conclusion we investigated back-parsing for amr-to-text generation by integrating the prediction of projected amrs into sentence decoding. the resulting model benefits from both richer loss and more structual features during decoding. experiments on two benchmarks show advantage of our model over a state-of-the-art baseline.