Generalized Shortest-Paths Encoders for AMR -to-Text Generation (2020)

Abstract: For text generation from semantic graphs, past neural models encoded input structure via gated convolutions along graph edges. Although these operations provide local context, the distance messages can travel is bounded by the number of encoder propagation steps. We adopt recent efforts of applying Transformer self-attention to graphs to allow global feature propagation. Instead of feeding shortest paths to the vertex self-attention module, we train a model to learn them using generalized shortest-paths algorithms. This approach widens the receptive field of a graph encoder by exposing it to all possible graph paths. We explore how this path diversity affects performance across levels of AMR connectivity, demonstrating gains on AMRs of higher reentrancy counts and diameters. Analysis of generated sentences also supports high semantic coherence of our models for reentrant AMRs. Our best model achieves a 1.4 BLEU and 1.8 chrF++ margin over a baseline that encodes only pairwise-unique shortest paths.

## related work initial work in amr-to-text generation relied on grammar-based approaches. @xcite used tree transducers to map between spanning tree representations of amrs and strings. these transducers generalize finite-state transducers (fsts) to recursively rewrite trees as strings. @xcite bypassed tree extraction and learned a synchronous node replacement grammar (snrg) to align graph and string fragments directly. though such approaches were well-founded, the grammars suffered from data sparsity due to the structural complexity of graphs relative to corpus size. amid the dominance of attentive recurrent models, @xcite linearized amrs and trained sequence transduction models in both the parsing and generation directions. they applied a paired training procedure such that the generator could be trained on synthetic data produced by the parser (and vice versa). gated gnns @xcite soon replaced the sequential encoders, reducing the need to compensate for noisy linearizations with synthetic data. @xcite evaluated the use of gnns over simpler sequential and tree encoders, showing that encoding reentrancies boosts generation performance. they further found that stacking a recurrent layer above a gnn is superior to previous gated gnns, verifying the importance of modeling amr sequential dependencies. recently, transformer self-attention has been adapted to encode amrs for generation tasks. as previously noted, the worst-case communication cost between vertices is constant in sans as opposed to linear in gnns. yet treating the input as a clique has the side-effect of weakening a model's grasp of amr structure and long-range paths. we build on the relation encoder developed by @xcite to bias self-attention on all paths in an amr. our methods mediate between the efficiency gains of san-based graph encoders and the need to capture fine-grained amr topology. progress in past models attests to the utility of high resolution path encoding for the language modeling objective. new lines of work have modified aspects of amr-to-text generation besides the graph encoder. @xcite explore autoencoder-style loss terms that encourage decoder states to reconstruct the graph. ribeiro et al. (2020) avoid graph encoding entirely, instead applying pretrained language models on linearized amrs. these additions of input reconstruction and pretrained embedding are orthogonal to the graph encoder changes we introduce. ## graph transformer here we establish a base adaptation of the transformer to a graph-to-sequence model. due to discrepancies between sequences and graphs, we highlight efforts to make the encoder san structure-aware. for sequence transduction let @xmath0. . . x n } denote source token embeddings and @xmath1. . . y m } the targets. for source token x j and context x i , the encoder san by @xcite applied dot-product attention between query vector q @xmath2 ## generalized shortest-paths encoders we now discuss how the relation encoder incorporates all possible amr paths using generalized shortestpaths algorithms. the goal is to learn comprehensive pairwise path embeddings that inform the graph encoder san. this approach relegates the choice of 'best' paths between vertices to the model instead of merely including them as model input. ## experiments we evaluate two generalized shortest-paths encoders, using the model by @xcite as a baseline. all components besides the relation encoder are held constant to test whether feeding the graph encoder a wider scope of path embeddings improves downstream language modeling. experiments are run on the ldc2017t10 dataset that consists of 36,521 training, 1,368 development, and 1,371 test amrsentence pairs. results are reported in terms of bleu @xcite and chrf++ @xcite for direct comparison to the baseline. ## conclusion we presented two relation encoders based on generalized shortest-paths algorithms for graph encoders. due to their expansive graph path encoding, we proposed that the encoders' more global context would especially benefit complexly structured amrs for text generation. experiments revealed that their performance gains are most consistent on higher-diameter amrs, verifying their efficacy on encoding longdistance vertex relations. our adjacency matrix multiplication relation encoder scores above the baseline shortest-paths model in terms of both bleu and chrf++. furthermore, sentence-level inspection showed that both our model variants produce more semantically sound output. encouraging the model to learn encoder-specific parameters (e.g., maximum path length) may improve its adaptivity to variations in amr structure and remedy its sensitivity to low amr reentrancy. we leave this experimentation for future work.