Structural Information Preserving for Graph-to-Text Generation (2020)

Abstract: The task of graph-to-text generation aims at producing sentences that preserve the meaning of input graphs. As a crucial defect, the current state-of-the-art models may mess up or even drop the core structural information of input graphs when generating outputs. We propose to tackle this problem by leveraging richer training signals that can guide our model for preserving input information. In particular, we introduce two types of autoencoding losses, each individually focusing on different aspects (a.k.a. views) of input graphs. The losses are then back-propagated to better calibrate our model via multi-task training. Experiments on two benchmarks for graph-to-text generation show the effectiveness of our approach over a state-of-the-art baseline.

## introduction many text generation tasks take graph structures as their inputs, such as abstract meaning representation (amr) @xcite , knowledge graph (kg) and database tables. for example, as shown in figure 1 (a), amr-to-text generation is to generate a sentence that preserves the meaning of an input amr graph, which is composed by a set of concepts (such as "boy" and "want-01") and their relations (such as ":arg0" and ":arg1"). similarly, as shown in figure 1 (b), kg-to-text generation is to produce a sentence representing a kg, which contains worldwide factoid information of entities (such as "australia" and "above the veil") and their relations (such as "followedby"). recent efforts on graph-to-text generation tasks mainly focus on how to effectively represent input graphs, so that an attention mechanism can better transfer input knowledge to the decoder when generating sentences. taking amr-to-text generation as an example, different graph neural networks (gnns) @xcite @xcite have been introduced to better represent input amrs than a sequence-to-sequence model @xcite , and later work @xcite @xcite showed that relationaware transformers can achieve even better results than gnns. these advances for encoding have largely pushed the state-of-the-art performance. existing models are optimized by maximizing the conditional word probabilities of a reference sentence, a common signal for training language models. as a result, these models can learn to produce fluent sentences, but some crucial input concepts and relations may be messed up or even dropped. taking the amr in figure 1 (a) as an example, a model may produce "the girl wants the boy to go", which conveys an opposite meaning to the amr graph. in particular, this can be very likely if "the girl wants" appears much more frequent than "the boy wants" in the training corpus. this is a very important issue, because of its wide existence across many neural graph-to-text generation models, hindering the usability of these models for real-world applications @xcite @xcite . a potential solution for this issue is improving the training signal to enhance preserving of structural information. however, little work has been done to explore this direction so far, probably because designing such signals is non-trivial. as a first step towards this goal, we propose to enrich the training signal with additional autoencoding losses @xcite . standard autoencoding for graph-to-sequence tasks requires reconstructing (parsing into) input graphs, while the parsing algorithm for one type of graphs (such as knowledge graphs) may not generalize to other graph types or may not even exist. to make our approach general across different types of graphs, we propose to reconstruct different views of each input graph (rather than the original graph), where each view highlights one aspect of the graph and is easy to produce. then through multi-task learning, the autoencoding losses of all views are back-propagated to the whole model so that the model can better follow the input semantic constraints. specifically, we break each input graph into a set of triples to form our first view, where each triple (such as "want-01 :arg0 boy" in figure 1(a) ) contains a pair of entities and their relation. as the next step, the alignments between graph nodes and target words are generated to ground this view into the target sentence for reconstruction. our second view is the linearization of each input graph produced by depth-first graph traversal, and this view is reconstructed token-by-token from the last decoder state. overall the first view highlights the local information of each triple relation, the second view focuses on the global semantic information of the entire graph. experiments on amr-to-text generation and webnlg @xcite show that our graph-based multi-view autoencoding loss improves the performance of a state-of-the-art baseline by more than 2 bleu points without introducing any parameter during decoding. besides, human studies show that our approach is indeed beneficial for preserving more concepts and relations from input graphs. ## related work previous work for neural graph-to-text generation @xcite @xcite @xcite @xcite @xcite @xcite @xcite mainly studied how to effectively represent input graphs, and all these models are trained only with the standard language modeling loss. as the most similar one to our work, @xcite proposed an encoder-decoder-reconstructor model for machine translation, which is trained not only to translate each source sentence into its target reference, but also to translate the target reference back into the source text (reconstruction). @xcite extended the reconstruction loss of @xcite on table-to-text generation, where a table contains multiple records that fit into several fields.we study a more challenging topic on how to reconstruct a complex graph structure rather than a sentence or a table, and we propose two general and effective methods that reconstruct different complementary views of each input graph. besides, we propose methods to breakdown the whole (graph, sentence) pair into smaller pieces of (edge, word) pairs with alignments, before training our model to reconstruct each edge given the corresponding word. on the other hand, neither of the previous efforts tried to leverage this valuable information. our work is remotely related to the previous efforts on string-to-tree neural machine translation (nmt) @xcite @xcite , which aims at generating target sentences with their syntactic trees. one major difference is that their goal is producing grammatical outputs, while ours is preserving input structures. besides, our multi-view reconstruction framework is a detachable component on top of the decoder states for training, so no extra error propagation (for structure prediction) can be introduced. conversely, their models generate trees together with target sentences, thus extra efforts @xcite are introduced to alleviate error propagation. finally, there exist transition-based algorithms @xcite to convert tree parsing into the prediction of transition actions, while we study reconstructing graphs, where there is no common parsing algorithm for all graph types. autoencoding loss by input reconstruction was mainly adopted on sequence labeling tasks, such as named entity recognition (ner) @xcite @xcite , simile detection @xcite and sentiment analysis @xcite . since input reconstruction is not intuitively related to these tasks, the autoencoding loss only serves as more training signals. different from these efforts, we leverage autoencoding loss as a means to preserve input knowledge. besides, we study reconstructing complex graphs, proposing a general multi-view approach for this goal. ## base: structure-aware transformer formally, an input for graph-to-text generation can be represented as @xmath0, ei, where v is the set of graph nodes and e corresponds to all graph edges. each edge e 2 e is a triple (v i , l, v j ), showing labelled relation between two connected nodes v i and v j . given a graph, we choose a recent relation-aware transformer model @xcite as our baseline to generate the ground-truth sentence @xmath1, . . . , y n ) that contain the same meaning as the input graph. it exhibits the state-ofthe-art performance for amr-to-text generation. ## conclusion we proposed reconstructing input graphs as autoencoding processes to encourage preserving the input semantic information for graph-to-text generation. in particular, the auxiliary losses for recovering two complementary views (triple relations and linearized graph) of input graphs are introduced, so that our model is trained to retain input structures for better generation. our training framework is general for different graph types. experiments on two benchmarks showed the effectiveness of our framework under both the automatic bleu metric and human judgements.