Graph-to-Text Generation with Dynamic Structure Pruning (2022)

Abstract: Most graph-to-text works are built on the encoder-decoder framework with cross-attention mechanism. Recent studies have shown that explicitly modeling the input graph structure can significantly improve the performance. However, the vanilla structural encoder cannot capture all specialized information in a single forward pass for all decoding steps, resulting in inaccurate semantic representations. Meanwhile, the input graph is flatted as an unordered sequence in the cross attention, ignoring the original graph structure. As a result, the obtained input graph context vector in the decoder may be flawed. To address these issues, we propose a Structure-Aware Cross-Attention (SACA) mechanism to re-encode the input graph representation conditioning on the newly generated context at each decoding step in a structure aware manner. We further adapt SACA and introduce its variant Dynamic Graph Pruning (DGP) mechanism to dynamically drop irrelevant nodes in the decoding process. We achieve new state-of-the-art results on two graph-to-text datasets, LDC2020T02 and ENT-DESC, with only minor increase on computational cost.

## introduction data-to-text task aims to generate a natural language description from structural or semi-structural data, such as tables @xcite , abstract meaning representation (amr) graphs @xcite , and knowledge graphs (kg) @xcite . it helps people get the key points of the input data and makes the stored information accessible to a broader audience of endusers. there have been several practical application scenarios in this field, such as biography generation @xcite , basketball news generation @xcite , and advertising text generation @xcite . this paper focuses on generation from graph structures in amr and kg, referred to as graph-to-text. in recent years, encoder-decoder with the crossattention mechanism has been the de facto framework for graph-to-text tasks (shown in figure 1(a) ). given an input graph, the encoder first computes vector representations for the graph nodes. on the decoding side, input graph (ig) context vector is obtained via cross-attention based on the partially generated text (gt) at each time step, then the next target token is finally predicted. unlike conventional text-to-text tasks, the structural nature of the input graph makes it unsuitable to naively apply sequential encoder-decoder architecture to the graph-to-text task. to alleviate this issue, recent studies @xcite @xcite proposed to utilize the graph encoder to capture the input graph structure. these works have demonstrated that explicitly modeling the graph structure can bring benefits to the model performance. although equipped with the structure-aware modeling, it is still hard for the encoder to capture all specialized information for graph-to-text generation. it is evidenced by recent studies @xcite that a vanilla structural encoder cannot capture the accurate semantic representation of the input structural data effectively. auxiliary supervision has been shown to be helpful, but effective auxiliary tasks are not easy to design and may not generalize well to different datasets. we suspect that it is challenging for the encoder to encode all relevant information into node representations in a single forward pass for all the decoding steps, especially if the input graph structure is complex. besides the encoder side, few works have focused on the decoder side for graph-to-text tasks. considering the ordinary cross-attention mechanism, the representations of input data obtained from the encoder are still treated as an unordered node representation sequence. we conjuncture that this plain cross-attention does not take full advantage of the input graph structure and therefore may harm the model performance. current models with graph encoder and crossattention may yield inaccurate input graph context representation due to the deficiency on both encoder and decoder as we discussed before. to tackle the above problems and avoid introducing auxiliary tasks, we propose a novel structure-aware cross-attention (saca) mechanism. apart from the plain cross-attention, our saca re-encodes the input graph conditioning on the newly generated context in a structure-aware fashion. other than a single forward pass, specialized representations from the source side are built adaptively at each decoding step, which makes the decoder easily exploit relevant-only information for prediction. more specifically, as shown in figure 1(b), we construct a joint graph, in which we explicitly treat the generated text context vector as an additional node and connect it with nodes in the input graph at each decoding step. we implement saca using the relational graph attention network (rgat, @xcite . furthermore, we stack multiple layers of sacas to perform deep interactions between the generated text context vector and input node representations. finally, we fetch the node representation corresponding to the newly added node as the structure-enhanced input graph context to predict the target token. in practice, we notice that some nodes become ir-relevant and uninformative as the decoding goes on. these nodes are distracting and can disturb the generation process. intuitively, the decoder should dynamically discard the unrelated parts of the graph at different decoding steps. in other words, the joint graph structure should be dynamically adjusted. to this end, we adapt saca and propose its variant dynamic graph pruning (dgp) mechanism (shown in figure 1(c) ). dgp prunes the structure of the joint graph via the gate mechanism to achieve sparse connections between the nodes based on the generated text context. we conduct experiments on two graph-to-text datasets, ldc2020t02 foot_0 and ent-desc @xcite , to verify the effectiveness of the proposed approach. empirical results show that our proposed methods achieve new state-of-the-art results on the two datasets. further experiments indicate that saca and dgp do not reduce the diversity of the generated text and can better handle complex graphs. meanwhile, additional investigation reveals that saca and dgp only bring minor increase on the model size and inference time. ## related works graph-to-text is a challenging task which aims at generating a descriptive text from the structured knowledge, such knowledge graph (kg), and abstract meaning representation (amr) graphs. it is helpful for interpretability of kgs in general @xcite and knowledge-based question answering @xcite @xcite . in recent years, most graph-to-text methods have been built based on the encoder-decoder architecture. this kind of method usually consists of a structural encoder and a decoder. the structural encoder aims to model the structure information into the representation of the input graph. @xcite first propose the graph recurrent networks (grns) to encode the amr node directly. and then, some works @xcite introduce the graph neural networks (gnns) as the structural encoder, which updates the representations of nodes based on their immediate neighbors. to integrate both local and non-local features and learn a better structural representation of a graph, @xcite introduce the dense connection, allowing deeper gcns. unlike the local information aggregation scheme, @xcite ; @xcite propose the graph transformer that uses explicit relation encoding and allows direct communication between two distant nodes. a recently proposed neural abstractive multi-document summarization (mds) model, graph-summ @xcite , also considers the input graph structure during decoding. the biggest difference between graphsum and our proposed saca is that the former only introduces one graph attention layer in each decoder layer. saca, on the other hand, injects graph structure into decoding by re-encoding the input graph. specifically, it re-computes the input graph representation by conditioning it on the newly generated text at each decoding step. recent approaches try to apply the pre-trained language models (plms) @xcite into the graph-to-text generation. particularly, @xcite propose to utilize the adapter method @xcite to encode graph structure into plms and only train graph structure-aware adapter parameters. in this way, they avoid catastrophic forgetting while maintaining the topological structure of the graph. ## approach we expect that developing graph-to-text generation should benefit from the recent advance on pre-trained language models (plms) @xcite . to explicitly encode the input graph structure into plms while alleviating the catastrophic forgetting problem, we consider sa- @xcite as our baseline model. sa-rgcn is an adapter method to encode graph structure into plms. the overall illustration of our model architecture is shown in figure 2(a) . in this section, we first introduce how to represent the input graph and the architecture of our baseline sa-rgcn. then, we depict our proposed structure-aware cross-attention (saca) in details. lastly, we adapt saca and propose its variant dynamic graph pruning (dgp). ## conclusions in this work, we make two main contributions. first, we propose structure-aware cross-attention (saca) to make decoder easily exploit relevant-only information for prediction. apart from the plain cross-attention, saca re-encodes the input graph conditioning on the newly generated context while explicitly considering the input graph structure. the second one is that we adapt saca and propose its variant dynamic graph pruning (dgp) mechanism. in detail, the dgp dynamically prunes the structure of the joint graph at different decoding steps according to the generated text. experimental results conducted on two graph-to-text datasets, ldc2020t02 and ent-desc, show the effectiveness of our method. the empirical and analysis results on both datasets show that the proposed methods can improve the model's performance on complex graphs while only bringing minor increase on the model size and inference time.