Lightweight, Dynamic Graph Convolutional Networks for AMR -to-Text Generation (2020)

Abstract: AMR-to-text generation is used to transduce Abstract Meaning Representation structures (AMR) into text. A key challenge in this task is to efficiently learn effective graph representations. Previously, Graph Convolution Networks (GCNs) were used to encode input AMRs, however, vanilla GCNs are not able to capture non-local information and additionally, they follow a local (first-order) information aggregation scheme. To account for these issues, larger and deeper GCN models are required to capture more complex interactions. In this paper, we introduce a dynamic fusion mechanism, proposing Lightweight Dynamic Graph Convolutional Networks (LDGCNs) that capture richer non-local interactions by synthesizing higher order information from the input graphs. We further develop two novel parameter saving strategies based on the group graph convolutions and weight tied convolutions to reduce memory usage and model complexity. With the help of these strategies, we are able to train a model with fewer parameters while maintaining the model capacity. Experiments demonstrate that LDGCNs outperform state-of-the-art models on two benchmark datasets for AMR-to-text generation with significantly fewer parameters.

## introduction graph structures play a pivotal role in nlp because they are able to capture particularly rich structural information. for example, figure 1 figure 1 : the concept (join-01) in vanilla gcns is that it only captures information from its immediate neighbors (first-order), while in ldgcns it can integrate information from neighbors of different order (e.g., second-order and third-order). in sans, the node collects information from all other nodes, while in structured sans it is aware of its connected nodes in the original graph. the realm of work on amr, we focus in this paper on the problem of amr-to-text generation, i.e. transducing amr graphs into text that conveys the information in the amr structure. a key challenge in this task is to efficiently learn useful representations of the amr graphs. early efforts @xcite neglect a significant part of the structural information in the input graph by linearizing it. recently, graph neural networks (gnns) have been explored to better encode structural information for this task @xcite @xcite . one type of such gnns is graph convolutional networks (gcns; @xcite . gcns follow a local information aggregation scheme, iteratively updating the representations of nodes based on their immediate (first-order) neighbors. intuitively, stacking more convolutional layers in gcns helps capture more complex interactions @xcite . however, prior efforts @xcite @xcite have shown that the locality property of existing gcns precludes efficient nonlocal information propagation. @xcite further proved that vanilla gcns are unable to capture feature differences among neighbors from different orders no matter how many layers are stacked. therefore, self-attention networks (sans; @xcite have been explored as an alternative to capture global dependencies. as shown in figure 1 (c), sans associate each node with other nodes such that we model interactions between any two nodes in the graph. still, this approach ignores the structure of the original graph. @xcite and @xcite propose structured sans that incorporate additional neural components to encode the structural information of the input graph. convolutional operations, however, are more computationally efficient than self-attention operations because the computation of attention weights scales quadratically while convolutions scale linearly with respect to the input length @xcite . therefore, it is worthwhile to explore the possibility of models based on graph convolutions. one potential approach that has been considered is to incorporate information from higher order neighbors, which helps to facilitate non-local information aggregation for node classification @xcite @xcite . however, simple concatenation of different order representations may not be able to model complex interactions in semantics for text generation @xcite . we propose to better integrate high-order information, by introducing a novel dynamic fusion mechanism and propose the lightweight, dynamic graph convolutional networks (ldgcns). as shown in figure 1 (b) , nodes in the ldgcn model are able to integrate information from first to thirdorder neighbors. with the help of the dynamic mechanism, ldgcns can effectively synthesize information from different orders to model complex interactions in the amr graph for text generation. also, ldgcns require no additional computational overhead, in contrast to vanilla gcn models. we further develop two novel weight sharing strategies based on the group graph convolutions and weight tied convolutions. these strategies allow the ldgcn model to reduce memory usage and model complexity. experiments on amr-to-text generation show that ldgcns outperform best reported gcns and sans trained on ldc2015e86 and ldc2017t10 with significantly fewer parameters. on the large-scale semi-supervised setting, our model is also consistently better than others, showing the effectiveness of the model on a large training set. we release our code and pretrained models at https://github.com/yanzhang92/ldgcns . foot_0 2 background graph convolutional networks our ldgcn model is closely related to gcns @xcite which restrict filters to operate on a first-order neighborhood. given an amr graph g with n concepts (nodes), gcns associate each concept v with a feature vector h @xmath0, where d is the feature dimension. g can be represented by concatenating features of all the concepts, i.e., @xmath1, ..., h vn ]. graph convolutions at l-th layer can be defined as: self-attention networks unlike gcns, sans @xcite capture global interactions by connecting each concept to all other concepts. intuitively, the attention matrix can be treated as the adjacency matrix of a fully-connected graph. formally, sans take a sequence of representations of n nodes @xmath2, ..., h vn ] as the input. attention score a uv between the concept pair (u,v) is: structured sans @xcite and cai and lam (2020) extend san s by incorporating the relation r uv between node u and node v in the w l denotes the trainable weights and × denotes matrix multiplication. vanilla gcns take the 1st-order adjacency matrix a 1 as the input, which only captures information from one-hop neighbors. ldgcns take k number of k-order adjacency matrix a k as inputs, w l is shared for all a k . k is set to 2 here for simplification. a dynamic fusion mechanism is applied to integrate the information from 1-to k-hop neighbors. ## dynamic fusion mechanism as discussed in section 2, gcns are generally more computationally efficient than structured sans as their computation cost scales linearly and no additional relation encoders are required. however, the locality nature of gcns precludes efficient non-local information propagation. to address this issue, we propose the dynamic fusion mechanism, which integrates higher order information for better non-local information aggregation. with the help of this mechanism, our model solely based on graph convolutions is able to outperform competitive structured sans. inspired by gated linear units (glus; @xcite , which leverage gating mechanisms @xcite to dynamically control information flows in the convolutional neural networks, we propose dynamic fusion mechanism (dfm) to integrate information from different orders. dfm allows the model to automatically synthesize information from neighbors at varying degrees of hops away. similar to glus, dfm retains non-linear capabilities of the layer while allowing the gradient to propagate through the linear unit without scaling. based on this non-linear mixture procedure, dfm is able to control the information flows from a range of orders to specific nodes in the amr graph. formally, graph convolutions based on dfm are defined as: computational overhead in practice, there is no need to calculate or store a k . a k h l is computed with right-to-left multiplication. specifically, if @xmath3, we calculate a 3 h l as (a(a(ah l ))). since we store a as a sparse matrix with m nonzero entries as vanilla gcns, an efficient implementation of our layer takes o(k max × m × d) computational time, where k max is the highest order used and d is the feature dimension of h l . under the realistic assumptions of k max m and d m, running an l-layer model takes o(lm) computational time. this matches the computational complexity of the vanilla gcns. on the other hand, dfm does not require additional parameters as the weight matrix is shared over various orders. deeper ldgcns to further facilitate the nonlocal information aggregation, we stack several ldgcn layers. in order to stabilize the training, we introduce dense connections @xcite , accordingly, h l in eq. 4 is replaced by ĥl . ## parameter saving strategies although we are able to train a very deep ldgcn model, the ldgcn model size increases sharply as we stack more layers, resulting in large model complexity. to maintain a better balance between parameter efficiency and model capacity, we develop two novel parameter saving strategies. we first reduce partial parameters in each layer based on group graph convolutions. then we further share parameters across all layers based on weight tied convolutions. these strategies allow the ldgcn model to reduce memory usage and model complexity. ## related work graph convolutional networks @xcite have been widely used as the structural encoder in various nlp applications including question answering @xcite , semantic parsing @xcite and relation extraction @xcite . early efforts for amr-to-text generation mainly include grammar-based models @xcite and sequence-based models @xcite @xcite , discarding crucial structural information when linearising the input amr graph. to solve that, various gnns including graph recurrent networks @xcite and graph convolutional networks @xcite have been used to encode the amr structure. though gnns are able to operate directly on graphs, the locality nature of them precludes efficient information propagation @xcite @xcite . larger and deeper models are required to model the complex non-local interactions @xcite . more recently, sanbased models @xcite @xcite outperform gnn-based models as they are able to capture global dependencies. unlike previous models, our local, yet efficient model, based solely on graph convolutions, outperforms competitive structured sans while using a significantly smaller model. ## conclusion in this paper, we propose ldgcns for amr-totext generation. compared with existing gcns and sans, ldgcns maintain a better balance between parameter efficiency and model capacity. ldgcns outperform state-of-the-art models on amr-to-text generation. in future work, we would like to investigate methods to stabilize the training of weight tied models and apply our model on other tasks in natural language generation.