Multi-hop Graph Convolutional Network with High-order Chebyshev Approximation for Text Reasoning (2021)

Abstract: Graph convolutional network (GCN) has become popular in various natural language processing (NLP) tasks with its superiority in long-term and non-consecutive word interactions. However, existing single-hop graph reasoning in GCN may miss some important non-consecutive dependencies. In this study, we define the spectral graph convolutional network with the high-order dynamic Chebyshev approximation (HDGCN), which augments the multi-hop graph reasoning by fusing messages aggregated from direct and long-term dependencies into one convolutional layer. To alleviate the over-smoothing in high-order Chebyshev approximation, a multi-vote-based cross-attention (MVCAttn) with linear computation complexity is also proposed. The empirical results on four transductive and inductive NLP tasks and the ablation study verify the efficacy of the proposed model.

## introduction graph neural networks (gnns) are usually used to learn the node representations in euclidean space from graph data, which have been developed to one of the hottest research topics in recent years @xcite . the primitive gnns relied on recursive propagation on graphs, which takes a long time to train @xcite . one major variant of gnns, graph convolutional networks (gcns) @xcite , takes spectral filtering to replace recursive message passing and needs only a shallow network to convergent, which have been used in various nlp tasks. for example, @xcite constructed the text as a graph and input it to a gcn. this method achieved better results than conventional deep learning models in text classification. afterward, the gcns have became popular in more tasks, such as word embedding @xcite , semantic analysis @xcite , document summarization @xcite , knowledge graph @xcite , etc. the spectral graph convolution in yao's gcn is a localized first-order chebyshev approximation. it is equal to a stack of 1-step markov chain (mc) layer and fully connected (fc) layer. unlike the multi-step markov chains, the message propagation in vanilla gcn lacks the node probability transitions. as a result, the multi-hop graph reasoning is very tardy in gcn and easily causes the suspended animation problem @xcite . however, the probability transition on the graph is useful to improve the efficiency in learning contextual dependencies. in many nlp tasks (like the question answering (qa) system and entity relation extraction), the features of the two nodes need to be aligned. as an example, figure 1 shows a simple graph where the node n4 is a pronoun of node n1. in this example, the adjacency matrix is masked on nodes n2, n3, n5 to demonstrate the message passing between n1 and n4. figure 1 (c ) and (d) plot the processes of feature alignment on two nodes without and with probability transitions respectively. in this example, the feature alignment process without probability transition needs 10 more steps than which with probability transition. it is shown that encoding the multi-hop dependencies through the spectral graph filtering in gcn usually requires a deep network. however, as well known that the deep neural network (dnn) is tough to train and easily causes the over-fitting problem @xcite . some newest studies to improve the multi-hop graph reasoning include graph attention networks (gats) @xcite , graph residual neural network (gresnet) @xcite , graph diffusive neural network (difnet) (zhang, 2020), tgmc-s @xcite and graph transformer networks @xcite . gats enhance the graph reasoning by implicitly re-defining the graph structure with the attention on the 1-hop neighbors, but there is equilibrial optimization on the whole graph. gresnet solves the suspended animation problem by creating extensively connected highways to involve raw node features and intermediate representations throughout all the model layers. however, the multi-hop dependencies are still reasoned at a slow pace. difnet introduces a new neuron unit, i.e., gdu (gated diffusive unit), to model and update the hidden node states at each layer. difnet replaces the spectral filtering with a recursive module and realizes the neural gate learning and graph residual learning. but the time cost is aggravated in difnet compared with gcn. tgmc-s stacks gcn layers on adjacent matrices with different hops of traffic networks. different from the ground-truth traffic network in tgmc-s, it is hard to construct the multi-hop word-word relationships objectively from the text. tgmc-s hadn't given a way to improve the multi-hop message passing in gcn. transformers @xcite and corresponding pre-trained models @xcite could be thought of as fully-connected graph neural networks that contain the multi-hop dependencies. they figure out the contextual dependencies on the fully-connected graph with the attention mechanism. the message propagation in transformers follows the relations self-adaptively learned from input sequence instead of the fixed graph structures. publications have shown that transformers outperform gcns in many nlp tasks. graph transformer (dwivedi and bresson, 2020) generalizes the transformer to arbitrary graphs, and improves inductive learning from laplacian eigenvectors on graph topology. however, due to the connections scale quadratically growth with node number n in graphs, things get out of hand for very large n . additionally, the fully-connected graph is not an interpretable architecture in practical tasks. for example, whether transformers are the best choice to bring the text in linguistic theory? 1 to improve the efficiency and performance of multi-hop graph reasoning in spectral graph convolution, we proposed a new graph convolutional network with high-order dynamic chebyshev approximation (hdgcn). a prime chebnet and a high-order dynamic (hd) chebnet are firstly applied to implement this chebyshev approximation. these two sub-networks work like a trade-off on low-pass signals (direct dependencies) and highpass signals (multi-hop dependencies) respectively. the prime chebnet takes the same frame as the convolutional layer in vanilla gcn. it mainly extracts information from direct neighbors in local contexts. the hd-chebnet aggregates messages from multi-hop neighbors following the transition direction adaptively learned by the attention mechanism. the standard self-attention @xcite has a o n 2 computation complexity and it is hard to be applied on long sequence. even the existing sparsity attention methods, like the star-transformer @xcite and extended transformer construction (etc) @xcite , have reduced the quadratic dependence limit of sequence length to linear dependence, but the fullyconnected graph structure cannot be kept. we design a multi-vote-based cross-attention (mvcattn) mechanism. the mvcattn scales the computation complexity o(n 2 ) in self-attention to o(n ). the main contributions of this paper are listed below: our work draws supports from the vanilla gcn and the attention mechanism, so we first give a glance at the paradigm of these models in this section. ## method in our model, the input graph @xmath0, e) takes the same form as the one in gcn. the nodes are attributed as @xmath1, and the adjacency matrix the k-th (@xmath2, to replace the parameters {θ i } k @xmath3, the kth-order chebyshev polynomials in equation 4 are approximated as: where the a is normalized adjacency matrix (as defined in equation 1 ). as the node state transition a 2k causes the over-smoothing problem @xcite , we take the dynamic pairwise relationship a d self-adaptively learned by the attention mechanism to turn the direction of node state transition. the powers of adjacency matrix a 2k in equation 5 can cause the over smoothing problem, we replace the a 2k with a k a k d . in our implementation, the first-order and higherorder chebyshev polynomials in equation 5 is approximated with a prime chebyshev network (chebnet) and high-order dynamic chebyshev networks (hd-chebnets) respectively. we generalize the graph convolution on kth-order dynamic chebyshev approximation (equation 5 ) to the layerwise propagation as follows, ## conclusions this study proposes a multi-hop graph convolutional network on high-order dynamic chebyshev approximation (hdgcn) for text reasoning. to improve the multi-hop graph reasoning, each convolutional layer in hdgcn fuses low-pass signals (direct dependencies saved in fixed graph structures) and high-pass signals (multi-hop dependencies adaptively learned by mvcattn) simultaneously. we also firstly propose the multi-votes based cross-attention (mvcattn) mechanism to alleviate the over-smoothing in high-order chebyshev approximation, and it just costs the linear computation complexity. our experimental results demonstrate that hdgcn outperforms compared sota models on multiple transductive and inductive nlp tasks.