Deep Attention Diffusion Graph Neural Networks for Text Classification (2021)

Abstract: Text classification is a fundamental task with broad applications in natural language processing. Recently, graph neural networks (GNNs) have attracted much attention due to their powerful representation ability. However, most existing methods for text classification based on GNNs consider only one-hop neighborhoods and low-frequency information within texts, which cannot fully utilize the rich context information of documents. Moreover, these models suffer from over-smoothing issues if many graph layers are stacked. In this paper, a Deep Attention Diffusion Graph Neural Network (DADGNN) model is proposed to learn text representations, bridging the chasm of interaction difficulties between a word and its distant neighbors. Experimental results on various standard benchmark datasets demonstrate the superior performance of the present approach.

## introduction text classification, as an important and fundamental task in the field of natural language processing, has attracted extensive attention from scholars for many years and has been widely used in various practical applications, such as topic labeling @xcite , computational phenotyping @xcite , question answering @xcite and dialog act classification @xcite . the performance of text classification relies heavily on the ability of the model to extract textual features from raw text. previous shallow learning-based text classification approaches mainly use hand-crafted sparse lexical features, such as bag-of-words (bow) or n-grams, for representing texts @xcite . since these features are predefined, the models do not take full advantage of the large amount of training data. deep learning architectures represented by convolutional neural networks (cnns) @xcite and * corresponding author. recurrent neural networks (rnns) @xcite are becoming more popular due to their strong performance in text mining. these models can capture semantic and syntactic information in local consecutive word sequences well. recently, graph neural networks (gnns) have attracted increasing attention due to their superiority in dealing with complex structured data and relations @xcite . gnns have achieved promising results in text classification tasks when modeling texts with graph structures due to their powerful expressiveness @xcite . despite the success of the mentioned models, several serious limitations of prevalent gnns hinder their performance, which is mainly attributed to the following factors: (i) restricted receptive fields. most previous approaches allow a word in the graph to access direct neighborhoods. in other words, the word embeddings depend solely on the influence of the representation of its neighboring words at each layer. moreover, the sliding window used to build word-word edges is typically not large. this makes it impossible to enable long-range word interactions. therefore, it is critical to increase the receptive field of target words to obtain precise text representations. (ii) shallow layers. most current graph-based models for text classification adopt fairly shallow settings, as they achieve the best performance given two layers. two-layer graph models aggregate nodes in two-hop neighborhoods and thus cannot extract information beyond two-hop neighbors. theoretically, we can capture long-range dependencies between words with a large number of layers. however, a common challenge faced by most gnns is that performance degrades severely when stacking multiple layers to exploit larger receptive fields @xcite . some researchers attribute this phenomenon to over-smoothing (indistinguishable representation of different classes of nodes) @xcite . it is still necessary to explore deeper gnns to obtain more latent features in text classification tasks, especially for short texts, because the available contextual information is limited. (iii) non-precision document-level representations. most graph-based models for text classification leverage simple pooling operations, such as summing or averaging all nodes in the graph, to obtain document-level representations. this weakens the effect of some key nodes and significantly reduces the expressiveness of the model since different words play distinct roles in the text. (iv) low-pass filters. essentially, existing graph-based methods for text classification are fixed coefficients low-pass filters @xcite . it has been confirmed that the low-pass filter in gnns mainly preserves the commonality of node features, ignoring the difference among them (oono and suzuki, 2020). therefore, the learned representations of connected nodes become indistinguishable when always adopting a low-pass filter. meanwhile, several studies have demonstrated the usefulness of high-frequency information in graph signals to enhance the discriminative power of the model especially when the network exhibits disassortativity @xcite . hence, it is necessary to design a filter that does not just exhibit low-pass properties for learning word embeddings. to overcome the limitations above, we propose a novel model named deep attention diffusion graph neural network (dadgnn) for text classification based on learned effective text representations. specifically, we use the attention diffusion technique to widen the receptive field of each word in the document, which can capture the longrange word interactions at each layer. moreover, to extract the deep hidden semantics of words, we decouple the propagation and transformation processes of gnns to train deeper networks. finally, we calculate the weight of each node to obtain precise document-level representations. our work's key contributions are as follows: ## methods the overall architecture of dadgnn is shown in fig. 1 . next, we will elaborate on the three aspects of text graph construction, key components and graph-level representation, respectively. ## spectral analysis in this section, we investigate the effectiveness of our model from the perspective of graph spectra. ## results and analysis one crucial reason why our model achieves more significant improvements is that the receptive field of the target node is enhanced by attention diffusion, which incorporates more informative messages (i.e., both low-frequency and high-frequency information) in the text. furthermore, it is possible to obtain distinguishable hidden features in deep graph networks by decoupling the procedures of feature transformation and propagation, which is extremely useful for short texts. most graph-based approaches achieve outstanding performance compared to the other two groups of baseline models in the topic classification datasets, such as r52, r8 and ag news. this phenomenon demonstrates the critical role of capturing non-consecutive and long-distance semantics in the document for text classification. however, for the sentiment classification task, the sequence-based models have more robust performance, which can be explained by the fact that word order modeling is crucial for capturing sentiment. since our model constructs directed graphs, it can obtain sequential context information ignored by most graph-based methods and achieves excellent results in sentiment classification. ## conclusion in this work, we propose a new graph-based model, named dadgnn, for text classification. our model decouples the necessary procedures of gnns (i.e., representation transformation and propagation) to train a deep neural network and introduces a attention diffusion mechanism to capture non-direct-neighbor context information in a single layer. furthermore, the node-level attention technique is introduced to obtain more precise document-level representations. using the techniques above, our model enhances the receptive field and increases the depth without suffering from over-smoothing problems. the theoretical analysis essentially shows that dadgnn can learn the optimal filter to adapt the dataset; i.e., it can preserve more relevant high-frequency information of nodes to further improve the expressiveness of the model. extensive experiments show that our model is much better than other competitive approaches. more importantly, our work not only provides a powerful baseline model for text classification but also contributes to graph representation learning.