Hierarchical Information Matters: Text Classification via Tree Based Graph Neural Network (2022)

Abstract: Text classification is a primary task in natural language processing (NLP). Recently, graph neural networks (GNNs) have developed rapidly and been applied to text classification tasks. As a special kind of graph data, the tree has a simpler data structure and can provide rich hierarchical information for text classification. Inspired by the structural entropy, we construct the coding tree of the graph by minimizing the structural entropy and propose HINT, which aims to make full use of the hierarchical information contained in the text for the task of text classification. Specifically, we first establish a dependency parsing graph for each text. Then we designed a structural entropy minimization algorithm to decode the key information in the graph and convert each graph to its corresponding coding tree. Based on the hierarchical structure of the coding tree, the representation of the entire graph is obtained by updating the representation of non-leaf nodes in the coding tree layer by layer. Finally, we present the effectiveness of hierarchical information in text classification. Experimental results show that HINT outperforms the state-of-the-art methods on popular benchmarks while having a simple structure and few parameters.

## introduction text classification is an essential problem in nlp. there are numerous applications of text classification, such as news filtering, opinion analysis, spam detection, and document organization @xcite . recently, gnns have developed rapidly. gnns learn the representation of each node by aggregating the information of neighboring nodes and can retain structural information in the graph embedding. therefore, many graph-based methods are applied to text classification and achieve good performance. @xcite proposed textgcn, which is the first method to employ a graph convolutional network (gcn) in the text classification task. they built a heterogeneous graph containing word nodes and document nodes for the corpus and transformed the text classification task into a node classification task. textgcn outperformed other traditional methods and attracted much attention, which has led to increasingly more applications of graphbased methods in text classification. in @xcite ; @xcite , the text classification task was converted into the graph classification task. they built text-level co-occurrence graphs for each data. @xcite employed a message passing mechanism (mpm) and outperformed textgcn. a gated graph neural network (ggnn) was employed in @xcite and achieved state-of-the-art performance. different from normal graph data, treestructured data is a simpler data structure with rich hierarchical information. in the text classification task, despite plenty of efforts that have been devoted to the adoption of gnns, none of them has realized the rich hierarchical information in text, which has already been employed in other nlp tasks. trdec was proposed for the nmt task to generate a target-side tree topology and uses the tree to help the translation process @xcite . in knowledge-based question answering tasks, @xcite treated the query as a tree and used the tree-based lstm to model the context of the entities or relationships in the query. @xcite clustered the labels in the extreme multi-label text classification task and built a label tree. @xcite proposed a rumor detection model based on tree transformer to better exploit user interactions in the dialogues. hierarchical information in text is likely to be helpful for text classification tasks, so it should be better utilized. graph construction document document document w 1 w 2 w 3 w 4 w 5 w 6 coding tree construction w 1 w 3 w 6 w 4 w 5 w 1 w 2 w 4 w 3 w 5 w 6 w 1 w 2 w 3 w 4 w 5 w 6 w 2 text representation learning w 1 w 3 w 6 w 2 w 4 w 5 w 1 w 2 w 4 w 3 w 5 w 6 w 1 w 2 w 3 w 4 w 5 w 6 model based on tree structure for text classification, named hint. structural entropy can measure the complexity of the hierarchical information of the graph, and decode its key structure. as shown in figure 1 , we first build individual graph for each document through dependency parsing. then the graphs are transformed into their corresponding coding trees by a structural entropy minimization algorithm. the coding tree not only retains the crucial features of data but also excludes many other features that worsen the model. the model classifies the entire document by learning the representation of the coding tree. so far, we not only make better use of the hierarchical information in the text data but also represent text with a simpler data structure (i.e., tree). we conduct several experiments to verify the advantages of our method over the baselines. to sum up, our contributions are as follows: ## related work in recent years, the graph-based text classification method transforms the text classification task into a graph classification task or a node classification task and has achieved good performance in the text classification task. different from traditional deep learning models @xcite , the graph-based text classification methods usually capture the rich relationships between nodes in the graph by constructing document graphs or corpus graphs, then apply gnn to learn the embedding of the document, and finally input the embedding to the classification layer. for graph construction, one is to construct a static graph for text or corpus. some methods are to construct a single heterogeneous graph for the entire corpus @xcite . other studies construct a separate graph for each document to handle the inductive learning setting @xcite . except for static graphs, the construction of dynamic graphs does not rely on prior knowledge and can be jointly learned with the model @xcite . for the learning of graph representation, various gnn models are used in text classification, such as gcn @xcite , ggnn @xcite , mpm @xcite , and gat-based model @xcite . tree structure data is a special kind of graph data with a simple structure and rich hierarchical information. with the development of deep learning, many models use trees to help solve nlp tasks. some methods process the data to get the tree structure data and use it to help the model. @xcite utilized the strong correlation between grammatical information and tree structure, and a treebased decoder trdec is proposed for the nmt task. trdec generates a target-side tree topology and uses the tree to guide the translation process. @xcite proposed a model based on label tree and attention mechanism for extreme multi-label text classification. the label tree is constructed by clustering the labels and the model is trained from top to bottom. in addition to using the generated tree, the deep learning model of the tree structure is also widely used. tree-structured multilinear principal component analysis (tmpca) used the pca of each layer to transfer the representation of two adjacent words to the next layer until the entire text is reduced to a single-word vector @xcite . @xcite treated the query as a tree and proposed a tree-based lstm in the knowledge-based question answering task to model the context of entities or relationships in the query. the tree structure can provide hierarchical information for the model, and the tree based deep learning model can make better use of the grammatical information through the hierarchy of the model. however, hierarchical information is not well studied in graph-based text classification methods. in this work, we aim to fill the gap in gnn-based text classification. ## method in this section, we will introduce hint in detail. first, we will explain the method of constructing a graph for each text. then, we introduce the coding tree construction algorithm that can decode the hierarchical information in text data. finally, we will show how the model learns the hierarchical information and text representation based on treestructured data and how to predict the label for a given text based on the learned representations. ## experiments in this section, we evaluate the effectiveness of hint foot_0 and report the experimental results. datasets. we utilize datasets including r8, r52, mr, and ohsumed. r8 and r52 are subsets of the reuters 21578 dataset. mr is a movie review dataset used for sentiment classification in which each review is a single sentence. the ohsumed corpus, which is designed for multilabel classification, is from the medline database. in this paper, we only use single-label data like other gnnbased text classification models @xcite . we employ stanfordnlp @xcite to build the dependency graphs for all datasets. the statistics of our datasets are summarized in lstm; (ii) word embedding methods, including fasttext @xcite and swem @xcite ; and (iii) graph-based methods for text classification, including the spectral approachbased textgcn , s 2 gc @xcite , and nonspectral method-based @xcite . settings. we randomly divide the training set into the training set and the validation set at a ratio of 9:1. we use the adam optimizer with an initial learning rate of 10 -3 and set the dropout rate to 0.5. the height of the coding tree is between 2 and 12. we set the sum or average function as the initial function of p ool in equation 6. for word embedding, we use pretrained glove with the dimension of 300, and the out-of-vocabulary (oov) words are randomly initialized from the uniform distribution [-0.01, 0.01]. for each position, we set this position in the vector to 1 and the rest to 0 to get the position encoding vector. we concatenate the glove vector and position encoding vector as the initial representation of graph nodes. ## conclusion in this paper, we proposed a novel method to address the limitation of previous works in text hierarchical information utilization. we build a dependency parsing graph for each text and construct a coding tree for each graph by structural entropy minimization algorithm. our model uses the hierarchy of the coding tree to learn the representation of each text. experimental results demonstrate the ability of hint to decode hierarchical information in text and show the positive effect of hierarchical information on text classification tasks. our model achieves state-of-the-art performance with a simple structure and few parameters.