Be More with Less: Hypergraph Attention Networks for Inductive Text Classification (2020)

Abstract: Text classification is a critical research topic with broad applications in natural language processing. Recently, graph neural networks (GNNs) have received increasing attention in the research community and demonstrated their promising results on this canonical task. Despite the success, their performance could be largely jeopardized in practice since they are: (1) unable to capture high-order interaction between words; (2) inefficient to handle large datasets and new documents. To address those issues, in this paper, we propose a principled model – hypergraph attention networks (HyperGAT), which can obtain more expressive power with less computational consumption for text representation learning. Extensive experiments on various benchmark datasets demonstrate the efficacy of the proposed approach on the text classification task.

## introduction text classification, as one of the most fundamental tasks in the field of natural language processing, has received continuous endeavors from researchers due to its wide spectrum of applications, including sentiment analysis @xcite , topic labeling @xcite , and disease diagnosis @xcite . inspired by the success of deep learning techniques, methods based on representation learning such as convolutional neural networks (cnns) @xcite and recurrent neural networks (rnns) @xcite have been extensively explored in the past few years. in essence, the groundbreaking achievements of those methods can be attributed to their strong capability of capturing sequential context information from local consecutive word sequences. more recently, graph neural networks (gnns) @xcite @xcite have drawn much attention and demonstrated their superior performance in the task of text classification @xcite @xcite . this line of work leverages the knowledge from both training and test documents to construct a corpus-level graph with global word co-occurrence and document-word relations, and consider text classification as a semi-supervised node classification problem. then with gnns, long-distance interactions between words could be captured to improve the final text classification performance. despite their promising early results, the usability of existing efforts could be largely jeopardized in real-world scenarios, mainly owing to their limitations in the following two aspects: (i) expressive power. existing gnn-based methods predominately focus on pairwise interactions (i.e., dyadic relations) between words. however, word interactions are not necessarily dyadic in natural language, but rather could be triadic, tetradic, or of a higher-order. for instance, consider the idiom "eat humble pie", whose definition is "admit that one was wrong" in common usage. if we adopt a simple graph to model the word interactions, gnns may misinterpret the word pie as "a baked dish" based on its pairwise connections to other two words (humblepie and eatpie), then further misunderstand the actual meaning of the whole idiom. hence, how to go beyond pairwise relations and further capture the high-order word interactions is vital for high-quality text representation learning, but still remains to be explored. (ii) computational consumption. on the one hand, most of the endeavors with gnn backbone tend to be memory-inefficient when the scale of data increases, due to the fact that constructing and learning on a global document-word graph consumes immense memory @xcite . on the other hand, the mandatory access to test documents during training renders those methods inherently trans-ductive. it means that when new data arrives, we have to retrain the model from scratch for handling newly added documents. therefore, it is necessary to design a computationally efficient approach for solving graph-based text classification. upon the discussions above, one critical research question to ask is "is it feasible to acquire more expressive power with less computational consumption?". to achieve this goal, we propose to adopt document-level hypergraph (hypergraph is a generalization of simple graph, in which a hyperedge can connect arbitrary number of nodes) for modeling each text document. the use of document-level hypergraphs potentially enables a learning model not only to alleviate the computational inefficiency issue, but more remarkably, to capture heterogeneous (e.g., sequential and semantic) high-order contextual information of each word. therefore, more expressive power could be obtained with less computational consumption during the text representation learning process. as conventional gnn models are infeasible to be used on hypergraphs, to bridge this gap, we propose a new model named hypergat, which is able to capture the encoded high-order word interactions within each hypergraph. in the meantime, its internal dual attention mechanism highlights key contextual information for learning highly expressive text representations. to summarize, our contributions are in three-fold: ## methodology in this section, we introduce a new family of gnn models developed for inductive text classification. by reviewing the existing gnn-based endeavors, we first summarize their main limitations that need to be addressed. then we illustrate how we use hypergraphs to model text documents for achieving the goals. finally, we propose the model hyper-gat based on a new dual attention mechanism and model training for inductive text classification. ## conclusion in this study, we propose a new graph-based method for solving the problem of inductive text classification. apart from the existing efforts, we propose to model text documents with document-level hypergraphs and further develop a new family of gnn model named hypergat for learning discriminative text representations. specifically, our method is able to acquire more expressive power with less computational consumption for text representation learning. by conducting extensive experiments, the results demonstrate the superiority of the proposed model over the state-of-the-art methods.