Text Graph Transformer for Document Classification (2020)

Abstract: Text classification is a fundamental problem in natural language processing. Recent studies applied graph neural network (GNN) techniques to capture global word co-occurrence in a corpus. However, previous works are not scalable to large-sized corpus and ignore the heterogeneity of the text graph. To address these problems, we introduce a novel Transformer based heterogeneous graph neural network, namely Text Graph Transformer (TG-Transformer). Our model learns effective node representations by capturing structure and heterogeneity from the text graph. We propose a mini-batch text graph sampling method that significantly reduces computing and memory costs to handle large-sized corpus. Extensive experiments have been conducted on several benchmark datasets, and the results demonstrate that TG-Transformer outperforms state-of-the-art approaches on text classification task.

## introduction text classification is a widely studied problem in natural language processing and has been addressed in many real-world applications such as news filtering, spam detection, and health record systems @xcite @xcite . the objective is to assign corresponding labels to textual units based on text representations. deep learning models like convolutional neural networks (cnn) @xcite and recurrent neural networks (rnn) @xcite have been applied for text representation learning instead of traditional hand-crafted features, such as n-gram and bag-of-words (bow) @xcite . researchers have recently turned to graph neural network (gnn) to exploit global features in text representation learning, which learns node embedding by aggregating information from neighbors through edges. @xcite first generalized cnn to graph for text classification task. then @xcite applied graph convolution network (gcn) @xcite on a corpus level heterogeneous text graph and achieved state-of-the-art performance. @xcite further improved classification accuracy by expanding the text graph with semantic and syntactic contextual information. however, these gcn-based models on heterogeneous text graphs suffer from two practical issues. firstly, none of these models are scalable to largesized corpus due to high computation and memory costs. calculation of all the nodes in the graph is required at each layer during training. secondly, all these models ignore the heterogeneity of the text graph, which consists of both document and word nodes. distinguishing nodes of different types will benefit node representation learning. to address the above problems, we propose a novel transformer-based heterogeneous gnn model, namely text graph transformer (tg-transformer). instead of learning based on the full text graph, we propose a text graph sampling method that enables subgraph mini-batch training. the significantly reduced computing and memory costs make the model scalable to large-sized corpus. moreover, we utilize transformer to aggregate information in subgraph batch with two proposed graph structural encodings. we also distinguish the learning process of different type nodes to fully utilize the heterogeneity of text graph. the main contributions of this work are as follows: ## methodology in this section, we introduce tg-transformer in great detail. first, we present how to construct a heterogeneous text graph for a given corpus. then, we introduce our text graph sampling method, which can generate subgraph mini-batch from the text graph. these subgraph batches will be fed into tg-transformer to learn efficient node representations for classification. the overall structure of our model is shown in fig. 1 . ## conclusion in this paper, we proposed a scalable heterogeneous graph model, tg-transformer, for text classification. experimental results prove its effectiveness and efficiency compared to state-of-the-art methods. it also enables parallelization and pre-training in gnn models for further research.