Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks (2020)

Abstract: Text classification is fundamental in natural language processing (NLP) and Graph Neural Networks (GNN) are recently applied in this task. However, the existing graph-based works can neither capture the contextual word relationships within each document nor fulfil the inductive learning of new words. Therefore in this work, to overcome such problems, we propose TextING for inductive text classification via GNN. We first build individual graphs for each document and then use GNN to learn the fine-grained word representations based on their local structure, which can also effectively produce embeddings for unseen words in the new document. Finally, the word nodes are aggregated as the document embedding. Extensive experiments on four benchmark datasets show that our method outperforms state-of-the-art text classification methods.

## introduction text classification is one of the primary tasks in the nlp field, as it provides fundamental methodologies for other nlp tasks, such as spam filtering, sentiment analysis, intent detection, and so forth. traditional methods for text classification include naive bayes @xcite , k-nearest neighbor @xcite and support vector machine @xcite . they are, however, primarily dependent on the hand-crafted features at the cost of labour and efficiency. there are several deep learning methods proposed to address the problem, among which recurrent neural network (rnn) @xcite and convolutional neural network (cnn) @xcite are essential ones. based on them, extended models follow to leverage the classification performance, for instance, textcnn @xcite , textrnn @xcite and textrcnn @xcite . yet they all focus on the locality of words and thus lack of long-distance and non-consecutive word interactions. graph-based methods are recently applied to solve such issue, which do not treat the text as a sequence but as a set of co-occurrent words instead. for example, @xcite employ graph convolutional networks @xcite and turns the text classification problem into a node classification one (textgcn). moreover, @xcite improve textgcn by introducing the message passing mechanism and reducing the memory consumption. however, there are two major drawbacks in these graph-based methods. first, the contextual-aware word relations within each document are neglected. to be specific, textgcn @xcite constructs a single graph with global relations between documents and words, where fine-grained text level word interactions are not considered @xcite . in @xcite , the edges of the graph are globally fixed between each pair of words, but the fact is that they may affect each other differently in a different text. second, due to the global structure, the test documents are mandatory in training. thus they are inherently transductive and have difficulty with inductive learning, in which one can easily obtain word embeddings for new documents with new structures and words using the trained model. therefore, in this work, we propose a novel text classification method for inductive word representations via graph neural networks, termed texting. in contrast to previous graph-based approaches with global structure, we train a gnn that can depict the detailed word-word relations using only training documents, and generalise to new documents in test. we build individual graphs by applying the sliding window inside each doc-ument @xcite . the information of word nodes is propagated to their neighbours via the gated graph neural networks @xcite , which is then aggregated into the document embedding. we also conduct extensive experiments to examine the advantages of our approach against baselines, even when words in test are mostly unseen (21.06% average gain in such inductive condition). noticing a concurrent work @xcite also reinforces the approach with a similar graph network structure, we describe the similarities and differences in the method section. to sum up, our contributions are threefold: ## method texting comprises three key components: the graph construction, the graph-based word interaction, and the readout function. the architecture is illustrated in figure 1 . in this section, we detail how to implement the three and how they work. ## experiments in this section, we aim at testing and evaluating the overall performance of texting. during the experimental tests, we principally concentrate on three concerns: (i) the performance and advantages of our approach against other comparable models, (ii) the adaptability of our approach for words that are never seen in training, and (iii) the interpretability of our approach on how words impact a document. ## conclusion we proposed a novel graph-based method for inductive text classification, where each text owns its structural graph and text level word interactions can be learned. experiments proved the effectiveness of our approach in modelling local word-word relations and word significances in the text.