Simplified Graph Learning for Inductive Short Text Classification (2022)

Abstract: Short text classification (STC) is hard as short texts lack context information and labeled data is not enough. Graph neural networks obtain the state-of-the-art on STC since they can merge various auxiliary information via the message passing framework. However, existing works conduct transductive learning, which requires retraining to accommodate new samples and takes large memory. In this paper, we present SimpleSTC which handles inductive STC problem but only leverages words. We construct word graph from an external large corpus to compensate for the lack of semantic information, and learn text graph to handle the lack of labeled data. Results show that SimpleSTC obtains state-of-the-art performance with lower memory consumption and faster inference speed.

## introduction short text classification (stc) is a fundamental task in many applications, such as sentiment analysis @xcite and query intent classification @xcite . due to the limited length, short texts lack context and strict syntactic structure, which makes them hard to understand @xcite . hence, existing stc methods strive to enrich short texts by incorporating various auxiliary information, including concepts of external knowledge bases @xcite @xcite , topics discovered in the corpus @xcite , partof-speech (pos) tags @xcite , and entities residing in knowledge graphs @xcite . another challenge faced by stc is the shortage of labeled data in practice @xcite . hence, recent stc methods @xcite @xcite further consider semi-supervised stc problem where only a small portion of the corpus is labeled. the state-of-the-art stc models are based on graph neural networks (gnns) @xcite . hgat @xcite applies a gnn with dual-level attention on a corpus-level graph containing topics, entities and texts. shine @xcite designs a hierarchical gnn to operate on a corpus-level hierarchical graph of word-level components including words, part-of-speech (pos) tags and entities, and a learned text graph. in summary, they conduct transductive learning, modeling the whole dataset (including the training and testing samples) as a large heterogeneous graph of various semantic concepts. hence, they both suffer from two problems: (i) require model retraining if new samples come and (ii) have high memory consumption to store the complete dataset. the recent hgat-inductive @xcite proposes to connect each new sample to training data and unlabeled data existing in the corpus, which shows potential of solving stc problem by inductive learning. in this paper, we propose a simple stc method called simplestc to handle inductive stc problem. inspired by the observation that semantic information contained in large datasets can be helpful to understand small datasets @xcite , we use external large corpus (which will be called global pool to avoid confusion with the dataset corpus) to construct the basic word graph of common words. upon the word graph, we represent short text embedding by pooling over words which appear in the short text, and dynamically learns the text graph which only connects related short texts. in a nutshell, the word graph constructed from the global pool compensates for the lack of semantic information, while the text graph handles the lack of labeled data. although existing stc methods leverage multiple auxiliary information, we find simplestc with word graph constructed using global pool is enough to provide semantic information and obtain state-of-the-art performance on benchmark stc datasets, using much lower memory consumption. incorporating auxiliary information such as entities or tags cannot further improve the performance. ## proposed method given a training set d @xmath0, y i )} i @xmath1, we aim at learning a predictive model to classify new samples. we present simplestc (figure 1 ) to solve this inductive stc problem. we first construct word graph using an external large corpus to compensate for the lack of semantic information. then, we learn text graph by connecting short texts to words appearing in them, and propagate the limited labeled information among connected texts. ## experiments we use a 24gb nvidia gtx 3090 gpu. all results are averaged over five runs. some results are put in appendix b due to space limit. ## limitations we consider solving inductive stc problem on a hierarchically organized word graph and text graph. we manage to supplement semantic information by word graph constructed from an external large corpus. nonetheless, how to dig out the most informative words instead of the common words from this external large corpus can be further explored. yufeng zhang, xueli yu, zeyu cui, shu wu, zhongzhen wen, and liang wang.