Hierarchical Heterogeneous Graph Representation Learning for Short Text Classification (2021)

Abstract: Short text classification is a fundamental task in natural language processing. It is hard due to the lack of context information and labeled data in practice. In this paper, we propose a new method called SHINE, which is based on graph neural network (GNN), for short text classification. First, we model the short text dataset as a hierarchical heterogeneous graph consisting of word-level component graphs which introduce more semantic and syntactic information. Then, we dynamically learn a short document graph that facilitates effective label propagation among similar short texts. Thus, comparing with existing GNN-based methods, SHINE can better exploit interactions between nodes of the same types and capture similarities between short texts. Extensive experiments on various benchmark short text datasets show that SHINE consistently outperforms state-of-the-art methods, especially with fewer labels.

## introduction short texts such as tweets, news feeds and web search snippets appear daily in our life @xcite . to understand these short texts, short text classification (stc) is a fundamental task which can be found in many applications such as sentiment analysis @xcite , news classification @xcite and query intent classification @xcite . stc is particularly hard in comparison to long text classification due to two key issues. the first key issue is that short texts only contain one or a few sentences whose overall length is small, which lack enough context information and strict syntactic structure to understand the meaning of texts @xcite . for example, it is hard to get the meaning of "birthday girl is an amusing ride" without knowing "birthday girl" is a 2001 movie. a harder case is to understand a web search snippet such as "how much tesla", which usually does not contain word order nor function words @xcite . in addition, real stc tasks usually only have a limited number of labeled data compared to the abundant unlabeled short texts emerging everyday @xcite . therefore, auxiliary knowledge is required to understand short texts, examples include concepts that can be found in common sense knowledge graphs @xcite , latent topics extracted from the short text dataset @xcite , and entities residing in knowledge graphs @xcite . however, simply enriching auxiliary knowledge cannot solve the shortage of labeled data, which is another key issue commonly faced by real stc tasks @xcite . yet the popularly used deep models require large-scale labeled data to train well @xcite . currently, graph neural networks (gnns) designed for stc obtain the state-of-the-art performance @xcite . they both take the stc as the node classification problem on a graph with mixed nodes of different types: hgat @xcite builds a corpus-level graph modeling latent topics, entities and documents and stgcn @xcite operates on a corpuslevel graph of latent topics, documents and words. in both works, each document is connected to its nodes of a different type such as entities and latent topics but not to other documents. however, they do not fully exploit interactions between nodes of the same type. they also fail to capture the similarities between short documents, which is both useful to understand short texts @xcite @xcite and and important to propagate few labels on graphs @xcite . besides, both works have large parameter sizes: hgat @xcite is a gnn with dual-level attention and stgcn @xcite merges the node representations with word embeddings obtained via a pretrained bert @xcite via a bidirectional lstm @xcite . to address the aforementioned problems, we propose a novel hierarchical heterogeneous graph representation learning method for stc called shine, which is able to fully exploit interactions between nodes of the same types and capture similarity between short texts. shine operates on a hierarchically organized heterogeneous corpuslevel graph, which consists of the following graphs at different levels: (i) word-level component graphs model interactions between words, partof-speech (pos) tags and entities which can be easily extracted and carry additional semantic and syntactic information to compensate for the lack of context information; and (ii) short document graph is dynamically learned and optimized to encode similarities between short documents which allows more effective label propagation among connected similar short documents. we conduct extensive experiments on a number of benchmark stc datasets including news, tweets, document titles and short reviews. results show that the proposed shine consistently outperforms the state-of-theart with a much smaller parameter size. ## proposed method as mentioned in section 2.2, gnn-based methods, i.e., hgat and stgcn, can classify short texts while hgat performs better. however, both works build a graph with mixed nodes of different types without fully exploiting interactions between nodes of the same type. besides, they fail to capture the similarities between short documents, which can be important to propagate few labels on graphs. here, we present the proposed shine which can address above limitations, thus is able to better compensate for the shortage of context information and labeled data for stc. given a short text dataset foot_1 s containing short documents, we model s as a hierarchically organized heterogeneous graph consisting of: (i) word-level component graphs: we construct wordlevel component graphs which model word-level semantic and syntactic information in order to compensate for the lack of context information; (ii) short document graph: we dynamically learn the short document graph via hierarchically pooling over word-level component graphs, such that the limited label information can be effectively propagated among similar short texts. a high-level illustration of shine is shown in figure 1 . in the sequel, vectors are denoted by lowercase boldface, matrices by uppercase boldface. for a vector x, [x] i denotes the ith element of x. for a matrix x, x i denotes its ith row, [x] ij denotes the (i, j)th entry of x. for a set s, |s| denotes the number of elements in s. ## conclusion in this paper, we propose shine, a novel hierarchical heterogeneous graph representation learning method for short text classification. it is particularly useful to compensate for the lack of context information and propagate the limited number of labels efficiently. specially, shine can effectively learn from a hierarchical graph modeling different perspectives of the short text dataset: word-level component graphs are used to understand short texts from the semantic and syntactic perspectives, and the dynamically learned short document graph allows efficient and effective label propagation among similar short documents. extensive experiments show that shine outperforms the others consistently.