K-hop neighbourhood regularization for few-shot learning on graphs: A case study of text classification (2023)

Abstract: We present FewShotTextGCN, a novel method designed to effectively utilize the properties of word-document graphs for improved learning in low-resource settings. We introduce K-hop Neighbourhood Regularization, a regularizer for heterogeneous graphs, and show that it stabilizes and improves learning when only a few training samples are available. We furthermore propose a simplification in the graph-construction method, which results in a graph that is ∼7 times less dense and yields better performance in little-resource settings while performing on par with the state of the art in high-resource settings. Finally, we introduce a new variant of Adaptive Pseudo-Labeling tailored for word-document graphs. When using as little as 20 samples for training, we outperform a strong TextGCN baseline with 17% in absolute accuracy on average over eight languages. We demonstrate that our method can be applied to document classification without any language model pretraining on a wide range of typologically diverse languages while performing on par with large pretrained language models.

## introduction text classification, a key task in natural language processing (nlp), has many real-world applications, including toxic comment identification, news categorization, spam detection and opinion mining. one popular approach to this problem relies on large-scale pretraining of transformer models @xcite @xcite , which have shown to be able to approach or even surpass human performance on many natural language understanding (nlu) benchmarks @xcite @xcite . while these results are impressive for the languages on which models are pretrained, performance tends to deteriorate on languages where no or little data is available @xcite . in practice, this means that these models are effective on a set of approximately 100 out of the 7000+ spoken languages in the world. next to the requirement for vast amounts of data for pretraining, transformer language models tend to be impractically large in terms of their number of parameters and have a high environmental footprint @xcite . recently, graph neural networks (gnns) have shown to be effective for text classification in both transductive @xcite @xcite and inductive @xcite learning settings -with promising results in both high-and low-resource settings. particularly in the transductive setting, the authors of textgcn @xcite show that graph convolutional networks (gcns) @xcite can outperform state-of-the-art methods for document classification on english datasets without any language model pretraining. they do so by modeling an entire corpus of documents simultaneously as one heterogeneous worddocument graph. the document classification task is formulated as a node-classification task over this graph. later work shows that (multilingual) pretrained language models (mplms) can be used to provide gnns used in transductive setting with rich representations of both words and documents, improving results further in both monolingual @xcite and cross-lingual settings @xcite . these works focus solely on high-resource settings and do not report any results on performance in low-resource settings. in this work, we propose a novel gnn-based method for learning document classification tasks on a range of languages without the need for any pretraining data (i.e., without utilizing any pretrained word embeddings or language models), and from few labeled samples only. to the best of our knowledge, we are the first to investigate few-shot graph-based transductive document classification in a range of languages other than english. we present fewshottextgcn, an improved version of the original textgcn model, where we exploit properties of the heterogeneous worddocument graph for improved learning from scratch and with only a few labels. more specifically, we: we compare fewshottextgcn to its predecessor and two strong plms on ten topic classification benchmarks comprising eight typologically diverse languages, and experiment with a range of lowresource settings, including using as little as 20 labeled samples to learn from, and without any other form of (pre-trained) knowledge about a language except for what constitutes a word (using word boundaries or a tokenizer). in our lowest-resource setting, our method outperforms textgcn with 4.6% and 17% points in absolute accuracy on average for reuters and mldoc, respectively -while having a substantially smaller computational and memory footprint. fewshottextgcn performs on par with large plms on the great majority of the considered benchmarks, without the need for any large-scale pretraining and at only a fraction of the parameter count of these plms -indicating that graph-based methods are an attractive alternative to using large plms for topic classification. all our code and models are released to facilitate further research on this topic. 1 graph neural networks (gnns) are a class of neural models designed to facilitate representation learning on geometric data -data that naturally occur in many situations/fields, such as chemistry, social networks, maps, visual meshes, etc. recently, there has been a great surge in research on gnns. gnns create new feature representations of nodes by aggregating the nodes' own feature representation and a message passed on from neighboring nodes. a graph @xmath0, e) is defined as a set of nodes v with edges e between them, typically represented as a square adjacency matrix a, where each entry holds the weight of the edge between node j and i. locality in graphs is defined by neighborhoods, where the neighbors of node i are defined as n @xmath1: (i, j) ∈ e ∀(j, i) ∈ e}. let be some permutation invariant aggregator such as sum, average or max, and let ψ and ϕ be two differentiable, learnable functions such as an mlp. using these ingredients, we can describe gnns by the way they do message passing. convolutional gnns use the weights c ij of the edge between nodes j and i to weigh the incoming messages. these weights are part of the definition of the graph, meaning they are statically defined. the input feature x i of node i is transformed to a latent representation h i by taking the first and most well-known convolutional gnn is the graph convolutional network (gcn) @xcite . ## related work our work is based on textgcn @xcite , which also serves as our baseline for all experiments. to the best of our knowledge, we are the first to investigate few-shot graph-based transductive learning from scratch for document classification in a range of languages other than english. since the scope of our work is few-shot document classification in many languages by learning from scratch, we do not consider clhg @xcite directly related work. the reasons being it models corpora in multiple languages jointly, whereas we learn each task in isolation, and relies on machine translation and mplms. similarly, mgl @xcite relies on mplms for encoding similar corpora in different languages into one embedding space, where consecutively a graph is dynamically constructed based on the similarity of the documents in the respective embedding space. finally, meta-learning is applied to learn to classify documents in one language, given a limited set of documents in at least three other languages. hence, we do not review these works in-depth. textgcn textgcn @xcite is the first application of gnns to transductive text classification, applied on english datasets without any language model pretraining. the great majority of experiments is performed in high-resource settings, but a small set of results on performance in lowresource settings is also provided -motivating us to further explore and expand upon this subject. the authors construct a heterogeneous graph containing both word and document nodes. word-word edges are weighed based on the pointwise mutual information (pmi) between the respective words, and word-document edges are created based on the tf-idf score of the word in the respective document. more specifically, the adjacency matrix a is defined as: document-document links are not considered. a one-hot encoding is used as input features for the nodes and a two-layer gcn is used to classify the document nodes. while this setup is relatively simple in terms of preprocessing, pretraining and the number of parameters in the model, the authors show that their method performs on par with stateof-the-art methods, even improving the state of the art for the 20news foot_1 dataset. bertgcn follow-up work on textgcn is that of bertgcn @xcite , where the authors leverage plms to initialize document-node features. more specifically, a bert-based model is used to encode the documents, and all other nodes are initialized with a one-hot vector. the bert model used for encoding documents is optimized both via gradients propagated through the gcn and via an auxiliary classifier that directly uses the bert embeddings to classify the documents. using bertgcn, the authors improve over textgcn on a variety of text classification tasks -especially on a sentiment analysis task, for which word order information is crucial for good performance @xcite . to be able to use bertgcn in a full-batch gradient descent method, the authors use a memory bank that allows decoupling the dictionary size from the mini-batch size. although the presented results are promising, a drawback of using large plms is the need for vast amounts of pretraining data, making these methods inaccessible for low-resource languages. ## data in this section, we give an overview of the datasets we use and the respective classification tasks. li (2018 per language, the train and test datasets contain 1k and 4k samples respectively. ## approach graph construction we follow the graph construction method as described in the original textgcn @xcite ) work except we deviate in two different directions. stopword removal is omitted, as this assumes knowledge of the language, whereas we aim for an approach that assumes no prior knowledge. furthermore, wordword edges are omitted too. omitting such edges results in a much less densely connected graph, making learning substantially less memory intensive. we argue that the added value of word-word edges in a word-document graph is minimal given 1) only global information of word co-occurrence is considered (i.e., co-occurrence over the whole corpus as opposed to within document co-occurrence), and 2) over the course of training, words co-occurring in a document can influence each other's representation through an n -layer gnn where @xmath2. we illustrate this with an example in appendix e, while in section 8, we experimentally demonstrate the limited effect of word-word edges using ablation studies. ## experimental setting throughout our experiments, textgcn is used as a directly comparable baseline. since our main goal is to develop a method that performs well in low-resource scenarios for many languages without the need of any knowledge of that languageapart from the ability to identify word boundaries in a sentence -our setup deviates from the original textgcn work. unlike the original work, we do not perform a grid-search of hyperparameter settings per experiment/language, but rather keep them fixed -which make our results not directly comparable to the original. similarly to the original textgcn work, we also consider the r8 and r52 datasets for an analysis of our approach on english (see section 4). additionally, we also provide results for two plms trained with the same amount of data. these results are not directly comparable, since the plms are trained in an inductive setting, but are included to provide better insight into the positioning of our method in the context of broader literature. ## ablation experiments the original textgcn implementation proposes to use edges between words based on their respective pmi. since pmi is calculated using a window size of 20, many extra edges are introduced. for the mldoc dataset, omitting word-word edges results in a graph that has, on average, only 15% of the amount of edges compared to the original graph (see no other edges exist on this graph and the classification task concerns the author nodes. in this case, we could apply k-nr on the author nodes based on the intuition that authors that publish a paper at the same venue are more similar to each other than authors that do not publish at the same venue. in order to get from the anchor author node to a positive author node, one has to traverse the graph by hopping to a neighboring paper, venue, paper and finally author node in that respective orderresulting in @xmath3. on this same graph, k-nr can be applied for paper nodes as well for @xmath4. in general, considering a graph with m different node types, k-nr can be applied if in terms of node types a symmetrical path with an odd number of nodes can be traversed. in this case, @xmath5. ## conclusion we introduced k-hop neighborhood regularization (k-nr), a contrastive learning method for heterogeneous graphs, and showed its implementation for word-document graphs (2-nr) is highly effective in improving learning from scratch in low-resource settings for a range of languages on topic classification tasks. we also showed that we can exploit properties of word-document graphs for improved learning in few-shot settings. we demonstrated that by simplifying the graph construction method via omitting word-word edges we can improve performance while reducing memory requirements in terms of total number of edges. additionally, we showed how pseudo-labeling can be successfully applied to word-document graphs. all approaches combined together form part of our new proposed method, fewshottextgcn, an improvement over textgcn for few-shot graph learning. fewshottextgcn performs on par with large plms across the considered benchmarks using only a fraction of the parameters and no pretraining whatsoever, showing that gnns are an attractive alternative for these transformer-based models. finally, using this method, we showed that transductive document classification can be performed successfully on a wide range of typologically diverse languages without any language model pretraining. in future work, we plan to explore the effectiveness of 2-nr on a large range of graphs, such as social networks, citation networks and product-user networks as well as adaptations of k-nr for @xmath6.