GAINER : Graph Machine Learning with Node-specific Radius for Classification of Short Texts and Documents (2024)

Abstract: Graphs provide a natural, intuitive, and holistic means to capture relationships between different text elements in Natural Language Processing (NLP) such as words, sentences, and documents. Recent advancements in the field of Graph Machine Learning (GML) have led to the development of numerous models to process text for various natural language applications, including but not limited to short-text classification, document classification, and others.At the heart of GML models, specifically those based on Graph Neural Networks (GNNs), lies the message passing operation which has shown to be an essential component for strong empirical performance in NLP.However, the number of message passing steps (often known as the radius) is \textit{fixed for all the nodes} in existing GML models for NLP.Fixing the radius poses a fundamental restriction as nodes exhibit diverse properties and varying amounts of informative local structures in the input graph.This paper presents GAINER, a novel framework called Graph mAchine learnIng with Node-spEcific Radius, aimed at graph-based NLP. We propose non-neural and novel neural approaches built on the core ideas of GAINER.Through rigorous experimentation, we demonstrate the efficacy of GAINER in various popular NLP tasks.

## introduction graphs present a natural, intuitive, and holistic representation for understanding the interactions that exist among different text elements, such as words, sentences, and documents. the use of graphs provides a wide array of options for effectively representing and tackling different problems in natural language processing (nlp). for instance, worldlevel, sentence-level, and document-level graphs capture various aspects of text datasets. recent breakthroughs in graph machine learning (gml), notably driven by the progress made in graph neural networks (gnns) @xcite @xcite , have led to the development of numerous models tailored for processing text. diverse nlp applications span a wide range @xcite , including but not limited to short-text classification and document classification at the core of gnns, the message passing operation @xcite plays a pivotal role in achieving remarkable success in nlp @xcite . however, in popular gnn models, the number of message passing steps, often known as the radius, is predetermined and remains fixed for every node in the input graph. for instance, in a three-hop gnn, each node gathers information from nodes that are within a three-hop radius. fixing the number of hops (i.e. radius) poses a fundamental restriction as nodes exhibit diverse properties and varying amounts of informative local structures in the input graph. in an intuitive sense, nodes with poor connectivity tend to derive greater advantages from a higher radius, whereas well-connected nodes may require only a limited radius. a gnn with a very small radius may not propagate enough information, resulting in limited smoothing effects for certain nodes. on the other hand, a gnn with a very large radius may oversmooth the information @xcite , leading to reduced node-specific characteristics. the prevalent approach in gnns for nlp research, including very recent publications @xcite , involves the application of a 2-layer graph convolutional network @xcite . while this method performs adequately for nodes with strong connections, it struggles with nodes having limited or weak connections, such as low-degree nodes connected to other low-degree nodes. inspired by the aforementioned fundamental limitations of existing gml models in graph nlp, our work makes the following contributions: ## related work we divide the related work into three subsections. ## preliminaries we present notation to introduce the method and discuss problems studied in the paper. ## proposed framework: gainer in the aformentioned examples, the existing literature employs an l-layer gcn, which considers l-hop information around each node to propagate and smooth information across edges. the number of layers l, is considered a hyperparameter, and empirical results suggest that setting @xmath0. figure 1 visually illustrates the primary contribution of gainer and highlights the distinctions from 2-layer gcns, that are commonly used. ## experiments in this section, we empirically validate gainer's efficacy by conducting extensive experiments including baseline comparison, training timeaccuracy tradeoff, memory consumption, sensitivity analyses, etc. the accuracy comparisons are shown in the main text while the other experiments are in the appendix. the tasks considered are 1. inductive short-text classification, 2. document classification on attributed graphs, 3. document coherence assessment. we have utilised an nvidia titan rtx gpu for training all the models. the training specifics are described in the appropriate subsection dedicated to the given task. additional details regarding graph construction procedures, datasets, baselines, hyperparameters, and more are given in the appendix following the references. ## conclusion we have introduced gainer, a novel graph-based learning framework that assigns a dedicated radius to each node, controlling information propagation depth. we propose simple-gainer and neural-gainer for graph nlp to harness the power of graph structures to advance graph nlp research. extensive experiments on short text classification, document classification, and coherence assessment demonstrate the significance of gainer.