Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification using Graph Neural Networks? (2023)

Abstract: Given the success of Graph Neural Networks (GNNs) for structure-aware machine learning, many studies have explored their use for text classification, but mostly in specific domains with limited data characteristics. Moreover, some strategies prior to GNNs relied on graph mining and classical machine learning, making it difficult to assess their effectiveness in modern settings. This work extensively investigates graph representation methods for text classification, identifying practical implications and open challenges. We compare different graph construction schemes using a variety of GNN architectures and setups across five datasets, encompassing short and long documents as well as unbalanced scenarios in diverse domains. Two Transformer-based large language models are also included to complement the study. The results show that i) although the effectiveness of graphs depends on the textual input features and domain, simple graph constructions perform better the longer the documents are, ii) graph representations are especially beneficial for longer documents, outperforming Transformer-based models, iii) graph methods are particularly efficient for solving the task.

## introduction document comprehension involves interpreting words that can alter the meaning of the text based on their placement. for example, in the sentence "the movie was boring, but i was surprised by the ending", the word but contrasts ideas. while traditional vector-based text representation methods lack the ability to capture the structural information of a text effectively, graph-based representation strategies explicitly seek to model relationships among different text elements (nodes) through associations between pairs of them (edges), capturing dependencies between text units and leveraging language structure. while such ideas have a long history @xcite , the rise of graph neural network (gnn) models in recent years has made it particularly appealing to convert even unstructured data into graphs. the model can then capture relevant patterns while accounting for dependencies between graph nodes via message passing for text classification, numerous graph-based text representation schemes have been proposed and demonstrated the efficacy of graphs. however, most of them were designed for particular domainspecific tasks and validated only on short documents using a restricted set of model architectures @xcite @xcite . moreover, some of these proposals predate the introduction of gnns and were validated using graph mining or classical machine learning models, making it challenging to determine the applicability and effectiveness of graphs in broader settings @xcite . text classification increasingly extends beyond simple topic classification tasks, encompassing real-world challenges such as noisy texts, imbalanced labels, and much longer documents consisting of more than just a few paragraphs. hence, a comprehensive assessment of the merits and drawbacks of different graph representations and methods in more diverse scenarios is needed. this work presents a thorough empirical investigation of previously proposed graph-based text representation methods, evaluating how graphs generalize across diverse text classification tasks. we analyze their effectiveness with several gnn-based architectures and setups across five prominent text classification datasets from a broad range of domains. unlike previous work @xcite , our study considers diverse datasets with both short and longer documents, as well as unbalanced classification scenarios. additionally, we evaluate the efficacy vs. efficiency of the proposals, an aspect usually neglected in previous studies. for each graph method, we conducted extensive experiments using 3 types of convolutional layers as different message-passing strategies for 12 gnn architecture variants, each using one out of 4 pretrained word embedding techniques as node feature vector initialization. this allows us to shed light on what are the most successful choices of gnn architectures for learning from them. our study finds that graph methods are a competitive and particularly efficient choice for solving classification tasks. this is because gnns can capture both local and global dependencies between structural components. therefore, they can capture rich semantic relationships and dependencies that are important for the task. additionally, unlike many sequence models, gnns can naturally handle variable-length inputs by operating on the graph structure, without any need to map every data sample to a fixed-sized vector or truncate them at a fixed maximum sequence length. while longer documents can be particularly challenging, our study finds that gnn methods hold particular promise for longer documents, an aspect unexplored in prior research. however, the graph's effectiveness depends on the textual input features and domain. based on our experimental results, we provide a discussion regarding what graph construction and gnn architecture choice is preferable depending on the task to be solved. surprisingly, although transformer-based language models (lms) yield outstanding results for the considered tasks, they often have difficulties converging when dealing with short texts. the study is structured around three research questions, which are discussed in section 4: ## prior work on graphs in nlp previous graph-based text representation methods can be categorized into three categories based on the nature of the underlying graph structure. early graph constructions primarily relied on cooccurrence and textual statistical patterns. subsequently, more advanced representations integrated linguistic features as graph components. recently, specialized graph constructions have evolved, entailing intricate structures that encompass the uti-lization of graph neural networks as essential components within the learning framework. ## comparing graph-based text representations to study the merits of prominent graph-based text representation strategies, we conducted comprehensive experiments on five well-known text classification datasets. for each task, we compare different graph construction schemes using a variety of gnn models to separate the effect of the graph construction strategy from that of the message-passing technique in the model. ## results and analysis a comparison with baselines such as bert is given in ## conclusion we present an empirical analysis of graph representations for text classification by comprehensively analyzing their effectiveness across several gnn architectures and setups. the experiments consider a heterogeneous set of five datasets, encompassing short and long documents. the results show that the strength of graph-based models is closely tied to the textual features and the source domain of documents. thus, the choice of nodes and edges is found to be crucial. despite this, intuitive graphs are shown to be a strong option, reaching competitive results across all considered tasks, especially for longer documents, exceeding those of bert and longformer. additionally, we observed that pre-trained static word embeddings, instead of bert vectors, allow reaching outstanding results on some tasks. we are enthusiastic about extending our study to further tasks in future work. to this end, we are releasing our code on github foot_3 and hope that it can grow to become a community resource. additionally, we will expand this study by exploring approaches for learning the graph structure to eliminate the need for picking a design manually, being less domain-dependent.