Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP (2022)

Abstract: Graph neural networks have triggered a resurgence of graph-based text classification methods, defining today’s state of the art. We show that a wide multi-layer perceptron (MLP) using a Bag-of-Words (BoW) outperforms the recent graph-based models TextGCN and HeteGCN in an inductive text classification setting and is comparable with HyperGAT. Moreover, we fine-tune a sequence-based BERT and a lightweight DistilBERT model, which both outperform all state-of-the-art models. These results question the importance of synthetic graphs used in modern text classifiers. In terms of efficiency, DistilBERT is still twice as large as our BoW-based wide MLP, while graph-based models like TextGCN require setting up an \mathcal{O}(N^2) graph, where N is the vocabulary plus corpus size. Finally, since Transformers need to compute \mathcal{O}(L^2) attention weights with sequence length L , the MLP models show higher training and inference speeds on datasets with long sequences.

## introduction text categorization is the task of assigning topical categories to text units such as documents, social media postings, or news articles. research on text categorization is a very active field as just the sheer amount of new methods in recent surveys shows @xcite @xcite @xcite . there are approaches based on a bag of words (bow) that perform text categorization purely on the basis of a multiset of tokens. among them are deep averaging networks (dan) @xcite , a deep multi-layer perceptron (mlp) model with n layers that relies on averaging the bow, simple word embedding models (swem) @xcite that explores different pooling strategies for pretrained word embeddings, and fasttext @xcite , which uses a linear layer on top of pretrained word embed-dings. these models count the occurrence of all tokens in the input sequence, while disregarding word position and order, and then rely on word embeddings and fully connected feedforward layer(s). we call these bow-based models. among the most popular recent methods for text categorization are graph-based models such as textgcn @xcite that first induce a synthetic word-document co-occurence graph over the corpus and subsequently apply a graph neural network (gnn) to perform the classification task. besides textgcn, there are follow-up works like hetegcn @xcite , tensorgcn @xcite , and hypergat @xcite ), which we collectively call graph-based models. finally, there is the well-known transformer @xcite universe with models such as bert @xcite and its sizereduced variants such as distilbert @xcite . here, the input is a (fixed-length) sequence of tokens, which is then fed into multiple layers of self-attention. lightweight versions such as distil-bert and others @xcite use less parameters but operate on the same type of input. together with recurrent models such as lstms, we call these sequence-based models. in this paper, we hypothesize that text categorization can be very well conducted by simple but effective bow-based models. we investigate this research question in three steps: first, we conduct an in-depth analysis of the literature. we review the key research in the field of text categorization. from this analysis, we derive the different families of methods, the established benchmark datasets, and identify the top performing methods. we decide for which models we report numbers from the literature and which models we run on our own. overall, we compare 16 different methods from the families of bow-based models (8 methods), sequence-based models (3 methods), and graphbased models (5 methods). we run our own experi-ments for 7 of these methods on 5 text categorization datasets, while we report the results from the literature for the remaining methods. the result is surprising: our own bow-based mlp, called the widemlp, with only one wide hidden layer, outperforms many of the recent graphbased models for inductive text categorization @xcite @xcite . moreover, we did not find any reported scores for bert-based methods from the sequence-based family. thus, we fine-tuned our own bert @xcite and distilbert @xcite . these models set a new state of the art. on a metalevel, our study shows that mlps have largely been ignored as competitor methods in experiments. it seems as if mlps have been forgotten as baseline in the literature, which instead is focusing mostly on other advanced deep learning architectures. considering strong baselines is, however, an important means to argue about true scientific advancement @xcite . simple models are also often preferred in industry due to lower operational and maintenance costs. below, we introduce our methodology and results from the literature study. subsequently, we introduce the families of models in section 3. thereafter, we describe the experimental procedure in section 4. we present the results of our experiments in section 5 and discuss our findings in section 6, before we conclude. bow-based models classical machine learning models that operate on a bow-based input are extensively discussed in two surveys @xcite and other comparison studies @xcite . @xcite proposed dan, which combine word embeddings and deep feedforward networks. it is an mlp with 1-6 hidden layers, non-linear activation, dropout, and ada-grad as optimization method. the results suggest to use pretrained embeddings such as glove @xcite over a randomly initialized neural bag of-words @xcite as input. in fasttext @xcite a linear layer on top of pretrained embeddings is used for classification. furthermore, @xcite explore embedding pooling variants and find that swem can rival approaches based on recurrent (rnn) and convolutional neural networks (cnn). we consider fasttext, swem, and a dan-like deeper mlp in our comparison. note that those approaches that rely on logistic regression on top of pretrained word embeddings, e. g., fasttext, share a similar architecture as an mlp with one hidden layer. however, the standard training protocol involves pretraining the word embedding on large amounts of unlabeled text and then freezing the word embeddings while training the logistic regression @xcite . graph-based models using graphs induced from text for the task of text categorization has a long history in the community. an early work is the term co-occurrence graph of the keygraph algorithm @xcite . the graph is split into segments, representing the key concepts in the document. co-occurence graphs have also been used for automatic keyword extraction such as in rake @xcite and can be also used for classification @xcite . modern approaches exploit this idea in combination with graph neural networks (gnn) @xcite . examples of gnn-based methods operating on a word-document co-occurence graph are textgcn @xcite and its successor tensorgcn @xcite as well as hete-gcn @xcite , hypergat @xcite , and dadgnn @xcite ). we briefly discuss these models: in textgcn, the authors set up a graph based on word-word connections given by window-based pointwise mutual information and word-document tf-idf scores. they use a one-hot encoding as node features and apply a two-layer graph convolutional network @xcite on the graph to carry out the node classification task. hetegcn combines ideas from predictive text embedding @xcite and textgcn and split the adjacency matrix into its word-document and word-word sub-matrices and fuse the different layers' representations when required. tensorgcn uses multiple ways of converting text data into graph data including a semantic graph created with an lstm, a syntactic graph created by dependency parsing, and a sequential graph based on word co-occurrence. hypergat extended the idea of text-induced graphs for text classification to hypergraphs. the model uses graph attention and two kinds of hyperedges. sequential hyperedges represent the relation between sentences and their words. semantic hyperedges for word-word connections are derived from topic models @xcite . finally, dadgnn is a graph-based approach that uses attention diffusion and decoupling techniques to tackle oversmoothing of the gnn and to be able to stack more layers. in textgcn's original transductive formulation, the entire graph including the test set needs to be known for training. this may be prohibitive in practical applications as each batch of new documents would require retraining the model. when these methods are adapted for inductive learning, where the test set is unseen, they achieve notably lower scores @xcite . gnns for text classification use corpus statistics, e. g., pointwise mutual information (pmi), to connect related words in a graph @xcite . when these were omitted, the gnns would collapse to bag-of-words mlps. thus, gnns have access to more information than bow-mlps. glove @xcite also captures pmi corpus statistics, which is why we include an mlp on glove input representations. sequence models: rnn and cnn recurrent neural networks (rnn) are a natural choice for any nlp task. however, it turned out to be challenging to find numbers reported on text categorization in the literature that can be used as references. the bidirectional lstm with two-dimensional max pooling blstm-2dcnn @xcite has been applied on a stripped-down to 4 classes version of the 20ng dataset. thus, the high score of 96.5 reported for 4ng cannot be compared with papers applied on the full 20ng dataset. also text-rcnn @xcite , a model combining recurrence and convolution uses only the 4 major categories in the 20ng dataset. the results of text-rcnn are identical with blstm-2dcnn. for the mr dataset, blstm-2dcnn provides no information on the specific split of the dataset. rnn-capsule @xcite is a sentiment analysis method reaching an accuracy of 83.8 on the mr dataset, but with a different train-test split. @xcite combine a 2d-cnn with bidirectional rnn. another work applying a combination of a convolutional layer and an lstm layer is by @xcite . the authors experiment with five english and two chinese datasets, which are not in the set of representative datasets we identified. the authors report that their approach outperforms existing models like fasttext on two of the five english datasets and both chinese datasets. sequence models: transformers surprisingly, only few works consider transformer models for text categorization. a recent work shows that bert outperforms classic tf-idf bow approaches on english, chinese, and portuguese text classification datasets @xcite . we have not found any results of transformer-based models reported on those text categorization datasets that are commonly used in the graph-based approaches. therefore, we fine-tune bert @xcite and distilbert @xcite on those datasets ourselves. bert is a large pretrained language model on the basis of transformers. dis-tilbert @xcite is a distilled version of bert with 40% reduced parameters while retaining 97% of bert's language understanding capabilities. tinybert @xcite and mo-bilebert @xcite would be similarly suitable alternatives, among others. we chose dis-tilbert because it can be fine-tuned independently from the bert teacher. its inference times are 60% faster than bert, which makes it more likely to be reusable by labs with limited resources. summary from our literature survey, we see that all recent methods are based on graphs. bow-based methods are hardly found in experiments, while, likewise surprisingly, transformer-based sequence models are extremely scarce in the literature on topical text categorization. the recent surveys on text categorization include both classical and deep learning models, but none considered a simple mlp except for the inclusion of dan @xcite in @xcite . ## models for text categorization we formally introduce the three families of models for text categorization, namely the bow-based, graph-based, and sequence-based models. ## conclusion we argue that a wide multi-layer perceptron enhanced with today's best practices should be considered as a strong baseline for text classification tasks. in fact, the experiments show that our widemlp is oftentimes on-par or even better than recently proposed models that synthesize a graph structure from the text.