Linear Classifier: An Often-Forgotten Baseline for Text Classification (2023)

Abstract: Large-scale pre-trained language models such as BERT are popular solutions for text classification. Due to the superior performance of these advanced methods, nowadays, people often directly train them for a few epochs and deploy the obtained model. In this opinion paper, we point out that this way may only sometimes get satisfactory results. We argue the importance of running a simple baseline like linear classifiers on bag-of-words features along with advanced methods. First, for many text data, linear methods show competitive performance, high efficiency, and robustness. Second, advanced models such as BERT may only achieve the best results if properly applied. Simple baselines help to confirm whether the results of advanced models are acceptable. Our experimental results fully support these points.

## introduction text classification is an essential topic in natural language processing (nlp). like the situations in most nlp tasks, nowadays, large-scale pre-trained language models (plms) such as bert (devlin et al., 2019) have become popular solutions for text classification. therefore, we have seen that many practitioners directly run pre-trained language models with a fixed number of epochs on their text data. unfortunately, this way may only sometimes lead to satisfactory results. in this opinion paper, through an intriguing illustration, we argue that for text classification, a simple baseline like linear classifiers on bag-of-words features should be used along with the advanced models for the following reasons. while various settings of bag-of-words features such as bi-gram or tri-gram can be considered, we advocate that simple uni-gram tf-idf features trained by linear classifiers can be a useful baseline to start with for text classification. for our illustration, we re-investigate an existing work (chalkidis et al., 2022) that evaluates both linear svm and pre-trained language models, but the authors pay more attention to the latter. the linear method is somewhat ignored even though the performance is competitive on some problems. we carefully design experiments to compare the two types of methods. our results fully demonstrate the usefulness of applying linear methods as simple baselines. some recent works (e.g., yu et al., 2022; gomes et al., 2021) have shown the usefulness of linear classifiers in the deep-learning era. however, they either consider sophisticated applications or investigate advanced settings in which linear methods are only one component. in contrast, in this paper, we consider the basic scenario of text classification. a more related work (wahba et al., 2023) has demonstrated the effectiveness of linear classifiers over plms on some problems. however, our investigation on linear methods is more comprehensive. the discussion also reminds us the trade-off between performance gain and the cost including running time, model size, etc. simple methods are useful to benchmark and justify the usage of advanced methods. this paper is organized as follows. in section 2 we take a case study to point out the needs of considering linear methods as a baseline for text classification. we describe the linear and bert-based methods used for investigation in section 3. the experimental results and main findings are in section 4, while section 5 provides some discussion. additional details are in appendix. programs used for experiments are available at https://github.com/jameslyc88/ text_classification_baseline_code . ## text classification these days: some issues in applying training methods large plms have shown dramatic progress on various nlp tasks. in the practical use, people often directly fine-tune plms such as bert on their data for a few epochs. however, for text classification, we show that this way may not always get satisfactory results. some simple baselines should be considered to know if the obtained plm model is satisfactory. we illustrate this point by considering the work on legal document classification by chalkidis et al. (2022), which evaluates the following sets. we focus on text classification in this work, so casehold is not considered. for each problem, training and test sets are available. @xcite the study in chalkidis et al. (2022) comprehensively evaluates both bert-based plms and linear svms. they use micro-f1 and macro-f1 to measure the test performance. @xcite in ## settings for investigation to better understand the performance of linear models and bert-based plms, we simulate how people work on a new data set by training these methods. we consider a text classification package lib-multilabel 3 because it supports both types of train-ing methods. ## experimental results and analysis in ## discussion and conclusions in our experiments, we encounter an issue of whether to incorporate the validation set for training the final model, which is used for predicting the test set. for linear methods, we follow the common practice to include the validation set for obtaining the final model. however, for bert or some other deep learning models, the validation set is often used only for selecting the best epoch and/or the best hyper-parameters. to fully use the available data, we have investigated how to incorporate the validation set for bert. experimental results and more details are in appendix h. for some text sets evaluated in this work, we have seen that simple linear methods give competitive performance. the reason might be that each document in these sets is not short. @xcite then tf-idf features are sufficiently informative so that linear methods work well. across all nlp areas, an important issue now is when to use plms and when not. we demonstrate that when plms may not perform significantly better, traditional methods are much simpler and require fewer resources. however, having a simple quantitative measurement to pre-determine when to use which remains a challenging future research problem. in summary, the study reminds us of the importance of employing simple baselines in nlp applications. parameter libmultilabel chalkidis et al. (2022) data pre-processing (tf-idf feature generation) stop_words none english ngram_range (1, 1) (1, 3) min_df 1 5 max_features none [10000, 20000, 40000] model loss squared_hinge ['hinge', 'squared_hinge'] solving primal/dual primal dual c 1.0 [0.1, 1.0, 10.0]