Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification (2020)

Abstract: Pre-trained language models provide the foundations for state-of-the-art performance across a wide range of natural language processing tasks, including text classification. However, most classification datasets assume a large amount labeled data, which is commonly not the case in practical settings. In particular, in this paper we compare the performance of a light-weight linear classifier based on word embeddings, i.e., fastText (Joulin et al., 2017), versus a pre-trained language model, i.e., BERT (Devlin et al., 2019), across a wide range of datasets and classification tasks. In general, results show the importance of domain-specific unlabeled data, both in the form of word embeddings or language models. As for the comparison, BERT outperforms all baselines in standard datasets with large training sets. However, in settings with small training datasets a simple method like fastText coupled with domain-specific word embeddings performs equally well or better than BERT, even when pre-trained on domain-specific data.

## introduction language models pre-trained on large amounts of text corpora form the foundation of today's nlp @xcite . they have proved to provide state-of-the-art performance against most standard nlp benchmarks @xcite . however, these models require large computational resources that are not always available and have important environment implications @xcite . moreover, there is limited research in the applicability of pre-trained models in classification tasks with small amount of labelled data. some related studies @xcite @xcite investigate whether it is helpful to tailor a pre-trained model to the domain while others @xcite @xcite analyse methods for fine-tuning bert to a given task. however, these studies perform evaluation on a limited range of datasets and classification models and do not consider scenarios with limited amounts of training data. in particular, this paper aims to estimate the role of labeled and unlabeled data for supervised text classification. our study is similar to @xcite where they investigate whether it is still helpful to tailor a pre-trained model to the domain of a target task. in this paper, however, we focus our evaluation on text classification and compare different types of classifiers on different domains (social media, news and reviews). unlike other tasks such as natural language inference or question answering that may require a subtle understanding, feature-based linear models are still considered to be competitive in text classification @xcite . however, to the best of our knowledge there has not been an extensive comparison between such methods and newer pre-trained language models. to this end, we compare the light-weight linear classification model fasttext @xcite , coupled with generic and corpus-specific word embeddings, and the pre-trained language model bert @xcite , trained on generic data and domain-specific data. specifically, we analyze the effect of training size over the performance of the classifiers in settings where such training data is limited, both in few-shot scenarios with a balanced set and keeping the original distributions. in both cases, our results show that a large pre-trained language model may not provide significant gains over a linear model that leverage word embeddings, especially when these belong to the given domain. ## experimental setting datasets. for our experiments we selected a suite of datasets with different domains and nature. these are: semeval 2016 task on sentiment analysis @xcite , semeval 2018 task on emoji predic-tion @xcite , ag news @xcite , newsgroups @xcite and imdb @xcite . the main features and statistics of each dataset are summarized in comparison models. as mentioned in section 2, our evaluation is focused on fasttext @xcite and bert @xcite . for completeness we include a simple baseline based on frequency-based features and a suite of classification algorithms available in the scikit-learn library @xcite , namely gaussian naive bayes (gnb), logistic regression and support vector machines (svm). of the three, the best results were achieved using logistic regression, which is the model we include in this paper as a baseline for our experiments. training. as pre-trained word embeddings we downloaded 300-dimensional fasttext embeddings trained on common crawl @xcite . in order to learn domain-specific word embedding models we used the corresponding training sets for each dataset, except for the twitter datasets where we leveraged an existing collection of unlabeled tweets from october 2015 to july 2018 to train 300-dimensional fasttext embeddings @xcite . word embeddings are then fed as input to a fasttext classifier where we used default parameters and softmax as the loss function. as for bert, we fine-tune it for the classification task using a sequence classifier, a learning rate of 2e-5 and 4 epochs. in particular, we made use of bert's hugging face default transformers implementation for classifying sentences @xcite and the hierarchical principles described in @xcite for pre-processing long texts before feeding them to bert. we used the generic base uncased pre-trained bert model and bert-twitter foot_1 , both from hugging face @xcite . evaluation metrics. we report results based on standard micro and macro averaged f1 @xcite . in our setting, since system provide outputs for all instances, micro-averaged f1 is equivalent to accuracy. ## analysis we perform two main types of analysis. first, we look at the effect of training size over the classifier's performance by randomly sampling different sized subsets from the original labeled datasets (section 4.1). then, we perform a few-shot experiment where we compare classifier's performance on different sizes of balanced subsets of the training data (section 4.2). ## conclusion and future work in this paper, we analyzed the role of training and unlabeled domain-specific data in supervised text classification. we compared both linear and neural models based on transformer-based language models. in settings with small training data, a simple method such as fasttext coupled with domain-specific word embeddings appear to be more robust than a more data-consuming model such as bert, even when bert is pre-trained on domain-relevant data. however, the same classifier with generic pretrained word embeddings does not perform consistently better than a traditional frequency-based linear baseline. 4 bert, pre-trained on domain-specific data (i.e., twitter) leads to improvements over generic bert, especially for few-shot experiments. for future work it would be interesting to further delve into the role of unlabeled data in text classification, both in terms of word embeddings (e.g., by making use of meta-embeddings @xcite ) and the data used to train language models @xcite . moreover, this quantitative analysis could be extended to more classification tasks and different models, e.g., larger language models such as roberta @xcite and @xcite , which appear to be more suited to few-shot experiments.