Benchmarking of Transformer-Based Pre-Trained Models on Social Media Text Classification Datasets (2020)

Abstract: Free text data from social media is now widely used in natural language processing research, and one of the most common machine learning tasks performed on this data is classification. Generally speaking, performances of supervised classification algorithms on social media datasets are lower than those on texts from other sources, but recently-proposed transformer-based models have considerably improved upon legacy state-of-the-art systems. Currently, there is no study that compares the performances of different variants of transformer-based models on a wide range of social media text classification datasets. In this paper, we benchmark the performances of transformer-based pre-trained models on 25 social media text classification datasets, 6 of which are health-related. We compare three pre-trained language models, RoBERTa-base, BERTweet and ClinicalBioBERT in terms of classification accuracy. Our experiments show that RoBERTa-base and BERTweet perform comparably on most datasets, and considerably better than ClinicalBioBERT, even on health-related datasets.

## introduction transformer-based pre-trained language models have proven to be effective for many natural language processing (nlp) tasks, such as text classification and question answering, and they have enabled systems to outperform previous state-ofthe-art approaches. a prime example of such language representation models is bidirectional encoder representations from transformers (bert), which was pre-trained on the book corpus and english wikipedia @xcite . since it was proposed, many efforts have attempted to improve upon it, and common strategies for doing so are to use more data and train longer @xcite , or to pre-train from scratch on domain-specific data @xcite . multiple variants of transformerbased models have been proposed, but there is currently limited information available about how the variants directly compare on a set of similar tasks. in this paper, we focus on text from a specific source, namely, social media, and the common task of text classification. we compare the performances of three pre-training methods. we chose text classification as our target task because it is perhaps the most common nlp-related machine learning task, and most of the publicly-available annotated datasets were prepared for it. we included 25 social media classification datasets, 6 of which are health-related. we compared three transformerbased models-roberta-base @xcite , bertweet @xcite , and clinicalbio-bert @xcite . our experiments show that roberta-base and bertweet perform comparably and are considerably better than clin-icalbiobert. in addition to comparing the performances of the models on all the datasets, we analyzed the differences in performances between domain-specific (medical), source-specific (social media), and generic pre-trained models. our empirical analyses suggest that roberta-base can capture general text characteristics, while bertweet can capture source-specific knowledge, and pretraining on large-scale source-specific data can improve the capabilities of models to capture general text features, potentially benefiting downstream source-specific tasks. ## related work the most relevant and recent related works are those by @xcite and @xcite . @xcite proposed the biomedical language understanding evaluation (blue) benchmark for the biomedical domain. the evaluations include five tasks with ten datasets covering both biomedical and clinical texts. the specific tasks in-clude named entity recognition, text classification and relation extraction. @xcite proposed the biomedical language understanding and reasoning benchmark (blurb) for pubmed-based biomedical nlp applications, with 13 biomedical nlp datasets in six tasks. to the best of our knowledge, there is no existing work that attempts to perform similar benchmarking for transformer-based approaches on social media data, and the results reported in this paper follow on the footsteps of the benchmarks referenced above. recent attempts at adaptation of transformerbased models are also relevant to our current work, since we wanted to include a domain-adapted and a source-adapted model in our comparisons. many domain adaptation efforts have been reported in the literature. biobert-generated by pre-training bert on biomedical corpora (e.g., pubmed abstracts)-was demonstrated to outperform bert on three representative biomedical text mining tasks @xcite . @xcite attempted to further adapt pre-trained models for clinical text by training biobert on clinical notes, resulting in the clinicalbiobert model. we included clinicalbiobert as an example of a domain-adapted pre-trained model in our comparisons. for source-adaptation (social media text), @xcite proposed bertweet by pretraining bert on a large set of english tweets. we include bertweet in our comparisons as an example of a source-adapted model. ## results and discussion the difference between two models is regarded as statistically significant if the p-@xmath0.05. bertweet achieves the highest accuracies on 16 out of 25 datasets, including health and non-healthrelated datasets from twitter, facebook, reddit, and youtube. the fact that bertweet performs well on non-tweet datasets suggests that bertweet can learn some universal characteristics of social media languages by pre-training on tweets. 3 on 5 datasets (specifically, smm4h-17-task2, trac-1-2, sarcasm-2, crowdflower, and semeval-18-j), the best-performing system obtained significantly better results than the next best system, and in all these cases, bertweet was the winner. there are, however, no significant differences between roberta-base and bertweet on most datasets, which shows that roberta-base can capture general text features and work well on social media tasks. the differences in the pretraining dataset sizes for roberta-base (160 gb) and bertweet (80 gb) suggest that pre-training on relatively small source-specific data may effectively benefit the downstream source-specific tasks. figure 2 visually illustrates the distribution of the accuracy scores and their 95% confidence intervals for all three models on our included datasets. from the figure, the relative underperformance of the clinicalbiobert is evident. clinicalbiobert does not appear to capture social media-specific characteristics of the data even for health-related classification datasets, although it is trained on clinical notes. this finding suggests that for social media-specific health-related research tasks, it might be better to choose a source-specific pretrained model (e.g., bertweet for social media) rather than a domain-specific one. it is possible that the gap between the language of clinical notes and social media text is large enough to negatively impact the social media text representation capability of the encoder. moreover, clinicalbiobert is pre-trained on tweets (twitter bert) on 3 tweet classification tasks. ## conclusion we benchmarked the performances of three transformer-based pre-trained models on 25 social media text classification datasets. we found that roberta-base and bertweet perform similarly on most datasets, consistently outperforming clinicalbiobert, even for health-related tasks. our experiments suggest that for social mediabased classification tasks, it might be best to use pre-trained models generated from large social media text. it might be possible to further improve the performance of bertweet by incorporating data from multiple social networks. dataset description source adr detection detect adverse reaction (adr) mentioned from text sarker and gonzalez (2015) breastcancer detect breast cancer patients based on their self-reports sarker et al. (2020) pm abuse identify prescription medication (pm) abuse on tweets ali al-garadi et al. (2020) 4 smm4h-17-task1 detect adverse reaction (adr) mentioned from text sarker et al. (2018) smm4h-17-task2 identify medication consumption from medication-mentioning tweets wnut-20-task2 identify informative covid-19 related tweets nguyen et al. (2020b) olid-1 identify offensive language from tweets zampieri et al. (2019) olid-2 olid-3 trac-1-1 detect aggressive information in social media kumar et al. (2018) trac-1-2 trac-2-1 detect aggressive language on social media text bhattacharya et al. (2020) trac-2-2 sarcasm-1 binary emotion classification of sarcasm ghosh et al. (2020) sarcasm-2 crowdflower multiclass emotion classification web foot_1 fb-arousal-1 classify the level of arousal preoţiuc-pietro et al. (2016) fb-arousal-2 fb-valence-1 classify the level of valence fb-valence-2 semeval-18-a emotion intensity ordinal classification of anger mohammad et al. (2018) semeval-18-f emotion intensity ordinal classification of fear semeval-18-j emotion intensity ordinal classification of joy semeval-18-s emotion intensity ordinal classification sadness semeval-18-v valence ordinal classification table a.1: data descriptions. data and systems for medication-related text classification and concept normalization from twitter: insights from the social media mining for health (smm4h)-2017 shared task. journal of the american medical informatics association, 25(10):1274-1283. abeed sarker and graciela gonzalez. 2015. portable automatic text classification for adverse drug reaction detection via multi-corpus training. j. of biomedical informatics, 53(c):196-207. zhilin yang, zihang dai, yiming yang, jaime carbonell, russ r salakhutdinov, and quoc v le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in h. wallach, h. larochelle, a. beygelzimer, f. dalché-buc, e. fox, and r. garnett, editors, advances in neural information processing systems 32, pages 5753-5763. curran associates, inc. marcos zampieri, shervin malmasi, preslav nakov, sara rosenthal, noura farra, and ritesh kumar. 2019. predicting the type and target of offensive posts in social media. in proceedings of naacl.