S oc BERT : A Pretrained Model for Social Media Text (2023)

Abstract: Pretrained language models (PLMs) on domain-specific data have been proven to be effective for in-domain natural language processing (NLP) tasks. Our work aimed to develop a language model which can be effective for the NLP tasks with the data from diverse social media platforms. We pretrained a language model on Twitter and Reddit posts in English consisting of 929M sequence blocks for 112K steps. We benchmarked our model and 3 transformer-based models—BERT, BERTweet, and RoBERTa on 40 social media text classification tasks. The results showed that although our model did not perform the best on all of the tasks, it outperformed the baseline model—BERT on most of the tasks, which illustrates the effectiveness of our model. Also, our work provides some insights of how to improve the efficiency of training PLMs.

## introduction in recent years, pretraining language models (plms) such as bert @xcite and roberta @xcite have proven to be effective for a wide range of natural language processing (nlp) tasks. domain adaptive pretraining (dapt), also known as pretraining on domain-specific data, has been a commonly employed approach to enhancing model performance on tasks that are specific to a particular domain @xcite . numerous efforts have been made to achieve this goal. for example, @xcite proposed biobert by pretraining bert on a large biomedical corpus of pubmed abstracts, and demonstrated that it outperformed bert on three representative biomedical text mining tasks. @xcite attempted to adapt pretrained models for clinical text by training biobert on clinical notes, resulting in the creation of bioclinical_bert @xcite . encouraged by the success of pretraining models in different domains, recent studies have developed pretrained models for social media nlp tasks. for example, @xcite built a model by further pretraining the model developed by @xcite by further training the model on english tweets. @xcite pretrained a transformer model named bertweet by training the model on a large scale of english tweets from scratch. however, these models only involve twitter data and may not be effective enough for social media data from other platforms such as reddit and facebook. to fill this gap, we trained a language model using both twitter and reddit data. we used 92gb text data including 20gb english tweets and 72gb reddit comments. our model was trained from scratch for 112k steps following the model architecture of roberta-base. for evaluation, we benchmarked our model and 3 transformer-based models-bert, bertweet, and roberta on 40 social media text classification tasks covering diverse health-related and non-health-related topics and from 6 social media platforms. the results showed that although our model did not perform the best on all of the tasks, it outperformed the baseline model-bert on most of the tasks. it showed that pretraining on the in-domain data can benefit the model on the downstream tasks. to sum up, our contributions are as follows: we call our final pretrained model socbert-an abbreviation of social media bert. ## benchmarking tasks we utilized a total of 40 social media text classification tasks to establish a benchmark, which represents the most extensive collection of social media text classification tasks currently available to us. manually annotated data for all these tasks were either publicly available or had been made available through shared tasks. the tasks covered diverse health-related and non-health-related topics including, but not limited to, adverse drug reactions (adrs) @xcite , cohort identification for breast cancer @xcite , non-medical prescription medication use (npmu) @xcite , informative covid-19 content detection @xcite , medication consumption @xcite , pregnancy outcome detection @xcite , symptom classification @xcite , suicidal ideation detection @xcite , identification of drug addiction and recovery intervention @xcite , signs of pathological gambling and selfharm detection @xcite , sentiment analysis and factuality classification in e-health forums (carrillo-de albornoz et al., 2018), offensive language identification @xcite , cyberbullying detection @xcite , sentiment analysis @xcite , and sarcasm language detection @xcite . the full details including the source, evaluation metric, training and test set sizes, the number of classes, and the inter-annotator agreement (iaa) for each task, if available, are shown in appendix a. seventeen tasks involved binary classification, 13 involved three-class classification, and 10 involved four-, five-, six-or nine-class classification each. the datasets combined included a total of 252,655 manually-annotated instances, with 204,989 (80%) instances for training and 47,666 (20%) for evaluation. the datasets involved data from multiple social media platforms-22 from twitter, 6 from medhelp foot_2 , 6 from reddit, 3 from facebook, 2 from youtube, and 1 from webmd foot_3 . for evaluation, we used the f 1 -score of the positive class for binary classification and the micro-averaged f 1 -score for other multi-class classification. ## discussion our work initially aimed to develop a plm which can efficiently work for the data from different social media platforms. however, the results showed that our model could not perform the best on all of the tasks compared to bertweet and roberta. the possible reason was that the training time of our model was not sufficient because of the limited computing resources. it revealed the dilemma for small labs in academia to develop large language models which has been studied since large language models became popular in the nlp field @xcite . however, our work can provide some insights for the nlp studies about developing and applying plms. first, training the model on a relatively small dataset for longer time might be more efficient than training the model on a large set of data for shorter time. second, pretraining the model on in-domain data may more efficiently improve the performance on downstream tasks than pretraining on out-of-domain data. also, the language models pretrained on sufficiently large open-domain data can be effective on domain-specific tasks. we released our model socbert-base foot_5 and socbert-final foot_6 via huggingface to help the nlp community conduct further studies in this field. ## conclusion in this work, we pre-trained a transformer-based model from scratch on social media data and benchmarked the model on 40 text classification tasks with social media data. although our model did not perform the best on all of the tasks, it outperformed the baseline model-bert on most of the benchmarking tasks. it showed that our model can be efficient for the text classification tasks with social media data. it may be possible to further improve the model performance if we continue training the model more efficiently. further work is required to improve the efficiency and reduce the cost of large language model training.