Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media (2020)

Abstract: Recent studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when models are pretrained on in-domain data. Often, the pretraining data used in these models are selected based on their subject matter, e.g., biology or computer science. Given the range of applications using social media text, and its unique language variety, we pretrain two models on tweets and forum text respectively, and empirically demonstrate the effectiveness of these two resources. In addition, we investigate how similarity measures can be used to nominate in-domain pretraining data. We publicly release our pretrained models at https://bit.ly/35RpTf0 .

## introduction sequence transfer learning @xcite , that pretrains language representations on unlabeled text (source) and then adapts these representations to a supervised task (target), has demonstrated its effectiveness on a range of nlp tasks @xcite @xcite . approaches vary in model, pretraining objective, pretraining data and adaptation strategy. we consider a widely used method, bert @xcite . it pretrains a transformer-based model using a masked language model objective and then fine-tunes the model on the target task. we investigate the impact of the domain (i.e., the similarity between the underlying distribution of source and target data) of pretraining data on the effectiveness of pretrained models. we also propose a cost-effective way to select pretraining data. recent studies on domain-specific bert models, which are pretrained on specialty source data, empirically show that, when in-domain data is used for pretraining, target task performance can be improved @xcite @xcite . these publicly available domain-specific bert models are valuable to the nlp community. however, the selection of in-domain data usually resorts to intuition, which varies across nlp practitioners @xcite . according to @xcite , the context specific usage of language is affected by three factors: field (the subject matter being discussed), tenor (the relationship between the participants in the discourse and their purpose) and mode (communication medium, e.g., 'spoken' or 'written'). 1 generally, the selection of pretraining data in existing domain-specific bert models is based on the field rather than the tenor. for example, biobert @xcite and scib-ert @xcite are both pretrained on scholar articles, but on different fields (biology and computer science). we conduct a case study of pretraining bert on social media text which has very different tenor from existing domain-specific bert models. our contributions are two-fold: (1) we release two pretrained bert models trained on tweets and forum text, and we demonstrate the effectiveness of these two resources on a range of nlp data sets using social media text; and, (2) we investigate the correlation of source-target similarity and task accuracy using different domain-specific bert models. we find that simple similarity measures can be used to nominate in-domain pretraining data (figure 1 ). ## related work selecting data to pretrain bert there are two known strategies: (1) collecting very large generic data, such as web crawl and news @xcite @xcite ; and, (2) selecting in-domain data, which we refer to as figure 1 : recent studies have demonstrated the effectiveness of domain-specific bert models. however, the selection of in-domain data usually resorts to intuition, which varies across nlp practitioners, especially regarding intersecting domains. we investigate the correlation of source-target similarity and the effectiveness of pretrained models. in other words, we aim to use simple similarity measures to nominate in-domain pretraining data. ## pretraining bert models we follow the practices used in other domainspecific bert models @xcite to pretrain our bert models. we use the original vocabulary of bert-base as our ## effectiveness of pretrained bert models to evaluate the effectiveness of our pretrained bert models, we experiment on a range of classification and named entity recognition (ner) data sets. both text classification and ner are fundamental nlp tasks that can employ generic architectures on top of bert. for the classification task, the representation of the first token (i.e., @xcite ) is fed into the output layer for the final prediction. for the ner task, the representations of the first sub-token within each token are taken as input to a token-level classifier to predict the token's tag. we did not explore more complex architectures, such as adding lstm or crf on top of bert @xcite , because our aim is to demonstrate the efficacy of domain-specific bert models and to observe the impact of pretraining data, rather than to achieve state-of-the-art performance on these data sets. our bert results follow the standard twostage approach of finetuning the pretrained model. domain-specific berts add a stage in the middle: finetuning bert on domain-specific unlabeled data (cf. figure 1 ). ## analysis after we empirically show the importance of selecting in-domain source data, the next question is: can we find a cost-effective way to nominate in-domain source data? ## summary we conduct a case study of pretraining bert on social media text. through extensive experiments, we show the importance of selecting in-domain source data. based on empirical analysis, we recommend measures to help select pretraining data for best performance on new applications.