BERT weet: A pre-trained language model for E nglish Tweets (2020)

Abstract: We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having the same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al., 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. We release BERTweet under the MIT License to facilitate future research and applications on Tweet data. Our BERTweet is available at https://github.com/VinAIResearch/BERTweet

## introduction the language model bert @xcite the bidirectional encoder representations from transformers @xcite -and its variants have successfully helped produce new stateof-the-art performance results for various nlp tasks. their success has largely covered the common english domains such as wikipedia, news and books. for specific domains such as biomedical or scientific, we could retrain a domainspecific model using the bertology architecture @xcite @xcite . twitter has been one of the most popular microblogging platforms where users can share realtime information related to all kinds of topics and events. the enormous and plentiful tweet data has been proven to be a widely-used and real-time source of information in various important analytic tasks @xcite . note that the characteristics of tweets are generally different from those of traditional written text such as wikipedia and news articles, due to the typical short length of tweets and frequent use of informal grammar as well as irregular vocabulary e.g. abbreviations, typographical errors and hashtags @xcite . thus this might lead to a challenge in applying existing language models pretrained on large-scale conventional text corpora with formal grammar and regular vocabulary to handle text analytic tasks on tweet data. to the best of our knowledge, there is not an existing language model pre-trained on a large-scale corpus of english tweets. to fill the gap, we train the first large-scale language model for english tweets using a 80gb corpus of 850m english tweets. our model uses the bert base model configuration, trained based on the roberta pre-training procedure @xcite . we evaluate our model and compare it with strong competitors, i.e. roberta base and xlm-r base @xcite , on three downstream tweet nlp tasks: part-of-speech (pos) tagging, named-entity recognition (ner) and text classification. experiments show that our model outperforms roberta base and xlm-r base as well as the previous state-of-the-art (sota) models on all these tasks. our contributions are as follows: ## bertweet in this section, we outline the architecture, and describe the pre-training data and optimization setup that we use for bertweet. ## experimental setup we evaluate and compare the performance of bertweet with strong baselines on three downstream nlp tasks of pos tagging, ner and text classification, using benchmark tweet datasets. ## conclusion we have presented the first large-scale language model bertweet pre-trained for english tweets. we demonstrate the usefulness of bertweet by showing that bertweet outperforms its baselines roberta base and xlm-r base and helps produce better performances than the previous sota models for three downstream tweet nlp tasks of pos tagging, ner, and text classification (i.e. sentiment analysis & irony detection). as of september 2020, we have collected a corpus of about 23m "cased" covid-19 english tweets consisting of at least 10 and at most 64 word tokens. in addition, we also create an "uncased" version of this corpus. then we continue pre-training from our pre-trained bertweet on each of the "cased" and "uncased" corpora of 23m tweets for 40 additional epochs, resulting in two bertweet variants of pre-trained "cased" and "uncased" bertweet-covid19 models, respectively. by publicly releasing bertweet and its two variants, we hope that they can foster future research and applications of tweet analytic tasks, such as identifying informative covid-19 tweets @xcite or extracting covid-19 events from tweets @xcite .