Bernice: A Multilingual Pre-trained Encoder for T witter (2022)

Abstract: The language of Twitter differs significantly from that of other domains commonly included in large language model training. While tweets are typically multilingual and contain informal language, including emoji and hashtags, most pre-trained language models for Twitter are either monolingual, adapted from other domains rather than trained exclusively on Twitter, or are trained on a limited amount of in-domain Twitter data.We introduce Bernice, the first multilingual RoBERTa language model trained from scratch on 2.5 billion tweets with a custom tweet-focused tokenizer. We evaluate on a variety of monolingual and multilingual Twitter benchmarks, finding that our model consistently exceeds or matches the performance of a variety of models adapted to social media data as well as strong multilingual baselines, despite being trained on less data overall.We posit that it is more efficient compute- and data-wise to train completely on in-domain data with a specialized domain-specific tokenizer.

## introduction bert @xcite has become a core part of modern natural language processing (nlp) pipelines. performance on downstream tasks depends on the quality of the included bert-style model, driven in large part by model size, vocabulary, and relevance of pre-training data. studies have shown that pre-training data should match the intended deployment domain to achieve optimal task performance @xcite @xcite @xcite . one of the most commonly studied domains for nlp research is twitter, which has led to numerous downstream applications. the unique nature of twitter data has prompted specialized applications, including sentiment prediction around elections and vaccines @xcite , stance prediction on gun control @xcite , civil unrest forecasting @xcite @xcite , and various mental health applications @xcite . while several different bert-style models have been developed for twitter @xcite @xcite , compared to bert models for other domains, these twitter-based models are smaller, use much less training data, and are often adapted from other domains instead of for twitter initially. this is especially problematic for the tokenizer, creating a mismatch that limits its efficacy on representing twitter data. furthermore, these models are primarily for english, despite the depth of multilingual work on twitter. we introduce bernice, the first multilingual pretrained encoder trained exclusively on twitter data with a custom tokenizer. 1 bernice uses more twitter data than most bert style models to date (2.5 billion tweets) foot_1 and utilizes a tokenizer trained exclusively on twitter data. we evaluate bernice on monolingual and mul-tilingual twitter benchmarks: tweeteval @xcite , unified multilingual sentiment analysis benchmark (umsab) @xcite , and multilingual hate speech @xcite . we also evaluate bernice's coverage on low-resource and twitter-specific language (e.g., hashtags) through an analysis of our tokenizer. we show that across these tasks, bernice does better than comparable models despite having much less data overall. 3 we discuss our results' implications for training in-domain models from scratch versus adapting out-of-domain models. ## background: twitter bert models several different bert models have been trained on twitter data. in this work we compare to @xcite , xlm-t @xcite , twhin-bert @xcite , and twhin-bert-mlm @xcite . while not a twitter-specific model, we also include xlm-r @xcite to compare to a model without any twitter pre-training. 6 we provide a brief overview and compare them with respect to multilinguality, pre-training data, and tokenizers, (see besides bertweet, which is trained on 850m english tweets, all comparison models are multilingual. also, all models besides xlm-r have been pre-trained on tweets. xlm-t is unique because @xcite continued training the last xlm-r base checkpoint on 198m multilingual tweets. thus, out of all the twitter models, xlm-t is domain-adapted to twitter and not trained completely on in-domain data. xlm-t has also seen the most data overall, when taking into consideration xlm-r's pre-training on 2.5tb of common-crawl (cc-100). xlm-r has seen the second-most amount of data, followed by both twhin-bert models (referred to as twhin-bert*) with 7b tweets, and then bernice with 2.5b tweets. the twhin-bert model is also notable because of its combined novel social pre-training objective and roberta mlm objective. twhin-bert-mlm is the same as twhin-bert, except only trained with the standard mlm objective. we compare bernice against both models in section 4 to evaluate the effects of more twitter pre-training data and the social objective. bertweet and bernice are the only models with tokenizers trained on tweets. xlm-t and twhin-bert* models use the xlm-r tokenizer, which was trained on out-of-domain common-crawl data and has the same vocabulary size as bernice (250k tokens), but covers 100 languages versus 66. 7 while trained on tweets, bertweet is monolingual and only has a vocabulary size of 64k tokens. in §4, we explore the difference in token coverage between the xlm-r and bernice tokenizers, and how it could impact downstream task performance. in addition, bernice, bertweet, and twhin-bert* models all have a smaller max sequence length to accommodate the short nature of tweets. their max sequence length is 128, a quarter of the length of xlm-t/xlm-r's max length of 512, foot_7 which enables faster training and inference. in summary, bernice is the only multilingual model trained from scratch on twitter data with a custom tokenizer. ## evaluation we benchmark bernice's performance on three datasets: tweeteval, umsab, and hate speech. we also compare tokenizer coverage of hashtags and emoji, which are unique to twitter. given our multilingual focus, we compare to xlm-t @xcite , xlm-r @xcite , twhin-bert @xcite , and twhin-bert-mlm @xcite which were previously shown to have competitive performance to monolingual english models for tweets @xcite . ## discussion across three benchmark datasets bernice consistently outperforms or matches its closest competitors xlm-t and twhin-bert-mlm, providing the community with a pre-trained multilingual encoder exclusively for twitter. what have we learned through the development of this model? first, bernice was much cheaper (faster) to train than its closest competitors. for xlm-t, @xcite continued training from the final xlm-r checkpoint by adding 198m multilingual tweets. while roughly 12x less twitter data than bernice, it is far more overall data as it includes the 2.5tb of cc-100 pre-training data for xlm-r @xcite . the total cost in terms of data and compute to get to a better result was less for bernice. we attribute this to the efficiency of in-domain pre-training data and smaller context size. despite using orders of magnitude less data, we have enough twitter data to learn a better model than through adaptation. a similar story is seen for the twhin-bert* models, where @xcite trained a model from scratch on 7b tweets. while still considerably less data than for xlm-t, twhin-bert* models were trained on almost 3x the number of tweets as for bernice. despite this large increase in in-domain data, twhin-bert* models perform similarity to bernice on the benchmarks evaluated here. this similar performance could be due to bernice's twitter-specific tokenizer. second, we made modeling choices for twitter specifically. we followed nguyen et al. ( 2020 ) and used a max sequence length of 128 subwords; longer contexts (512 for xlm-t) are unnecessary for individual tweets (maximum 280 characters). the limited context led to more efficient training and inference. 12 our other modeling choice was in how we developed the vocabulary. we demonstrated in section 4.4 that bernice's tokenizer is better suited for twitter data than xlm-r's, which is used by xlm-t and twhin-bert* models. 13 finally, we purposely used a random selection of tweets for pre-training, rather than selection by location/topic, to produce a general purpose multilingual twitter encoder with utility for as many tasks as possible. we encourage users to fine-tune on specific tasks and to continue pre-training on relevant twitter data @xcite ). foot_12 ## conclusion despite the prominence of nlp for twitter data, there are few language models designed especially for this domain. bernice is the first bert style multilingual model trained from scratch for twitter on 2.5b tweets with a custom tokenizer. bernice outperforms comparative models with significantly less overall training by focusing exclusively on indomain twitter pre-training data.