I ndo BERT weet: A Pretrained Language Model for I ndonesian T witter with Effective Domain-Specific Vocabulary Initialization (2021)

Abstract: We present IndoBERTweet, the first large-scale pretrained model for Indonesian Twitter that is trained by extending a monolingually-trained Indonesian BERT model with additive domain-specific vocabulary. We focus in particular on efficient model adaptation under vocabulary mismatch, and benchmark different ways of initializing the BERT embedding layer for new word types. We find that initializing with the average BERT subword embedding makes pretraining five times faster, and is more effective than proposed methods for vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based datasets.

## introduction transformer-based pretrained language models @xcite @xcite have become the backbone of modern nlp systems, due to their success across various languages and tasks. however, obtaining high-quality contextualized representations for specific domains/data sources such as biomedical, social media, and legal, remains a challenge. previous studies @xcite @xcite have shown that for domain-specific text, pretraining from scratch outperforms off-the-shelf bert. as an alternative approach with lower cost, @xcite demonstrated that domain adaptive pretraining (i.e. pretraining the model on target domain text before task fine-tuning) is effective, although still not as good as training from scratch. the main drawback of domain-adaptive pretraining is that domain-specific words that are not in the pretrained vocabulary are often tokenized poorly. for instance, in biobert @xcite , immunoglobulin is tokenized into {i, ##mm, ##uno, ##g, ##lo, ##bul, ##in}, despite being a common term in biology. to tackle this problem, @xcite ; @xcite proposed simple methods to domain-extend the bert vocabulary: @xcite initialize new vocabulary using a learned projection from word2vec @xcite , while @xcite use random initialization with weight augmentation, substantially increasing the number of model parameters. new vocabulary augmentation has been also conducted for language-adaptive pretraining, mainly based on multilingual bert (mbert). for instance, @xcite replace 99 "unused" wordpiece tokens of mbert with new common tokens in the target language, while wang et al. (2020) extend mbert vocabulary with nonoverlapping tokens (|v mbert -v new |). these two approaches use random initialization for new word-piece token embeddings. in this paper, we focus on the task of learning an indonesian bert model for twitter, and show that initializing domain-specific vocabulary with average-pooling of bert subword embeddings is more efficient than pretraining from scratch, and more effective than initializing based on word2vec projections @xcite . we use in-dobert @xcite , a monolingual bert for indonesian as the domain-general model to develop a pretrained domain-specific model in-dobertweet for indonesian twitter. there are two primary reasons to experiment with indonesian twitter. first, despite being the official language of the 5th most populous nation, indonesian is underrepresented in nlp (notwithstanding recent indonesian benchmarks and datasets @xcite ). second, with a large user base, twitter is often utilized to support policymakers, business @xcite , or to monitor elections @xcite or health issues @xcite . note that most previous studies that target indonesian twitter tend to use traditional machine learning models (e.g. n-gram and recurrent models @xcite ). to summarize our contributions: (1) we release indobertweet, the first large-scale pretrained indonesian language model for social media data; and (2) through extensive experimentation, we compare a range of approaches to domain-specific vocabulary initialization over a domain-general bert model, and find that a simple average of subword embeddings is more effective than previouslyproposed methods and reduces the overhead for domain-adaptive pretraining by 80%. ## experimental setup we accumulate gradients over 4 steps to simulate a batch size of 2048. when pretraining from scratch, we train the model for 1m steps, and use a learning rate of 1e-4 and the adam optimizer with a linear scheduler. all pretraining experiments are done using 4×v100 gpus (32gb). for domain-adaptive pretraining (using in-dobert model), we consider three benchmarks: (1) domain-adaptive pretraining without domainspecific vocabulary adaptation (v @xmath0; (2) applying the new vocabulary adaptation approaches from section 2.3 without additional domain-adaptive pretraining; and (3) applying the new vocabulary adaptation approaches from section 2.3 with 200k domain-adaptive pretraining steps. ## experimental results finally, we pretrain indobert @xcite for 200k steps in the target domain, after performing vocabulary adaptation. we see a strong improvement for all initialization methods, with the embedding averaging method once again per- ## discussion given these positive results on indonesian, we conducted a similar experiment in a second language, english: we follow nguyen et al. (2020) in adapting roberta foot_7 for twitter using the embedding averaging method to initialize new vocabulary, and compare ourselves against bertweet (trained from scratch on 845m english tweets). a caveat here is that bertweet (nguyen et al., 2020) and roberta @xcite use different tokenization methods: byte-level bpe vs. fastbpe @xcite . because of this, rather than replacing roberta's vocabulary with bertweet's (like our indonesian experiments), we train roberta's bpe tokenizer on english twitter data (described below) to create a domainspecific vocabulary. this means that the two models (bertweet and domain-adapted roberta model average roberta 72.9 bertweet (nguyen et al., 2020) 76.3 roberta + vocabulary adaptation + 200k steps of domain-adaptive pretraining 74.1 following nguyen et al. (2020), we download 42m tweets from the internet archive 10 over the period july 2017 to october 2019 (the first two days of each month), which we use for domain-adaptive pretraining. note that this pretraining data is an order of magnitude smaller than that of bertweet (42m vs. 845m). we use spacy 11 to filter english tweets, and follow the same preprocessing steps and downstream tasks as nguyen et al. (2020) (7 tasks in total; see the appendix for details). we pretrain roberta for 200k steps using the embedding averaging method. in lastly, we argue that the different tokenization settings between indobertweet and bertweet (ours) may also contribute to the difference in results. the differences include: (1) uncased vs. cased; (2) wordpiece vs. fastbpe tokenizer; and (3) vocabulary size (32k vs. 50k) between both models. in figure 1 , we present the frequency distribution of #subword of new types in both models after tokenizing by each general-domain tokenizer. interestingly, we find that bertweet has more new types than in-dobertweet, with #subword after tokenization being more varied (average length of #subword of new types are 2.6 and 3.4 for indobertweet and bertweet, respectively). ## conclusion we present the first large-scale pretrained model for indonesian twitter. we explored domain-adaptive pretraining with domain-specific vocabulary adaptation using several strategies, and found that the best method -averaging of subword embeddings from the original model -achieved the best average performance across 7 tasks, and is five times faster than the dominant paradigm of pretraining from scratch.