Domain adaptation challenges of BERT in tokenization and sub-word representations of Out-of-Vocabulary words (2020)

Abstract: BERT model (Devlin et al., 2019) has achieved significant progress in several Natural Language Processing (NLP) tasks by leveraging the multi-head self-attention mechanism (Vaswani et al., 2017) in its architecture. However, it still has several research challenges which are not tackled well for domain specific corpus found in industries. In this paper, we have highlighted these problems through detailed experiments involving analysis of the attention scores and dynamic word embeddings with the BERT-Base-Uncased model. Our experiments have lead to interesting findings that showed: 1) Largest substring from the left that is found in the vocabulary (in-vocab) is always chosen at every sub-word unit that can lead to suboptimal tokenization choices, 2) Semantic meaning of a vocabulary word deteriorates when found as a substring in an Out-Of-Vocabulary (OOV) word, and 3) Minor misspellings in words are inadequately handled. We believe that if these challenges are tackled, it will significantly help the domain adaptation aspect of BERT.

## introduction bert is one of the prominent models used for a variety of nlp tasks. with the masked language model (mlm) method, it has been successful at leveraging bidirectionality while training the language model. the bert-base-uncased model has 12 encoder layers, with each layer consisting of 12 self-attention heads. the word representations are context-dependent 768 dimensional dynamic embeddings. in order to leverage the learnings of such pre-trained networks, fine tuning is commonly done while building nlp applications in industries.the bert-base-uncased vocabulary has a size of 30522 with only 994 unused slots (in comparison, bert-base-cased has only 101 unused slots). while the unused slots in the vocabulary can be used to include domain specific words, the representations of these will have to be fine tuned with domain specific corpus before they can be utilized. hence, it is essential that the tokenization algorithm performs well to handle domain specific oov words. bert relies on the wordpiece algorithm @xcite to create the vocabulary, that chooses those sub-word units for the vocabulary towards maximising the language model likelihood. however, the tokenization using this vocabulary is not done semantically. this leads to a poor tokenization that induces a semantic information loss in terms of dealing with oov words for domain centric downstream tasks. while the largest substring tokenization problem can be alleviated to a large extent by integrating recent algorithms like bpe-dropout @xcite or sen-tencepiece @xcite , that use frequency and/or language model based tokenization, the remaining aforementioned challenges still persist. in the following section, we discuss these challenges with respect to two categories: tokenization and sub-word representations. ## experiments the experiments we performed are using the pretrained bert-base-uncased model without any domain specific fine tuning as the examples were chosen with the purpose of highlighting the challenges with bert across various domains. the 12 th encoder layer dynamic embeddings were used for all the analysis tasks. in case a word was oov, the average of its sub-word units embeddings was considered as its embedding. otherwise, the embedding for the word was considered as it is. in all cases, we ignore the [cls] and [sep] embeddings while computing the embedding of a par ## conclusion in this paper we highlighted various challenges in the bert model which if solved could significantly boost the models accuracy, especially in domain specific applications. these are mainly due to bert lacking a semantic tokenization algorithm and its semantic information loss from sub-word representations in oov scenarios.