C haracter BERT : Reconciling ELM o and BERT for Word-Level Open-Vocabulary Representations From Characters (2020)

Abstract: Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite it not being intrinsically linked to the notion of Transformers. While this system is thought to achieve a good balance between the flexibility of characters and the efficiency of full words, using predefined wordpiece vocabularies from the general domain is not always suitable, especially when building models for specialized domains (e.g., the medical domain). Moreover, adopting a wordpiece tokenization shifts the focus from the word level to the subword level, making the models conceptually more complex and arguably less convenient in practice. For these reasons, we propose CharacterBERT, a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level, and open-vocabulary representations.

## introduction pre-trained language representations from transformers @xcite have become arguably the most popular choice for building nlp systems 1 . among all such models, bert @xcite has probably been the most successful, spawning a large number of new improved variants @xcite @xcite @xcite . as a result, many of the recent language representation models inherited bert's subword tokenization system which relies on a predefined set of wordpieces @xcite , supposedly striking a good balance between the flexibility of characters and the efficiency of full words. while current research mostly focuses on improving language representations for the default "generaldomain", there seems to be a growing interest in building suitable word embeddings for more specialized domains @xcite @xcite . however, with the growing complexity of recent representation models, the default trend seems to favor re-training general-domain models on specialized corpora rather than building models from scratch with a specialized vocabulary (e.g., bluebert @xcite and biobert @xcite ). while these methods undeniably produce good models 2 , a few questions remain: how suitable are the predefined general-domain vocabularies when used in the context of specialized domains (e.g., the medical domain)? is it better to train specialized models with specialized subword units? do we induce any biases by training specialized models with general-domain wordpieces? in this paper, we propose characterbert, a possible solution for avoiding any biases that may come from the use of a predefined wordpiece vocabulary, and an effort to revert back to conceptually simpler word-level models. this new variant does not rely on wordpieces but instead consults the characters of each token to build representations similarly to previous word-level open-vocabulary systems @xcite @xcite . in practice, we replace bert's wordpiece embedding layer with elmo's @xcite character-cnn module while keeping the rest of the architecture untouched. as a result, characterbert is able to produce word-level contextualized representations and does not require a wordpiece vocabulary. furthermore, this new model seems better suited than vanilla bert for training specialized models, as evidenced by an evaluation on multiple tasks from the medical domain. finally, as expected from a character-based system, characterbert is also seemingly more robust to noise and misspellings. to the best of our knowledge, this is the first work that replaces bert's wordpiece system with a word-level character-based system. our contributions are the following: this work has only focused on the english language and the medical (clinical and biomedical) domain. the generalization to other languages and specialized domains is left to future work. ## general-domain wordpieces in specialized domains since many specialized versions of bert come from re-training the original model on a set of specialized texts, we carry out a couple of preliminary experiments to gauge the effect of using a general-domain wordpiece vocabulary in a specialized domain. here we focus on the medical domain for which we learn foot_1 a new wordpiece vocabulary using mimic-iii clinical notes @xcite and pmc oa foot_2 biomedical article abstracts. we then process a sample (1m tokens) of the medical corpus with either the medical vocabulary or bert's original vocabulary and examine the difference. looking at the frequency of splitting an unknown token into multiple wordpieces (cf. figure 1 ) we see that the medical vocabulary produces overall less wordpieces than the general version, both at occurrence and type levels. moreover, we see that ≈ 13% of occurrences are never split as they are already part of the medical vocabulary but are decomposed into two or more wordpieces by the general vocabulary. when looking closer at the quality of the produced wordpieces (cf. ## characterbert characterbert is similar in every way to vanilla bert but uses a different method to construct initial context-independent representations: while the original model consults its vocabulary to split unknown tokens into multiple wordpieces then embeds each unit independently using a wordpiece embedding matrix, characterbert uses a character-cnn module @xcite which consults the characters of a token to produce a single representation (see figure 2 ). in this illustration, bert splits the word "apple" into two wordpieces then embeds each unit separately. characterbert produces a single embedding for "apple" by consulting its sequence of characters. ## experiments we compare bert and characterbert on multiple medical tasks to evaluate the impact of using a character-cnn module instead of wordpieces. in an attempt to dissociate this impact from any other effects that may be related to training models in our own specific settings, we train each characterbert model alongside a bert counterpart in the exact same conditions. ## conclusion the overall strategy when building specialized versions of bert seems to be re-training the original model on a specialized corpus. this implies keeping a general-domain wordpiece vocabulary that may not be suited for the domain of interest. our main contribution is characterbert, a variant of bert that drops the wordpiece system altogether in favor of a character-cnn. this module represents tokens by consulting their characters, allowing our model to produce word-level open-vocabulary representations. we evaluate characterbert and show that it globally outperforms bert when specialized for the medical domain while at the same time being more robust to misspellings.