P atch BERT : Just-in-Time, Out-of-Vocabulary Patching (2020)

Abstract: Large scale pre-trained language models have shown groundbreaking performance improvements for transfer learning in the domain of natural language processing. In our paper, we study a pre-trained multilingual BERT model and analyze the OOV rate on downstream tasks, how it introduces information loss, and as a side-effect, obstructs the potential of the underlying model. We then propose multiple approaches for mitigation and demonstrate that it improves performance with the same parameter count when combined with fine-tuning.

## background the advent of large scale language models pretrained with large unannotated corpora has shown significant advancements in the domain of natural language processing, especially by demonstrating their effectiveness through transfer learning for downstream tasks @xcite @xcite , analogous to imagenet @xcite pre-trained backbones in the domain of computer vision. in the domain of natural language processing, new methods have made it possible to use internet corpora as a nearly free source for increasing the amount of data at an unprecedented scale during pre-training. additionally, new tokenization methods such as byte-pair encoding (bpe, @xcite ), wordpiece @xcite , sentencepiece @xcite , which break the lexicons into smaller subwords, have shown to be effective when applied to alphabetic languages to reduce the size of the vocabulary while increasing the robustness against out-of-vocabulary (oov) in downstream tasks. this is especially powerful when combined with transfer learning. however, these tokenizers still operate at unicode character levels -contrary to the names suggesting bytelevel (which would completely mitigate oov, as studied in @xcite ). hence, the minimum size of the vocabulary is twice the size of all unique characters in the corpus, as subword tokenizers store each character in prefix and suffix form in the vocabulary. as oov was a much more prevalent problem in the context of lexiconbased methods, there have been many methods, such as dictionary-based postprocessing @xcite and distributional representation based substitution @xcite . recently this has not been as actively studied in the context of subword tokenization as latin languages are no longer affected. for these reasons, when trained against a diverse set of languages, the vocabulary size increases proportionally to the number of languages supported. existing models have sampled portions of entire corpora or relaxed constraints on character level coverage for these languages, to prevent the vocabulary from growing to an unmanageable scale. as of today, this is an unavoidable trade-off when training multilingual models. this introduces a bottleneck for downstream tasks since any character omitted causes information loss. training models for each language is the most straightforward possible mitigation. however, the downside is the cost for pre-training; acquiring a large corpus is a daunting task, and training a large model for many researchers can be financially infeasible. the high upfront cost leaves transfer learning on an open, multilingual model as an economically attractive alternative. unfortunately, due to corpus imbalance during pre-training, minor languages, especially those with a diverse character set (such as cjk languages), oov is likely to surface. our motivation is to improve performance for these languages, without significantly increasing the computation cost when using open-source pre-trained models. in our work, we propose multiple approaches to mitigate oov during fine-tuning. we compare each approach with no oov mitigation, along with increasing vocabulary size as a secondary baseline. ## approach the multilingual bert model bert-basemultilingual-cased @xcite we used performs two-phase tokenization, first with whitespace followed by wordpiece. the tokenizer was modified to support a secondary vocabulary which points new words to existing words for our experiments, connected to a transformers library @xcite bert model. the approach consists of three steps. first, we perform a complete corpus analysis and search for all oov surfaces by tokenizing the task corpus. an oov surface in the context of bert is an entire space tokenized token. whenever oov occurs, we keep a record of the entire oov surface, along with the context. for each oov surface, we brute-force search to find the maximally specific oov subword surface. an oov subword surface is an actual subword missing in an oov surface. in this step, we compute a frequency table for both oov and invocabulary subwords for a preference mechanism in the mitigation strategy. we observed that most cases of the oov subword surface were caused by one character missing in the vocabulary during our experiments, which is a result of incomplete character coverage from the corpora used for pretraining. finally, we use this information to build a mitigation strategy for the oov subwords. here, we evaluate different algorithms for oov mitigation, each of which we discuss in the individual method sections below. after applying oov mitigation, we then optionally perform fine-tuning and evaluate against the baseline. substitution to mitigate oov has been studied in this example, we assume î is a missing subword. in @xcite . this method depends on part-of-speech tagging or a secondary corpus and model for similarity computation, which is challenging to apply in a subword model. the significance of our approach is that it works for subword models and it's practical applicability, as only a downstream task corpus and a pre-trained model is required. ## datasets for our experiments, we used four datasets for evaluation. for all tasks, we first learn oov words, perform fine-tuning, then evaluate. the oov rates noted for each dataset is the ratio of sentences containing at least one oov token. ## results the evaluation was done through the sst-2 glue task metrics @xcite for the sentiment analysis tasks, and em/f1 evaluation from the squad metrics for korquad, as the two tasks are compatible. each model used the same dataset and training parameters as the baseline, only with different oov mitigation methods. additionally, while chinese and japanese are both scriptio continua languages, bert's tokenizer treats cjk ideograph text differently and breaks at every character. this makes the affected surface from oov significantly smaller, resulting in less information loss. for these reasons, we expect to see larger gains in korean, as the per-character break is not enabled. ## conclusions after demonstrating examples (1) and the effects of oov triggered information loss, we propose multiple methods for mitigating oov during downstream task fine-tuning. we then demonstrate and compare with no mitigation, mitigation through network modification, and surrogates, which require no network modification, and show how each approach affects downstream tasks. in particular, we show that vocabulary surrogates can provide performance boosts with no additional computation cost, especially when paired with fine-tuning. we also empirically show that tasks with lower oov suffer less when compared to languages that do not, as seen in