AV oca D o: Strategy for Adapting Vocabulary to Downstream Domain (2021)

Abstract: During the fine-tuning phase of transfer learning, the pretrained vocabulary remains unchanged, while model parameters are updated. The vocabulary generated based on the pretrained data is suboptimal for downstream data when domain discrepancy exists. We propose to consider the vocabulary as an optimizable parameter, allowing us to update the vocabulary by expanding it with domain specific vocabulary based on a tokenization statistic. Furthermore, we preserve the embeddings of the added words from overfitting to downstream data by utilizing knowledge learned from a pretrained language model with a regularization term. Our method achieved consistent performance improvements on diverse domains (i.e., biomedical, computer science, news, and reviews).

## introduction a language model (lm) is pretrained with a large corpus in a general domain and then is fine-tuned to perform various downstream tasks, such as text classification, named entity recognition, and question answering. however, fine-tuning the lm is challenging when the downstream domain is significantly different from the pretrained domain, requiring domain adaptation to improve the downstream performance @xcite @xcite . prior approaches conducted additional training with a large domain-specific corpus in between pretraining and fine-tuning. in these approaches, the pretrained vocabulary remains unchanged, although the model is being adapted to a downstream domain, such as biomedicine or politics. we argue that the vocabulary should also be adapted during the fine-tuning process towards downstream data. recent studies (e.g., scib-ert @xcite ) showed that using an fig1 current version optimized vocabulary for a particular downstream domain is more effective than using the vocabulary generated in pretraining stage. however, these approaches required a large domain-specific corpus additional to the downstream data in order to construct optimized vocabulary for the downstream domain. ## related work as transfer learning has shown promising results in natural language processing (nlp), recent work leveraged the knowledge learned from the pretrained model, such as bert @xcite in various domains. scibert @xcite trains a language model with the large domain-specific corpus from scratch, showing that the vocabulary constructed from the domain-specific corpus contributes to improving performance. @xcite and @xcite conducted additional training on a pretrained lm with a large domainspecific corpus before fine-tuning. on the other hand, exbert @xcite extended the pretrained model with new vocabulary to adapt to biomedical domain. similarly, @xcite and @xcite proposed to expand vocabulary and leverage external domain-specific corpus to train new embedding layers. on the contrary, avocado requires only downstream dataset in domain adaptation. furthermore, our method selects a subset of domain-specific vocabulary considering the relative importance of words. ## methods in avocado, we generate domain-specific vocabulary based on the downstream corpus. the subset of the generated vocabulary is merged with the original pretrained vocabulary. the size of subset is controlled by the fragment score. afterwards, we apply a regularization term during fine-tuning to prevent the embeddings of added words from overfitting to the downstream data. ## experimental settings datasets we conducted experiments on four domains that are significantly different from the pretraining domain; biomedical (biomed) papers, computer science (cs) papers, news, and amazon reviews (reviews). chemprot @xcite , acl-arc @xcite , hyperparti-san @xcite , and amazon @xcite datasets are used in respective domains. target task for each dataset is text classification. appendix c describes more details. ## conclusion in this paper, we demonstrate that a pretrained vocabulary should be updated towards a downstream domain when fine-tuning. we propose a fine-tuning strategy called avocado that adapts the vocabulary to the downstream domain by expanding the vocabulary based on a tokenization statistic, and by regularizing the newly added words. our approach shows consistent performance improvements in diverse domains on various pretrained language models. avocado is applicable to a wide range of nlp tasks in diverse domains without any restrictions, such as massive computing resources or a large domain-specific corpus. sung electronics co., ltd. we thank the anonymous reviewers for their helpful feedback and discussions. we also thank seong-su bae for his insight and helpful opinion.