Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation (2021)

Abstract: Large pre-trained models such as BERT are known to improve different downstream NLP tasks, even when such a model is trained on a generic domain. Moreover, recent studies have shown that when large domain-specific corpora are available, continued pre-training on domain-specific data can further improve the performance of in-domain tasks. However, this practice requires significant domain-specific data and computational resources which may not always be available. In this paper, we aim to adapt a generic pretrained model with a relatively small amount of domain-specific data. We demonstrate that by explicitly incorporating multi-granularity information of unseen and domain-specific words via the adaptation of (word based) n-grams, the performance of a generic pretrained model can be greatly improved. Specifically, we introduce a Transformer-based Domain-aware N-gram Adaptor, T-DNA, to effectively learn and incorporate the semantic representation of different combinations of words in the new domain. Experimental results illustrate the effectiveness of T-DNA on eight low-resource downstream tasks from four domains. We show that T-DNA is able to achieve significant improvements compared to existing methods on most tasks using limited data with lower computational costs. Moreover, further analyses demonstrate the importance and effectiveness of both unseen words and the information of different granularities. Our code is available at https://github.com/shizhediao/T-DNA .

## introduction pre-trained language models have achieved great success and shown promise in various application scenarios across natural language understanding @xcite @xcite and generation @xcite @xcite . normally applying pre-trained language models to different applications follows a two-stage paradigm: pre-training on a large unlabeled corpus and then fine-tuning on a downstream task dataset. however, when there are domain gaps between pre-training and fine-tuning data, previous studies @xcite have observed a performance drop caused by the incapability of generalization to new domains. towards filling the gaps, the main research stream @xcite @xcite on adapting pre-trained language models starts from a generic model (e.g., bert, roberta) and then continues pre-training with similar objectives on a large-scale domain-specific corpus. however, without providing sufficient understanding of the reason for the performance drop during the domain shift, it is prone to failure of adaptation. therefore, many aspects of continuous pre-training are expected to be enhanced. first, although generic pre-trained models offer better initialization for continuous pre-training models, it still costs considerable time (and money) that are beyond the reach of many institutions. 2 second, it is clumsy to pre-train domain-specific models repeatedly for each domain on large-scale corpora. 3 therefore, it is helpful to have an efficient and flexible method for being able to adapt pre-trained language models to different domains requiring limited resources. starting from the observed vocabulary mismatch problem @xcite , we further show empirically that the domain gap is largely caused by domain-specific n-grams. 4 motivated by this find-ing, we propose a light-weight transformer-based domain-aware n-gram adaptor (t-dna) by incorporating n-gram representations to bridge the domain gap between source and target vocabulary. specifically, the proposed model is able to explicitly learn and incorporate better representations of domain-specific words and phrases (in the form of n-grams) by the adaptor networks with only requiring small pieces of data. with this adaptor, once entering a new domain, one can choose to train the adaptor alone or train it with a transformer-based backbone (e.g., bert) together, where the joint training paradigm could provide more improvement. in addition, although it is designed for a lowresource setting, the adaptor is still able to work with enough data, which ensures its generalization ability in different scenarios. experimental results demonstrate that t-dna significantly improves domain adaptation performance based on a generic pre-trained model and outperforms all baselines on eight classification tasks (on eight datasets). the results confirm that incorporating domain-specific n-grams with the proposed t-dna is an effective and efficient solution to domain adaptation, showing that the information carried by larger text granularity is highly important for language processing across domains. moreover, further analyses investigate the factors that may influence the performance of our model, such as the amount of available data, the training time cost and efficiency, and the granularity of domain-specific information, revealing the best way and setting for using the model. ## the motivation as observed in @xcite , the transfer gain of domain-specific pre-training becomes increasingly significant when the source and target domain are vastly dissimilar in terms of the vocabulary overlap. motivated by this association between transfer gain and vocabulary distribution, we further investigate the shift of words and phrases across domains and attempt to alleviate the degradation of language models without large domainspecific corpora. in particular, we start with a roberta-base model from the generic domain and then fine-tune it on the imdb @xcite dataset. we investigate the outputs predicted by the [cls] embedding on the imdb development set and divide them into two categories: correct predictions (true positive/negative) and false predictions (false positive/false negative). to examine the vocabulary mismatch problem during the domain shift, we extract the top 1k most frequent n-grams foot_4 from these two categories respectively. we identify the n-grams not in the top 10k most frequent n-grams of source data foot_5 as domain-specific n-grams. as revealed in figure 1 , a larger proportion of domainspecific n-grams are captured when the model is misled to make wrong predictions, which suggests that the shifts in semantic meaning for both words and phrases might account for the domain shift. furthermore, we conjecture that the representations of domain-specific n-grams are unreliable, which exacerbates the model degradation. while more details will be presented in §6.3, we briefly mention here that the tokens usually improperly attend to other tokens in the sentence but omit the most important words and phrases. in light of this empirical evidence, we are motivated to design a framework to not only capture the domain-specific n-grams but also reliably embed them to extrapolate in the novel domain. ## the t-dna our approach follows the standard recipe of pretraining and fine-tuning a language model, which receives a sentence @xmath0, and outputs the representation of each token. the overall architecture of our approach is shown in figure 2 . in the middle, a generic pre-trained encoder, such 3338 token embedding layer ## experimental results we compare the performance of the roberta model with and without t-dna on the aforementioned datasets. in both fine-tuning and task adaptive pre-training experiments, t-dna shows significant improvements over the pre-trained generic roberta. ## analyses we analyze several aspects of t-dna, including the effects of different granularities and the effects ## related work a large performance drop of pre-trained models caused by domain shift has been observed and many domain-specific bert models @xcite @xcite have been introduced to bridge the domain gap. for example, scibert @xcite is trained on 1.14m scientific papers from semantic scholar corpus @xcite for 7 days on tpu v3-8 machine and biobert @xcite ) is trained on pubmed abstracts and pmc full text articles for 23 days on eight nvidia v100 gpus. clinicalbert @xcite is trained on about 2 million notes in the mimic-iii v1.4 database @xcite for 17-18 days on a single geforce gtx titan x 12 gb gpu. however, they all incur a huge computational cost, which is not affordable for many university labs or institutions. this is precisely why we believe that our efficient adaptor is useful to the community. although @xcite introduced task-adaptive pre-training (tapt) to save time by training on unlabeled downstream task data, we demonstrate that our plug-in adaptor is faster and more effective because of the explicit learning strategy and efficient model architecture. out of vocabulary (oov) words refer to those words that are not in the vocabulary list and have received a lot of attention in recent years. one way to handle oov words is to simply utilize and learn an "unknown" embedding during training. another way is to add in-domain words into the original vocabulary list and learn their representation by pretraining from scratch @xcite , which requires substantial resources and training data. moreover, scibert @xcite found that in-domain vocabulary is helpful but not significant while we attribute it to the inefficiency of implicit learning of in-domain vocabulary. to represent oov words in multilingual settings, the mixture mapping method @xcite utilized a mixture of english subwords embedding, but it has been shown useless for domain-specific words by @xcite . exbert @xcite applied an extension module to adapt an augmenting embedding for the in-domain vocabulary but it still needs large continuous pre-training. similar to our work, they highlight the importance of the domain-specific words but all of these work neither explore the understanding of performance drop during a domain shift nor examine the importance of multi-grained information. large granularity contextual information carried by spans or n-grams has proven to be helpful to enhance text representation for chinese @xcite @xcite @xcite @xcite @xcite and english @xcite @xcite . in addition to text encoders on pre-training, the knn-lm @xcite proposes to augment the language model for effective domain adaptation, by varying the nearest neighbor datastore of similar contexts without further training. however, all of the previous studies focused on either general pre-training procedures or different tasks (e.g., language modeling), and did not explore the effectiveness of multigrained information for domain adaptation. we hence view them as orthogonal to our work. ## conclusion in this work, we first reveal a novel discovery behind the performance drop during a domain shift, demonstrating that an unreliable representation of domain-specific n-grams causes the failure of adaptation. to this end, we propose an innovative adaptor network for generic pre-trained encoders, supporting many training strategies such as taskadaptive pre-training and fine-tuning, both leading to significant improvements to eight classification datasets from four domains (biomedical, computer science, news and reviews). our method is easy to implement, simple but effective, implying that explicitly representing and incorporating domainspecific n-grams offer large gains. in addition, further analyses consistently demonstrate the importance and effectiveness of both unseen words and the information carried by coarse-grained n-grams.