TADPOLE : T ask AD apted P re-Training via A n O ma L y D E tection (2021)

Abstract: The paradigm of pre-training followed by finetuning has become a standard procedure for NLP tasks, with a known problem of domain shift between the pre-training and downstream corpus. Previous works have tried to mitigate this problem with additional pre-training, either on the downstream corpus itself when it is large enough, or on a manually curated unlabeled corpus of a similar domain. In this paper, we address the problem for the case when the downstream corpus is too small for additional pre-training. We propose TADPOLE, a task adapted pre-training framework based on data selection techniques adapted from Domain Adaptation . We formulate the data selection as an anomaly detection problem that unlike existing methods works well when the downstream corpus is limited in size. It results in a scalable and efficient unsupervised technique that eliminates the need for any manual data curation. We evaluate our framework on eight tasks across four different domains: Biomedical, Computer Science, News, and Movie reviews, and compare its performance against competitive baseline techniques from the area of Domain Adaptation. Our framework outperforms all the baseline methods. On small datasets with less than 5K training examples, we get a gain of 1.82% in performance with additional pre-training for only 5% steps compared to the originally pre-trained models. It also compliments some of the other techniques such as data augmentation known for boosting performance when downstream corpus is small; highest performance is achieved when data augmentation is combined with task adapted pre-training.

## introduction pre-trained language models such as elmo @xcite , gpt @xcite , @xcite , transformer-xl @xcite and xlnet @xcite have become a key component in solving virtually all natural language tasks. these models are pre-trained on large amount of cross-domain data ranging from wikipedia to book corpus to news articles to learn powerful representations. a generic approach for using these models consists of two steps: (a) pretraining: train the model on an extremely large general domain corpus, e.g. with masked language model loss; (b) finetuning: finetune the model on labeled task dataset for the downstream task. even though the approach of pre-training followed by fine-tuning has been very successful, it suffers from domain shift when applied to tasks containing text from a domain that is not sufficiently represented in the pre-training corpus. an immediate way of solving the problem is to pre-train the model on task domain data instead of the general domain data. for a handful of very popular task domains, the research community invested time and resources to collect a large domain-specific data corpus and pre-train a language model on it. the models include biobert pre-trained on biomedical text @xcite , clinicalbert pretrained on clinical notes @xcite , scib-ert pre-trained on semantic scholar corpus @xcite , and finbert pre-trained on financial documents @xcite . these models achieve significant gain in performance over a model trained on general domain data, when the downstream task belongs to the respective domains. these papers demonstrate how useful it can be to shift the domain of the pre-trained model. however, the approach is expensive and time consuming as it requires collecting gigabytes of domain data for each new task. the long-tail of domains remains left behind without a realistic solution. to mitigate this, in absence of the huge task domain data, a different known approach is to collect a medium (mbs, not gbs) amount of unlabeled task data, and adapt the pre-trained (on general data) model by e.g. extending the pre-training procedure on the unlabeled data @xcite . such task adapted pre-training approach achieves relatively smaller gain but is less expensive. although this approach is cheaper in terms of manual labor when compared to domain adapted bert, it still requires an effort to collect unlabeled data. it requires much more data than what is needed for only fine-tuning. this is often impossible to achieve, for example when data is highly sensitive. in this paper we propose a solution to this challenging problem, providing domain adaptation to the pre-trained model without the need for any manual effort of data collection. the high level idea is quite intuitive. given a generic pre-training data containing text from multiple domains, we filter the available general domain to contain only pieces that are similar to the downstream task corpus. by continuing the pretraining process on this adapted corpus we achieve a better tuned pre-trained model. figure 1 illustrate the feasibility of this approach with an example downstream task from a medical domain and highlighted text from a news article available in a general domain corpus. the key for a successful implementation is finding the best way of evaluating the similarity of a given snippet to the downstream task. ## related work since our focus is on data selection methods, we only discuss the related work on data selection in domain adaptation here. we discuss the other domain adaptation techniques in appendix a. ## tadpole: task adapted pre-training via anomaly detection language model and downstream tasks. a generic approach for using state-of-the-art language models such as elmo, gpt, bert, and xlnet is to pre-train them on an extremely large general domain corpus and then finetune the pretrained model on the downstream labeled task data. there is evident correlation between model's pretraining loss and its performance on the downstream task after finetuning @xcite . our design is motivated by an observation, backed by empirical evidence, that the correlation is even stronger if we consider the pre-training loss not on the pre-training data but the downstream task data. to make this distinction formal, let d, d in be the pre-training and task data. let θ denote the parameters of the language model and ℓ lm denote the language model loss function. the pre-training loss on pre-training data (l lm (θ)) and target data (l in lm (θ)) are defined as follows: we observe in figure 3 that if we have two equally good language models (similar language model loss on general domain), the language model which is better (tailored) for the task domain has significantly better downstream performance. we conclude that in order to ensure success in the downstream task, we should aim to minimize l in lm (θ). a first attempt would be to pre-train or finetune the language model on d in . however, training a language model such as elmo, gpt, bert or xlnet requires a large corpus with several gbs of text and the available domain specific corpus d in is often just the task data which has few mbs of text. training on such a small dataset would introduce high variance. we reduce this variance by taking training examples from the general domain corpus d, but control the bias this incurs by considering only elements having high relevance to the domain d in . formally, we optimize a weighted pre-training loss function ## experiments ## conclusion domain shift in finetuning from pre-training can significantly impact the performance of deep learning models. we address this issue in the most reasonable setting when we only have access to the labeled task data for finetuning. we adapt data selection methods from domain adaptation to adapt pre-training for the downstream task. the existing methods either require sufficiently large task data, or are based on adhoc techniques that do not generalize well across tasks. our major contribution is providing a new data selection technique that performs well even with very little task data, and generalizes well across tasks.