TADA : Efficient Task-Agnostic Domain Adaptation for Transformers (2023)

Abstract: Intermediate training of pre-trained transformer-based language models on domain-specific data leads to substantial gains for downstream tasks. To increase efficiency and prevent catastrophic forgetting alleviated from full domain-adaptive pre-training, approaches such as adapters have been developed. However, these require additional parameters for each layer, and are criticized for their limited expressiveness. In this work, we introduce TADA, a novel task-agnostic domain adaptation method which is modular, parameter-efficient, and thus, data-efficient. Within TADA, we retrain the embeddings to learn domain-aware input representations and tokenizers for the transformer encoder, while freezing all other parameters of the model. Then, task-specific fine-tuning is performed. We further conduct experiments with meta-embeddings and newly introduced meta-tokenizers, resulting in one model per task in multi-domain use cases. Our broad evaluation in 4 downstream tasks for 14 domains across single- and multi-domain setups and high- and low-resource scenarios reveals that TADA is an effective and efficient alternative to full domain-adaptive pre-training and adapters for domain adaptation, while not introducing additional parameters or complex training steps.

## introduction pre-trained language models @xcite utilizing transformers @xcite have emerged as a key technology for achieving impressive gains in a wide variety of natural language processing (nlp) tasks. however, these pre-trained transformer-based language models (ptlms) are trained on massive and heterogeneous corpora with a focus on generalizability without addressing particular domain-specific concerns. in practice, the absence of such domainrelevant information can severely hurt performance in downstream applications as shown in numerous studies (i.a., @xcite @xcite . to impart useful domain knowledge, two main methods of domain adaptation leveraging transformers have emerged: (1) massive pre-training from scratch @xcite relies on large-scale domain-specific corpora incorporating various self-supervised objectives during pre-training. however, the extensive training process is time-and resource-inefficient, as it requires a large collection of (un)labeled domainspecialized corpora and massive computational power. (2) domain-adaptive intermediate pretraining @xcite is considered more light-weight, as it requires only a small amount of in-domain data and fewer epochs continually training on the ptlm from a previous checkpoint. however, fully pre-training the model (i.e., updating all ptlm parameters) may result in catastrophic forgetting and interference @xcite , in particular for longer iterations of adaptation. to overcome these limitations, alternatives such as adapters @xcite , and sparse finetuning @xcite have been introduced. these approaches, however, are still parameter-and time-inefficient, as they either add additional parameters or require complex training steps and/or models. in this work, we propose task-agnostic domain adaptation for transformers (tada), a novel domain specialization framework. as depicted in figure 1 , it consists of two steps: (1) we conduct intermediate training of a pre-trained transformerbased language model (e.g., bert) on the unlabeled domain-specific text corpora in order to inject domain knowledge into the transformer. here, we fix the parameter weights of the encoder while updating only the weights of the embeddings (i.e., embedding-based domain-adaptive pre-training). ## methods for domain specialization to inject domain-specific knowledge through domain-adaptive pre-training into ptlms, these models are trained on unlabeled in-domain text corpora. for this, we introduce a novel embeddingbased intermediate training approach as an alternative to fully pre-training and adapters ( § 2.1), and further study the effects of domain-specific tokenization ( § 2.2). we then utilize multiple domainspecialized embeddings with our newly proposed meta-tokenizers and powerful meta-embeddings in multi-domain scenarios ( § 2.3 and § 2.4). ## experimental setup this section introduces four downstream tasks with their respective datasets and evaluation metrics. we further provide details on our models, their hyperparameters, and the baseline systems. ## evaluation results for each downstream task, we first conduct experiments in a single-domain scenario, i.e., training and testing on data from the same domain, to show the advantages of our proposed approach of task-agnostic domain-adaptive embedding-based pre-training and tokenizers ( § 4.1). we further consider the combination of domain-specialized embeddings with meta-embeddings variants @xcite in a multidomain scenario, where we jointly train on data from all domains of the respective task ( § 4.2). ## analysis to more precisely analyze the advantages of our proposed embedding-based domain-adaptive pretraining methods and tokenizers, we study the following: few-shot transfer capability ( § 5.1), the effect of domain-specialized tokenizers on subword tokens ( § 5.2), and the combinations of multiple domain-specialized tokenizers with metatokenizers in multi-domain scenarios ( § 5.3). ## conclusions in this paper, we introduced tada -a novel task-agnostic domain adaptation method which is modular and parameter-efficient for pre-trained transformer-based language models. we demonstrated the efficacy of tada in 4 downstream tasks across 14 domains in both single-and multi-domain settings, as well as high-and low-resource scenarios. an in-depth analysis revealed the advantages of tada in few-shot transfer and highlighted how our domain-specialized tokenizers take the domain vocabularies into account. we conducted the first study on meta-tokenizers and showed their potential in combination with meta-embeddings in multi-domain applications. our work points to multiple future directions, including advanced meta-tokenization methods and the applicability of tada beyond the studied tasks in this paper. future research builds on top of our findings and extends the research toward more domains, more languages, more tasks, and specifically with the meta-tokenizers for efficiency concerns of domain adaptation approaches.