Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories (2023)

Abstract: Pre-trained language models (PLMs) demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain. Although continued pre-training on a large domain-specific corpus is effective, it is costly to tune all the parameters on the domain. In this paper, we investigate whether we can adapt PLMs both effectively and efficiently by only tuning a few parameters. Specifically, we decouple the feed-forward networks (FFNs) of the Transformer architecture into two parts: the original pre-trained FFNs to maintain the old-domain knowledge and our novel domain-specific adapters to inject domain-specific knowledge in parallel. Then we adopt a mixture-of-adapters gate to fuse the knowledge from different domain adapters dynamically. Our proposed Mixture-of-Domain-Adapters (MixDA) employs a two-stage adapter-tuning strategy that leverages both unlabeled data and labeled data to help the domain adaptation: i ) domain-specific adapter on unlabeled data; followed by ii ) the task-specific adapter on labeled data. MixDA can be seamlessly plugged into the pretraining-finetuning paradigm and our experiments demonstrate that MixDA achieves superior performance on in-domain tasks (GLUE), out-of-domain tasks (ChemProt, RCT, IMDB, Amazon), and knowledge-intensive tasks (KILT).Further analyses demonstrate the reliability, scalability, and efficiency of our method.

## introduction pre-trained language models (plms) have achieved a multitude of successful applications in natural language understanding @xcite @xcite and generation @xcite @xcite . the predominant methodology for domain adaptation is fine-tuning on labeled domain-specific data or continued pre-training @xcite on unlabeled domain-specific data. although effective, both fine-tuning and continued pre-training methods require tuning all the parameters of a plm, raising high costs beyond many institutions' reach. to mitigate this, multiple parameter-efficient fine-tuning (peft) methods are proposed, including prompt-based tuning @xcite @xcite @xcite , and adapter-based tuning @xcite @xcite . however, they are more concerned about task adaptation and it is still unclear how to regularly, and inexpensively inject domain knowledge into plms for different domain-specific tasks. moreover, directly tuning plms on a domain-specific corpus with peft methods will lead to the catastrophic forgetting problem @xcite . these limitations highlight an important research question: how to adapt plms with the new domain knowledge while keeping the old-domain knowledge unmodified? inspired by the recent studies @xcite @xcite that found knowledge is stored in feed-forward networks (ffns), we decouple the ffns into two parts: the original pre-trained ffns to maintain the olddomain knowledge and our novel domain-specific adapters to inject domain-specific knowledge in parallel. specifically, we propose mixture-of-domain-adapters (mixda), a mixture of several domain adapters to inject domain-specific knowledge without affecting the old-domain knowledge. our model has two stages: piq domain-specific tuning multiple knowledge adapters on unlabeled data and then piiq task-specific tuning adapters on labeled data. in the first stage, we train several domain adapters on both domain-specific corpus and pre-training corpus simultaneously while keeping the original feed-forward networks unchanged. in the second stage, we train a mixture-of-adapters gate to dynamically select the desired knowledge adapter and a task-specific adapter for task adaptation. we conduct experiments on a broad range of tasks, including 4 out-of-domain datasets, 9 in-domain datasets, and 2 knowledge-intensive datasets. our experimental results demonstrate the effectiveness of mixda on 15 datasets, spanning biomedical, computer science publications, news, and reviews. further analysis displays three key properties of our proposed approach: piq reliability: it shows superior performance on both in-domain and out-of-domain tasks. piiq scalability: it scales well to the increasing number of domains. piiiq efficiency: it adds only a small number of parameters per domain. we claim that these properties are helpful for language models as a service, where a frozen plm is served, and multiple adapters are inserted to support different customized services. ## related work in this section, we will review four research lines related to injecting domain knowledge into pretrained language models: knowledge injection, domain adaptation, parameter-efficient fine-tuning, and mixture-of-adapters. ## approach given a pre-trained language model m, the input is a sentence x " t 1 t 2 ¨¨¨t i ¨¨¨t t (t i indicates the i-th token) and the output is the representation of each token. the overall architecture of our model is shown in figure 1 . the training process is divided into two-stage. in stage 1 (figure 1 (a)), we inject new feed-forward networks (ffns) (namely domain-adapter) paralleled to the original pre-trained ffns in some transformer layers, acting as a key-value memory. the newly injected domain-adapter is trained on both domain-specific unlabeled data and original pre-training unlabeled data to store new factual associations while keeping old-domain ones. all modules are frozen except domain-adapter in this stage. in stage 2 (figure 1 (b) ), we train a mixture-of-adapters (moa) gate and a task-adapter on downstream tasks with labeled data, and only these two new modules are updated. the moa gate receives outputs from the old-domain ffns and domain-adapter, then outputs a weighted sum of them. an additional taskadapter is inserted in each transformer block to facilitate downstream tasks. figure 1 (c) shows the structures of the domain-adapter and the moa gate. in this section, we first introduce domainadapter, which learns and stores domain-specific knowledge, and then describe task-adapters that perform the downstream task. finally, we discuss how the moa gate integrates the outputs from the ffn and the domain-adapter. ## experimental settings in this section, we first introduce the datasets, then the baseline models, the evaluation metrics, and implementation details in the following four subsections, respectively. ## experimental results we compare the performance of mixda with our baselines on 15 datasets. first, we train the domainadapter for each domain individually and then perform each task with its corresponding domainadapter, which shows significant improvement over our baselines. next, we plug in several domainadapters trained on different domains parallelly to verify the scalability of our model. ## analysis in this section, we analyze the respective contributions of each part of mixda through detailed analysis, including the stage 1 training, task-adapters in stage 2, and the mixture-of-adapters gate. ## conclusion in this paper, we proposed mixda, a mixture of adapters for domain adaptation. we first decouple the knowledge modules (i.e., ffns) into the old-domain and domain-specific ffns. then we propose a two-stage adapter tuning strategy: first tuning the domain adapter on each domain and then tuning the task adapter on each task. moreover, our model could be scaled to multiple domains easily with the introduction of the mixture-of-adapters gate. empirically, mixda achieved significant improvement over in-domain tasks, out-of-domain tasks, and knowledge-intensive tasks. further analyses demonstrate the reliability, scalability, and efficiency of our method.