Efficient Hierarchical Domain Adaptation for Pretrained Language Models (2022)

Abstract: The remarkable success of large language models has been driven by dense models trained on massive unlabeled, unstructured corpora. These corpora typically contain text from diverse, heterogeneous sources, but information about the source of the text is rarely used during training. Transferring their knowledge to a target domain is typically done by continuing training in-domain. In this paper, we introduce a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. Our method is based on the observation that textual domains are partially overlapping, and we represent domains as a hierarchical tree structure where each node in the tree is associated with a set of adapter weights. When combined with a frozen pretrained language model, this approach enables parameter sharing among related domains, while avoiding negative interference between unrelated ones. Experimental results with GPT-2 and a large fraction of the 100 most represented websites in C4 show across-the-board improvements in-domain. We additionally provide an inference time algorithm for a held-out domain and show that averaging over multiple paths through the tree enables further gains in generalization, while adding only a marginal cost to inference.

## introduction pretrained language models (plms) @xcite @xcite , trained on massive general-domain corpora, have enabled great progress in many natural language processing (nlp) benchmarks @xcite . nonetheless, continuing pretraining (as a dense model) a plm on a narrower domain @xcite is beneficial, although computationally expensive (maronikolakis and schütze, 2021), which indicates * work done while an intern at allenai. figure 1: we model domains as a hierarchical tree structure that associates adapters with nodes, allowing parameter sharing among related domains. internet domains appear as leaf nodes. during training, we activate the adapters along the path to a leaf to specialize a language model to the domain corresponding to it. prior work typically assumes that individual domains are distinct, and models them accordingly. for example, @xcite train one model for each textual domain, either in a dense or sparse manner. this is related to data selection @xcite @xcite , which aims to select the best matching data for a new domain. this process does not scale to multiple domains efficiently, as the parameters grow linearly with the domains. it also does not allow sharing representations among related domains during training, as each domain is modeled with a separate set of parameters. at the other extreme, training one model on all domains as is common during unsupervised pretraining does not account for their similarities and differences and might hinder the model's generalization ability due to negative interference. as an alternative, we start with the observation that the term "domain" typically denotes a distribution over language characterizing a given topic or genre, and that domains are partially overlapping. for example, a sentiment model processing hotel reviews could be expected to benefit by also including data from restaurant reviews, which might in turn benefit from cooking recipes, but combing hotel reviews and recipes may be detrimental. we want to model the relations between domains and selectively share information, so that we allow positive transfer and avoid negative interference. to this end, we propose a data-driven approach to modeling domains that automatically clusters them in a tree using plm representations. we then introduce an efficient method that specializes a plm in a number of domains leveraging their hierarchical structure. our approach allows parameter sharing among related domains using adapters @xcite , which are lightweight layers, added after each transformer @xcite layer. each node in the tree is associated with a separate set of adapters, that are only activated for a particular domain. for instance, data from booking.com activates parameters in nodes 3, 6, and 7, allowing parameter sharing with the highly related yelp.com through nodes 6 and 7 (figure 1 ). we verify the efficacy of our approach in two settings. first, we manually define a tree structure, using websites as the leaves. in this first few-domain setting, our method outperforms prior work including single and multi-domain adapters added to gpt-2 @xcite when tested in-domain. we further show that our method generalizes better to held-out websites than the baselines. we then scale our model to a many-domain setting across almost 100 websites. we induce the hierarchical structure in an unsupervised way using representations from gpt-2 with a gaussian mixture model (gmm) @xcite and hierarchical clustering, similar to das gupta et al. @xcite . in this way, the clusters model textual domains and the gmm provides a mechanism to automatically find the closest training websites to any held-out website. empirical results show across-the-board improvements over strong baselines when evaluated in-domain. we also show that an efficient inference-time algorithm that averages over multiple paths through the tree improves generalization when tested on held-out websites. foot_0 ## hierarchical representation of domains in this section, we provide a formal problem definition and the intuition for a hierarchical ordering of domains. we then describe how we add a hierarchical structure to a plm and present the training process. additionally, we show how a path in the tree is selected to evaluate the in-domain and out-ofdomain sets. we finally discuss the computational cost of our approach compared to the baselines and our experimental setup. ## hierarchical domain adaptation with a manually created tree in this section, we implement the model described in the prior section for a very limited number of domains (few-domain setup), to comprehensively examine design choices and verify the performance, before moving to a large-scale setting in §4. ## related work our approach draws on prior work in domain adaption and efficient language model fine-tuning. ## conclusion & future work in this paper, we present a novel approach for efficient domain adaptation on multiple domains using hierarchical adapters that encode the similarities and differences of domains, allowing parameter sharing but avoiding negative transfer. we start with a manually defined tree and then scale to a large tree, created in an unsupervised way. we also provide an evaluation-time algorithm that can combine paths to best adapt to an unseen domain. in the future, we would like to investigate a more efficient evaluation-time approach, using only a few tokens of an unseen domain. it would also be interesting to extend our model to a multi-lingual setup. finally, we would like to use our method to control language generation of plms, in order to avoid generating hate speech or toxic text.