Efficient Domain Adaptation of Sentence Embeddings Using Adapters (2023)

Abstract: Sentence embeddings enable us to capture the semantic similarity of short texts. Most sentence embedding models are trained for general semantic textual similarity tasks. Therefore, to use sentence embeddings in a particular domain, the model must be adapted to it in order to achieve good results. Usually, this is done by fine-tuning the entire sentence embedding model for the domain of interest. While this approach yields state-of-the-art results, all of the model’s weights are updated during fine-tuning, making this method resource-intensive. Therefore, instead of fine-tuning entire sentence embedding models for each target domain individually, we propose to train lightweight adapters. These domain-specific adapters do not require fine-tuning all underlying sentence embedding model parameters. Instead, we only train a small number of additional parameters while keeping the weights of the underlying sentence embedding model fixed. Training domain-specific adapters allows always using the same base model and only exchanging the domain-specific adapters to adapt sentence embeddings to a specific domain. We show that using adapters for parameter-efficient domain adaptation of sentence embeddings yields competitive performance within 1% of a domain-adapted, entirely fine-tuned sentence embedding model while only training approximately 3.6% of the parameters.

## introduction learning sentence embeddings is an essential task in natural language processing (nlp) and has already been extensively investigated in the literature @xcite @xcite @xcite @xcite @xcite . sentence embeddings are especially useful in information retrieval @xcite ; by injecting domain-specific knowledge of adapters into the sentence embedding model, we can efficiently adapt the resulting representations for semantic textual similarity tasks in different domains. @xcite or unsupervised text classification settings @xcite . lately, the most popular approach for sentence embedding learning is to fine-tune pretrained language models with a contrastive learning objective @xcite @xcite @xcite . while this approach provides state-of-theart results, all of the model's weights are updated during fine-tuning, making this method resourceintensive. this is a problem, particularly when domain-specific models are needed. then, a specialized model must be trained for each domain of interest, resulting in resource-intensive training. recently, adapters have emerged as a parameterefficient strategy to fine-tune language models (lms). adapters do not require fine-tuning of all parameters of the pretrained model and instead introduce a small number of task-specific parameters while keeping the underlying pretrained lan-guage model fixed @xcite . they enable efficient parameter sharing between tasks and domains by training many task-specific, domainspecific, and language-specific adapters for the same model, which can be exchanged and combined post-hoc @xcite . therefore, many different adapter architectures have been proposed for various domains and tasks @xcite @xcite @xcite @xcite . however, to the best of our knowledge, no method currently exists for efficient domain adaptation of sentence embeddings using adapters. in this paper, we aim to bridge this gap by proposing approaches for adapter-based domain adaptation of sentence embeddings, allowing us to train models for many different domains efficiently. therefore, we investigate how to adapt general pretrained sentence embedding models to different domains using domain-specific adapters. as shown in figure 1 , this allows always using the same base model to adapt sentence embeddings to a specific domain and only needing to exchange the domain-specific adapters. accordingly, we train lightweight adapters for each domain and avoid expensive training of entire sentence embedding models. ## related work adapters have been introduced by @xcite as a parameter-efficient alternative for taskspecific fine-tuning of language models. since their introduction, adapters have been used to fine-tune models for single tasks as well as in multi-task settings @xcite . usually, adapters are used to solve tasks such as classification @xcite , machine translation @xcite , question answering @xcite , or reasoning @xcite . while there exist adapters for semantic textual similarity (sts) tasks on the adapterhub @xcite , these are trained on general sts datasets using a task-unspecific pretrained language model as a basis. we, however, focus on adapting pretrained sentence embedding models to specific domains using adapters. ## method we assume we have a base sentence embedding model from the source domain and labeled datasets for each target domain. instead of fine-tuning the entire sentence embedding model for each target domain individually, we train lightweight adapters for each domain. this domain-specific fine-tuning with adapters involves adding a small number of new parameters to the sentence embedding model. during training, the parameters of the sentence embedding model are frozen, and only the weights of the adapters are updated. formally, we adopt the general definition for adapter-based fine-tuning of @xcite as follows: for each of the n domains, the sentence embedding model is initialized with parameters θ 0 . additionally, a set of new and randomly initialized adapter parameters φ n are introduced. the parameters θ 0 are fixed and only the parameters φ n are trained. given training data d n and a loss function l, the objective for each domain @xmath0, ..., n is of the form: usually, the adapter parameters φ n are significantly less than the parameters θ 0 of the base model @xcite , e.g., only 3.6% of the parameters of the pretrained model in @xcite . ## experiments in this section, we describe the used adapter architectures, loss functions, and datasets. in all experiments, we use simcse sup-bert-base @xcite as the base sentence embedding model. it is trained on natural language inference (nli) datasets @xcite for sts tasks in the general domain. we fine-tune all models and adapters for five epochs using a learning rate of 1e -5 . ## evaluation the evaluation reveals that adapter-based domain adaptation yields competitive results compared to fine-tuning the entire base model. in particular, the houlsby and pfeiffer adapters perform very well with both loss functions, even though they use only a fraction of the parameters of the upper bounds. the slightly larger k-adapter, however, performs considerably worse than the other adapters investigated. we conclude that the bottleneck architecture is more suitable than the external plug-in architecture for domain adaptation of sentence embedding models. in particular, the houlsby adapter, although the smallest among the adapters investigated, yields the best results for both loss functions. using the out-of-the-box simcse model without domain adaptation results in considerably worse performance, indicating the overall importance of domain-specific fine-tuning for sentence embedding models. furthermore, the contrastive loss function ℓ 2 performs consistently better than ℓ 1 . our results align with the observations of @xcite who conclude that the contrastive objective ensures a distribution of embeddings around the entire embedding space. in contrast, ℓ 1 may yield learned representations occupying a narrow vector space cone, which severely limits their expressiveness. from the obtained results, we conclude that using the houlsby-adapter architecture together with the contrastive objective ℓ 2 is most suitable for parameter-efficient domain adaptation of sentence embedding models. this adapter approach shows performance that is within 1% of the supervised, entirely fine-tuned simcse model, while only training approximately 3.6% of the parameters. ## conclusion in this work, we proposed the use of adapters for parameter-efficient domain adaptation of sentence embedding models. in contrast to fine-tuning the entire sentence embedding model for a particular domain, adapters add a small number of new parameters that are updated during training while the weights of the sentence embedding model are fixed. we showed that adapter-based domain adaptation of sentence embedding models yields competitive results compared to fine-tuning the entire model, although only a fraction of the parameters are trained. in particular, we show that using the houlsby-adapter architecture together with a contrastive objective yields promising results for parameter-efficient domain adaptation of sentence embedding models.