A da S ent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification (2023)

Abstract: Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. In this work, we investigate strategies for domain-specialization in the context of few-shot sentence classification with SEs. We first establish that unsupervised Domain-Adaptive Pre-Training (DAPT) of a base Pre-trained Language Model (PLM) (i.e., not an SE) substantially improves the accuracy of few-shot sentence classification by up to 8.4 points. However, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domain) Sentence Embedding Pre-Training (SEPT). On the other hand, applying general-domain SEPT on top of a domain-adapted base PLM (i.e., after DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top of a DAPT-ed PLM of each domain. As a solution, we propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent’s effectiveness in extensive experiments on 17 different few-shot sentence classification datasets. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs. The code for AdaSent is available.

## introduction few-shot learning aims at training an effective model with a few labeled examples, reducing the cost of developing models for new domains and tasks. in recent work, setfit @xcite achieves strong performance in few-shot classification by contrastively fine-tuning @xcite figure 1 : training diagram of adasent. trainable parameters are marked in green. after domain-adaptive pre-training (dapt) on the pre-trained language model (plm) and sentence-embedding pre-training (sept) with an adapter, the two parts are assembled together to perform setfit for few-shot classification. despite its success, setfit fine-tunes a sentence encoder with only a few labeled samples without leveraging unlabeled data from the target-task domain, which are easy to obtain. it is well-known that domain-adaptive pre-training (dapt) foot_0 on a vanilla plm with unlabeled in-domain data can significantly improve its downstream performance @xcite . however, it is ineffective to apply dapt on sentence encoders, i.e. vanilla plms that have undergone sentence embedding pre-training (sept, @xcite in general domain, as dapt messes up the effects of sept and disrupts the model's ability to semantically accurately embed sentences. though dapt before sept is effective in contrast @xcite , it is computationally inefficient as the generaldomain sept has to be done all over again on every domain-adapted plm if we have more than one domain. to create a domain-specialized sentence encoder for few-shot sentence classification both efficiently and effectively, we propose adasent, which combines dapt and sept in a modular fashion. specifically, it stores the sentence-specialization abilities -obtained via a single sept procedure in the general domain -into an adapter. this sentenceencoding adapter is trained once regardless of the number of domains, and can be plugged into domain-adapted plms from various domains to make them domain-specialized sentence encoders, on which setfit is carried out to do downstream classification training (figure 1 ). our experiments show that adasent can match or surpass the inefficient "full sept after dapt" approach's performance on 17 sentence classification tasks from various domains. the contribution of adasent is two-fold: ## method as illustrated in figure 1 , our method for few-shot classification with domain-adapted sentence embeddings consists of three parts of training: (1) dapt on the base plm with task-specific unlabeled data, (2) sept on an adapter module with labeled sentence pairs from the general domain and (3) setfit on the whole architecture (i.e. both the plm and the adapter) with few-shot labeled data. in the first part, specifically, we continue to train a base plm like distilroberta on unlabeled target task data with the mlm loss to learn domainspecific language knowledge. in another separate procedure, sept is done by tuning an adapter on a frozen base transformer (the same plm as in dapt) without any domain adaptation. once the domain-independent sentence encoding adapter is trained, it can be easily inserted into different dapt models, ready for the few-shot classification task learning via setfit in the third part. compared to the previous approach described in subsection 3.3, adasent is more efficient for three data all paraphrase nli+sc+se nli data size 1b 86m 0.6m 0.3m accuracy 68.6 70.0 70.0 68.8 given the extensive number of experiments in this study, we use a mixture of three datasets for sept, dubbed nli+sc+se, consisting snli (bowman et al., 2015) + multinli @xcite , sentence compression @xcite and stackexchange duplicate questions, for the sake of simplicity. this is a much smaller subset of the 1 billion sentence pairs on which the popular off-the-shelf sentence embedding models 5 are pre-trained. we found that these three sept datasets transfer the best for the downstream clas-sification tasks 6 , and are adequate to train a model that performs on par with or even better than offthe-shelf models as shown in ## training cost ## conclusion we introduce an efficient method to obtain domainadapted sentence embeddings for few-shot classification. we found that setfit, the previous state-of-the-art approach, can be significantly improved by introducing a simple domain-adaptive pre-training (dapt) stage before its sentence-embedding pre-training (sept). however, this @xmath1, resulting in computational inefficiency. we propose a novel approach, adasent, to address this issue by storing the sept knowledge in an adapter that is trained on an unadapted plm and insertable into any dapt-ed plm. adasent matches or surpasses the performance of @xmath2, while significantly reducing the training cost of sept. we attribute the success of adasent to the generalization ability of the sept adapter to work with plm parameters trained on data from different domains with a consistent mlm objective.