Generalizability of Mixture of Domain-Specific Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning (2024)

Abstract: Several parameter-efficient fine-tuning methods based on adapters have been proposed as a streamlined approach to incorporate not only a single specialized knowledge into existing Pre-Trained Language Models (PLMs) but also multiple of them at once. Recent works such as AdapterSoup propose to mix not all but only a selective sub-set of domain-specific adapters during inference via model weight averaging to optimize performance on novel, unseen domains with excellent computational efficiency. However, the essential generalizability of this emerging weight-space adapter mixing mechanism on unseen, in-domain examples remains unexplored. Thus, in this study, we conduct a comprehensive analysis to elucidate the generalizability of domain-specific adapter mixtures in in-domain evaluation. We also provide investigations into the inner workings of the mixture of domain-specific adapters by analyzing their weight signs, yielding critical analysis on the negative correlation between their fraction of weight sign difference and their mixtures’ generalizability. The code is available at Github.

## introduction recently, several parameter-efficient fine-tuning methods that are based on adapters have been introduced as a streamlined approach for fine-tuning pretrained language models (plms) to equip them with new, specialized knowledge or domain. several algorithms have been proposed to train a distinct adapter for each new domain @xcite @xcite . to further improve a model's generalizability, existing works @xcite @xcite mostly focus on training multiple adapters for multiple tasks and continuously adding more adapters for incoming new tasks. this can be inefficient for the new domain tasks that have only a few examples, making the learning among the tasks unequal. thus, more recent works such as @xcite ; @xcite ; @xcite ; @xcite opt for weight-space averaging of model and/or adapters trained on different domains, resulting in so-called mixture of expert adapters. one recent notable work in this space is adapter-soup @xcite , which proposes to merge the weights of a fixed-size, selective subset of different domain-specific adapters via an averaging function to accommodate unseen tasks or domains during inference. such weight-space merging mechanism on adapters is efficient in practice as one can efficiently train a small, additional adapter and plug it into existing plms to incorporate new knowledge. although the work reported favorable evaluation results on unseen, novel domains, it is unclear to what extent such weight-space merging mechanism on domainspecific adapters can generalize in an in-domain evaluation setting-i.e., how well it makes predictions on unseen examples of domains already seen during training. moreover, to the best of our knowledge, no existing works comprehensively study the generalization of the mixture of adapters in the indomain setting. this literature gap seems counterintuitive because in-domain evaluation is fundamental and should precede out-of-domain evaluation. moreover, in real-world applications, model owners have incentives to utilize as much as possible available information to improve their models over time. with the availability of parameterefficient finetuning methods that are fairly easy to adopt with minimal space and runtime cost, the model owners are then incentivized to quickly finetune their models on a few examples collected from a new domain on an additional adapter to optimize the performance (rather than totally relying on outof-domain prediction capability). as a result, although in-domain evaluation seems trivial, it is fundamental as one must ensure that the mixture of adapters works well on the tasks they have already been trained on. furthermore, several key questions regarding the resulting mixtures of domainspecific adapters remain unanswered, especially those regarding their generalizability and their adversarial robustness when mixing adapters trained from very different tasks. therefore, borrowing the pop-culture saying that "mixed drinks and cocktails aren't actually the same thing", in contrast from existing works, we hypothesize that not all mixture of expert adapters are created equal and all have superior performance. then, through an array of comprehensive experiments, we attempt to give answers to questions about when and what to mix when it comes to domain-specific adapters. we found that the weight-space merging mechanism suffers from performance degradation in terms of both generalizability and adversarial robustness even with inputs from domains it already trains on. moreover, we also attempt to explain such performance degradation by revealing a critical negative correlation between signed directions of adapter weights during mixing and domainspecific predictive performance (fig. 1 ). although simple, this intuitive and novel observation also allows us to select "when and what adapters to mix?" and design a more effective model pruning as a by-product application. overall, our study does not focus on proposing a new mechanism, algorithm, or method. instead, we focus on analyzing and bringing understanding of an existing and emerging paradigm of mixing multiple domain-specific adapters that was previously introduced @xcite . specifically, we focus on in-domain prediction when mixing adapters from different domains as an emerging and potential paradigm for the deployment of plms in practice. our contributions are summarized as follows. adapter fine-tuning. the primary method for adapting general-purpose plms to downstream tasks is via full fine-tuning, which requires adjusting all models' parameters @xcite . however, this results in redundant copies of fine-tuned models for each task, posing a significant memory challenge. thus, various parameter-efficient fine-tuning methods have been proposed, including prompt-based @xcite and adapter-based fine-tuning @xcite @xcite . among adapter-based fine-tuning methods, houlsby @xcite introduces two adapter blocks with bottleneck networks in each transformer block of a plm. similarly, the pfeiffer @xcite adapter differs in architecture, incorporating only one adapter layer in each transformer block, in contrast to the two layers introduced by houlsby @xcite . @xcite takes a distinctive approach by freezing the mlp modules of transformers and representing updates to attention weights with two low-rank matrices to optimize space while effectively retaining model performance. in this work, we focus on analyzing adapter-based fine-tuning methods as they are more popular and effective. mixture of expert adapters. additionally, several approaches @xcite @xcite have been proposed to further optimize their adapters for various downstream tasks by maintaining a set of adapters and combine them during inference. particularly, adamix wang et al. (2022) fine-tunes so-called mixture of experts (moes) with adapters on a downstream task and averaging their weights during inference. in addition, li et al. (2022) explores performance in novel domains through weight averaging on entire language models. similarly, m n l i m r p c q n l i q q p r t e s s t 2 r c t t w e e t s i m d b a g f i n a n a u t h o r w i k i mnli mrpc qnli qqp rte sst2 rct tweets imdb ag finan author wiki adaptersoup @xcite opts for weight-space averaging of adapters trained on different domains. among these methods, weightspace averaging is identified as the most intuitive method for mixing different adapters @xcite and @xcite . nevertheless, none of these works comprehensively evaluates and analyzes the generalizability and adversarial robustness of the resulting mixture of adapters under different mixtures of domainspecific knowledge, which is necessary to answer the question "when and what to mix?". ## comprehensive in-domain evaluation to evaluate the in-domain performance of adapter mixtures, we train several adapters with domainspecific knowledge and mix them in different combinatorial combinations. then, we evaluate each combination on different downstream tasks on two aspects: (1) generalizability on unseen in-domain examples and (2) adversarial robustness under adversarial text attacks. ## greedy adapter mixing with fsd our observations from sec. 5 reveal that two adapters with minimal disparity in fsd can yield competitive performance when combined. therefore, to demonstrate the utility of fsd, in this section, we design a mixing strategy, so-called greedy adapter mixing (alg. 1), that utilizes fsd to decide which domain-specific adapters to mix by minimizing the overall fsd in a greedy manner to get a final mixed model with competitive performance. this algorithm is also based on the hypothesis that mixing adapters with minimal fsd can yield mixture models of better generalizability. we proceed by evaluating model performance across various adapter combinations. fig. 7 shows the generalizability performance of a mixture of @xmath0, 13] domain-specific adapters. overall, greedy adapter mixing resulted in very competitive performance compared with empirical upper-bound accuracy. in contrast, using the same algorithm but maximizing fsd resulted in performance close to the empirical lower-bound accuracy (fig. 7 ). however, in both two cases, mixing adapters with fsd cannot achieve the empirical upper-bound and lower-bound performance. thus, greedily mixing adapters to minimize fsd does not totally prevent knowledge loss in the adapter mixtures. nevertheless, fsd is still useful as a guidance measure to effectively mix domain-specific adapters. ## towards effective model pruning to further demonstrate the utility of our fsd analysis in sec. 5 and sec. 6, in this section, we leverage fsd information to reduce knowledge loss through the development of a pruning algorithm guided by fsd insights. specifically, fig. 5 shows that predictive performance experiences a significant drop when integrating adapters with pronounced disparities in weight signs-i.e., positive v.s. negative signs. moreover, neural network pruning indicates that only a limited number of weights significantly contribute to task performance, suggesting redundancy within the weights that can be pruned without compromising the original task performance @xcite @xcite . thus, to mitigate the impact of weight sign differences in adapter mixtures, we propose mixing only the sparse versions of the adapters' weights. different from alg. 1, this strategy indirectly reduces the fraction of weight sign conflicts. intuitively, by minimizing the fsd, the mixing process becomes more resilient to the inadvertent elimination of important weights by less significant or redundant weights. this phenomenon is visually depicted in step 2 of fig. 1 , where only significant weights in the two adapters need to be preserved, and small or unimportant weights of opposing signs can be eliminated. ## conclusion this work provides a comprehensive empirical indomain evaluation of the emerging mechanism of mixing domain-specific adapters. we also provide insights into the inner workings of the mixture of domain-specific adapters by analyzing their weight signs, yielding critical observations on the negative correlation between the fraction of sign difference among adapters and their mixtures' generalizability. by examining the signed directions of adapter weights, we also offer the readers valuable advice on the optimal selection of adapters to mix to achieve competitive performance. such examination also helps enhance our understanding of the interconnected role of weight sign difference in the context of sparse neural networks.