What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition (2024)

Abstract: The knowledge encapsulated in a model is the core factor determining its final performance on downstream tasks. Much research in NLP has focused on efficient methods for storing and adapting different types of knowledge, e.g., in dedicated modularized structures, and on how to effectively combine these, e.g., by learning additional parameters. However, given the many possible options, a thorough understanding of the mechanisms involved in these compositions is missing, and hence it remains unclear which strategies to utilize. To address this research gap, we propose a novel framework for zero-shot module composition, which encompasses existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. Focusing on the scenario of domain knowledge and adapter layers, our framework provides a systematic unification of concepts, allowing us to conduct the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. In particular, we test two module combination methods and five selection and weighting strategies for their effectiveness and efficiency in an extensive experimental setup. Our results highlight the efficacy of ensembling but also hint at the power of simple though often-ignored weighting methods. Further in-depth analyses allow us to understand the role of weighting vs. top-k selection, and show that, to a certain extent, the performance of adapter composition can even be predicted.

## introduction pre-trained language models (plms), e.g., the gptfamily @xcite , determine the current state-of-the-art in natural language processing (nlp), which has often been attributed to the rich knowledge they encapsulate in their parameters (e.g., @xcite . previous research has heavily focused on utilizing the plms' knowledge in various scenarios, figure 1 : our unified framework for on-demand module composition consisting of three steps: selection, weighting, and final combination. we show the example of zero-shot domain adaptation with adapter layers. besides the numerous practical advantages of knowledge modularization -such as parameterefficiency @xcite , avoiding catastrophic forgetting @xcite , and reducing negative interference @xcite -researchers have shown the benefits of re-using and re-combining already existing modules @xcite . based on this idea, a particularly attractive scenario is the on-demand selection and combination of knowledge modules at inference time. to do so, there exist a plethora of potential strategies: mod-ules can be selected by computing sentence similarities and domain clusters @xcite , domain priors @xcite , and model entropy @xcite . then, they can be combined with a weight space averaging, following the idea of a "model soup" @xcite , or output vector ensembling @xcite . however, despite the existence of a variety of knowledge composition methods, there is (a) no comprehensive overview and evaluation of those methods, and (b) no unified view on knowledge composition that could facilitate this process. the composition methods introduced for various objectives have not been tested in a comparable setup (e.g., @xcite , do not focus on zero-shot domain adaptation, in contrast to @xcite ), and various factors (e.g., the number of modules to select, and whether to additionally weight each module in the composition) have not been systematically taken into account. we shed light on these, focusing on the specific case of zero-shot domain adaptation with adapter layers. given a series of adapters originating from domain-specific training, we address the problem of how to choose and combine adapters to improve the performance on unseen evaluation domains. contributions. our contributions are three-fold: our experiments show that w.r.t. combination strategies, output vector ensembling is often superior to parameter averaging, supporting findings from recent work @xcite . importantly, we observe that corpus-based weighting and selection strategies (tf-idf and sentence simi-larity) often outperform more complex modelbased approaches, while also being more efficient. our study on meta-regression shows that zero-shot domain adaptation performance is partially predictable, particularly for specific adapter combinations. we hope that our work will advance efficient and effective nlp. for full reproducibility, we release all code publicly under https: //github.com/uhhds/whattheweight . ## a unified composition framework in this section, we present our unified framework for knowledge module composition. we base our explanation on the scenario of domain adaptation using adapters as the underlying module. our framework is, however, generic and can be applied to various composition scenarios. the problem of composing knowledge boils down to the following: let θ i be the parameters of n adapters trained via language modeling on n domains d 1 , ..., d n while the original model parameters ϕ are kept frozen. given an unseen evaluation domain d n+1 , the task is to effectively adapt to d n+1 via an optimal domain composition. as illustrated in figure 1 , our approach to such a composition relies on three steps: (1) identify k suitable adapters; (2) apply a weighting to the selected adapters; (3) perform the final combination. in the following, we describe the scoring and the combination strategies, implemented in our framework and used for conducting the experiments. ## benchmarking composition strategies we use our framework to benchmark module composition strategies for zero-shot domain adaptation. ## meta-regression in §3, we have shown that adding more adapters (i.e., increasing k) often does not lead to performance gains, and that the effectiveness of the scoring strategies varies across models and evaluation domains. motivated by these results, here, we analyze to what extent we are able to predict the expected performance for particular compositions. ## related work we cover the related literature concerning the topics of knowledge modularization and knowledge composition. for a thorough overview of modular deep learning, we refer to @xcite . modularizing knowledge. famously, @xcite proposed to use adapter layers @xcite as a more efficient alternative to full task-specific fine-tuning. subsequently, researchers in nlp explored adapters for various purposes, e.g., domain adaptation (e.g., @xcite @xcite , bias mitigation (e.g., @xcite @xcite , language adaptation (e.g., @xcite , and for the injection of various other types of knowledge, such as common sense @xcite , factual @xcite , and sociodemographic knowledge @xcite . similarly, much effort has been spent designing new adapter variants with the aim of further increasing their efficiency or effectiveness (e.g., @xcite @xcite . alternatives to adapters that support modularity include subnetworks @xcite obtained via sparse fine-tuning, prefix tuning @xcite , and mixture-of-expert (moe; @xcite models. the latter, exemplified by switch transformers @xcite , integrate a learned gating mechanism to channel inputs to appropriate expert modules. like other modularization techniques, moes have been studied extensively for a wide range of problems (e.g., @xcite @xcite . most relevant to us, they have also been used to modularize different types of domain knowledge @xcite . in this context, recent studies have considered experts as entirely autonomous models, challenging prevailing efficiency paradigms @xcite @xcite . composing knowledge. the composition of knowledge modules can be conducted via optimizing additional parameters (e.g., @xcite , or in a zero-shot manner (e.g., @xcite . falling under the first category of approaches, @xcite proposed the fusion of adapters based on weights obtained via learned attention matrices. the same mechanism has been adopted by @xcite , dubbed knowledge controller. in a similar vein, @xcite ensemble the output vectors of multiple language adapters and optimize the respective ensemble weights. @xcite and @xcite compose moe models by learning to route the input to the right modules. most recently, @xcite propose to directly learn scaling parameters for efficient knowledge composition in task transfer. in this work, we are interested in zeroshot knowledge composition. in this realm, @xcite rely on weight space averaging and simple selection strategies. @xcite and @xcite compare ensembling and averaging for composing domain plms, relying on domain prior for selection. until now, a unified view is missing. ## conclusion in this work, we proposed a unified framework providing an interoperable notion of zero-shot knowledge composition. using our framework, we analyzed the effectiveness of different knowledge module selection, weighting, and combination strategies. we studied the problem of domain adaptation with adapters and showed, for instance, that ensembling generally yields better results than parameter averaging. examining five different scoring strategies, we found that even simple approaches can deliver strong results. our findings also suggest that the number of adapters selected is generally more important than the weights assigned to them. while we have chosen the popular scenario of zero-shot domain adaptation with adapter layers, we are convinced that our framework is applicable to many other problems and modularization techniques (e.g., moes, entire models). overall, we believe that our results will fuel future research in effective knowledge composition by providing a consolidated perspective on zeroshot module composition.