Client-Customized Adaptation for Parameter-Efficient Federated Learning (2023)

Abstract: Despite the versatility of pre-trained language models (PLMs) across domains, their large memory footprints pose significant challenges in federated learning (FL), where the training model has to be distributed between a server and clients. One potential solution to bypass such constraints might be the use of parameter-efficient fine-tuning (PEFT) in the context of FL. However, we have observed that typical PEFT tends to severely suffer from heterogeneity among clients in FL scenarios, resulting in unstable and slow convergence. In this paper, we propose Client-Customized Adaptation (C2A), a novel hypernetwork-based FL framework that generates client-specific adapters by conditioning the client information. With the effectiveness of the hypernetworks in generating customized weights through learning to adopt the different characteristics of inputs, C2A can maximize the utility of shared model parameters while minimizing the divergence caused by client heterogeneity. To verify the efficacy of C2A, we perform extensive evaluations on FL scenarios involving heterogeneity in label and language distributions. Comprehensive evaluation results clearly support the superiority of C2A in terms of both efficiency and effectiveness in FL scenarios.

## introduction the advent of large-scale pre-trained language models (plms) for natural language processing (nlp) has led to exceptional performance across a broad spectrum of domains. however, the high memory requirements for plms impede their applicability to resource-constrained environments. these challenges are particularly evident in federated learning (fl), where model weights are transmitted between the server and clients to preserve data privacy (konečný et al., 2016; mcmahan et al., * these authors contributed equally to this work. 1 our code is available at https://github.com/ yeachan-kr/c2a 2017). while recent fl studies have expanded the application of plms in various tasks, such as text classification @xcite @xcite , language modeling @xcite , and question answering @xcite , communicating the training model among clients requires huge computational resources and bandwidth, presenting a significant challenge in terms of practicality. parameter-efficient fine-tuning (peft) approach is thereby a promising strategy for reducing communication costs in fl. through tuning only a small fraction of parameters, such as adapter-based tuning @xcite @xcite , bias tuning @xcite , and prompt-tuning @xcite , peft approaches significantly enhance the memory efficiency in centralized scenarios. however, the feasibility of peft in decentralized scenarios has not been well explored. hence, we investigate the applicability of typical peft approaches in fl scenarios. specifically, we measure the performance and client drifts @xcite of peft approaches in fl. our discoveries are as follows: (1) typical peft approaches show large performance degradation in fl scenarios as the degree of non-iid increases; (2) these approaches usually suffer from large client drifts in non-iid scenarios, resulting in slow convergence and detrimental model performance. the above observations reveal that adopting peft in fl is not trivial, and posing the necessity to address large client drift. to overcome the identified limitations, we propose a novel hypernetwork-based fl framework, client-customized adaptation (c2a), that leverages the information of different data distributions on clients. our key idea is to generate the adapter parameters tailored to each client via hypernetworks by taking the information of client data distribution, rather than naively fitting a single global adapter to all heterogeneous data distributions (figure 1 ). by learning to adopt the different data distributions to generate adapters for each client, c2a enables robust training for various non-iid conditions while sharing knowledge among clients. moreover, in order to manage the large number of parameters associated with hypernetworks, we introduce factorized hypernetworks, thereby significantly reducing the number of parameters without sacrificing the performance. we carefully design the experimental setting to verify the efficacy of c2a on realistic fl scenarios, considering on both label and language heterogeneous. the experimental results show clearly that c2a can be robust to the heterogeneity of clients, thereby leading to the state-of-the-art results on diverse non-iid setups. in addition, our framework shows a significant enhancement in training efficiency across a range of downstream tasks. finally, we demonstrate that our c2a successfully mitigates the large client drifts among local clients in non-iid scenarios. a summary of our main contributions is as follows: ## c2a: client-customized adaptation in this section, we elaborate on the proposed framework in detail. the core strategy is to generate customized peft modules tailored to each client to mitigate the negative impact of heterogeneity among clients. to achieve this, we first derive latent vectors to represent the data distribution of each client (section 3.2). the resulting embeddings are then conditioned on the hypernetworks so as to generate parameters of the peft modules tailored to each client (section 3.3). regarding on the large number of parameters induced from hypernetworks, we effectively factorize the weights of the hypernetworks (section 3.4). ## evaluation in this section, we evaluate the efficacy of our c2a on two realistic fl scenarios: 1) heterogeneity in label distributions, and 2) heterogeneity in both label and language distributions. ## further analysis on c2a in order to gain a deeper understanding of the benefits of c2a, we perform a series of analytical experiments utilizing xglue-nc with a value of β = 0.5, which represents the most challenging setup within our experimentation. ## conclusion in this paper, we have observed significant performance degradation for typical peft approaches in decentralized scenarios. by carefully designed analysis, we have also shown that typical peft suffers from large client drifts, resulting in slow convergence and performance degradation. to address these issues, we have proposed c2a, a novel hypernetwork-based fl framework, which generates client-customized adapters by incorporating the data distribution of each client. our experimental results show that c2a achieves state-of-the-art results in various decentralized scenarios. moreover, we have verified that c2a successfully mitigates the large client drift problem among local clients in fl scenarios.