Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models (2024)

Abstract: As a promising paradigm to collaboratively train models with decentralized data, Federated Learning (FL) can be exploited to fine-tune Large Language Models (LLMs). While LLMs correspond to huge size, the scale of the training data significantly increases, which leads to tremendous amounts of computation and communication costs. The training data is generally non-Independent and Identically Distributed (non-IID), which requires adaptive data processing within each device. Although Low-Rank Adaptation (LoRA) can significantly reduce the scale of parameters to update in the fine-tuning process, it still takes unaffordable time to transfer the low-rank parameters of all the layers in LLMs. In this paper, we propose a Fisher Information-based Efficient Curriculum Federated Learning framework (FibecFed) with two novel methods, i.e., adaptive federated curriculum learning and efficient sparse parameter update. First, we propose a fisher information-based method to adaptively sample data within each device to improve the effectiveness of the FL fine-tuning process. Second, we dynamically select the proper layers for global aggregation and sparse parameters for local update with LoRA so as to improve the efficiency of the FL fine-tuning process. Extensive experimental results based on 10 datasets demonstrate that FibecFed yields excellent performance (up to 45.35% in terms of accuracy) and superb fine-tuning speed (up to 98.61% faster) compared with 17 baseline approaches).

## introduction as a promising paradigm to collaboratively train models with decentralized data, federated learning (fl) can be exploited to fine-tune large language models (llms) without aggregating the raw † equal contribution. * corresponding author: jiliuwork@gmail.com, yangzhou@auburn.edu data from a large number of devices @xcite @xcite @xcite @xcite . a number of stringent legal regulations (official journal of the european union, 2016; californians for consumer privacy, 2020) have been set up in order to protect the security and the privacy of personal data, which hinders the aggregation of the decentralized raw data. fl typically utilizes a parameter server @xcite @xcite to aggregate the distributed model updates in devices, which only transfers the parameters or the gradients of the updated models in replace of the raw personal data. by leveraging the distributed raw data of end users, large language models (llms) can be trained on devices with excellent performance @xcite . while @xcite has achieved remarkable progress, llms @xcite @xcite @xcite have attracted extensive attention. the size of llms ranges from several million parameters, e.g., roberta large @xcite , to several hundreds billion parameters @xcite . while the large scale brings strong capability in various natural language processing (nlp) tasks @xcite , the pre-training and the fine-tuning process of llms significant communication and computation costs @xcite . two types of parameter-efficient approaches exist for reducing the number of parameters within the fine-tuning process of llms. prompt tuning @xcite @xcite can dynamically adjust the prompts to finetune llms with only a few trainable parameters, which may introduce performance degradation. although low-rank adaptation (lora) @xcite can significantly reduce the scale of parameters to update in the fine-tuning process of llms so as to enable the training of llms on edge devices @xcite , it still takes unaffordable time to update the low-rank parameters of all the layers in llms when dealing with decentralized data. while being an effective method to improve the efficiency and effectiveness of training process, curriculum learning @xcite is exploited to train large-scale models @xcite . inspired by the learning strategy of starting small @xcite , the curriculum training process starts with easier data and then gradually increase the difficulty. instead of randomly sampling the batch from training dataset, curriculum learning allows the model to gradually learn from easy samples to hard samples during the training or the fine-tuning process. existing approaches generally measure the complexity of samples based on heuristic methods @xcite or a simple mode-based method @xcite , both of which cannot provide an accurate estimation of difficulty of data samples and cannot be directly applied in fl. in the context of fl, the data is generally non-independent and identically distributed (non-iid), which requires an adaptive difficulty evaluation approach for diverse devices @xcite . model compression methods, e.g., pruning @xcite @xcite or sparse training @xcite , are exploited in fl to reduce computation and communication costs. sparse training can achieve personalization so as to further improve the performance of fl @xcite . however, the pruning or sparse training incurs severe accuracy degradation due to lossy strategies or simple component (neuron) selection mechanisms. in addition, the model can be split into two parts, i.e., the server part and the device part, in order to achieve both the generalization and personalization capability @xcite . fisher information can be exploited to accelerate the training process of llms @xcite @xcite . fisher information is defined as the amount of information carried by a random variable corresponding to some unknown parameters @xcite . as a measure of the local curvature @xcite , fisher information matrix (fim) defines the riemannian metric of the parameter space @xcite , which can indicate the difficulty of data samples and the importance of each component of the network along with the generalization performance @xcite . in this paper, we propose fibecfed, i.e., a fisher information-based efficient curriculum federated learning framework. fibecfed is composed of two novel methods, i.e., adaptive federated curriculum learning and efficient sparse parameter update. to the best of our knowledge, we are among the first to exploit the fisher information to perform curriculum learning and sparse training at the same time within fl settings. we summarize out major contributions as follows: the rest of the paper is organized as follows. the related work is presented in section 2. we formulate the problem to address in section 3. we present the architecture of fibecfed and propose the adaptive federated curriculum learning and the efficient sparse parameter update method in section 4. we demonstrate the experimental results in section 5. finally, section 6 concludes. ## related word & preliminaries inspired by the learning strategy of starting small @xcite , curriculum learning @xcite is exploited in large-scale model training @xcite . existing works measure the complexity of samples based on static characteristics of data samples, e.g., sequence length @xcite . although a simple global mode-based method is proposed to predict the performance improvement based on several training states @xcite , it still cannot provide an accurate estimation of difficulty of data samples due to non-iid data in fl settings. direct evaluation based on the inference loss of models @xcite cannot well explore the impact on the generalization of the training process. the attention scores can analyze the dependency among diverse layers, but varies significantly between heads (vig and belinkov, 2019), which cannot be directly utilized in fl settings. while a sharpness-aware minimization method @xcite can help minimize loss value and loss sharpness to improve model generalization, it does not consider federated fine-tuning settings of llms. fim can be exploited to enable the second-order optimization so as to improve the training process @xcite and to compute a global posterior for federated learning @xcite . in addition, continual learning can be used to improve the performance of trained models while addressing the forgetting problem @xcite . different from @xcite @xcite , we exploit the sum of diagonal of fim to evaluate the difficulty of samples within the efficient curriculum learning method and to calculate the importance score of each layer and neuron within the llm. model compression methods @xcite are exploited in fl to reduce both computation and communication costs. although pruning methods can reduce the size of large models @xcite @xcite , it is complicated to choose a proper pruning rate and may incur inferior performance in terms of accuracy @xcite . sparse training can achieve personalization @xcite @xcite while addressing the client shift problem brought by the non-iid data @xcite . however, the existing sparse training methods may incur severe accuracy degradation with poor generalization capacity due to simple component selection mechanisms. the model can be split into a server part and a device part to achieve both generalization and personalization capability @xcite , which still incurs severe computation and communication costs in the fl settings of llms. please note our approach is orthogonal with model compression methods. for nlp tasks, prompt tuning @xcite @xcite can fine-tune llms with only a few parameters. with prompt tuning, an extra network is exploited to generate proper prompts or prefix, which is concatenated with the input to guide llms to generate proper answers. furthermore, lora updates trainable rank decomposition matrices while freezing the parameters of the original network, which can significantly reduce the scale of parameters to update @xcite . however, both the prompt tuning and lora still incur significant communication costs due to the update for all the layers. ## problem formulation in this paper, we delve into the problem of how to efficiently fine-tune a large language model within a fl setting. the fl setting is composed of a parameter server and k devices. we assume that the data samples are distributed among the devices, each of which contains a dataset d @xmath0, m i } n k with s i , m i , and n k referring to a data sample, the corresponding label, and the cardinality of d k . we denote the cardinality of the whole dataset we consider a llm m of l layers, each layer contains a full parameter matrix w l o . we exploit the lora method to reduce the parameters to update in this paper @xcite , and denote the lora parameters of the the llm m by p with p l representing the set of lora parameters in layer l of m. we denote the updated lora parameters on device k by p k . then, we formulate the problem to address in this paper as how to efficiently update p so as to minimize the global loss: ## efficient curriculum federated learning in this section, we first explain the system model. then, we propose the adaptive federated curriculum learning method. afterward, we further detail the efficient sparse tuning method. ## experiments in this section, we demonstrate extensive experimentation with 17 baseline approaches and 10 nlp tasks to reveal the advantages of fibecfed. ## conclusion in this paper, we propose an original fisher information-based efficient curriculum federated learning, i.e., fibecfed. within fibecfed, we propose an adaptive federated curriculum learning method and an efficient sparse parameter update method. we exploit fisher information to calculate the difficulty scores of data samples and propose the an original curriculum data selection strategy. in the sparse parameter update method, we propose a new sensitivity-based important layer selection technique and a novel fisher information-based important parameter technique method while freezing the remaining parameters to achieve both efficiency and effectiveness. we demonstrate the results of extensive experimentation to compare fibecfed with 17 baseline approaches based on 10 nlp tasks, which reveal significant advantages of fibecfed in terms of accuracy (up to 45.35%) and fine-tuning speed (up to 98.61% faster).