MEFT : Memory-Efficient Fine-Tuning through Sparse Adapter (2024)

Abstract: Parameter-Efficient Fine-tuning (PEFT) facilitates the fine-tuning of Large Language Models (LLMs) under limited resources. However, the fine-tuning performance with PEFT on complex, knowledge-intensive tasks is limited due to the constrained model capacity, which originates from the limited number of additional trainable parameters. To overcome this limitation, we introduce a novel mechanism that fine-tunes LLMs with adapters of larger size yet memory-efficient. This is achieved by leveraging the inherent activation sparsity in the Feed-Forward Networks (FFNs) of LLMs and utilizing the larger capacity of Central Processing Unit (CPU) memory compared to Graphics Processing Unit (GPU). We store and update the parameters of larger adapters on the CPU. Moreover, we employ a Mixture of Experts (MoE)-like architecture to mitigate unnecessary CPU computations and reduce the communication volume between the GPU and CPU. This is particularly beneficial over the limited bandwidth of PCI Express (PCIe). Our method can achieve fine-tuning results comparable to those obtained with larger memory capacities, even when operating under more limited resources such as a 24GB memory single GPU setup, with acceptable loss in training efficiency. Our codes are available at https://github.com/CURRENTF/MEFT.

## introduction as large language models (llms) scale up, their fine-tuning becomes increasingly resourceintensive @xcite . to address this, parameter efficient fine-tuning (peft) methods, which add few trainable modules that only fraction of the llms' full parameters (e.g., 0.1%), have shown promising results on many scenarios, often competitive to full finetuning @xcite ; li and liang, 2021; @xcite . for instance, lora employs matrix decomposition parameters to leverage the low-rank characteristics of llm parameters, while parallel adapter introduces an additional ffn adapter that operates parallel to original ffn layer. however, the number of new parameters required can vary depending on the task. for knowledge-intensive tasks, such as qa, it becomes necessary to fine-tune a greater number of parameters to increase the model's capacity for new knowledge memorization @xcite . figure 1 illustrates the performance of llama-7b with parallel adapter on nq datasdet @xcite . a clear upward trend in answer accuracy is observed as adapter's parameter count increases. concretely, for achieving optimal results, approximately 10% of the parameters are updated, which corresponds to more than 24g of gpu memory usage (the orange part in the figure). therefore, reducing the memory cost when the trainable parameters increase is important and has attached research attentions. among them, cpuoffload @xcite offloads model parameters to cpu memory, which are then returned to the gpu only when computation reaches the corresponding layer. however, cpu-offload transfers all layer parameters, which is inflexible and incurs a high communication overhead. to address these challenges, we propose meft, which introduces sparse activation to reduce memory usage. for a given input, sparse activation implies that only a few neurons are activated, as evidenced by various empirical results @xcite and our pilot experiments detailed at section 2. inspired by this, meft places all trainable parameters in cpu memory to leverage the advantage of the large capacity of cpu memory. for a given input, meft retrieves a few neurons from cpu memory that are highly relevant to input, and moves them to gpu to complete the computation. in this way, meft significantly reducing gpu memory usage. furthermore, to reduces the computational burden on the cpu, we introduce a novel mixture of experts (moe)-based adapter to partition parameters. specifically, given an input, a router is used to activate a subset of the large-scale network and conduct further computation only on these activated subset. this approach effectively reduces the computational complexity from o(dn m ) to o(dn √ m ) to complete the retrieval of relevant parameters. we conduct experiments on two models (i.e., llama and mistral) and four datasets (i.e., nq, squad, toolbench, and gsm8k). experimental results on knowledge-intensive tasks show that our method achieves the best results under resourcerestricted conditions. specifically, meft reduces gpu memory usage by 50% (e.g., reducing from 48g to 24g) while achieving comparable benchmark results. our method also outperforms other peft methods like parallel adapter and lora when using the same memory capacity. in summary, our contributions are as follows: (i) we propose meft, a novel method that utilizes sparse activations and moe for memory-efficient fine-tuning (ii) meft reduces communication overhead by artificially limiting number of activated neurons copied to gpu memory from cpu memory. (iii) we propose a key-experts mechanism to partition a large number of parameters, using a router to allocate inputs to corresponding experts, reducing unnecessary computations and lowering cpu computational burden. (iv) our experimental results on knowledge-intensive tasks using two popular llms demonstrate that our method can achieve the best results under resource-constrained conditions, and the results are comparable to those in relatively resource-rich situations. ## preliminary peft concentrates on fine-tuning llms under resource-restricted conditions. to examine performances of different peft on knowledge-intensive tasks, we fine-tune llama-7b on natural questions with lora, adapterh, and parallel adapter. figure 1 shows the results of the experiments. we can see from the left blue part that it is challenging to inject knowledge into the model when the number of trainable parameters is limited (e.g., 0.1%). parallel adapter demonstrates the best scaling results, but for achieving optimal accuracy, approximately 10% of the parameters are updated. therefore, we argue that to memorize more knowledge required by downstream tasks, the number of trainable parameters needs to be increased accordingly. our study is based on parallel adapter due to its superior performance. next, we introduce parallel adapter and our motivation for making it more efficient. ## method we propose meft, which dynamically loads parameters from cpu memory to gpu to train a larger size adapter. in section 3.1, we introduce sparse activation to reduce communication overhead, followed by the key-experts mechanism in section 3.2 to reduce the computational complexity of cpu operations. we further analyze the efficiency of our method in section 3.3. ## related work parameter-efficient fine-tuning (peft) can be categorized into four main types: (1) adapter-based tuning @xcite , (2) prompt-based tuning @xcite @xcite , (3) sparse tuning @xcite , and (4) reparametrizationbased tuning @xcite . there are several studies that are relevant to our work. @xcite utilized a unified perspective of previous peft methods to propose the parallel adapter, which has demonstrated superior performance among most popular peft methods @xcite . in our study, we found the parallel adapter to be highly effective in knowledge memorization. @xcite proposed adamix as a method that tunes a mixture of adaptation modules but only with 0.1 -0.2% of llm's parameters. furthermore, zhang et al., 2023 preserved the w a in the lora module @xcite in a frozen state, which resulted in reduced memory usage but didn't try to scale up. mixture of experts is a computationally efficient architecture that only activates a subset of a large-scale neural network compared to a dense model of the same size @xcite @xcite . we leverage the computational efficiency of moe to alleviate the issue of insufficient cpu flops and take advantage of the large memory capacity of the cpu to compensate for its tendency to occupy more memory. additional minor related works can be found in appendix e. ## conclusion in this paper, we found that the parallel adapter can continuously improve performance on knowledgeintensive tasks as parameters increase. we then proposed a memory-efficient training method by leveraging sparsity and moe. this method significantly reduces the demand for gpu memory and reduces the computational pressure on the cpu. experimental results show that our method achieves the best results under resource-restricted conditions.