Sparsity-Accelerated Training for Large Language Models (2024)

Abstract: Large language models (LLMs) have demonstrated proficiency across various natural language processing (NLP) tasks but often require additional training, such as continual pre-training and supervised fine-tuning. However, the costs associated with this, primarily due to their large parameter count, remain high. This paper proposes leveraging sparsity in pre-trained LLMs to expedite this training process. By observing sparsity in activated neurons during forward iterations, we identify the potential for computational speed-ups by excluding inactive neurons. We address associated challenges by extending existing neuron importance evaluation metrics and introducing a ladder omission rate scheduler. Our experiments on Llama-2 demonstrate that Sparsity-Accelerated Training (SAT) achieves comparable or superior performance to standard training while significantly accelerating the process. Specifically, SAT achieves a 45% throughput improvement in continual pre-training and saves 38% training time in supervised fine-tuning. It offers a simple, hardware-agnostic, and easily deployable framework for additional LLM training.

## introduction large language models (llms), such as gpt-4 @xcite , mistral @xcite , and llama-2 @xcite , have demonstrated remarkable capabilities across numerous nlp tasks @xcite . in general, all these models are initially pre-trained on massive data by unsupervised learning. furthermore, such models regularly necessitate additional training for two primary scenarios: 1) continual pre-training on new data as the pool of available pre-training data continuously expands (gupta * the corresponding authors are lu chen and kai yu. ## methodology in this section, we first give an overview of sat. next, more details are provided within the context of transformers, which is the predominant architecture of recent llms. finally, we discuss our implementation and conduct a theoretical efficiency analysis for the sat. ## experiments we conduct comprehensive experiments to address the following three questions: q1: what is the optimal configuration, including assessment of neurons importance (across various metrics) and neurons selection (top-k vs. sampling), with lors for sat? q2: does sat harm the performance of llms in both continual pre-training and supervised fine-tuning scenarios? q3: what is the efficiency achieved by sat over popular llms in practice? ## related works transformer pruning pruning has demonstrated considerable potential in augmenting the inference speed and reducing the memory footprint of transformers, as highlighted in prior research @xcite . broadly speaking, transformer pruning techniques can be delineated into two principal categories @xcite : unstructured pruning and structured pruning. unstructured pruning methods @xcite @xcite possess the capability to substantially diminish the number of parameters due to their fine-grained pruning granularity. nonetheless, they frequently encounter challenges in achieving significant improvements in inference speed across diverse hardware platforms @xcite . hence, numerous researchers are embracing structured pruning methods at the granularity level of entire layers, filters, channels, or heads @xcite . for instance, liu et al. prunes the unimportant heads within multi-head attention layers and insignificant channels in feed-forward networks on bert (kenton and toutanova, 2019). liu et al. and song et al. extend such a method and scale it for large language models. in this study, distinct from their focus on the inference of large language models (llms), we draw upon the principles of structured pruning and apply them to the training of llms. sparse fine-tuning sparse fine-tuning is a technique aimed at expediting model fine-tuning and diminishing model memory demands by selectively updating only a small subset of parameters @xcite . for relatively small language models, such as bert, guo et al. devise a binary mask controlled by a regularization term to manage the parameters for updating. @xcite induce the mask by retaining parameters exhibiting higher fisher information over numerous iterations and subsequently maintain the mask to sparsely fine-tune models. additionally, ansell et al. retain parameters that undergo the most significant changes during an initial round of full fine-tuning. with the widespread adoption of llms, @xcite scaling sparse fine-tuning to llms. however, to mitigate the memory overhead of llms, they both incorporate the parameter-efficient fine-tuning (peft) methods @xcite @xcite , which may not always be entirely suitable @xcite . in this work, instead of employing the peft methods, we place greater emphasis on conventional sparse training techniques, making it applicable to common scenarios such as continual pre-training and supervised fine-tuning for llms. ## conclusion in this paper, we aim to accelerate two regular additional training scenarios of llms, i.e., continual pre-training and supervised fine-tuning. our main intuition is to leverage the sparsity of neurons in large models, as discovered by previous researchers, to accelerate training by disregarding computations for unimportant neurons and propose a sparsity-accelerated training (sat) framework. extensive experiments on several benchmarks demonstrate sat can accelerate the training of llms and maintain performance simultaneously.