L lama F actory: Unified Efficient Fine-Tuning of 100+ Language Models (2024)

Abstract: Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It provides a solution for flexibly customizing the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at https://github.com/hiyouga/LLaMA-Factory and received over 25,000 stars and 3,000 forks.

## introduction large language models (llms) @xcite present remarkable reasoning capabilities and empower a wide range of applications, such as question answering @xcite , machine translation @xcite , and information extraction @xcite . subsequently, a substantial number of llms are developed and accessible through open-source communities. for example, hugging face's open llm leaderboard @xcite boasts over 5,000 models, offering convenience for individuals seeking to leverage the power of llms. fine-tuning extremely large number of parameters with limited resources becomes the main challenge of adapting llm to downstream tasks. a popular solution is efficient fine-tuning @xcite @xcite , which reduces the training cost of llms when adapting to various tasks. however, the community contributes various methods for efficient finetuning, lacking a systematic framework that adapts and unifies these methods to different llms and provides a friendly interface for user customization. ## related work with the rapid increase in demand for fine-tuning llms, numerous frameworks for adapting llms to specific purposes have been developed. llama-adapter @xcite efficiently fine-tunes the llama model @xcite using a zero-initialized attention. fastchat @xcite is a framework focused on training and evaluating llms for chat completion purposes. litgpt (ai, 2023) provides the implementation of generative models and supports various training methods. open-instruct @xcite provides recipes for training instruct models. colossal ai @xcite takes advanced parallelism strategies for distributed training. lmflow @xcite supports training llms for specialized domains or tasks. gpt4all @xcite allows llms to run on consumer devices, while also providing fine-tuning capabilities. compared with existing competitive frameworks, llamafactory supports a broader range of efficient fine-tuning techniques and training approaches. we list the features among representative frameworks in ## efficient fine-tuning techniques efficient llm fine-tuning techniques can be divided into two main categories: those focused on optimization and those aimed at computation. the primary objective of efficient optimization techniques is to fine-tune the parameters of llms while keeping costs to a minimum. on the other hand, efficient computation methods seek to decrease the time or space for the required computation in llms. the methods included in llamafactory are listed in ## llamafactory framework llamafactory consists of three main modules: model loader, data worker, and trainer. the model loader manipulates various model architectures for fine-tuning, supporting both large language models (llms) and vision language models (vlms). the data worker processes data from different tasks through a well-designed pipeline, supporting both single-turn and multi-turn dialogues. the trainer applies efficient fine-tuning techniques to different training approaches, supporting pretraining, instruction tuning and preference optimization. beyond that, llamaboard provides a friendly visual interface to access these modules, enabling users to configure and launch individual llm fine-tuning instance codelessly and monitor the training status synchronously. we illustrate the relationships between these modules and the overall architecture of llamafactory in figure 1 . ## empirical study we systematically evaluate llamafactory from two perspectives: 1) the training efficiency in terms of memory usage, throughput and perplexity. 2) the effectiveness of adaptation to downstream tasks. ## conclusion and future work in this paper, we demonstrate llamafactory, a unified framework for the efficient fine-tuning of llms. through a modular design, we minimize dependencies between the models, datasets and training methods and provide an integrated approach to fine-tune over 100 llms with a diverse range of efficient fine-tuning techniques. additionally, we offer a flexible web ui llamaboard, enabling customized fine-tuning and evaluation of llms without coding efforts. we empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. we will consistently keep llamafactory synchronous with the state-of-the-art models and efficient fine-tuning techniques. we also welcome contributions from the open-source community. the road map of llamafactory including: