LL a MA Pro: Progressive LL a MA with Block Expansion (2024)

Abstract: Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with an expansion of Transformer blocks. We tune the expanded blocks using only new corpus, efficiently and effectively improving the model’s knowledge while mitigating forgetting. In this paper, we experiment on the corpus of code and math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro and its instruction-following counterpart (LLaMA Pro - Instruct) achieve advanced performance among various benchmarks, demonstrating superiority over existing open models in the LLaMA family and the immense potential of reasoning and addressing diverse tasks as an intelligent agent. Our findings provide valuable insights into integrating natural and programming languages, laying a solid foundation for developing advanced language agents that operate effectively in various environments.

## introduction the advent of large language models (llms) has revolutionized the field of natural language processing, exhibiting remarkable proficiency in a variety of real-world tasks @xcite . despite the versatility, llms still fall short in certain domains, for example, programming, mathematics, biomedical, or finance. this limitation impedes the progress of developing generic language agents for broader applications. existing works @xcite @xcite attempted to improve the multifaceted capabilities of pre-trained llms with tai-* correspondence to yixiaoge@tencent.com. lored data recipes. while feasible, they require substantial computational resources and vast amounts of data, which poses a challenge to the democratization of llm research. consequently, another line of research, known as domain-adaptive pretraining, focuses on post-pretraining with domainspecific corpora @xcite . these approaches have demonstrated efficacy in adapting various llms to specific domains @xcite @xcite , resulting in enhanced performance on downstream domain-specific tasks at a reduced computational cost. nonetheless, a considerable obstacle emerges in catastrophic forgetting @xcite . post-pretraining often leads to a decline in the model's original general abilities, inhibiting the fine-tuned performance of the model on diverse tasks @xcite . this necessitates a method that can inject domainspecific knowledge into llms while preserving their general abilities, thereby enhancing their com-prehensive capabilities. towards this end, we introduce a simple yet effective post-pretraining method, termed block expansion. we expand the off-the-shelf pre-trained llm using copied transformer blocks, as illustrated in figure 2 . the newly added blocks, whose linear layers are zero-initialized to enable identity mapping, are further tuned with only domainspecific corpus while the remaining blocks are frozen. after tuning, the extended pre-trained model excels in both general and domain-specific tasks. in practice, we extend the pre-trained llama2-7b @xcite by eight more blocks, yielding llama pro, a foundation model with 8.3b parameters, and enhanced performance in programming, coding, and reasoning. we pre-train llama pro's expanded blocks on 80b tokens using open-source code and math data for 2830 gpu hours (16 nvidia h800 gpus for about 7 days). we further perform supervised instruction tuning (fully fine-tuning of all the blocks, aka sft) on llama pro with approximately 80m tokens, yielding llama pro -instruct. it is noted that pre-trained models produced by our block expansion method are well-compatible with the subsequent sft techniques without specific modification. as shown in figure 1 , llama pro -instruct reaches state-of-the-art performance across a broad range of general, code (i.e., hu-maneval), and math (i.e., gsm8k) tasks. furthermore, we assess the capabilities of llama pro -instruct as a language agent across various scenarios (i.e., mint-bench), with a focus on the tool usage abilities and the capacity to ground in environmental and human feedback. we also employ gpt-4 (openai, 2023) automatic evaluation to assess llama pro's ability to serve as an effective assistant (i.e., mt-bench). comprehensive experimental results indicate the superiority of llama pro -instruct over other models from the llama family on both benchmarks and practical applications. our contributions are three-fold: post-pretraining. language model applications typically involve a two-step process: generaldomain pretraining followed by domain-specific training @xcite . fine-tuning often aims to enhance instruction-following abilities @xcite @xcite or align model outputs with human preferences @xcite @xcite . some research explores parameter-efficient fine-tuning methods for adapting pretrained models to new domains @xcite @xcite , while others focus on continual learning post-pretraining @xcite @xcite . parameterefficient tuning methods like adaptor and lora are generally applied during the instruction tuning phase rather than the pretraining phase. in contrast, our focus is on enhancing the capacity of llms by increasing their depth during continued pretraining. our work proposes an adaptation strategy that combines continued training with general capability maintenance, allowing llms to specialize without sacrificing overall performance. progressive learning. progressive training has gained attention for accelerating large-scale model training in computer vision @xcite and nlp research @xcite . @xcite proposed a stacking method doubling model depth successively. compound-grow @xcite extends stacking with feed-forward network expansion in schedule design. @xcite introduced a staged method supporting hidden size expansion. bert2bert @xcite and ligo @xcite accommodate all growth dimensions. our method utilizes depth growth to maintain general performance while adapting to specific domains. ## experiments this section presents our key experimental findings. we begin with experimental settings (described in sec. 4.1), and then verify the effectiveness of block expanded tuning after pretraining (described in sec. 4.2). next, we give the supervised finetuning (sft) results (described in sec. 4.3). finally, ablation studies of the key design choices are presented (described in sec. 4.5). ## scope and limitations although our study presents a promising method for balancing general and domain-specific capabilities in llms, its scope is limited to the language modality, especially programming language and english. future research could explore extending the application of our block expansion method to other domains, such as maintaining original language ability in multimodal large language models @xcite , and multi-lingual domains. ## conclusion in this study, we introduced a novel block expansion method for large language models (llms) post-pretraining, aiming to enhance domain-specific abilities while preserving the original general capabilities. our approach effectively balances the model's performance across both general and domain-specific tasks. we demonstrated the effectiveness of our method through llama pro, an llm initialized from llama2-7b with 8 added blocks, which outperformed other llamaseries models on comprehensive benchmarks.