Llama SL ayer 8 B : Shallow Layers Hold the Key to Knowledge Injection (2024)

Abstract: As a manner to augment pretrained large language models (LLM), knowledge injection is critical to develop vertical domain large models and has been widely studied. While most current approaches, including parameter-efficient fine-tuning (PEFT) and block expansion methods, uniformly apply knowledge across all LLM layers, it raises the question: are all layers equally crucial for knowledge injection? We embark upon evaluating the importance of each layer to locate the optimal layer range for knowledge injection. Intuitively, more important layers should play more critical roles in knowledge injection and deserve denser injection. We observe performance dips in question-answering benchmarks after the removal or expansion of the shallow layers, and the degradation shrinks as the layer gets deeper, indicating that the shallow layers hold the key to knowledge injection. This insight leads us to propose the S strategy, a post-pretraining strategy of selectively enhancing shallow layers while pruning the less effective deep ones. Based on this strategy, we introduce Llama Slayer 8B. We experimented on the corpus of code & math and demonstrated the effectiveness of our strategy. Further experiments across different LLM, Mistral-7B, and a legal corpus confirmed the approach’s general applicability, underscoring its wide-ranging efficacy.

## introduction large language models (llms) have significantly revolutionized the natural language processing area, showcasing unparalleled abilities across various tasks @xcite . despite their versatility, llms exhibit limitations in specialized areas such as mathematics, programming, etc., which * work done during an internship at alibaba cloud. to address these gaps, existing work @xcite has sought to enhance the diverse skills of pre-trained llms through customized data strategies. however, they require extensive computational efforts and massive data volumes, challenging the widespread accessibility of llm research. furthermore, while parameter-efficient fine-tuning (peft) techniques offer a reduction in training requirements, their effectiveness tends to diminish @xcite compared to traditional fine-tuning methods, especially as the size of the model and the dataset grows. subsequently, another line of research emerged, focusing on methods such as model merging @xcite and model expansion @xcite @xcite . model merging methods strive to synthesize a multifaceted model that amalgamates insights from various pre-trained domain-specific llms, potentially crafting a model adept at addressing a multitude of tasks concurrently. however, the process of training multiple domain-specific llms is resource-intensive. on the other hand, model expansion methods, exemplified by llama pro, seek to refine pre-trained models for domain-specific applications in the post-pretraining phase by only fine-tuning the expanded layers. therefore, it can employ significantly fewer trainable parameters than full model fine-tuning. however, present model expansion methods generally treat each part of llms equally, although different layers may exhibit varying sensitivity to incorporated knowledge. this lack of differentiation can result in less-than-ideal knowledge injection results. an intuitive idea is to inject knowledge into the most important layers so that the llm can more sufficiently leverage the new knowledge without the overhead of redundant adjustments across all layers. to this end, we select three different metrics to evaluate the importance of each layer to find which part of the llm is more important to knowledge injection. our findings suggest that the shallow layers are more important compared to the last few layers, as the drop in precision -whether through the removal or addition of the last few layers -is markedly less significant than the drops induced by manipulating the shallow layers. drawing on this insight, we propose s strategy, a novel strategy to knowledge injection that concentrates on enriching the shallow layers while deleting the least important deepest layers. based on the proposed strategy, we further introduce llama slayer-8b and llama slayer-8b-instruct, versatile llms that excelling in programming, and mathematics and general language tasks. figure 2 displays the superiority of llama slayer-8b-instruct. the main contributions of this paper can be summarized to three aspects: ## related works here we introduce four prevalent types of methods for injecting domain-specific knowledge, including full fine-tuning, parameter-efficient fine-tuning, model merging, and model expansion. ## experiments in this section, we detail our key experimental findings. initially, we discuss our experimental settings (described in sec. 4.1), and then provide our postpretraining results (cpt+sft) (described in sec. 4.2). finally, ablation studies of the different block expansion mode, different llm and data corpus are presented (described in sec. 4.3). ## conclusion we propose s strategy to inject domain-specific knowledge into llms via block expansion based on layer importance in the post-pretraining phase. this strategy prioritizes knowledge injection to the important shallow layers while pruning the ineffective deep layers, and can not only bolster the model's specific-domain capabilities but also maintain its general proficiency. based on the proposed strategy, we introduce llama slayer-8b and llama slayer-8b-instruct, llms that derive from the base model llama2-7b. the two models surpass various predecessors in the llama series across a wide array of benchmarks, evidenc-ing the superior performance of our strategy.