SOLAR 10.7 B : Scaling Large Language Models with Simple yet Effective Depth Up-Scaling (2024)

Abstract: We introduce SOLAR 10.7B, a large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale LLMs, we present a method for scaling LLMs called depth up-scaling (DUS), which encompasses depthwise scaling and continued pretraining. In contrast to other LLM up-scaling methods that use mixture-of-experts, DUS does not require complex changes to train and inference efficiently. We show experimentally that DUS is simple yet effective in scaling up high-performance LLMs from small ones. Building on the DUS model, we additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for instruction-following capabilities, surpassing Mixtral-8x7B-Instruct. SOLAR 10.7B is publicly available under the Apache 2.0 license, promoting broad access and application in the LLM field.

## introduction the field of natural language processing (nlp) has been significantly transformed by the introduction of large language models (llms), which have enhanced our understanding and interaction with human language @xcite . these advancements bring challenges such as the increased need to train ever larger models @xcite @xcite @xcite owing to the performance scaling law @xcite @xcite . to efficiently tackle the above, recent works in scaling language models such as a mixture of experts (moe) @xcite have been proposed. while those approaches are able to effi-ciently and effectively scale-up llms, they often require non-trivial changes to the training and inference framework @xcite , which hinders widespread applicability. effectively and efficiently scaling up llms whilst also retaining the simplicity for ease of use is an important problem @xcite @xcite . inspired by @xcite , we present depth up-scaling (dus), an effective and efficient method to up-scale llms whilst also remaining straightforward to use. dus consists of scaling the number of layers in the base model and continually pretraining the scaled model. unlike @xcite , dus does not scale the model using moe and rather use a depthwise scaling method analogous to @xcite which is adapted for the llm architecture. thus, there are no additional modules or dynamism as with moe, making dus immediately compatible with easy-to-use llm frameworks such as hug-gingface @xcite with no changes to the training or inference framework for maximal efficiency. furthermore, dus is applicable to all transformer architectures, opening up new gateways to effectively and efficiently scale-up llms in a simple manner. using dus, we release so-lar 10.7b, an llm with 10.7 billion parameters, that outperforms existing models like llama 2 @xcite and mistral 7b @xcite in various benchmarks. we have also developed solar 10.7b-instruct, a variant fine-tuned for tasks requiring strict adherence to complex instructions. it significantly outperforms the mixtral-8x7b-instruct model across various evaluation metrics, evidencing an advanced proficiency that exceeds the capabilities of even larger models in terms of benchmark performance. by releasing solar 10.7b under the apache 2.0 license, we aim to promote collaboration and innovation in nlp. this open-source approach allows for wider access and application of these models by researchers and developers globally. ## depth up-scaling to efficiently scale-up llms, we aim to utilize pretrained weights of base models to scale up to larger llms @xcite . while existing methods such as @xcite use moe @xcite to scale-up the model architecture, we opt for a different depthwise scaling strategy inspired by @xcite . we then continually pretrain the scaled model as just scaling the model without further pretraining degrades the performance. base model. any n-layer transformer architecture can be used but we select the 32-layer llama 2 architecture as our base model. we initialize the llama 2 architecture with pretrained weights from mistral 7b, as it is one of the top performers compatible with the llama 2 architecture. by adopting the llama 2 architecture for our base model, we aim to leverage the vast pool of community resources while introducing novel modifications to further enhance its capabilities. depthwise scaling. from the base model with n layers, we set the target layer count s for the scaled model, which is largely dictated by the available hardware. with the above, the depthwise scaling process is as follows. the base model with n layers is duplicated for subsequent modification. then, we remove the final m layers from the original model and the initial m layers from its duplicate, thus forming two distinct models with nm layers. these two models are concatenated to form a scaled model with @xmath0. note that @xmath1, i.e., fitting between 7 and 13 billion parameters. naturally, this leads to the removal of @xmath2. the depthwise scaling process with @xmath3, @xmath4, and @xmath5: depthwise scaling' of fig. 1 . we note that a method in the community that also scale the model in the same manner foot_0 as 'step 1: depthwise scaling' of fig. 1 has been concurrently developed. continued pretraining. the performance of the depthwise scaled model initially drops below that of the base llm. thus, we additionally apply the continued pretraining step as shown in 'step 2: continued pretraining' of fig. 1 . experimentally, we observe rapid performance recovery of the scaled model during continued pretraining, a phenomenon also observed in @xcite . we consider that the particular way of depthwise scaling has isolated the heterogeneity in the scaled model which allowed for this fast performance recovery. delving deeper into the heterogeneity of the scaled model, a simpler alternative to depthwise scaling could be to just repeat its layers once more, i.e., from n to 2n layers. then, the 'layer distance', or the difference in the layer indices in the base model, is only bigger than 1 where layers n and n + 1 are connected, i.e., at the seam. however, this results in maximum layer distance at the seam, which may be too significant of a discrepancy for continued pretraining to quickly resolve. instead, depthwise scaling sacrifices the 2m middle layers, thereby reducing the discrepancy at the seam and making it easier for continued properties training datasets instruction alignment alpaca-gpt4 openorca synth. math-instruct orca dpo pairs ultrafeedback cleaned synth. math-alignment total # samples 52k 2.91m 126k 12.9k 60.8k 126k maximum # samples used 52k 100k 52k 12.9k 60.8k 20.1k open source o o ✗ o o ✗ comparison to other up-scaling methods. unlike komatsuzaki et al. (2022), depthwise scaled models do not require additional modules like gating networks or dynamic expert selection. consequently, scaled models in dus do not necessitate a distinct training framework for optimal training efficiency, nor do they require specialized cuda kernels for fast inference. a dus model can seamlessly integrate into existing training and inference frameworks while maintaining high efficiency. ## training details after dus, including continued pretraining, we perform fine-tuning of solar 10.7b in two stages: 1) instruction tuning and 2) alignment tuning. instruction tuning. in the instruction tuning stage, the model is trained to follow instructions in a qa format @xcite . we mostly use open-source datasets but also synthesize a math qa dataset to enhance the model's mathematical capabilities. a rundown of how we crafted the dataset is as follows. first, seed math data are collected from the math @xcite dataset only, to avoid contamination with commonly used benchmark datasets such as gsm8k @xcite . then, using a process similar to metamath @xcite , we rephrase the questions and answers of the seed math data. we use the resulting rephrased question-answer pairs as a qa dataset and call it 'synth. math-instruct'. alignment tuning. in the alignment tuning stage, the instruction-tuned model is further fine-tuned to be more aligned with human or strong ai (e.g., gpt4 (openai, 2023)) preferences using sdpo @xcite , an improved version of direct preference optimization (dpo) @xcite . similar to the instruction tuning stage, we use mostly open-source datasets but also synthesize a math-focused alignment dataset utilizing the 'synth. math-instruct' dataset mentioned in the instruction tuning stage. the alignment data synthesis process is as follows. we take advantage of the fact that the rephrased question-answer pairs in synth. math-instruct data are beneficial in enhancing the model's mathematical capabilities (see sec. 4.3.1). thus, we speculate that the rephrased answer to the rephrased question is a better answer than the original answer, possibly due to the interim rephrasing step. consequently, we set the rephrased question as the prompt and use the rephrased answer as the chosen response and the original answer as the rejected response and create the {prompt, chosen, rejected} dpo tuple. we aggregate the tuples from the rephrased question-answer pairs and call the resulting dataset 'synth. math-alignment'. ## conclusion we introduce solar 10.7b and its fine-tuned variant solar 10.7b-instruct, which are depth upscaled (dus) models with 10.7 billion parameters foot_1 . they show superior performance over models like llama 2, mistral 7b, and mixtral-7b-instruct in essential nlp tasks while maintaining computational efficiency. thus, dus is effective in scaling-up highly performant llms from smaller ones. with more exploration, dus could be further improved, paving a new path to efficiently scaling llms.