content: text: A Contributions

content: text: The contributions of this study are as follows:

content: text: · Introduction of the SOLAR 10.7 Billion-
Parameter Model: We have released the SO-
LAR 10.7B model, which is not only depth-
wise scaled but also continually pretrained.
The availability of SOLAR 10.7B under the
Apache 2.0 license permits commercial us-
age, enabling the integration of this advanced
model into a diverse range of products and ser-
vices. This bridges the gap between academic
research and practical applications, fostering
wider accessibility and utility in various fields.

content: text: · Superior Performance Across Diverse
Benchmarks: SOLAR 10.7B excels in var-
ious benchmarks, outperforming established
models like Llama 2 and Mistral 7B in reason-
ing, mathematics, and the MMLU framework.

content: text: · Advancement in Instruction-Following Ca-
pabilities: The introduction of SOLAR 10.7B-
Instruct, a variant fine-tuned for enhanced
instruction-following abilities, marks a sig-
nificant improvement in the model's ability to
understand and execute complex instructions.

content: text: Dahyun Kim, Chanjun Park, Sanghoon Kim,
and Wonsung Lee contributed equally to this pa-
per. Sanghoon Kim led the Foundation Model part,
with Dahyun Kim, Wonho Song, Yunsu Kim, and
Hyeonwoo Kim. Chanjun Park led the Data and
Evaluation (Data-Centric LLM) part, with Yungi
Kim, Jihoo Kim, Changbae Ahn, Seonghoon Yang,
Sukyung Lee, and Hyunbyung Park. Wonsung Lee
led the Adaptation Modeling part, with Gyoungjin
Gim, Hyeonju Lee, and Mikyoung Cha. Hwalsuk
Lee performed the role of the overall project op-
eration. All these individuals contributed to the
creation of SOLAR 10.7B.

content: text: B Related Works and Background

content: text: B.1 Large Language Models

content: text: Following the advent of context-based language
models, various studies have revealed a "scaling
law" (Kaplan et al., 2020; Hernandez et al., 2021;
Anil et al., 2023), demonstrating a positive corre-
lation between the size of model and training data
and model performance. This has led to the emer-
gence of Large Language Models (LLMs). Un-
like previous language models, LLMs possess the

content: text: ability for In-context learning, including Zero-shot
learning (Radford et al., 2019) and Few-shot learn-
ing (Brown et al., 2020), allowing them to perform
new tasks without updating model weights. These
capabilities of LLMs, not evident in smaller mod-
els, are referred to as Emergent abilities (Wei et al.,
2022a).

content: text: B.2 Mixture of Experts

content: text: In the landscape of machine learning architectures,
the Mixture of Experts (MoE) models like (Shazeer
et al., 2017; Shen et al., 2019; Komatsuzaki et al.,
2022) has gained attention for its capability to ad-
dress the challenges posed by complex and hetero-
geneous data. MoE models offer notable benefits,
including enhanced output diversity, allowing for
the capture of intricate patterns within the input
space. Moreover, their computational efficiency,
especially when implemented in a sparse form, has
made them valuable in scenarios where resource
constraints are a consideration (Shazeer et al., 2017;
Komatsuzaki et al., 2022).

content: text: However, efficient implementation of MoE mod-
els poses a considerable challenge, primarily due to
the intricacies associated with dynamic routing and
load-imbalanced computation (Gale et al., 2023).
Existing hardware and software for deep learning,
such as TPUs and XLA compilers, often demand
static knowledge of tensor shapes, making MoE
implementation on TPU challenging.

content: text: While GPU implementation offers more flexi-
bility, sparse computation compatibility becomes
a hurdle. Striking the right balance between fix-
ing the size of each expert to facilitate efficient
computation and maintaining model quality creates
a tradeoff between information preservation and
hardware efficiency. This tradeoff, in turn, necessi-
tates careful consideration during hyperparameter
tuning, adding a layer of complexity to the imple-
mentation of MoE models, potentially offsetting
their advantages. Given the formidable challenges
in MoE model implementation, it becomes almost
inevitable for researchers and practitioners to re-
sort to specialized tools and frameworks, such as
Tutel (Hwang et al., 2023) or Megablocks (Gale
et al., 2023).

content: text: Departing from the horizontal expansion char-
acteristic of MoE models, the DUS method intro-
duces model scaling in the vertical dimension. No-
tably, DUS does not introduce dynamism in the
scaled model, which significantly reduces the com-