content: text: Step 1-1 Step 1-2
Output Output Output
Output Output Output
24 Layers 24 Layers
Merge
8 Layers
48 Layers
Copy
8 Layers Continued
32 Layers 32 Layers Pretraining
24 Layers
24 Layers Input
Input Input Input Input Input
Step 1. Depthwise Scaling Step 2. Continued Pretraining

content: text: Figure 1: Depth up-scaling for the case with n = 32, s = 48, and m = 8. Depth up-scaling is achieved through a
dual-stage process of depthwise scaling followed by continued pretraining.

content: text: for wider access and application of these models
by researchers and developers globally.

content: text: 2 Depth Up-Scaling

content: text: To efficiently scale-up LLMs, we aim to utilize pre-
trained weights of base models to scale up to larger
LLMs (Komatsuzaki et al., 2022). While exist-
ing methods such as Komatsuzaki et al. (2022) use
MoE (Shazeer et al., 2017) to scale-up the model ar-
chitecture, we opt for a different depthwise scaling
strategy inspired by Tan and Le (2019). We then
continually pretrain the scaled model as just scaling
the model without further pretraining degrades the
performance.

content: text: Base model. Any n-layer transformer architec-
ture can be used but we select the 32-layer Llama
2 architecture as our base model. We initialize the
Llama 2 architecture with pretrained weights from
Mistral 7B, as it is one of the top performers com-
patible with the Llama 2 architecture. By adopting
the Llama 2 architecture for our base model, we
aim to leverage the vast pool of community re-
sources while introducing novel modifications to
further enhance its capabilities.

content: text: Depthwise scaling. From the base model with n
layers, we set the target layer count s for the scaled
model, which is largely dictated by the available
hardware.

content: text: With the above, the depthwise scaling process
is as follows. The base model with n layers is
duplicated for subsequent modification. Then, we
remove the final m layers from the original model
and the initial m layers from its duplicate, thus
forming two distinct models with n - m layers.
These two models are concatenated to form a scaled
model with s = 2·(n-m) layers. Note that n = 32
from our base model and we set s = 48 considering

content: text: our hardware constraints and the efficiency of the
scaled model, i.e., fitting between 7 and 13 billion
parameters. Naturally, this leads to the removal of
m = 8 layers. The depthwise scaling process with
n = 32, s = 48, and m = 8 is depicted in 'Step 1:
Depthwise Scaling' of Fig. 1.

content: text: We note that a method in the community that also
2
scale the model in the same manner as 'Step 1:
Depthwise Scaling' of Fig. 1 has been concurrently
developed.

content: text: Continued pretraining. The performance of the
depthwise scaled model initially drops below that
of the base LLM. Thus, we additionally apply
the continued pretraining step as shown in 'Step
2: Continued Pretraining' of Fig. 1. Experimen-
tally, we observe rapid performance recovery of
the scaled model during continued pretraining, a
phenomenon also observed in Komatsuzaki et al.
(2022). We consider that the particular way of
depthwise scaling has isolated the heterogeneity
in the scaled model which allowed for this fast
performance recovery.

content: text: Delving deeper into the heterogeneity of the
scaled model, a simpler alternative to depthwise
scaling could be to just repeat its layers once more,
i.e., from n to 2n layers. Then, the 'layer distance',
or the difference in the layer indices in the base
model, is only bigger than 1 where layers n and
n + 1 are connected, i.e., at the seam.

content: text: However, this results in maximum layer distance
at the seam, which may be too significant of a
discrepancy for continued pretraining to quickly
resolve. Instead, depthwise scaling sacrifices the
2m middle layers, thereby reducing the discrep-
ancy at the seam and making it easier for continued

content: text: 2https://huggingface.co/Undi95/
Mistral-11B-v0.1