content: html: <tr><td rowspan="3">Properties</td><td colspan="6">Training Datasets</td></tr><tr><td colspan="3">Instruction</td><td colspan="3">Alignment</td></tr><tr><td>Alpaca-GPT4</td><td>OpenOrca</td><td>Synth. Math-Instruct</td><td>Orca DPO Pairs</td><td>Ultrafeedback Cleaned</td><td>Synth. Math-Alignment</td></tr><tr><td>Total # Samples</td><td>52K</td><td>2.91M</td><td>126K</td><td>12.9K</td><td>60.8K</td><td>126K</td></tr><tr><td>Maximum # Samples Used</td><td>52K</td><td>100K</td><td>52K</td><td>12.9K</td><td>60.8K</td><td>20.1K</td></tr><tr><td>Open Source</td><td>O</td><td>O</td><td>X</td><td>O</td><td>O</td><td>X</td></tr>

content: text: Table 1: Training datasets used for the instruction and alignment tuning stages, respectively. For the instruction
tuning process, we utilized the Alpaca-GPT4 (Peng et al., 2023), OpenOrca (Mukherjee et al., 2023), and Synth.
Math-Instruct datasets, while for the alignment tuning, we employed the Orca DPO Pairs (Intel, 2023), Ultrafeedback
Cleaned (Cui et al., 2023; Ivison et al., 2023), and Synth. Math-Alignment datasets. The 'Total # Samples' indicates
the total number of samples in the entire dataset. The 'Maximum # Samples Used' indicates the actual maximum
number of samples that were used in training, which could be lower than the total number of samples in a given
dataset. 'Open Source' indicates whether the dataset is open-sourced.

content: text: pretraining to quickly recover performance. We
attribute the success of DUS to reducing such dis-
crepancies in both the depthwise scaling and the
continued pretraining steps. We also hypothesize
that other methods of depthwise scaling could also
work for DUS, as long as the discrepancy in the
scaled model is sufficiently contained before the
continued pretraining step.

content: text: Comparison to other up-scaling methods. Un-
like Komatsuzaki et al. (2022), depthwise scaled
models do not require additional modules like gat-
ing networks or dynamic expert selection. Conse-
quently, scaled models in DUS do not necessitate
a distinct training framework for optimal training
efficiency, nor do they require specialized CUDA
kernels for fast inference. A DUS model can seam-
lessly integrate into existing training and inference
frameworks while maintaining high efficiency.

content: text: 3 Training Details

content: text: After DUS, including continued pretraining, we
perform fine-tuning of SOLAR 10.7B in two stages:
1) instruction tuning and 2) alignment tuning.

content: text: Instruction tuning. In the instruction tuning
stage, the model is trained to follow instructions in
a QA format (Zhang et al., 2023b). We mostly use
open-source datasets but also synthesize a math QA
dataset to enhance the model's mathematical capa-
bilities. A rundown of how we crafted the dataset is
as follows. First, seed math data are collected from
the Math (Hendrycks et al., 2021) dataset only, to
avoid contamination with commonly used bench-
mark datasets such as GSM8K (Cobbe et al., 2021).
Then, using a process similar to MetaMath (Yu
et al., 2023), we rephrase the questions and an-
swers of the seed math data. We use the resulting
rephrased question-answer pairs as a QA dataset

content: text: and call it 'Synth. Math-Instruct'.

content: text: Alignment tuning. In the alignment tuning stage,
the instruction-tuned model is further fine-tuned to
be more aligned with human or strong AI (e.g.,
GPT4 (OpenAI, 2023)) preferences using direct
preference optimization (DPO) (Rafailov et al.,
2023). Similar to the instruction tuning stage, we
use mostly open-source datasets but also synthe-
size a math-focused alignment dataset utilizing the
'Synth. Math-Instruct' dataset mentioned in the
instruction tuning stage.

content: text: The alignment data synthesis process is as
follows. We take advantage of the fact that
the rephrased question-answer pairs in Synth.
Math-Instruct data are beneficial in enhancing the
model's mathematical capabilities (see Sec. 4.3.1).
Thus, we speculate that the rephrased answer to the
rephrased question is a better answer than the orig-
inal answer, possibly due to the interim rephrasing
step. Consequently, we set the rephrased question
as the prompt and use the rephrased answer as the
chosen response and the original answer as the re-
jected response and create the {prompt, chosen,
rejected} DPO tuple. We aggregate the tuples from
the rephrased question-answer pairs and call the
resulting dataset 'Synth. Math-Alignment'.

content: text: 4 Results

content: text: 4.1 Experimental Details

content: text: Training datasets. We present details regarding
our training datasets for the instruction and align-
ment tuning stages in Tab. 1. We do not always
use the entire dataset and instead subsample a set
amount. Note that most of our training data is
open-source, and the undisclosed datasets can be
substituted for open-source alternatives such as the
MetaMathQA (Yu et al., 2023) dataset.