content: html: <tr><td>Model</td><td>H6 (Avg.)</td><td>ARC</td><td>HellaSwag</td><td>MMLU</td><td>TruthfulQA</td><td>Winogrande</td><td>GSM8K</td></tr><tr><td>Cand. 1</td><td>73.73</td><td>70.48</td><td>87.47</td><td>65.73</td><td>70.62</td><td>81.53</td><td>66.57</td></tr><tr><td>Cand. 2</td><td>73.28</td><td>71.59</td><td>88.39</td><td>66.14</td><td>72.50</td><td>81.99</td><td>59.14</td></tr>

content: text: Table 6: Performance comparison amongst the merge candidates. 'Cand. 1' and 'Cand. 2' are trained using the
same setting as 'DPO v2' and 'DPO v3', respectively, but with slightly different hyper-parameters. The best scores
for H6 and the individual tasks are shown in bold.

content: html: <tr><td>Model</td><td>Merge Method</td><td>H6 (Avg.)</td><td>ARC</td><td>HellaSwag</td><td>MMLU</td><td>TruthfulQA</td><td>Winogrande</td><td>GSM8K</td></tr><tr><td>Merge v1</td><td>Average (0.5,0.5)</td><td>74.00</td><td>71.16</td><td>88.01</td><td>66.14</td><td>71.71</td><td>82.08</td><td>64.90</td></tr><tr><td>Merge v2</td><td>Average (0.4, 0.6)</td><td>73.93</td><td>71.08</td><td>88.08</td><td>66.27</td><td>71.89</td><td>81.77</td><td>64.52</td></tr><tr><td>Merge v3</td><td>Average (0.6, 0.4)</td><td>74.05</td><td>71.08</td><td>87.88</td><td>66.13</td><td>71.61</td><td>82.08</td><td>65.50</td></tr><tr><td>Merge v4</td><td>SLERP</td><td>73.96</td><td>71.16</td><td>88.03</td><td>66.25</td><td>71.79</td><td>81.93</td><td>64.59</td></tr>

content: text: Table 7: Ablation studies on the different merge methods used for obtaining the final model. We use 'Cand. 1'
and 'Cand. 2' from Tab. 6 as our two models for merging. We name the merged models with the 'Merge' prefix to
indicate they are merged. The best scores for H6 and the individual tasks are shown in bold.

content: text: tively impacted by adding Synth. Math-Alignment.
Thus, we can conclude that adding Synth. Math-
Alignment is beneficial for H6.

content: text: Then, we experiment whether merging 'DPO
v1' and 'DPO v2' is beneficial. Unfortunately,
'DPO v1+v2' scores 73.21 in H6, which is worse
than 'DPO v2'. More importantly, the gain in
the GSM8K score from adding Synth. Math-
Alignment is gone, which is undesirable. One
reason for this could be that 'DPO v2' is a strict
improvement over 'DPO v1', unlike the case for
merging 'SFT v3' and 'SFT v4' where the models
had different strengths and weaknesses.

content: text: Ablation on the SFT base models. When ap-
plying DPO, we start from a model that is already
instruction tuned ,i.e., the SFT base model and ab-
late on using different SFT base models. We use
Ultrafeedback Clean and Synth. Math-Alignment
datasets for this ablation. Each of the ablated mod-
els is trained as follows. 'DPO v2' uses 'SFT v3'
as the base SFT model, while 'DPO v3' uses 'SFT
v3+v4' as the SFT base model instead.

content: text: Note that 'SFT v3+v4' has higher scores on all
tasks compared to 'SFT v3', and the gap is espe-
cially large for ARC (+1.45) and GSM8K (+2.43).
Surprisingly, the two models perform similarly in
terms of H6. A closer look at the scores for the
individual tasks shows only a small margin in the
GSM8K scores, and other task scores show little
difference. Thus, the performance gaps in certain
tasks in the SFT base models do not always carry
over to the alignment-tuned models.

content: text: Ablation on different merge methods. From
Tab. 3, we saw that merging two models that have
different strengths can be beneficial to performance.

content: text: To utilize this for the alignment-tuned model as
well, we train two models named 'Cand. 1' and
'Cand. 2' using the same training dataset and SFT
base model as 'DPO v2' and 'DPO v3' but with dif-
ferent hyper-parameters to maximize each model's
respective strengths. We compare 'Cand. 1' and
'Cand. 2' in Tab. 6 where we can see that 'Cand. 1'
has high GSM8K scores but relatively low scores
for the other tasks, whereas 'Cand. 2' has low
scores for GSM8K but high scores for the other
tasks. We merge these two models using various
methods and ablate the results in Tab.. 7.

content: text: We use two merge methods: 1) Average (a, b),
where a and b denote the weighting for 'Cand.
1' and 'Cand. 2' when averaging weights and 2)
SLERP (Shoemake, 1985). We use (0.5, 0.5), (0.4,
0.6), and (0.6, 0.4) for Average (a, b). From Tab. 7,
we can see that the different merge methods have
little effect on the H6 scores. The scores for the
individual tasks also do not differ by much, suggest-
ing that as long as the merge candidates have suffi-
ciently different strengths, the exact merge method
may not be as crucial. Thus, we chose 'Merge v1'
as our SOLAR 10.7B-Instruct model.

content: text: 5 Conclusion

content: text: We introduce SOLAR 10.7B and its fine-tuned vari-
ant SOLAR 10.7B-Instruct, which are depth up-
scaled (DUS) models with 10.7 billion parameters.
They show superior performance over models like
Llama 2, Mistral 7B, and Mixtral-7B-Instruct in es-
sential NLP tasks while maintaining computational
efficiency. Thus, DUS is effective in scaling-up
highly performant LLMs from smaller ones. With
more exploration, DUS could be further improved,
paving a new path to efficiently scaling LLMs.