content: html: <tr><td>Model</td><td>Alpaca-GPT4</td><td>OpenOrca</td><td>Synth. Math-Instruct</td><td>H6 (Avg.)</td><td>ARC</td><td>HellaSwag</td><td>MMLU</td><td>TruthfulQA</td><td>Winogrande</td><td>GSM8K</td></tr><tr><td>SFT v1</td><td>O</td><td>X</td><td>X</td><td>69.15</td><td>67.66</td><td>86.03</td><td>65.88</td><td>60.12</td><td>82.95</td><td>52.24</td></tr><tr><td>SFT v2</td><td>O</td><td>O</td><td>X</td><td>69.21</td><td>65.36</td><td>85.39</td><td>65.93</td><td>58.47</td><td>82.79</td><td>57.32</td></tr><tr><td>SFT v3</td><td>O</td><td>O</td><td>O</td><td>70.03</td><td>65.87</td><td>85.55</td><td>65.31</td><td>57.93</td><td>81.37</td><td>64.14</td></tr><tr><td>SFT v4</td><td>O</td><td>X</td><td>O</td><td>70.88</td><td>67.32</td><td>85.87</td><td>65.87</td><td>58.97</td><td>82.48</td><td>64.75</td></tr><tr><td>SFT v3 + v4</td><td>O</td><td>O</td><td>O</td><td>71.11</td><td>67.32</td><td>85.96</td><td>65.95</td><td>58.80</td><td>2.08</td><td>66.57</td></tr>

content: text: Table 3: Ablation studies on the different datasets used for instruction tuning. 'SFT v3+v4' indicates that the model
is merged from 'SFT v3' and 'SFT v4' by simply averaging the model weights. The best scores for H6 and the
individual tasks are shown in bold.

content: html: <tr><td>Model</td><td>Ultrafeedback Clean</td><td>Synth. Math-Alignment</td><td>H6 (Avg.)</td><td>ARC</td><td>HellaSwag</td><td>MMLU</td><td>TruthfulQA</td><td>Winogrande</td><td>GSM8K</td></tr><tr><td>DPO v1</td><td>O</td><td>X</td><td>73.06</td><td>71.42</td><td>88.49</td><td>66.14</td><td>72.04</td><td>81.45</td><td>58.83</td></tr><tr><td>DPO v2</td><td>O</td><td>O</td><td>73.42</td><td>71.50</td><td>88.28</td><td>65.97</td><td>71.71</td><td>82.79</td><td>60.27</td></tr><tr><td>DPO v1 + v2</td><td>O</td><td>O</td><td>73.21</td><td>71.33</td><td>88.36</td><td>65.92</td><td>72.65</td><td>82.79</td><td>58.23</td></tr>

content: text: Table 4: Ablation studies on the different datasets used during the direct preference optimization (DPO) stage.
'SFT v3' is used as the SFT base model for DPO. We name ablated models with the 'DPO' prefix to indicate the
alignment tuning stage. 'DPO v1+v2' indicates that the model is merged from 'DPO v1' and 'DPO v2' by simply
averaging the model weights. The best scores for H6 and the individual tasks are shown in bold.

content: html: <tr><td>Model</td><td>Base SFT Model</td><td>H6 (Avg.)</td><td>ARC</td><td>HellaSwag</td><td>MMLU</td><td>TruthfulQA</td><td>Winogrande</td><td>GSM8K</td></tr><tr><td>DPO v2</td><td>SFT v3</td><td>73.42</td><td>71.50</td><td>88.28</td><td>65.97</td><td>71.71</td><td>82.79</td><td>60.27</td></tr><tr><td>DPO v3</td><td>SFT v3 + v4</td><td>73.58</td><td>71.33</td><td>88.08</td><td>65.39</td><td>72.45</td><td>81.93</td><td>62.32</td></tr>

content: text: Table 5: Ablation studies on the different SFT base models used during the direct preference optimization (DPO)
stage. Ultrafeedback Clean and Synth. Math-Alignment datasets are used. We name ablated models with the 'DPO'
prefix to indicate the alignment tuning stage. The best scores for H6 and the individual tasks are shown in bold.

content: text: indicate that using OpenOrca results in a model that
behaves differently from using only Alpaca-GPT4.

content: text: Second, we investigate whether Synth. Math-
Instruct dataset is beneficial. For 'SFT v3', we
add the Synth. Math-Instruct dataset, which boosts
GSM8K scores to 64.14 and achieves comparable
scores for the other tasks. Interestingly, when we
add the Synth. Math-Instruct dataset to 'SFT v1'
to train 'SFT v4', we get our highest H6 score of
70.88 with higher scores than 'SFT v3' for all tasks.
From the above, we can see that adding the Synth.
Math-Instruct dataset is helpful.

content: text: Lastly, we see whether merging models trained
with and without OpenOrca can boost performance.
In the first analysis, we saw that using OpenOrca re-
sulted in a model that behaved differently from the
model that was trained without OpenOrca. Build-
ing on this intuition, we merge 'SFT v3' and 'SFT
v4' as they are the best-performing models with
and without OpenOrca. To our surprise, the result-
ing merged model 'SFT v3+v4' retains the high
scores for non-GSM8K tasks from 'SFT v4' but
also achieves a higher GSM8K score than 'SFT v3'
or 'SFT v4'. Thus, we see that merging models
that specialize in different tasks is a promising way
to obtain a model that performs well generally.

content: text: 4.3.2 Alignment Tuning

content: text: As we utilize DPO for practical alignment tuning,
there are additional aspects to ablate such as the
SFT base models used. Thus, we present ablations
for the different training datasets used for training,
the different SFT base models to initialize the DPO
model, and finally, the model merging strategy to
obtain the final alignment-tuned model.

content: text: Ablation on the training datasets. We ablate on
the different alignment datasets used during DPO
in Tab. 4. We use 'SFT v3' as the SFT base model
for DPO. 'DPO v1' only uses the Ultrafeedback
Clean dataset while 'DPO v2' also used the Synth.
Math-Alignment dataset.

content: text: First, we test how Ultrafeedback Clean and
Synth. Math-Alignment impacts model perfor-
mance. For 'DPO v1', it achieves 73.06 in H6,
which is a substantial boost from the SFT base
model score of 70.03. However, we note that while
scores for tasks like ARC, HellaSwag, and Truth-
fulQA all improved by good margins, the score
for GSM8K is 58.83, which is lower than the
SFT base model score of 64.14. Adding Synth.
Math-Alignment to train 'DPO v2', we see that
the GSM8k score improves to 60.27, which is
lower than the SFT base model but still higher
than 'DPO v1'. Other task scores are also not nega-