Paper: Unified Language Model Alignment with Demonstration and Point-wise Human Preference (ICLR-2024)

Reviewer comment (Evaluation):
The paper lacks empirical analysis and provides limited experimental results to justify the design of each component in Equation (9). It is difficult to interpret how GPT-4–evaluated win rates correlate with human judgment or actual quality.

Author response:
**Q2:** "lacks empirical analysis with limited experimental results to justify the design of each component"

**A2:** In the revised manuscript, we have conducted an ablation study to verify the design of the hybrid objective formulation in ULMA. Specifically, we first use positive samples as demonstration data to compare ULMA with point-wise DPO (which adopts KL regularization for these samples) to evaluate the effectiveness of ULMA for learning from high quality (positive) demonstration data. Then we use negative samples as dispreferred demonstration data to compare ULMA with the counterpart algorithm without KL regularization (i.e., Unlearning) to evaluate the ability of ULMA for learning from possibly noisy (negative) preference data. As presented in Table 3, the results show that ULMA can better exploit the high quality positive samples by applying SFT loss to these samples, while preventing from overfitting to possibly noisy negative samples via adding regularization to negative samples. Please refer to the revised manuscript for more detailed empirical results.

Reviewer comment (Evaluation):
The baseline for comparison appears to be the chosen answer in the dataset, considered golden samples for preference learning. This makes the win-rate numbers in Tables 1 and 2 seem somewhat weird and vague, since there should be a substantial proportion of tie cases as reported in previous works.

Author response:
**Q3:** "the numbers of win-rates in Tables 1 and 2 are somewhat weird and vague... The win-rates, especially in Golden HH, are close to 100 percents"

**A3:** The numbers reported in Table 1 and 2 are actually the **win \& tie** rate [1, 2] of the harmful metric. We had made a typo in the original captions of Table 1 and 2. Sorry for causing your confusion! In the revised manuscript, beyond the win \& tie rate presented in Table 1 and 2, we also report the detailed win, tie and loss rates respectively in Figure 1 and Figure 2 in Appendix D.

We also note that it is reasonable that the win \& tie rates on 
*Golden HH* are close to 100\%. Here we explain the reason. Since *Golden HH* is constructed based on *HH*, to make the results on *Golden HH* and *HH* comparable, the generated answers are **compared to the chosen sample in the original *HH* dataset** when evaluating the harmful score on *Golden HH*. However, the quality of the chosen samples in *HH* is not very high. When employing SFT on *Golden HH* where the new chosen data are of much higher quality, the answers generated on *Golden HH* is very likely to be at least no worse than the original chosen data in *HH*. As a consequence, the win \& tie rates on *Golden HH* are close to 100\%. Note that the above phenomenon actually implies that the quality of the chosen samples in *Golden HH* is much higher than that in *HH* (also see Table 5 in Appendix C for more detailed comparison of both datasets).

Reviewer comment (Evaluation):
The evaluation should be extended to other benchmarks, such as RealToxicityPrompts, to compare effectiveness at least in the domain of harmlessness.

Author response:
**Q4:** "It is also important to extend the evaluation to other benchmarks" 

**A4:** We have added two more datasets *QA-feedback* and *red-team* in the revised manuscript. The first dataset *QA-feedback* is a point-wise QA dataset with binary labels, in which the perplexity and the helpful score are evaluated. The second dataset *red-team* is a point-wise dataset with continuous labels on robustness of LLMs to red teaming attacks, in which we report the perplexity and the harmful score. The empirical results are presented in Table 1 and Table 2 in the updated manuscript, from which we also observe that our proposed ULMA method outperforms other compared methods on both datasets. Please refer to the manuscript for empirical results and detailed discussions.

Reviewer comment (Experiments):
The experiments section is not very detailed.

Author response:
We have added two more datasets *QA-feedback* and *red-team* in the revision. The former dataset is a point-wise dataset with binary labels, and the second dataset is a point-wise dataset with continuous labels. The empirical results are presented in Table 1 and Table 2 in the updated manuscript. Please refer to the manuscript for empirical results and detailed discussions.

In particular, on the *red-team* dataset, we give an example of expanding the methodology of ULMA, i.e., using a hybrid objective formulation for different types of data, to handle point-wise dataset with continuous labels. Specifically, *red-team* is a point-wise dataset on LLM's robustness to red teaming attacks. It consists of samples scoring from 0 to 4, among which those rated 0 can be considered as high quality demonstration data. To better exploit these data, we introduce a hybrid loss by adding SFT loss of the samples rated 0 to the original MSE loss of all samples.

Reviewer comment (Experiments):
Expanding the methodology to more datasets would be nice.

Author response:
We have added two more datasets *QA-feedback* and *red-team* in the revision. The former dataset is a point-wise dataset with binary labels, and the second dataset is a point-wise dataset with continuous labels. The empirical results are presented in Table 1 and Table 2 in the updated manuscript. Please refer to the manuscript for empirical results and detailed discussions.

In particular, on the *red-team* dataset, we give an example of expanding the methodology of ULMA, i.e., using a hybrid objective formulation for different types of data, to handle point-wise dataset with continuous labels. Specifically, *red-team* is a point-wise dataset on LLM's robustness to red teaming attacks. It consists of samples scoring from 0 to 4, among which those rated 0 can be considered as high quality demonstration data. To better exploit these data, we introduce a hybrid loss by adding SFT loss of the samples rated 0 to the original MSE loss of all samples.

Reviewer comment (Theory):
In many cases, there can be a mapping function between pair-wise preference and point-wise preference. The authors do not discuss these cases.

Author response:
Admittedly, there exists some relation between pair-wise data and point-wise data as you have suggested, but we shall note that it is **not** a one-to-one correspondence, and the transformation may suffer from information loss. Now we explicate the above point in two folds. 

(i) "Pair-wise to point-wise" transformation may lose some pair-wise comparison information. For example, consider two pair-wise datasets $\mathcal D_1=\\{x_1\succ x_2, x_2\succ x_3, x_3\succ x_4\\}$ and $\mathcal D_2=\\{x_1\succ x_3, x_3\succ x_2, x_2\succ x_4\\}$. Obviously, the two datasets differ from the pair-wise relation between $x_2$ and $x_3$. These two datasets will be transformed to the same point-wise dataset $\mathcal D_3=\\{(x_1,1), (x_2,0), (x_3,0), (x_4,-1)\\}$, in which $x_2$ and $x_3$ have the same score. In this case, the transformation drops the relation between $x_2$ and $x_3$, which incurs loss in information.

(ii) "Point-wise to pair-wise" transformation is unsuitable to some dataset where no comparison can be made, e.g., all samples have the same score or there is only one sample for some prompt. One example is the *red-team* dataset, which only has one sample for each prompt. This dataset cannot be transformed into a pair-wise dataset, and pair-wise methods such as RLHF and DPO are inapplicable. Please see the revised manuscript for more detailed descriptions and empirical results. 

Therefore, the conversion between pair-wise and point-wise datasets is not always possible, and may suffer from information loss in many cases. In the revised manuscript, we have compared the performance of point-wise DPO and point-wise DPO on pair-wise dataset *Golden HH* and point-wise dataset *QA-feedback*, respectively. We observe that pair-wise DPO performs slightly worse than point-wise DPO on point-wise dataset, and vice versa, which supports our above claim. 

In addition to the above difference, we shall emphasize that **our investigation on point-wise dataset has motivated the design of more flexible tuning method**. Specifically, the absolute labeling on point-wise dataset motivates us to treat different types of samples (e.g., high quality positive data and noisy negative data) in different ways, which results in the ULMA framework. Our investigation on point-wise dataset and proposed new dataset may motivate more elaborate algorithmic design for point-wise preference learning.

Reviewer comment (Experiments):
It would be great to have more experimental results in terms of more LLM-based tasks.

Author response:
We have added two more datasets *QA-feedback* and *red-team* in the revision. The former dataset is a point-wise dataset with binary labels, and the second dataset is a point-wise dataset with continuous labels. In particular, *red-team* only has a single answer for each prompt and no comparison can be made, hence it cannot be transformed into a pair-wise dataset and pair-wise methods such as RLHF and DPO are inapplicable. The empirical results are presented in Table 1 and Table 2 in the updated manuscript. Please refer to the manuscript for more details.

Reviewer comment (Evaluation):
There is no significance test in the tables.

Author response:
In the revised manuscript, for each  harmful/helpful score presented in Table 1-3, we report its 95-percent confidence interval after repeating the model training procedure for three times beyond its mean value. The results show that, on the *red-team* and *QA-feedback* datasets, ULMA performs **significantly** better than other methods; on the *HH* and *Golden HH* datasets, though the confidence intervals are overlapped, the mean value of ULMA is still better. Please refer to the manuscript for detailed empirical results.

Reviewer comment (Novelty):
One essential issue is that in many cases, there can be a mapping function between pair-wise preference and point-wise preference. For example, from pair-wise to point-wise: you can directly enumerate how many positive preferences have been received for each document, and then rank the documents according to the numbers and assign a ranking score to each document. Or, a simpler way is to use the number of (positive num – negative num) preferences as the point-wise preference.

Author response:
Admittedly, there exists some relation between pair-wise data and point-wise data as you have suggested, but we shall note that it is **not** a one-to-one correspondence, and the transformation may suffer from information loss. Now we explicate the above point in two folds. 

(i) "Pair-wise to point-wise" transformation may lose some pair-wise comparison information. For example, consider two pair-wise datasets $\mathcal D_1=\\{x_1\succ x_2, x_2\succ x_3, x_3\succ x_4\\}$ and $\mathcal D_2=\\{x_1\succ x_3, x_3\succ x_2, x_2\succ x_4\\}$. Obviously, the two datasets differ from the pair-wise relation between $x_2$ and $x_3$. These two datasets will be transformed to the same point-wise dataset $\mathcal D_3=\\{(x_1,1), (x_2,0), (x_3,0), (x_4,-1)\\}$, in which $x_2$ and $x_3$ have the same score. In this case, the transformation drops the relation between $x_2$ and $x_3$, which incurs loss in information.

(ii) "Point-wise to pair-wise" transformation is unsuitable to some dataset where no comparison can be made, e.g., all samples have the same score or there is only one sample for some prompt. One example is the *red-team* dataset, which only has one sample for each prompt. This dataset cannot be transformed into a pair-wise dataset, and pair-wise methods such as RLHF and DPO are inapplicable. Please see the revised manuscript for more detailed descriptions and empirical results. 

Therefore, the conversion between pair-wise and point-wise datasets is not always possible, and may suffer from information loss in many cases. In the revised manuscript, we have compared the performance of point-wise DPO and point-wise DPO on pair-wise dataset *Golden HH* and point-wise dataset *QA-feedback*, respectively. We observe that pair-wise DPO performs slightly worse than point-wise DPO on point-wise dataset, and vice versa, which supports our above claim. 

In addition to the above difference, we shall emphasize that **our investigation on point-wise dataset has motivated the design of more flexible tuning method**. Specifically, the absolute labeling on point-wise dataset motivates us to treat different types of samples (e.g., high quality positive data and noisy negative data) in different ways, which results in the ULMA framework. Our investigation on point-wise dataset and proposed new dataset may motivate more elaborate algorithmic design for point-wise preference learning.

Reviewer comment (Evaluation):
To test the performance of an LLM, there are many evaluation metrics and many LLM-based tasks. Therefore, the authors are expected to test the LLM on more tasks and metrics.

Author response:
We have added two more datasets *QA-feedback* and *red-team* in the revision. The former dataset is a point-wise dataset with binary labels, and the second dataset is a point-wise dataset with continuous labels. In particular, *red-team* only has a single answer for each prompt and no comparison can be made, hence it cannot be transformed into a pair-wise dataset and pair-wise methods such as RLHF and DPO are inapplicable. The empirical results are presented in Table 1 and Table 2 in the updated manuscript. Please refer to the manuscript for more details.

Reviewer comment (Evaluation):
For the reported tables, many numbers are quite close, and it is necessary to have a significance test to see whether the proposed method is better, or alternatively, the mean and standard deviation for multiple runs could be reported.

Author response:
In the revised manuscript, for each  harmful/helpful score presented in Table 1-3, we report its 95-percent confidence interval after repeating the model training procedure for three times beyond its mean value. The results show that, on the *red-team* and *QA-feedback* datasets, ULMA performs **significantly** better than other methods; on the *HH* and *Golden HH* datasets, though the confidence intervals are overlapped, the mean value of ULMA is still better. Please refer to the manuscript for detailed empirical results.

Reviewer comment (Experiments):
Though the experimental results look promising to demonstrate framework’s effectiveness, more human preference datasets to align LLM should be included, such as datasets provided and used in [1] and [2].

Author response:
**A1:** We have added two more datasets *QA-feedback* and *red-team* in the revision. The former dataset is a point-wise dataset with binary labels, and the second dataset is a point-wise dataset with continuous labels. The new empirical results are presented in Table 1 and Table 2 (together with the original results on *HH* and *Golden HH*). Please refer to the revised manuscript for detailed descriptions and empirical results.

Reviewer comment (Experiments):
The proposed framework should be able to be generalized to more complex metrics (such as the discussion to handle continuous labels) but the datasets used in the experiment are only in binary classes, which is not enough to support the capability of its generalization.

Author response:
**A2:** In fact, the core concept of our proposed ULMA framework, i.e., using a hybrid objective formulation for different types of data, is capable of generalizing to more general metrics. On the new *red-team* dataset in the revision, we provide an example of how ULMA can be generalized to handle continuous labels. 

Specifically, *red-team* is a point-wise dataset on LLM's robustness to red teaming attacks, which consists of samples scoring from 0 to 4. The scores are rated by human and indicate how successful the attacks are, among which those rated 0 can be considered as high quality demonstration data. To better exploit these data, we introduce a hybrid loss by adding SFT loss of the samples rated 0 to the original MSE loss of all samples. 

In experiment, we observe that ULMA performs better than Unlikelihood and point-wise RLHF on *red-team* (note that pair-wise methods such as RLHF and DPO cannot be applied to this dataset since each prompt only has a single sample), which supports the capability of ULMA's generalization to more complex metrics.

Reviewer comment (Writing):
The generalization to other metrics with positive and negative samples needs further description in details.

Author response:
**A3:** Good suggestion! We have detailedly discussed how to generalize ULMA to datasets with continuous labels in Section 4.3 in the revised manuscript (at the end of page 6). Specifically, for these datasets, there is no direct separation of positive and negative samples. To apply ULMA, we can specify some sample as high quality data (e.g., the most helpful or harmless ones) and treat these high quality samples as "positive" demonstration data. Then we can adopt a hybrid loss by combining the SFT loss on these demonstration data with the original loss (e.g., MSE) on preference data. A concrete example on the *red-team* dataset can be seen in **A2**.

Reviewer comment (Reproducibility):
In this case, how does the framework deal with positive and negative samples?

Author response:
**A4:** The objective loss for continuous preference labels is a hybrid of SFT loss on high quality demonstration data and the original loss (e.g., MSE) on preference data. In this case, some samples in the dataset (e.g., the most helpful or harmless ones) are treated as the "positive" demonstration data and applied to the SFT loss. An example on the *red-team* dataset is discussed and empirical evaluated in the revised manuscript. Please refer to our response in **A2** and **A3** for more details.