Paper: Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis (ICLR-2024)

Reviewer comment (Presentation):
Section 4 is hard to read but it shouldn't be. The first subsection is the background on SFT, the next three sections are about the specific novel training approach proposed in the paper. The fifth subsection talks about inference. Finally there is a discussion section about the mathematical intuition on the exact mechanisms. I strongly recommend the authors to give this section a clear structure to inform readers the background, exact contribution and discussions.

Author response:
**W1: Clearer structure of Section 4.**
Thanks for your advice. Following your suggestions, we have updated the structure of Section 4 by separating the background notations, the proposed method and the discussions. See details in the revision.

Reviewer comment (Writing):
I suggest avoiding using "it still remains oblivious" in the introduction, as we are not sure what "oblivous" exactly means in this context.

Author response:
**W2: Inappropriate wording and typos.**

Thank you for pointing that out. We have correct all the mentioned inappropriate wording and typos in the revision.

Reviewer comment (Writing):
In induction success rate, "augments mistake induction induction" seems to be a typo.

Author response:
**W2: Inappropriate wording and typos.**

Thank you for pointing that out. We have correct all the mentioned inappropriate wording and typos in the revision.

Reviewer comment (Experiments):
Do authors perform experiments to ground the mathematical explanations? More precisely, after applying the proposed alignment training method, have the authors observed that harmless tag prediction is improved?

Author response:
**Q1: Verifying Equation (2) via harmless tag prediction.**

Following your suggestions, we conduct an additional experiment evaluating the LLM's ability to discern the harmfulness of responses. Specifically, we utilize the test set of PKU-SafeRLHF dataset, each instruction paired with a response and the corresponding binary harmful or harmless tag, and then instruct LLMs to distinguish whether the provided response is harmful or not. The accuracy is reported in Table R5. The LLM's accuracy in identifying the harmfulness of responses improves from 54.5% for vanilla Aplaca-7B to 72.6% after applying our method, revealing the significant improvement of discrimination ability.

▾ Table R5. **Comparison between discrimination ability with respect to the binary classification accuracy on the test set of the PKU-SafeRLHF dataset.**

|              | Vanilla Alpaca | SFT         | Ours             |
| :------------: | :--------------: | :-----------: | :----------------: |
| Accuracy (%) | 54.5           | 54.9 (+0.4) | **72.6 (+18.1)** |

Reviewer comment (Writing):
While the core idea of the paper is sound, the explanations were confusing and scattered, and the experimental results were not convincing.

Author response:
**W1: The reason for discussing the relative ease of discrimination compared to generation, which is already a core principle of RLHF.**

Thank you for your valuable feedback. The discussion on generation versus discrimination actually sets the foundation for our methodology, which utilizes the inherent discriminative power of LLMs. Our method, leveraging the LLM's self-critique for improved generation via mistake analysis, stands out from RLHF's reliance on external human annotations and reward models. This difference emphasizes the importance of easier discrimination over generation by the model itself. 

According to your comments, we realize that the success of RLHF may also benefit from increasing the discrimination ability of models with external reward models. On the other hand, our method boosts this ability through mistake analysis, which provides more informative feedback in natural language compared to the relative quality ranking in RLHF (see Figure 1). This explains why our model achieves superior performance over RLHF even without external reward models (refer to Table 1).

Reviewer comment (Presentation):
Figure 2, which discusses generation vs discrimination, has a very low-information caption that is largely redundant with the figure itself: "Each histogram is composed of three segments with distinct colors, labeled with three score numbers...".

Author response:
**W3 and Q1: Experimental results with the same mistake source for baseline methods.**

As demonstrated in the 4th-6th rows in Table 1 and the 3rd-5th rows in Table 2, we have conducted experiments for all baseline methods with the **original mistakes** provided by the datasets for a fair comparison, revealing the superiority of our mistake analysis. Following your suggestions, we further provide the experimental results of all baselines with **induced mistakes** below and also in the revision, which is consistent with our previous observation.

▾ Table R3. **Comparative results of LLM alignment across various methods with induced mistakes.**

| Method          | Mistake Source | Analysis Source | Helpful Score | Harmless Score | Harmless Rate (%) | Harmless Helpful |
| :---------------: | :--------------: | :---------------: | :-------------: | :--------------: | :-----------------: | :----------------: |
| Critique-Revise | Alpaca         | -               | 6.11          | 6.17           | 61.3              | 4.56             |
| CoH             | Alpaca         | -               | 6.28          | 6.87           | 65.7              | 5.29             |
| Ours            | Alpaca         | Alpaca          | **6.38**      | **7.41**       | **72.4**          | **5.39**         |

▾ Table R4. **Comparative results of defense against attacks across various methods with induced mistakes.**

| Method          | Mistake Source | Analysis Source | Goal Hijacking Score | Goal Hijacking Rate (%) |
| :---------------: | :--------------: | :---------------: | :--------------------: | :-----------------------: |
| Critique-Revise | ChatGLM        | -               | 7.42                 | 76.5                    |
| CoH             | ChatGLM        | -               | 7.86                 | 79.4                    |
| Ours            | ChatGLM        | ChatGLM         | **8.14**             | **85.3**                |

Reviewer comment (Experiments):
In Table 1, both rows showing improvement over baselines use Alpaca as the Mistake Source. For a fair comparison, the baselines should also make use of mistakes generated by Alpaca.

Author response:
**W3 and Q1: Experimental results with the same mistake source for baseline methods.**

As demonstrated in the 4th-6th rows in Table 1 and the 3rd-5th rows in Table 2, we have conducted experiments for all baseline methods with the **original mistakes** provided by the datasets for a fair comparison, revealing the superiority of our mistake analysis. Following your suggestions, we further provide the experimental results of all baselines with **induced mistakes** below and also in the revision, which is consistent with our previous observation.

▾ Table R3. **Comparative results of LLM alignment across various methods with induced mistakes.**

| Method          | Mistake Source | Analysis Source | Helpful Score | Harmless Score | Harmless Rate (%) | Harmless Helpful |
| :---------------: | :--------------: | :---------------: | :-------------: | :--------------: | :-----------------: | :----------------: |
| Critique-Revise | Alpaca         | -               | 6.11          | 6.17           | 61.3              | 4.56             |
| CoH             | Alpaca         | -               | 6.28          | 6.87           | 65.7              | 5.29             |
| Ours            | Alpaca         | Alpaca          | **6.38**      | **7.41**       | **72.4**          | **5.39**         |

▾ Table R4. **Comparative results of defense against attacks across various methods with induced mistakes.**

| Method          | Mistake Source | Analysis Source | Goal Hijacking Score | Goal Hijacking Rate (%) |
| :---------------: | :--------------: | :---------------: | :--------------------: | :-----------------------: |
| Critique-Revise | ChatGLM        | -               | 7.42                 | 76.5                    |
| CoH             | ChatGLM        | -               | 7.86                 | 79.4                    |
| Ours            | ChatGLM        | ChatGLM         | **8.14**             | **85.3**                |

Reviewer comment (Experiments):
In Table 2, the row showing improvement over baselines uses ChatGLM as the Mistake Source. For a fair comparison, the baselines should also make use of mistakes generated by ChatGLM.

Author response:
**W3 and Q1: Experimental results with the same mistake source for baseline methods.**

As demonstrated in the 4th-6th rows in Table 1 and the 3rd-5th rows in Table 2, we have conducted experiments for all baseline methods with the **original mistakes** provided by the datasets for a fair comparison, revealing the superiority of our mistake analysis. Following your suggestions, we further provide the experimental results of all baselines with **induced mistakes** below and also in the revision, which is consistent with our previous observation.

▾ Table R3. **Comparative results of LLM alignment across various methods with induced mistakes.**

| Method          | Mistake Source | Analysis Source | Helpful Score | Harmless Score | Harmless Rate (%) | Harmless Helpful |
| :---------------: | :--------------: | :---------------: | :-------------: | :--------------: | :-----------------: | :----------------: |
| Critique-Revise | Alpaca         | -               | 6.11          | 6.17           | 61.3              | 4.56             |
| CoH             | Alpaca         | -               | 6.28          | 6.87           | 65.7              | 5.29             |
| Ours            | Alpaca         | Alpaca          | **6.38**      | **7.41**       | **72.4**          | **5.39**         |

▾ Table R4. **Comparative results of defense against attacks across various methods with induced mistakes.**

| Method          | Mistake Source | Analysis Source | Goal Hijacking Score | Goal Hijacking Rate (%) |
| :---------------: | :--------------: | :---------------: | :--------------------: | :-----------------------: |
| Critique-Revise | ChatGLM        | -               | 7.42                 | 76.5                    |
| CoH             | ChatGLM        | -               | 7.86                 | 79.4                    |
| Ours            | ChatGLM        | ChatGLM         | **8.14**             | **85.3**                |

Reviewer comment (Writing):
There are a number of typos throughout the paper, including:  
- Bottom of page 2: "into precious instruction tuning corpus"  
- Bottom of page 5: "Dissusion"  
- Bottom of page 5: "Bayesian’s Theorem".

Author response:
Thank you for your thorough review of the manuscript. We have carefully checked the entire paper and made the following changes:

1. Corrected grammatical errors and awkward phrasings;

2.  Reorganized the paper, particularly the introduction and preliminary sections, to enhance the overall coherence.

The revised version has been updated. We hope that it can address your concerns adequately.

Reviewer comment (Evaluation):
May not generalize well to diverse contexts beyond the training distributions.

Author response:
**(1) Generalization beyond training distributions.** We acknowledge that no alignment method, including ours, can fully address all out-of-distribution scenarios. Nonetheless, our model exhibits enhanced generalization capabilities by learning from a wide array of mistakes and their analyses, rather than solely relying on human-labeled instruction-response pairs. As depicted in Figure 5, our model effectively rejects unsafe user instructions in "Goal Hijacking" scenarios for topics not in the training data, a feat not achieved by methods like SFT. This demonstrates our model’s advanced understanding of such complicated mistakes and its improved generalization ability.

Reviewer comment (Experiments):
Hard to control mistake induction to cover all failure modes.

Author response:
**(2) Addressing diverse failure modes and mistakes of alignment models.** Admittedly, even meticulously aligned LLMs face challenges in completely avoiding unsafe responses in novel or unanticipated situations. This necessitates an adaptable "debugging" methodology for addressing numerous corner cases. Section 5.2 details our approach’s efficacy in defending against new-coming instructional attacks. After applying our method, the safety-aligned ChatGLM model demonstrates significant post-alignment improvement, with its Harmless Rate in resisting "Goal Hijacking" attacks increasing from 68.4% to 85.3%. This positions it as the second rank only under the OpenAI's GPT-3.5, as per the SAFETYPROMPTS benchmark [1, 2], while maintaining its helpfulness performance.

Reviewer comment (Evaluation):
Still possible for aligned models to make mistakes during inference.

Author response:
**(2) Addressing diverse failure modes and mistakes of alignment models.** Admittedly, even meticulously aligned LLMs face challenges in completely avoiding unsafe responses in novel or unanticipated situations. This necessitates an adaptable "debugging" methodology for addressing numerous corner cases. Section 5.2 details our approach’s efficacy in defending against new-coming instructional attacks. After applying our method, the safety-aligned ChatGLM model demonstrates significant post-alignment improvement, with its Harmless Rate in resisting "Goal Hijacking" attacks increasing from 68.4% to 85.3%. This positions it as the second rank only under the OpenAI's GPT-3.5, as per the SAFETYPROMPTS benchmark [1, 2], while maintaining its helpfulness performance.

Reviewer comment (Evaluation):
Limited analysis on how mistake analysis exactly improves alignment.

Author response:
**W4 & Q4: Theoretical analysis on how mistake analysis improves alignment.**

In Section 4, Equation (2), we explore the role of mistake analysis in improving LLM alignment from a Bayesian perspective, considering that LLMs operate as conditional generators. Mistake analysis involves examining the model's harmful responses to identify patterns leading to such errors. To train the model in this way, we construct triplets of the instruction $X$, the response $Y$, and the mistake analysis $T$ indicating whether the response is harmful or harmless. Such data refines the model's understanding of the conditional probabilities $p(T|Y, X)$, *i.e.*, the likelihood of a response being harmful or harmless given a certain instruction. Subsequently, we enhance its ability to generate appropriate and harmless responses $p(Y|X, T)$ through guided inference, where the model is required to produce harmless responses during inference (see Figure 1). By focusing on optimizing conditional probabilities through the analysis of mistakes, we enhance the model's ability to generate safe, coherent, and contextually appropriate responses. This is why mistake analysis benefits model alignment. See Section 4.2 and Appendix D for more details.

Reviewer comment (Experiments):
How resilient is the method to adversarial instructions aimed at bypassing the alignment?

Author response:
**Q1: Resilience against adversarial instruction attacks.**

Recent studies [3, 4] demonstrate that even well-aligned models, such as OpenAI's GPT-4 and Anthropic's Claude2, are vulnerable to adversarial instruction attacks, presenting a challenge distinct from aligning with human values. Our research primarily focuses on aligning LLM with human values. We have observed the effectiveness of our method in defending against delicately designed instruction attacks, such as "Goal Hijacking" with superior generalization ability (see Section 5.2). This suggests the potential effectiveness of our method in enhancing resilience against adversarial attacks. Further investigation will be considered in our future work.

Reviewer comment (Experiments):
Is the quality or diversity of mistake analysis critical for alignment improvements?

Author response:
**(1) Quality of mistake analysis.** Yes, the quality of mistake analysis significantly impacts alignment performance. As Table 1 shows, using GPT-3.5-turbo for mistake analysis results in better alignment than using Alpaca-7B itself (see rows 7 and 8). Additionally, guided analysis outperforms unguided analysis, as shown in Table 3 (rows 3 and 4).

Reviewer comment (Experiments):
Could you conduct an ablation study focused on mistake analysis components?

Author response:
**(2) More ablation study on mistake analysis.** In Section 5.3, we present a comprehensive ablation study on the essential components of mistake analysis, including (i) quality of mistake analysis (ii) quantity of mistake analysis and (iii) the source of responses for constructing analysis data. We find that high-quality analysis, whether guided or from advanced models like GPT, leads to better alignment. On the other hand, increasing the volume of analysis data does not always improve results. We observe a decrease in performance when multiple analyses are applied to the same instructions, possibly due to conflicting interpretations in the analyses of the same instruction. Furthermore, we observe that the alignment effectiveness is maximized when the bad cases are generated by the model itself. See Section 5.3 for detailed experimental results.

Reviewer comment (Theory):
Is there a theoretical understanding of why mistake analysis provides strong alignment signal?

Author response:
**W4 & Q4: Theoretical analysis on how mistake analysis improves alignment.**

In Section 4, Equation (2), we explore the role of mistake analysis in improving LLM alignment from a Bayesian perspective, considering that LLMs operate as conditional generators. Mistake analysis involves examining the model's harmful responses to identify patterns leading to such errors. To train the model in this way, we construct triplets of the instruction $X$, the response $Y$, and the mistake analysis $T$ indicating whether the response is harmful or harmless. Such data refines the model's understanding of the conditional probabilities $p(T|Y, X)$, *i.e.*, the likelihood of a response being harmful or harmless given a certain instruction. Subsequently, we enhance its ability to generate appropriate and harmless responses $p(Y|X, T)$ through guided inference, where the model is required to produce harmless responses during inference (see Figure 1). By focusing on optimizing conditional probabilities through the analysis of mistakes, we enhance the model's ability to generate safe, coherent, and contextually appropriate responses. This is why mistake analysis benefits model alignment. See Section 4.2 and Appendix D for more details.

Reviewer comment (Evaluation):
The claim about generation and discrimination in Section 3.1 is confusing. The authors use GPT-4 scores (plus human verification) on generated responses / analysis to demonstrate that discrimination is easier than generation. But in my view, discrimination requires to output a definite classification result (such as safe or unsafe), whose performance can be easily assessed via objective metrics like accuracy. The analysis may help the model to conduct discrimination. But its generation quality is not equal to discrimination results, especially the quality is measured via GPT-4 in a “continuous” scale of 1-10.

Author response:
**W1: Justification for comparing LLM's "discrimination" ability with generation" ability.**

**(1) Definitions of generalization and discrimination.** Thank you for your valuable feedback. To clarify, in our paper, "generation" is defined as the LLM's ability to produce helpful and harmless responses to user instructions, while "discrimination" refers to its capacity to identify potential errors in responses, which requires not only a binary classification between "harmful" and "harmless" but also to encompass a more nuanced analysis of the response quality, as illustrated in Figure 7 of the Appendix. Therefore, the continuous score is considered as a better metric than the simple binary classification accuracy here. The preliminary aims to show the observation that LLMs are generally more adept at identifying potential mistakes than directly generating high-quality responses. This distinction is vital for the subsequent method design, where we use self-critique data to enhance the model's generation capabilities.

**(2) Experiments of binary harmful/harmless classification.** However, we acknowledge your concern about regarding the potential oversight of the binary classification accuracy. Following your suggestions, we conduct an additional experiment evaluating the LLM's ability to discern the harmfulness of responses. Specifically, we utilize the test set of PKU-SafeRLHF dataset, each instruction paired with a response and the corresponding binary harmful or harmless tag, and then instruct LLMs to distinguish whether the provided response is harmful or not. The accuracy is reported in Table R1. The LLM's accuracy in identifying the harmfulness of responses improves from 54.5% for vanilla Alpaca-7B to 72.6% after applying our method, revealing the significant improvement of discrimination ability.

▾ Table R1. **Comparison between discrimination ability with respect to the binary classification accuracy on the test set of the PKU-SafeRLHF dataset.**

|              | Vanilla Alpaca |           SFT | Ours |
| :--------: | :-----------------: | :-------------: | :-----: |
| Accuracy (%) | 54.5           | 54.9 (+0.4) | **72.6 (+18.1)** |

Reviewer comment (Theory):
The derivation in Equation (2) of Section 4 is not rigorous. The authors get a conclusion that $p(T|Y,X)$ is proportional to $p(Y|X,T)$. This indicates that the gap between $\log p(T|Y,X)$ and $\log p(Y|X,T)$ is a scalar. Following this, if we train a safety discrimination model with the loss of $\log p(T|Y,X)$, we can directly obtain a safe response generation model. This is counter-intuitive and needs more explanations. I’m afraid that the authors miss some important assumptions in their derivation.

Author response:
**W2: More explanation about Equation (2) which implies a direct link between training a safety discrimination model and obtaining a safe response generation model.**

Thanks for your valuable comment. We have revised the deviation of Equation (2) in Section 4 and Appendix D. In summary, your observation about the counter-intuitive nature of directly obtaining a safe response generation model from a safety discrimination model is astute when only mistake analysis data is utilized in fine-tuning. However, this scenario is different from the discussion of Equation (2), where the mistake analysis data is incorporated together with the normal SFT data, as discussed in the **"Step 2: Guided analysis generation"** paragraph of Section 4.1.

Let us start with the Bayes' Theorem $p(T|Y, X) = [p(Y|X, T)p(X|T)p(T)]/[p(Y|X)p(X)]$, which simplifies to $p(T|Y, X) \propto p(Y|X, T)$ under the assumption that $p(Y|X)$ remains relatively stable during the fine-tuning process. To maintain the model's capability to produce a response given an instruction (*i.e.*, preserving $p(Y|X)$）, we combine original SFT data, both helpful ($D_{\text{helpful}}$) and harmless ($D_{\text{harmless}}$), with mistake analysis data during fine-tuning, rather than solely relying on the mistake analysis data. Therefore, Equation (2) actually aims to explore how the additional mistake analysis data, along with the original SFT instruction-response pairs, can enhance alignment performance. 

Moreover, our ablation study in Section 5 demonstrates that an excess of mistake analysis data, particularly when it doubles the SFT data, actually leads to poorer performance (3rd vs. 5th rows), which indicates a nuanced relationship between different types of data in fine-tuning and the model's ability to balance between generation and discrimination tasks effectively. We hope this can clarify the assumption and methodology underlying our approach and address your concerns.

Reviewer comment (Experiments):
In Section 5.1, the authors only use Alpaca as their base model in the experiment. Since Alpaca is a weak base model, improving its performance may be relatively easy. I wonder whether the proposed method can still work well on stronger base models such as Vicuna.

Author response:
**W3: Justification about whether the method effective on the weak Alpaca model in Section 5.1 is also applicable to stronger models like Vicuna.**

**(1) Reason for choosing Alpaca.** We select Alpaca-7B as the baseline in Section 5.1 due to its absence of prior safety alignment. This characteristic renders it an ideal candidate for evaluating the effectiveness of alignment methods for an unaligned LLM from scratch. Our results, as elaborated in Table 1 and Figures 12-13, demonstrate that even unaligned LLMs like Alpaca-7B can effectively leverage  mistake analysis to enhance their safety instruction-following capabilities.

**(2) Effectiveness on SOTA models.** Moreover, to address concerns about the method's effectiveness on stronger, safety-aligned models, we have expanded our experiments to include ChatGLM-6B, the highest-ranked open-source model on the C-Eval leaderboard [1]. As shown in Table 2 and Section 5.2, after incorporating the mistake analysis to defend against a novel type of instruction attack known as "Goal Hijacking", ChatGLM ranks the second on the SAFETYPROMPTS benchmark [2, 3]. Notably, it is only surpassed by OpenAI's GPT-3.5 in defending against the "Goal Hijacking" attack, while concurrently preserving its helpfulness performance.

**(3) Additional experiments on Vicuna.** In line with your suggestion, we further conduct experiments on Vicuna-7B (v1.1) following the exact setting as in Section 5.1. As reported in Table R2, our mistake analysis achieves consistent improvement regradless of the baseline models.

▾ Table R2. **Comparative results of LLM alignment across various methods on Vicuna-7B.**

| Method | Helpful Score | Harmless Score | Harmless Rate (%) | Harmless Helpful |
| :------: | :-------: | :----: | :--------: | :----: |
| Vicuna (vanilla) | 7.17          | 9.62           | 95.9              | 8.85 |
| SFT | **7.33**      | 9.63           | 96.1              | 8.84 |
| Ours | 7.30          | **9.74**          | **97.1**              | **8.90** |

[1] C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. https://cevalbenchmark.com/.

[2] SAFETYPROMPTS Benchmark. http://115.182.62.166:18000/.

[3] Sun H, Zhang Z, Deng J, et al. Safety assessment of Chinese large language models. arXiv preprint arXiv:2304.10436, 2023.