Paper: Making Large Language Models Better Reasoners with Alignment (ICLR-2024)

Reviewer comment (Novelty):
The proposed approach is very similar to [LARGE LANGUAGE MODELS CAN SELF-IMPROVE (Huang et al., 2022)](https://arxiv.org/pdf/2210.11610.pdf), which came out a year ago. Since the authors did not cite it, I assume that they were not aware of it, but in terms of novelty there is a significant overlap.

Author response:
***Q1: Question about the novelty: the proposed approach is very similar to [Huang et al., 2022)](https://arxiv.org/pdf/2210.11610.pdf) [1], which severely hurts the paper in terms of novelty.***

A1: It is essential to point out that the contributions between [1] and our work are **totally different.**

[1] aim to achieve self-improvement of LLMs with two steps:

1. Using LLM to generate solutions for a given question with chain-of-thought promoting and select high-confidence solutions with self-consistency techniques.
2. Fine-tuning the LLMs on the selected self-generated solutions with a next-token prediction objective to achieve self-improvement.

**Totally Different** from [1]:

1. We discover that LLMs fine-tuned by the vanilla next-token prediction objective **suffer** from an assessment misalignment problem.
2. We propose an alignment fine-tuning (AFT) paradigm to address our identified problem. Our proposed AFT method consists of novel and effective alignment loss to train LLMs, which is totally different from the next-token prediction objective used in [1].
3. We delve deeply into recent ranking-based alignment methods (which are truly similar works to ours) and discover that the constraint, which has been overlooked by these approaches, is also crucial for their performance.

Given such **obvious differences in the contributions** between [1] and our work. We can confirm that the contributions of these two works are orthogonal.

Reviewer comment (Evaluation):
The motivation in Table 1 is unclear. T-Accuracy is ~40% but ~A-Accuracy is ~70% – is it a surprising result? The paper says that:  
>These results show that the assessment ability of VFT-LLMs is far from expected, as they cannot  
accurately discern the quality of various COTs of previously learned questions.  
The reviewer is not sure they agree with this interpretation and questions what other results the authors would expect.

Author response:
***Q2: Question about the motivation: The results of our pilot experiment (Table 1) are trivial and not surprising. What are the expected results of our pilot experiment, and what is the motivation derived from the pilot experiment?***

A2: The results of our pilot experiment inspired us to propose an effective alignment fine-tuning paradigm to improve the reasoning ability of LLMs, which is not trivial for the following reasons:

In this paper, we define two important abilities of LLM reasoners: 1) **reasoning ability**: LLMs can generate the right solution **for the unlearned question**; 2) **assessment ability**: LLMs can evaluate the quality of different candidate solutions **for the learned question**, i.e, they can assign higher scores (lower perplexity) to right solutions compared with wrong solutions.

Previous studies have shown that the vanilla chain-of-thought fine-tuning (VFT) can significantly improve the reasoning abilities of LLMs. However, is VFT sufficient for teaching LLM reasoning ability? We conducted a pilot experiment to explore this question.

If VFT is sufficient enough, we **expect that LLMs trained with VFT will demonstrate strong assessment ability**. This is based on a reasonable assumption that LLMs that truly learn to solve a reasoning question should be able to assess the quality of different candidate solutions to the learned question.

Contrary to our expectations, our pilot experiments show that VFT-LLMs lack this desired assessment ability: the assessment accuracy on **their learned questions** is just around 70% on GSM8K and 62% on ECQA.

**Their assessment accuracy is surprising (NOT trivial) and far from our expectations because**:

1. the assessment task in our pilot experiments is a binary classification task where a random selection baseline can have up to 50% assessment accuracy.
2. The questions of our pilot assessment task are from the training set, and LLMs have already been trained on these questions.
3. With the right training in problem-solving, we as humans can achieve near-perfect assessment accuracy. Take Figure 1 as an example, after learning to generate the reference solution for the given question, it becomes very easy for humans to evaluate the quality of two candidate solutions for the learned question. However, this level of assessment is currently beyond the capabilities of VFT-LLMs.

In addition, the pilot experiment also shows that **the reasoning ability on unlearned questions has a strong positive correlation to the assessment ability of learned questions**.

Therefore, our pilot experiences inspire us to improve the reasoning ability of LLMs by proposing an assessment alignment loss to improve their assessment and reasoning ability.

Reviewer comment (Novelty):
The paper says that:  
>We discover that LLMs fine-tuned by the vanilla fine-tuning ... frequently assign lower scores to high-quality COTs compared to low-quality ones  
While this is correct, the reviewer questions whether it is trivial — noting that in any machine learning model, sometimes the model assigns higher probability to the wrong output and lower probability to the correct output, which is the source of any mistake.

Author response:
***Q2: About the assessment ability for VFT-LLMs.***

The reviewer writes:
```
sometimes the model assigns higher probability to the wrong output and low probability to the correct output? Isn't this the source of any kind of mistake in any machine learning model? isn't it trivial?
```
It is trivial for the model to assign a higher probability to the wrong output for their **unlearned question**. However, we believe doing so for their **learned questions** is not trivial. If we think VFT is good enough to teach LLM reasoning, it will be really surprising when VFT-LLMs stumble on extremely easy assessment tasks for their learned questions, such as Figure 1.

The reviewer writes:

```
Why are these numbers surprising? The number "70%" on its own is meaningless without comparison to any alternative (that is non-human and non-random). Did the authors expect 100%? What about 80% or 90%, are they close to expected or not?
```

In Table 4, we have compared the assessment ability of VFT with that of our AFT (a non-human and non-random alternative). We include the results here for your convenience:

| Methods | Reasoning Accuracy | Assessment Accuracy |
| --- | --- | --- |
| VFT | 20.82 | 68.72 |
| AFT | 26.08 | 81.36 |

Please note that the COTs used to measure the assessment ability here are newly sampled COTs, distinct from the COTs used to train AFT. 

This result demonstrates that there is substantial room for improvement in the assessment ability of VFT, further affirming the validity of our motivation: **Enhancing the reasoning ability of LLMs by addressing the assessment misalignment problem in VFT.**

We would greatly appreciate it if you could let us know whether you are satisfied with our response. We will be happy to address any remaining concerns.

Reviewer comment (Theory):
Some claims in the paper are inaccurate. For example:  
>Intuitively, the MLE objective seeks to exclusively allocate probability mass to the reference COT  
The reviewer argues that it does not "exclusively" allocate probability mass to the reference COT, since a lot of mass remains for other possible CoTs.  
As another example:  
>As demonstrated by our pilot experiment, VFT-LLMs fail to give reasonable scores to COTs in GP and GN.  
The reviewer asks what are "reasonable scores" and what scores the authors expected.

Author response:
***Q3: What are "reasonable scores"? What scores did the authors expect? (Question to our statement “As demonstrated by our pilot experiment, VFT-LLMs fail to give reasonable scores to chain-of-thought (COTs) in GP and GN.”)***

A3: The “reasonable scores” imply that VFT-LLMs should give lower perplexity (high score) to right COTs in GP compared to wrong COTs in GN for their learned questions. However, our pilot experiments reveal that VFT-LLMs fail to give such “reasonable scores”, i.e., they have poor assessment ability. In addition, our pilot experiment also revealed that the assessment ability has a strong correlation to the final reasoning ability. Therefore, we propose an alignment fine-tuning paradigm to enhance their assessment ability to improve the reasoning ability of LLMs.

Reviewer comment (Experiments):
The experiments were performed across multiple benchmarks (which is great), using the 7B and 13B versions of LLama 1 and 2. However, the reviewer believes that these models were only pretrained, without instruction tuning or RLHF, and suggests that it would be great if the authors could also experiment with the "Chat" version of Llama 2 (of the same sizes).

Author response:
***Q5: Experiment results on the Chat version of Llama 2.***

A5: As you suggest, we conduct further experiments on the Chat version of LLM, i.e., LLama-2-7B-chat and LLama2-13B-chat on GSM8K:

| Methods (on 7B-Chat) | Accuracy | Methods (on 13B-Chat) | Accuracy |
| --- | --- | --- | --- |
| VFT | 39.03 | VFT | 47.19 |
| AFT (DC) | 43.82 | AFT (DC) | 50.32 |
| AFT (BC) | 44.04 | AFT (BC) | 51.01 |

As is shown, our AFT significantly outperforms VFT with the Chat version models as the backbone.

[1] Large Language Models Can Self-Improve. arXiv:2210.11610

Reviewer comment (Novelty):
The main approach is too similar to a previous work that came out a year ago (and was not cited), which severely hurts the paper in terms of novelty. The reviewer would vote for rejection unless convinced that there is a significant difference they have missed.

Author response:
***Q1: Regarding the empirical comparison to Huang et al., (2022). — Your main concern***

A1: The review writes:
```
The main concern that I have is the lack of conceptual and empirical comparison to Huang et al., (2022).
```
```
The fact that the loss proposed in this work is "different" is not sufficient; maybe the part of generating various CoT solutions (which as far as I understand, is common to the two papers) is the key, and then training on these generated solutions can be done even with the standard next-token loss?
```

I agree with you that generating various CoT solutions is important, and training LLMs on these generated solutions with a standard next-token loss is a necessary and strong baseline. **It is essential to point out that we have compared our AFT with such baseline, i.e., Rejective Sample Fine-tuning (RFT, a concurrent work to ours)**. RFT further trains LLMs on the high-quality generated solutions with the correct answer using the next token prediction. **Our experiments have demonstrated that AFT outperforms RFT, especially in the ranking situation.**

**Although Huang et al., (2022) focus on the self-improvement of LLMs on unlabeled datasets, which is totally different from ours**, we have also followed your suggestion to replicate Huang et al., (2022)'s approach. This replication involves training LLama-7B on their generated CoTs with the majority voting answer.

| Methods | GSM8K | GSM8K-RANK |
| --- | --- | --- |
| VFT | 36.48 | 20.82 |
| Huang et al., (2022) | 38.23 | 22.34 |
| RFT | 39.75 | 25.09 |
| AFT | 40.26 | 26.08 |

As is shown, our AFT outperforms Huang et al., (2022).  **More importantly, from the principles of the methods, AFT can better utilize the feedback compared with RFT or Huang et al., (2022).** Specifically, AFT has the ability to help LLMs recognize quality differences among any given COT pair in a ranking context, while the RFT or Huang et al., (2022)'s approach can not achieve this because they only try to optimize the probability of the chosen COTs.

Based on the experimental results and the underlying principles of these methods, it's clear that AFT is essential. We hope these results can address your main concern.

Reviewer comment (Evaluation):
It would be great if the authors could explain why the performance drop for other baseline methods when comparing to vanilla finetuning.

Author response:
As illustrated in Sections 6.1 and 6.2, the performance of other ranking-based methods drops because they lack constraint in their losses and over-punish negative COTs. After adding our constraint, they outperform the vanilla fine-tuning.

Reviewer comment (Experiments):
I also wonder how the quality of LLM-generated COTs impact the performance of AFT. For example, how large is the variance using 3 generated examples?

Author response:
Our AFT enhances the reasoning capabilities of LLMs by improving their assessment ability on sampled candidate COTs. We discovered that the diversity of these sampled candidates is crucial. For instance, we found that data sampled from models fine-tuned over two epochs outperforms data sampled from models fine-tuned over three epochs, with a noteworthy accuracy difference approaching 1%.

On your suggestion, we sampled 3 groups of generated COTs (each group comprising 3 COTs per question) on GSM8K from the fine-tuned Llama-7B model. We then used this sample data to further fine-tune the Llama-7B model using AFT. The standard deviations measured were 0.42 and 0.53 for AFT-DC and AFT-BC, respectively. The results appear stable, which we believe can be attributed to the homogeneity in the quality of samples drawn from the same model.

The question of how to define quality, and how to sample high-quality solutions from fine-tuned LLMs, remains a valuable area of exploration, which we leave for future work.

Reviewer comment (Experiments):
There is no clear argumentation why other mathematical datasets are not used or referenced in order to back up the design decision for the chosen datasets. It would be good/important to introduce a clear argumentation or reference why these datasets have been chosen, as there are other/more datasets in this field.

Author response:
***Q2: Why are the chosen evaluation datasets sufficient for your claims? Are these the main datasets of other related works in the field or are other reasoning datasets "easier" than the chosen ones?***

A2: We focus on two typical reasoning tasks, including mathematical reasoning and commonsense reasoning. For the mathematical reasoning task, we employ GSM8K and AQUA-RAT as our benchmarks, while we use ECQA for the commonsense reasoning task. All of these benchmarks are broadly recognized and used in relevant research works [1] [2] [3] [4] [5] [6] [7].

We observe the assessment misalignment problem of VFT on both mathematical and commonsense reasoning tasks, and our proposed AFT significantly outperforms VFT on all benchmarks. These results are sufficient for our claims.

Reviewer comment (Experiments):
There is no evaluation against some of the direct competitors, such as the referenced Li et al., 2023. It would be important to argue why these models have not been chosen for comparison—maybe it is not required. Otherwise, it is difficult to understand for the reader if the proposed approach supersedes the current State-of-the-Art. As the approach of the paper can be applied to other/general fine-tuning problems, the added value could also be shown by comparing on more general datasets.

Author response:
***Q3: Are the empirical results on-par with other referenced works in the field, such as Li et al., 2023?***

A3: Li et al., 2023 [6] is a chain-of-thought prompting work, which is orthogonal work to ours. They find that better reasoning performance of LLMs can be achieved by a step voting strategy.

Our work focuses on revealing the assessment misalignment problem of the vanilla chain-of-thought fine-tuning (VFT) paradigm, and proposes an alignment fine-tuning (AFT) paradigm to address the find problem. Therefore, we choose VFT and recent alignment methods as our baselines.  Our experiments demonstrate the importance of assessment misalignment problems and the effectiveness of our AFT.

Reviewer comment (Evaluation):
Are the empirical results on par with other referenced works in the field, such as Li et al., 2023?

Author response:
***Q3: Are the empirical results on-par with other referenced works in the field, such as Li et al., 2023?***

A3: Li et al., 2023 [6] is a chain-of-thought prompting work, which is orthogonal work to ours. They find that better reasoning performance of LLMs can be achieved by a step voting strategy.

Our work focuses on revealing the assessment misalignment problem of the vanilla chain-of-thought fine-tuning (VFT) paradigm, and proposes an alignment fine-tuning (AFT) paradigm to address the find problem. Therefore, we choose VFT and recent alignment methods as our baselines.  Our experiments demonstrate the importance of assessment misalignment problems and the effectiveness of our AFT.

Reviewer comment (Experiments):
How would standard RLHF perform here? It would be an interesting baseline, as no constraints on the ranking loss are put and it is simpler than PRO.

Author response:
***Q4: How would standard RLHF perform here? It would be an interesting baseline, as no constraints on the ranking loss are put and it is simpler than PRO.***

A4: Using standard Reinforcement Learning algorithms such as Proximal Policy Optimization (PPO) requires a well-trained Reward Model which needs careful human annotation and data quality control, as well as significantly more computational resources, which is beyond the scope of this paper and left for further work.

In addition, our proposed AFT is a ranking-based preference alignment fine-tuning method, and the most relevant baselines of our work are other ranking-based methods such as RRHF and PRO that are analyzed in our paper.

Reviewer comment (Theory):
Although the intention is to improve the "reasoning" capability of the model, the additional loss function makes use of the slightly risky assumption that generated outputs with the correct final answer should be assigned higher score than those with the wrong final answer. One could argue that the chain of thoughts itself is perhaps more important than the final answer and some negative examples should still be scored higher than positive examples with "wrong" reasoning steps. Obviously this cannot be done without additional annotation and the proposed approach seems to work fine despite the risk.

Author response:
***Q1: The slightly risky assumption that generated outputs with the correct final answer should be assigned a higher score than those with the wrong final answer.***

A1: Thank you for your valuable review comments. In this paper, we manually check 50 pairs of incorrect-correct pairs, discovering that in 48 out of these 50 pairs, the solution with the correct answer surpassed the one with the incorrect answer in terms of quality.  In addition, our proposed alignment fine-tuning (AFT) paradigm significantly outperforms the vanilla chain-of-thought fine-tuning (VFT) paradigm. These evidences substantiate that it is reasonable to consider solutions with correct answers superior to those with incorrect ones.

Moreover, the main contribution of our paper does not reside in the development of the feedback signal. Rather, it fundamentally revolves around pinpointing the assessment misalignment problem within VFT and introducing AFT as a solution to this issue. We believe that with more precise feedback, our AFT will yield more significant improvement.