Paper: Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICLR-2024)

Reviewer comment (Novelty):
The novelty is overstated. In essence, the primary experiment merely involves using different random seeds, prompting ChatGPT to generate varied responses, and then refining them. Although there's an additional experiment that uses different language models like ChatGPT and Bard, the test set is very limited.

Author response:
Thank you for your detailed comments. Please see our clarifications below.

**Novelty.** The primary novelty of the paper is proposing the methodology of multi-agent debate as an inference-time approach to improving the performance of LLMs across both factuality and reasoning. While prior works have explored how both using ensembles of models improves performance as well using self-reflection of a single model, our paper illustrates how both ideas may be jointly combined to generate more accurate responses.  

This combination goes beyond simply having an ensemble of self-reflection agents, as the debate procedure lets each agent reflect on both the generations of the original language model and that of other models. We illustrate the power of the debate procedure in new Figure XXXI ([screenshot here](https://ibb.co/426vcgb)) and Figure XXXII ([screenshot here](https://ibb.co/0FSWZFz)) in the updated paper, where we show how debate can be used to self-correct two initially incorrect answers.

We have further provided an additional experiment using debate on the open-source chat-Llama 7B model. Below, we find that similarly, debate improves final performance.

 | Model | Arithmetic | GSM  | MMLU |
| -------- | ------- | ------- | ------- |
|Single Agent  |  9.0 +- 1.6  | 20.7 +- 2.3 | 41.0 +- 2.8 |
|Single Agent (Reflection) | 10.7 +- 1.7 | 21.0 +- 2.3  | 39.7 +- 2.8 |
|Multi-Agent (Majority) | 11.0 +- 1.8 |  25.7 +- 2.5 | 43.3 +- 2.9|
|Multi-Agent (Debate) | 13.3 +- 1.9  | 29.3 +- 2.6 | 47.7 +- 2.9  |

We have added this to Appendix A.1 of the paper.

Reviewer comment (Experiments):
The proposed method is resource-intensive and not suitable for lengthy questions or answers. Employing multiple agents and several rounds of debate can lead to very long conversations, potentially exceeding the context limit and reducing performance.

Author response:
**Computational Expense.**  While our approach is computationally expensive compared to a single forward pass to a language model, it can be used as a method to take a computationally cheap model (i.e. GPT-3.5) and have it perform similarly to a significantly more computationally expensive model (i.e GPT-4 which is over 100 times more expensive), providing computational savings. In addition, the multiagent debate procedure can be distilled down into another model both for faster inference as well as a method for self-improvement in a language model.

Reviewer comment (Evaluation):
Although the performance improved, it is still not clear why "debating" would improve model performance. Especially, for uncertain answers (examples in Figure 9), it is not convincing why referencing each others' answers would generate more factual answers, as reasoning is not very relevant. Further exploration is required, in particular with more agents. Moreover, even for reasoning tasks (in Figure 2 Round 2), it does not seem that much reasoning is involved. The model is mostly copying, or echoing what the correct answer is, rather than having in-depth "debate", or reasoning, about why one answer is correct and the other is wrong. Without more analysis, this contribution is limited.

Author response:
**Understanding Gains in Debate.**  The Multiagent Debate procedure in the paper tightly couples two sources of performance gains in LLMs, (1) wisdom of the crowds, where multiple instances of LLMs can jointly leverage to improve performance (i.e. from majority voting)  (2) self-reflection, the ability for LLMs to use previous generations to improve their performance. However, the procedure goes beyond directly combining these two aspects because individual LLMs can reflect not only based on their thought process but also that of other agents (which is referred to as debate in the paper). The core reason multiagent debate improves This allows LLMs to recover and generate correct answers in a subsequent round of reasoning even if all answers in the current round are incorrect. This is illustrated in Figure 11 ([screenshot here](https://ibb.co/YDM7ZGc)) of the original paper and in Figure A.XXXI ([screenshot here](https://ibb.co/426vcgb)) and A.XXXII ([screenshot here](https://ibb.co/0FSWZFz)) in the updated paper.

To more clearly understand the performance of debate, we also consider a new baseline of running ensembles over self-reflection, where we instantiate multiple instances of a self-reflection agent and then take a final ensemble vote over generated self-reflection results.  We find that debate outperforms this baseline, indicating that discourse between agents is helpful overall beyond ensembling answers from a self-reasoning language model.

 | Model | Arithmetic | GSM  | MMLU |
| -------- | ------- | ------- | ------- |
|Single Agent  | 67.0 +- 4.7  | 77.0 +- 4.2 | 63.9 +- 4.8 |
|Single Agent (Reflection) | 72.1 +- 2.4  | 75.0 +- 4.3 | 57.7 +- 5.9 |
|Multi-Agent (Majority) | 75.0 +- 3.9 | 81.0 +- 3.9  | 67.0 +- 4.7 |
|Multi-Agent (Reflection) | 76.0 +- 4.3 | 80.0 +- 4.0 | 65.0 +- 4.7 |
|Multi-Agent (Debate) | 81.8 +- 2.3 | 84.0 +- 2.1  | 71.1 +- 4.6 |

To further understand the performance of debate with more agents, we also consider using a total of 10 agents for debate across 2 rounds of debate in the table below compared to majority vote over 50 agents. We find that having more agents in debate continues to improve the performance.


| Model | Arithmetic | GSM  | MMLU |
| -------- | ------- | ------- | ------- |
| Majority Vote (50 agents) | 92.0+- 2.7  | 85.0 +- 3.6 |  67.0 +- 4.7 |
| Debate (10 agents 2 rounds) | 96.0+- 1.9 |  89.0 +- 3.1 | 71.0 +- 4.5 |

Reviewer comment (Presentation):
Can you clarify what short and long represent in Figure 3?

Author response:
**Short vs Long.** In Figure 3, short refers to a prompt to a large language model that induces debates to be shorter in length, while long refers to a prompt to a large language model that induces debates to be longer in length. We have updated the paper to make this clearer.

Reviewer comment (Experiments):
When would the model agents be more "agreeable"? Would they be more agreeable when there are more similar answers (regardless of correctness), or would then tend to be more agreeable when the answer is correct? Can you verify the relationship between "agreeable" and "a result of instruction tuning or RLHF" using pre-aligned checkpoints?

Author response:
**Analysis of Agreeableness.** In our experiments, we found that agents were agreeable regardless of whether they knew the answer was correct so long as there were similar answers from other agents. To verify the relation between “agreeable” and “a result of instruction tuning or RLHF”, we consider the LLama-2 7B and the RLHF chat-LLama-2 7B models. On the GSM8K task, we find that  LLama-2 7B model achieves a consensus of 51.3% after debate compared to consensus of 62.7% with aligned chat-LLama 2 model. On the MMLU task, we find that  LLama-2 7B model achieves a consensus of 74.3% after debate compared to consensus of 100.0% with aligned  chat-LLama 2 model. This supports the conclusion that RLHF models tend to be more agreeable than the previous unaligned checkpoints. We have added this experiment to Appendix A.1.