Paper: ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (ICLR-2024)

Reviewer comment (Experiments):
The experiments are conducted on relatively simple reasoning tasks, which seem inconsistent with the claim of "solving complex reasoning problems" made in the introduction. Does ReConcile enhance performance on more complex reasoning tasks, such as logical reasoning, the MATH dataset, CommonsenseQA?

Author response:
> **The experiments are conducted on relatively simple reasoning tasks.**

We respectfully disagree with the reviewer here because the tasks considered in our study are not simple. StrategyQA, CommonsenseQA, GSM8k, and AQuA are some of the most commonly used benchmarks in almost all prior works on reasoning. That said, following your suggestions, we conducted experiments on a few other datasets to further demonstrate the capabilities of ReConcile. 

* **CommonsenseQA**: Please note that CommonsenseQA experiments were already part of our paper and were referred to as ECQA. This is because we leverage convincing human explanations in ReConcile, which come from the ECQA benchmark (short for Explanations for CommonsenseQA) [1]. ReConcile obtains a 6.4% improvement in ECQA, as per Table 2 in the original version of the paper.
* **Logical Reasoning**: StrategyQA is listed as a logical reasoning task as per the Big-Bench categorization of tasks [5], for which we already have results in the paper. We also experimented with another logical reasoning task, namely the “Date Understanding” task [2]. See results in the common response.
* **MATH**: Following your suggestion, we also tested ReConcile on MATH. See results in the common response.

Reviewer comment (Experiments):
The proposed method consists of multi-model predictions. An ablation study (Table 5) shows that the performance significantly decreases without the multi-model approach. It is unclear whether the performance gain is caused by the multi-model ensemble. A comparison with an ensemble baseline should be considered.

Author response:
> **It is Unclear whether the performance gain is caused by the multi-model ensemble.**

Please refer to our analysis and discussion in the common response.

> **A comparison with an ensemble baseline should be considered.**

Apart from the vanilla single-agent methods, all of our baselines in Table 2 are already indeed ensemble-based (either involving a single agent like self-consistency or involving multiple agents like the Multi-agent Debate). Moreover, even for ReConcile, we compare different mechanisms for obtaining the "Team Answer" in Table 3 (e.g., majority vote, weighted vote, maximum confidence), all of which should be seen as ensemble methods for aggregating the answers from different models. In summary, most of the pertinent baselines in our paper are ensemble-based.

Reviewer comment (Experiments):
There are many hyperparameters in ReConcile (number of discussion rounds, voting weight). It would be interesting to explore if these hyperparameters can be dynamically generated during the roundtable discussions and if they are robust across tasks.

Author response:
> **There are many hyperparameters in ReConcile (number of discussion rounds, voting weight).**

This is **not** true because the number of discussion rounds is not a hyperparameter. As Algorithm 1 of our paper shows, the number of discussion rounds is dynamically determined based on when all agents reach a consensus. R is just the "maximum" number of rounds, which can be any arbitrarily large value.

As for the voting weights, we use the exact same weights across all seven tasks in the paper and in this rebuttal. This is a testament to its robustness. Please refer to our common response for a more comprehensive discussion of our recalibration strategy and voting weights.

In summary, ReConcile does not have many hyperparameters, and our choice of voting weights is also adequately backed up by extensive robustness and ECE-style experiments.

Reviewer comment (Experiments):
It is unclear whether a weak model could negatively impact the performance.

Author response:
> **It is unclear whether a weak model could negatively impact the performance.**

Note that we do have experimental results in the paper using agents of varying strengths. In particular, Table 3 shows the results with GPT4 as one of the agents. We'd like to point you to the paragraph "_Using GPT-4 as an agent in ReConcile._" in Section 5.1 for details (and Reviewer 5C9y also acknowledges this in their strengths). Similarly, in Appendix B.2 in our original version, we have experiments with LLaMA-2 as one of the agents. Our conclusion is that through the discussion process in ReConcile, all agents individually improve (e.g., see GPT-4 improvement by 10% in Table 4). If the three agents have similar strengths, the team performance improves beyond all individual results, but if one agent is significantly stronger than others, the team answer expectedly converges to the strongest agent.

Reviewer comment (Novelty):
There is a line of work improving the reasoning ability of LLMs [1]. It would be interesting to investigate whether ReConcile outperforms these methods both theoretically and empirically.

Author response:
> **There is a line of work improving the reasoning ability of LLMs. It would be interesting to investigate whether ReConcile outperforms these methods both theoretically and empirically.**

Thanks for the reference! We note that a direct comparison between WizardMath and ReConcile is not relevant in the context of our work because the former is a single model while the latter is a generic reasoning framework with multiple models. Being a multi-model framework, ReConcile allows the flexibility of adding any model as an agent, including WizardMath.

Reviewer comment (Evaluation):
How to calculate the performance of ChatGPT? The performance reported in Table 2 seems lower than previous work [2].

Author response:
> **How to calculate the performance of ChatGPT? The performance reported in Table 2 seems lower than previous work.**

We compute ChatGPT’s performance with zero-shot Chain-of-Thought prompting (see Sec 5.1 first bullet). Thanks for the reference although it does not seem to report ChatGPT’s performance but does so for text-davinci-002, which is an older version of the GPT series. In general, the performance of API-based models can always vary a bit because of their non-deterministic nature and also because different papers experiment with varying numbers of samples due to the computation cost [3, 4, 5]. In our paper, we make sure that our experiments are fair by (1) following the experimental setup of past work (Multi-agent Debate [3]), (2) reporting variance in accuracy for at least three separate runs, and (3) using a similar configuration (same rounds and same number of agents) of ChatGPT in baselines and ReConcile for fair comparisons.

Reviewer comment (Experiments):
It would be useful to provide individual ablations for demonstrations of answer-rectifying human explanations on each of the models and more qualitative examples of the same to disentangle its effect from diverse model responses.

Author response:
Thanks for the suggestion! We did not do this experiment specifically because "convincing or answer-rectifying human explanations" are used to prompt models to generate explanations that can convince _other_ agents. Hence, when there is only a single model, the role of convincing samples of other agents is unclear. That said, we experimented with including ChatGPT's _own_ convincing samples for its own predictions and observe moderate improvement in accuracy (from 67.3 to 69.0).  
Thanks for the suggestion! Please check Table 11 in the updated version of our paper which shows some examples of generated model reasoning with and without convincing samples. Our analysis suggests that in the presence of convincing samples, agents tend to become more confident of their reasoning and generate explanations that indeed sound more "_convincing_" to other agents. For example, the model starts its reasoning with factual statements like "_Mount Fuji is the highest mountain in Japan_" and avoids phrases like "_It is possible that …_", which may suggest uncertainty.

Reviewer comment (Experiments):
There are many moving pieces in the proposed method, especially the convincing samples and team answer generation heuristics with confidence estimation. In particular, the method requires choosing samples "from the training set for which the agent's initial answer is wrong" which brings a very strong bias compared to the baseline methods. Although there are ablation studies, it is still not very convincing what "complementary benefits" are provided across different LLMs, and how exactly the multi-agent method is fundamentally better than previous debate or self-consistency. This is more concerning because in Table 3, it seems that although Weighted Vote saturates with 3 rounds, majority vote results keep improving. Furthermore, the proposed confidence rescaling heuristics seem brittle and it is not clear what the impact of this is from tasks to tasks.

Author response:
> **Choosing samples "from the training set for which the agent's initial answer is wrong" brings a very strong bias compared to the baseline methods.**

We want to highlight a few results here that will show that this is not the case.

- First, note that ReConcile has many components to it and "choosing convincing samples" is only one of them. Even without these samples, ReConcile outperforms multi-agent baselines. In particular, Table 2 shows that even without access to these convincing samples, ReConcile achieves an accuracy of 74.2% on StrategyQA, substantially higher than single ChatGPT (67.3%) or multi-agent Debate with ChatGPT (66.7%) results. This demonstrates the utility of ReConcile beyond the usage of convincing samples.

- Second, the utility of convincing samples goes beyond the specifics of the ReConcile framework and hence the concept of answer-rectifying explanations, in isolation, should be seen as an important contribution of our paper. Specifically, Table 6 demonstrates that convincing samples provide gains to baselines like multi-agent debate as well, improving the accuracy from 66.7% to 69.5%.

- Third, this argument of bias also holds for papers like Chain-of-Thought prompting, which curate human-written demonstrations of step-by-step reasoning but then compare to "no-explanation" baselines. 

- Fourth, even if one were to consider usage of convincing samples as bias against the baselines, we only leverage 4 convincing samples in our experiments and hence the cost associated with obtaining those is negligible. 

>  **Although there are ablation studies, it is still not very convincing what "complementary benefits" are provided across different LLMs.**

Conceptually, employing agents belonging to diverse LLMs brings in diversity in their reasoning processes (and the corresponding answers). This, in turn, fosters a better discussion, because if all agents are mostly agreeing on the same answer, there wouldn’t be any discussion. Better discussion leads to better consensus and improved accuracy. Please see our common response for more analysis of this complementarity and diversity of multiple models.

> **How exactly the multi-agent method is fundamentally better than previous debate or self-consistency.**

Figure 1 and Table 1 clearly explain the fundamental differences between ReConcile and other baselines. As also explained in Section 4, ReConcile differs from multi-agent debate in three distinct aspects: (1) usage of convincing samples (2) discussion between agents belonging to diverse models, and (3) usage of confidence estimation to quantify model uncertainty. Compared to self-consistency, ReConcile is a discussion framework while self-consistency employs a majority vote between different solutions from the same underlying model. Hence there is no notion of communication of explanations or iteratively improving/refining them in self-consistency. The ablation study presented in Table 4 methodically evaluates all key components by isolating one at a time, and the results show a notable drop in performance when any component is removed.

> **In Table 3, it seems that although Weighted Vote saturates with 3 rounds, majority vote results keep improving.**

Majority vote does not keep improving. We just chose to show results for 3 rounds in the paper but the results below should clarify that both voting mechanisms saturate in 3-4 rounds.

|            | StrategyQA    |             | GSM8K        |             |
|------------|--------------|--------------|--------------|--------------|
| Round      | Majority     | Weighted     | Majority     | Weighted     |
| Round 0    | 74.2         | 74.3         | 79.3         | 79.6         |
| Round 1    | 76.3         | 77.0         | 81.0         | 82.7         |
| Round 2    | 77.1         | 79.0         | 82.7         | 85.0         |
| Round 3    | 78.0         | 78.7         | 83.3         | 84.3         |
| Round 4    | 77.3         | 78.7         | 83.0         | 84.0         |

Reviewer comment (Evaluation):
Why is the margin over baselines much larger for StrategyQA and ECQA (while being almost the same on GSM8K and AQuA)?

Author response:
> **Why is the margin over baselines much larger for StrategyQA and ECQA (while being almost the same on GSM8K and AQuA)?**

We think that this is because in StrategyQA, all individual models are mostly equally performant. Hence, ReConcile is able to better exploit the complementary strengths of these models to obtain bigger gains. On the other hand, in GSM8k, ChatGPT significantly outperforms Bard. Hence, multi-agent debate with ChatGPT also works really well for GSM8k, making the effect size with ReConcile comparatively smaller.

Reviewer comment (Evaluation):
Are the results in Table 2 and Table 3 consistent (on StrategyQA)? Why is the weighted vote score 79.0 in Table 2 and 78.7 in Table 3?

Author response:
> **Are the results in Table 2 and Table 3 consistent (on StrategyQA)?**

Yes, they are! In Table 2, we report the accuracy after round 2, which matches with Table 3.

Reviewer comment (Writing):
Can you clarify how grouping works? What are "distinct categories"?

Author response:
> **Can you clarify how grouping works? What are "distinct categories"?**

The distinct categories are the different answers produced by the different agents. So, in StrategyQA, if two agents have predicted the answer as "yes" and one has predicted a "no", there would be two groups. This enables us to consider the explanations generated by the agents at the abstraction of each group. Hence, during the discussion, we aggregate all agents' answers and explanations as "_there are 2 agents that think the answer is yes (with explanation x and y), and 1 agent that thinks the answer is no (with explanation z)_".

Reviewer comment (Evaluation):
From Table 4, why do you think GPT4 + Bard/Claude2 hurt the performance (of GPT-4 + GPT-4) significantly? i.e., why would the powerful model be dramatically impacted by smaller models?

Author response:
> **From Table 4, why do you think GPT4 + Bard/Claude2 hurt the performance (of GPT-4 + GPT-4) significantly?**

Note that the GPT4+Bard+Claude2 accuracy for ReConcile is actually higher (89.0%) than the multi-agent debate baseline using GPT4 (78.0% accuracy, which has three GPT4 instances). This shows that GPT4 benefits from the diversity of the reasoning generated by somewhat less powerful models like Bard and Claude2.

Reviewer comment (Experiments):
In Section 5.1, how is 3 rounds of Debate with GPT-4 different from RECONCILE with GPT-4?

Author response:
> **How is 3 rounds of Debate with GPT-4 different from ReConcile with GPT-4?**

ReConcile with GPT4 is different from Debate with GPT-4 in the following three aspects: (1) it uses GPT-4, Bard, and Calude2 as the three agents (while Debate uses three instances of GPT-4 as three agents), (2) it makes use of "convincing samples" (there is no such thing in Debate), and (3) it uses uncertainty estimation for answers (again there is no such thing in Debate).