Paper: Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View (ICLR-2024)

Reviewer comment (Evaluation):
The experiment results exhibit significant variance, making it challenging to derive meaningful insights and conclusions. The W-T metric, which the authors use as a complementary measure, also fails to reveal a consistent pattern.

Author response:
Regarding "significant variance", we conducted the ANOVA analysis of significance, and the results shown in Appendix F.3. We also made revisions to some descriptions of experiments in the previous manuscript, highlighted in blue font.

Reviewer comment (Experiments):
The fact that all experiments were conducted on a single model, ChatGPT, further hurts the generalizability of the findings. Moreover, as ChatGPT is a closed proprietary model that silently gets frequent updates, replicating the results will be considerably challenging.

Author response:
Your point about the generalizability of findings due to the sole use of ChatGPT is well-taken. Future work will aim to include a broader range of models. The issue with ChatGPT’s closed nature and frequent updates is indeed a hurdle.  However, we believe that our findings still provide valuable preliminary insights that can inform subsequent studies with similar models. We also provide the code and experimental results in the supplementary materials to facilitate replication. 

Moreover, ChatGPT is frequently updated, a phenomenon referred to as the “replication crisis” in social psychology [1]. Similarly, human individuals also continually change over time. The solution to this issue is to replicate experiments and investigate more human subjects [1]. The conclusions of our paper may also evolve as LLMs advance, but as mentioned in other researches [2][3], once a test setup for machine psychology is established, researchers can investigate the development of LLMs over time by applying the same tasks repeatedly, thus generating longitudinal data. This data can serve as a benchmark to infer trends in the development of LLMs.

Reviewer comment (Experiments):
The societal setup is oversimplified in terms of the number of traits and the size of the society. As a preliminary study, it is a good start. However, this is not clearly acknowledged in the paper.

Author response:
We recognize that the simplification of societal setups and trait numbers was a necessary limitation for this preliminary investigation, and we have included this in the section of Limitations in Appendix E. 

Human societies are vast and diverse, necessitating efficient modes of collaboration. For machine societies, an increase in scale brings about an increase in the length of context, which currently remains a significant challenge. We believe that a complex society functions through specialization, where small groups work collaboratively, much like how a large problem needs to be broken down into smaller problems for resolution. 
*Our current research can only offer insights on how to solve small-scale problems. In the future, we plan to study larger-scale societies, a more diverse range of agents, and explore collaboration between different social groups.*

Reviewer comment (Experiments):
The study involves two identical language models interacting with each other, essentially sharing a common knowledge base. This setup differs from a typical human societal arrangement, and the impact of this factor is not explicitly addressed. For instance, it is unclear what would be the impact of using different language models for the agents.

Author response:
The use of two identical LLMs was indeed a deliberate choice for the initial study phase to control for variables related to the language processing abilities of the agents. Our setup follows the work of [1][2], and our framework is promising to effectively support agents driven by different LLMs. We agree that exploring the impact of heterogeneous LLMs could add depth to our understanding and will highlight this as an avenue for future research. 

There is existing research [3] that has begun to explore the impact of agents based on different LLMs on task performance, but the question of which is more optimal remains to be resolved. 
Here are our personal views: In the long term, using different LLMs seems to be a transitional solution rather than a final one. The premise of using different LLMs is that a single model may not fully grasp all the knowledge of human society, and using different LLMs can effectively compensate for this weakness, as they are trained on different data and methods. However, this assumption should be increasingly addressable with ongoing research. In the short term, collaboration between agents driven by different LLMs is an interesting and worthwhile topic of study.

Reviewer comment (Experiments):
They argue that scaling is not the key and support their claim with intuitive explanations. However, the scaling is limited to 3-4 agents and 2-4 rounds, which makes the observations seem a bit far-fetched.

Author response:
Our claim that scaling is not a key driver was based on our initial observations, which we acknowledge may appear speculative given the limited scope of agent numbers and collaboration rounds. We intend to explore a broader range of configurations in subsequent studies to test the validity of this claim more robustly. 

In the revised manuscript, we added experiments with two agents, as shown in Tables 8 and 9 of the appendix. We found that three agents is a more suitable choice among 2-4 agents, with detailed analysis in Appendix F.3. As for the number of collaboration rounds, we tracked the consistency changes of different collaborative strategies with increasing rounds, as shown in Figure 12 at the appendix. We also have revised our original statement; discussing the number of rounds without considering the collaborative strategy is unconscionable. We believe that the number of rounds is closely related to consistency. If a society achieves strong consistency after a round of collaboration, then adding more debates will not lead to performance improvement. For a detailed analysis, refer to Appendix F.3.

Reviewer comment (Experiments):
How would a potential non-simplified collaboration setup look like and how would this influence the observations made in the paper?

Author response:
A more complex collaboration setup might involve a larger number of agents with a greater diversity of traits and interaction rounds. We hypothesize that such a setup would provide more nuanced insights into the dynamics of agent collaboration and competition. Including more complex societal structures, such as hierarchical or network-based interactions, could also significantly impact the results and the generalizability of our observations.

Previous research [4] outlines several branches of psychological research, such as social psychology, group psychology, moral psychology, and the psychology of judgment and decision-making. Due to constraints in assessment, context length, modality, and datasets, many aspects of machine psychology cannot be immediately studied. Therefore, in the foreseeable future, we anticipate more specific setups in two main dimensions:

(1) *A Richer Society*:
- An environment with feedback. Feedback plays a crucial role in human interaction. Studying machine society based on feedback phenomena and agents' responses is a fascinating research direction.
- As you mentioned, a richer variety of agent personalities, more role-playing, agents driven by different LLMs, larger-scale machine societies, and collaboration between machine societies would be more challenging settings.

(2) *More Detailed Collaborative Strategies*:
- Expanding methods of reflection. Reflection can take various forms, such as convergent thinking, divergent thinking, critical thinking, etc., which can be referred to [5]. Notably, Chain of Thought [6] can also be seen as a form of self-reflection.
- Human-computer interaction is a future trend. It's promising to introduce human-computer collaboration to observe human and agent behaviors.
- Introducing an external third-party agent to determine the appropriate thinking pattern and timing for each participating agent.

Moreover, these research extensions on "Non-Simplified Collaboration" could have potential impacts, such as:
- They might uncover more efficient collaboration techniques that swiftly pinpoint correct answers, diverging from the methods currently discussed in our paper. 
- Furthermore, employing agents powered by various LLMs could result in fascinating behavioral dynamics, such as increased overconfidence during collaboration. It's also possible that reflection doesn’t invariably result in model hallucinations, opening new avenues for exploration.

Reviewer comment (Experiments):
What is your motivation to choose agents backed by the same language model rather than different models? How do you think using different models would influence the current experimental setup?

Author response:
(1) *The motivation behind using agents backed by the same language model*:
- To ensure consistency in language understanding and response generation capabilities across agents. This homogeneity allows us to isolate the effects of the designed societal roles and collaborative strategies. 
- Many studies [1][2] have also adopted settings of using the same language model, and we just follow the commonly used settings. 
- We lack access to APIs of other LLMs like Bard, Claude, and lack enough computing resources to conduct all the experiments on open-source LLMs at the time of writing, thus we have no choice but to use the same language model tentatively. 

(2) *The influence of using different models on the current experimental setup*: 
Employing different models could introduce variability in basic language processing capabilities, which would be an interesting dimension to explore in understanding how model-specific characteristics influence collaborative outcomes. Different models use varied training methods and data, so they differ in the breadth and depth of knowledge, and in their ability to follow instructions.
- Regarding the experimental setup, our framework still supports this, and it can be easily implemented with a few lines of code.
- Regarding the experimental results, ideally, agents might show overconfident behavior in collaboration, leading to absolute obedience between agents. Additionally, agents could complement each other through collaboration to arrive at more accurate answers. We look forward to the results of such experiments.

Reviewer comment (Novelty):
If the agent composition of society doesn’t have a marked difference then what is the utility of having such composition?

Author response:
The composition of agents with different 'traits' (easy-going or overconfident) in the society, even without marked differences, provides a nuanced understanding of social dynamics, with Sherif’s Classic Autokinetic Effect Study[1] being a representative piece of research. It serves to model real-world scenarios where subtle variations in behavior or capability can significantly impact collective outcomes. For example, as can be seen from Table 2, the performance exhibited by different societies is not always exactly consistent and still present differences. 
Our design of society’s agent composition initially did not account for the lack of significant differences among different societies. However, this phenomenon became apparent after our simulation experiments. 
Besides, Reviewer wLih thinks the assertion that collaborative strategies overshadow the influence of agent composition may be premature.  

Moreover, our supplementary experiments in the Appendices (as illustrated in Figure 8, where we conducted a word frequency analysis on two societies with the most distinct compositions, S1 and S4) reveal that all societies tend to harmonize. This suggests a propensity of agents to exhibit easy-going rather than overconfident traits. 
We speculate that this could be due to issues with the prompt or the model (similar to what you mentioned in Q4), involving two potential scenarios:  
- (1) The model itself is not at fault; rather, the ineffectiveness of the prompt is the issue. 
- (2) The prompt is adequate, but the model does not adhere to it. 
We lean towards the scenario (2) as the more probable cause, and you can check our response to Q4 for more details, where we found that the prompt is effective enough. 

Furthermore, we relate this phenomenon to the alignment of LLMs, which discourages the adoption of traits like overconfidence that conflict with human preferences. This indicates a need for LLMs to strike a balance between "following instructions" and "maintaining non-toxicity", instead of only trying to mirror human preferences.

Reviewer comment (Evaluation):
On MMLU S1 is doing better on min for example but not on the avg. How can we be confident that the differences are attributed to not by chance and are truly significant?

Author response:
"on MMLU S1" you mentioned is just one case and cannot adapt to all of the scenarios. To ensure the differences in performance are statistically significant and not by chance, we conduct additional ANOVA analysis, to provide a more rigorous significance test for these comparisons in the main experiment of section 3.1. These findings have been included in Table 4, Appendix F.1 of the revised manuscript. 

A notable observation is that the p-value associated with the "collaborative strategy" is significantly below the 0.05 threshold, indicating its substantial impact. In contrast, the p-value of the other two factors, "society" and "collaborative strategy and society", is obviously greater than 0.05. This corroborates our earlier conclusion in section 3.1, emphasizing that **the influence of collaborative strategy outweighs that of society, and performance and collaborative strategy are significantly correlated.**

Reviewer comment (Evaluation):
Starting with P0 helps but for best performance you need to add at least 1 P1 at end. Even with same two P0 and one P1. Why might that be the case?

Author response:
Based on the results in Table 2, it's evident that the collaborative strategy $p_0p_0p_1$ is not always superior to $p_0p_0p_0$, but there are some collaborative strategies that achieve performance close to $p_0p_0p_0$. On MATH S3, $p_0p_0p_1$ actually performs worse. Only for MMLU dataset, $p_0p_0p_1$ tends to be the optimal choice, as shown in Table 2 and Figure 4(d). We observe that in comparison to $p_0p_0p_0$, $p_0p_0p_1$ increases the proportion of `correcting mistakes` while maintaining the rate of `changing correct answers`. Additionally, Figure 12 illustrates that $p_0p_0p_1$ achieves a consistency of 2.8 in the first two rounds. At this point, the magnitude of changes in answers triggered by one reflection ($p_1$) is relatively minor (as shown in Figure 4(d)). This scenario can be akin to trials in human society (assuming the stability of human society), where new trials invariably lead to fresh discoveries and innovations, thereby advancing human science and technology. 
The analysis in our response to Q7 also helps to understand this issue. 

\\

References

[1] https://opentext.wsu.edu/social-psychology/chapter/module-7-social-influence/

Reviewer comment (Reproducibility):
How are the agents initialized and conditioned? Is the prompt indicating how agents should behave enough? What specific LLM settings were used and how does it personify how agents are implemented for this study (are overconfident agents default to 0 shot prompting and easy going multi shot?)

Author response:
We enhanced our experiments to validate the effectiveness of the Prompt and Model, maintaining alignment with the main experiments detailed in this paper. 
Detailed experimental setup is available in Appendix C: for the prompt, refer to Table 3 in the Appendix; and for complete dialogue examples, refer to Figures 9 and 10 in the Appendix. The experiments used the gpt-3.5-turbo model, with both overconfident and easy-going agents using 0-shot prompting. 

**Effectiveness of the Prompt**: Analysis of the word cloud in Figure 8 suggests that the prompt designed for easy-going traits is successful. Consequently, our attention turns to confirming the effectiveness of the prompt for overconfident traits. In the absence of a robust validation approach, we draw upon our existing experimental results and practical experience to assess the prompt's effectiveness from four key angles: 

- *Behavior Explanation*: We identified two behaviors, "confident in your answer" and "persuades other agents to believe in you", which align with the behavioral display of overconfidence. 
- *Refusal to Accept*: We employ a role-play method to prompt the model, and then ask if the model comprehends the prompt (if a prompt induces harmful content output, the model will refuse it). When checking our logs, the model didn't refuse our prompts and replied affirmatively with 'OK'.
- *Re-asking*: Building on the previous step, we posed another query: "What should you do if someone else's answer differs from yours?". The model's reply, "I will convince them", proves the effectiveness of our prompt tailored for overconfident traits.
- *Example Analysis*: Delving deeper, we contextualize the "re-asking" with a practical scenario. Even if the model's own response is wrong and our prompted answer is correct, the model still insists on its viewpoint. This observation further substantiates the effectiveness of our prompt tailored for overconfident traits. 

**In summary, these experiments collectively affirm the effectiveness of the prompts we have developed for both easy-going and overconfident traits.**

Reviewer comment (Experiments):
What were the task details for subject of this study? If we just have one agent (either of two types) how many prompts (in case of reflection) does it take for it to get to the answer? Are the tasks based on knowledge (like valid chess move pieces) that may be available to the agents as part of their training data? If yes, why will it change on reflection? And what is the source of this change (is there a query in background that fills in the context)?

Author response:
(1) For *details regarding the experimental setup and task specifics*, please refer to our response to Q4, with extensive information located in Appendix C. 

(2) Regarding *the number of prompts required for the agent to reach an answer during reflection*, there is no definitive count. The nature of reflection, devoid of direct feedback and influenced by human biases, results in the agent's answers evolving throughout the reflection process.

(3) On the matter of *whether the test data was exposed during the training phase*, this topic has been extensively debated. For further insights, refer to [2]. The training of GPT3.5 model involves three distinct phases: pre-training, Supervised Fine-Tuning (SFT), and Alignment, detailed at https://openai.com/blog/chatgpt. It is possible that certain stages of training could lead to some information being forgotten. In a hypothetical extreme scenario where test data is present at all three stages, the likelihood of answer variation for a single-turn dialogue in response to a query is low, typically yielding a high accuracy rate. However, by introducing a reflection prompt, which creates a multi-turn dialogue, the experimental findings indicate that the reflection process induces notable alterations in the responses. 

(4) The *main hypothesized causes for these changes during reflection* are twofold: (i) the absence of constructive feedback during self-reflection, and (ii) a propensity for altering initial responses during the alignment phase when prompted to reflect. As illustrated in Figure 12, there is a marked reduction in answer consistency on the Chess dataset following reflection. This is further evidenced by the prevalent use of the word 'apology' in the word cloud depicted in Figure 8. In essence, when prompted to reflect, the model tends to reassess its own responses for potential errors, a tendency that becomes more evident when the model encounters unfamiliar question types.

Reviewer comment (Evaluation):
It seems there are differences between all three tasks and not just Math vs MMLU and Chess move validity.

Author response:
Sure, the three datasets exhibit substantial differences. MMLU is characterized by knowledge-based, discrimination, and straightforward questions. The MATH dataset comprises reasoning, generative, and challenging problems. Meanwhile, Chess dataset features spatial imagination, generative, and complex problems, including those with multiple potential answers. As for *the differences across the three datasets*, we compare MATH vs MMLU and Chess move validity, not one vs another one between all three tasks, due to space limits.

Reviewer comment (Evaluation):
Behavior in section 3.2 is contradictory. It would be good to have a comprehensive analysis of how many times collaboration helped arrive at the correct answer across all tasks. The qualitative explanation can go only so far. Also, a key point needing details is around the factual information (where the information may be a fact that can be recalled/learned from training data) vs the problem where the answer may not be factual but available as a set of rules with multiple possibilities.

Author response:
(1) The behavior described in section 3.2 does indeed present a contradiction, mirroring a similar phenomenon in human society. 

(2) In terms of *collaboration times*, as elaborated in section 3.2 and illustrated in Figures 4(a-c), multiple collaboration proves to be counterproductive for MMLU tasks, as accuracy drops after 3-round collaboration. However, for more complex tasks, such as MATH, 3-round collaboration can be advantageous. We also conducted additional experiments to thoroughly analyze the evolution of conformity phenomenon across varying collaboration times, and this analysis is detailed in Figure 11 of the Appendix. For this analysis, we focused specifically on the turn of debate ($p_0$), where conformity is more prevalent, and compiled statistics based on individual participants. 
Firstly, a broad view reveals that the prevalence of conformity is over 50%, suggesting its widespread nature. Secondly, we classified the shifts in the correctness of answers, pre- and post-conformity, into four categories: True->False, False->True, True->True, and False->False. The results from Figure 11 reveals that the proportion of correct answers post-conformity outweighs that of incorrect answers, indicating the potential benefits of conformity. However, an interesting trend emerges with an increasing number of turns/rounds: the proportion of True->False answers rises, while the False->True ratio remains constant. Moreover, the increment in True->True answers is less than that in False->False, suggesting a diminishing return with an excess of interaction turns/rounds. 

(3) For *factual information can be recalled/learn from training data vs the problem where answer may not be factual but available as a set of rules*, we guess you mean how to achieve the final answers if there are multiple candidate answers that may contain factual information and rules of induction. An explanation for this can be found in Appendix C.2. Taking the Chess dataset as an example, which permits multiple correct answers, we initially evaluate the accuracy of each individual’s response. Subsequently, a majority vote based on these accuracy evaluations is conducted. For instance, if the correct solutions to a problem are [e1, e3], and the answers provided by three agents at a certain moment are [e1, e3, b1], we classify these as [Correct, Correct, Wrong], and then decide the overall correctness of the discussion. In the case of the MATH dataset, assume the correct answer is 1/4; however, answers such as 2/8 or 0.25 are also deemed correct. This judgment is reached through a combination of manual review and rule-based matching.

Reviewer comment (Experiments):
Section 4.1 "We utilize the majority vote (Li et al., 2022; Cobbe et al., 2021) method to determine the answer for each round.": How much of this is if you do multishot prompting on an LLM and use an ensemble of LLMs? Is there a society aspect to this? Or is it just that the ensemble of LLMs gives a better answer?

Author response:
We utilized a zero-shot setting for our experiment. To determine the answer for a specific round, we relied on a majority vote, taking into account the responses from all agents within a society for that round. This approach essentially harnesses the collective wisdom of LLMs to yield better answers.

Moreover, there are various methods to decide on an answer. It could either be determined by one of the participants or an external observer. We opted for the latter, choosing majority vote for two key reasons: firstly, this method is widely used in numerous studies [3]; secondly, graph theory [4] suggests that when over two-thirds of the participants concur on an answer, it often leads to a more optimal result, a concept also prevalent in most electoral systems. 

Your idea of tailoring the answer determination method to the specific composition of a society is intriguing. Our present experiments indicate no substantial differences between different societies, thereby justifying the use of majority vote. Should notable variances between societies arise in the future, we might consider adapting our decision-making strategy to reflect these societal compositions. For example, increasing the weight of answers from overconfident agents could be one such adaptation.

Reviewer comment (Evaluation):
Section 4.1 "Wavering Answers resemble model hallucination due to the occurrence of self-contradictory answers.": How can we be sure this is from hallucination and not the model changing its output to what the user prefers from a set of possibilities?

Author response:
*"How can we be sure this is from hallucination?"*: In Section 4, we define TFTF, FTFT, TFFT, and FTTF as "Wavering Answers", as they represent answers that continually oscillate (taking MMLU as an example, the answer may be A at one moment, then B at another, and possibly C later). This kind of conflict is considered a manifestation of model hallucination [5]. 
- *"Not model changing it’s output to what user prefers from set of possibilities?"*: Our tasks don’t fall under creative endeavors such as novel writing. Rather, it is characterized by definitive answers with a limited set of possible outcomes. In our evaluation process, the emphasis is placed on the accuracy of the answers, not on the composition of the sentences. Given that the answers are objective in nature, they do not accommodate subjective human preferences.

Reviewer comment (Theory):
"We group samples from different societies under the same strategy because the effect of society is minimal": What is the scientific evidence to prove that this is indeed the case?

Author response:
We conducted a two-way ANOVA analysis of significance test (before that we ensure that it conformed to normal distribution and variance homogeneity), and the results are shown in Table 4 at Appendix. We found that only the collaborative strategy had a significant impact, while the influence of society was not significant.

Reviewer comment (Evaluation):
Collaborative strategies play a significant role in performance: I don’t see a significance test to lay this claim.

Author response:
It seems there was a misunderstanding due to previous statements. The conclusion refers to the observation that, for 'answers wavering', the proportion of continuous reflection $p_1$ (which implies more severe model hallucination) is the highest. 
- On MMLU dataset, the top three proportions are $p_1p_1p_1$、$p_1p_0p_1$、$p_1p_1p_0$; 
- For MATH dataset, the top three are $p_1p_1p_1$, $p_1p_1p_0$, $p_0p_1p_1$; 
- On Chess dataset, the leading three are $p1p_0p_1$, $p_1p_1p_1$, $p_0p_1p_1$.

Reviewer comment (Evaluation):
Section 4.1 Conclusion 2 — For continuous reflection strategies, the proportion of “Wavering Answers” occurrences is the highest among all strategies as seen: Doesn’t seem to be the case from figure 4 d–f.

Author response:
We've made some clarifications to address potential ambiguities in the original text. Debates are shown to effectively reduce model hallucinations. When examining the strategies with the lowest rates of wavering answers, on MMLU, MATH, and Chess datasets, the strategies with the lowest are respectively $p_0p_0p_0$, $p_1p_0p_0$, $p_0p_0p_0$. Notably, for the MATH dataset, the difference between $p_0p_0p_0$ and $p_1p_0p_0$ is minimal. Furthermore, in terms of consistency, as depicted in Figure 12 at Appendix, on all three datasets, the consistencies of $p_0p_0p_0$ and $p_1p_1p_1$ represent two contrasting extremes.

Reviewer comment (Evaluation):
Section 4.1 Conclusion 2 — The strategy of “Pure Debate” (i.e., P0 P0 P0) can effectively reduce this fluctuation (hallucination): Doesn’t seem to be exclusive to pure debate from figure 4 d–f.

Author response:
In the revised version, we have changed the original description to "debates can effectively reduce hallucinations." 
As for "hallucination lower for $P_0P_1P_1$ or $P_0P_1P_0$", we guess you mean that based on Figure 4(d-f), it is observed that for different collaborative strategies, $p_0p_1p_1$ and $p_0p_1p_0$ tend to have lower hallucinations on MMLU dataset. This conclusion should be evaluated comparatively. In the case of these two strategies (both starting with $p_0p_1$ interactions in the first two rounds), they align with the observed trend. 
- The $p_0p_1p_0$ strategy, not involving continuous reflection, exhibits a lesser proportion of hallucinations compared to $p_0p_1p_1$. 
- Conversely, the $p_0p_1p_1$ strategy, which includes continuous reflection, shows a higher proportion of hallucinations compared to $p_0p_1p_0$. This pattern is consistently observed across all three datasets.

Reviewer comment (Evaluation):
Section 4.1 Conclusion 2: If that is true why is hallucination lower for P0P1P1 or P0P1P0?

Author response:
Regarding *the Number of Agents*, we conducted a one-way ANOVA analysis of significance, as shown in Table 5 at Appendix. We found that the increase in the number of agents does not present a significant impact.

Reviewer comment (Evaluation):
Section 4.2 Number of agents: Doesn’t seem to be always the case on both accounts.

Author response:
Regarding "Different Rounds," we employed a one-way ANOVA analysis for testing, as detailed in Table 6 at Appendix. From the combined insights of Figure 6 and Figure 12, we noted that strategies with lower consistency and greater variability are more sensitive to the number of rounds. This is especially true for the continuous reflection strategy on Chess dataset, where continuous reflection results in a gradual decrease in consistency, meaning lesser fluctuations and more uniformity, thus showing less sensitivity to the number of rounds. Strategies with the p-value greater than 0.05 generally exhibit higher consistency. However, increasing the number of rounds does not always equate to improved outcomes. For strategies that already have high consistency, adding more rounds can actually decrease performance, as illustrated in Figure 11 in Appendix. This is also discussed in our response to Q7.

Reviewer comment (Evaluation):
Section 4.2 More rounds: Doesn’t seem to be always the case. The pattern seems inconsistent as is the definition of good or bad. Also, the improvement needs statistical significance.

Author response:
Regarding "Other Collaborative Strategies," we applied a one-way ANOVA analysis, as indicated in Table 7 at Appendix. In this context, 'performance' pertains to accuracy, with higher accuracy denoting better performance. Our findings reveal that collaborative strategies significantly affect Chess dataset. In contrast, their impact on the MMLU dataset is less pronounced. This is primarily because collaboration doesn't substantially benefit the MMLU dataset, as shown in Figure 4(a), and as further elaborated in our response to Q7.