Paper: MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (ICLR-2024)

Reviewer comment (Experiments):
Most of the experiment are on GPT4, which is expensive to access, how is the performance on the benchmarks or real development demands when used with open-source LLMs? Can you share some insight on which abilities of the LLMs matters most for the success of using this multi-agent framework and how to choose proper LLMs for use?

Author response:
**Q1.  Can you share some insight on which abilities of the LLMs matter most for the success of using this multi-agent framework and how to choose proper LLMs for use?**

In our experience, coding tasks need GPT-4, while GPT-3.5 suffices for other tasks without obvious performance impact. We recommend using models fine-tuned for specific skills in future developments. For example, a model specialized in code generation can be an engineer's LLM, optimizing cost and role performance.

Reviewer comment (Experiments):
Does the framework have strong generalizability when switched to GPT 3.5 or other open-source models? how far can the framework or practical use go with open-source models?

Author response:
**Q2. Most of the experiments are on GPT4, which is expensive to access, how is the performance on the benchmarks or real development demands when used with open-source LLMs?  Does the framework have strong generalizability when switched to GPT 3.5 or other open-source models? how far can the framework or practical use go with open-source models?**

Following your suggestions, we evaluated the HumanEval & MBPP datasets and a few SoftwareDev tasks to assess using GPT-3.5 and Deepseek Coder 33B as backends. The results indicate that although MetaGPT can complete tasks with these LLMs, using GPT-4 as the backend yields superior performance.


|Model|HumanEvel|MBPP|
|:--------------------|:----------------|:----------------|
|GPT-3.5|$48.1$|$52.2$|
|MetaGPT (w/ GPT-3.5)| $62.8 (+14.7)$|$74.7 (+21.5)$|
|GPT-4| $67.0$|-|
|MetaGPT (w/ GPT-4)|$85.9 (+18.4)$|$87.7$|


| Model| Open source|Running time(s)|Total Code Lines|Executability |Human Revision Cost(times)|
|---|---|:---:|:---:|:---:|:---:|
|MetaGPT (w/ GPT-3.5)|✗|$75.18$|$161.6$|$2.8$|$2.4$|
|MetaGPT (w/ GPT-4)|✗|$552.94$|$178.2$|$3.8$|$1.2$|
|MetaGPT (w/ Deepseek Coder 33B)|✓|$1186.20$|$120.2$|$1.4$|$2.6$|

Reviewer comment (Evaluation):
In table 3, In this multi-agent collaboration with different roles, do you have some qualitive analysis or case study to show where each role contributes to the final performance, except for an executability score? Can you provide more detailed analysis?

Author response:
**Q3: In Table 3, In this multi-agent collaboration with different roles, do you have some qualitative analysis or case study to show where each role contributes to the final performance, except for an executability score? Can you provide more detailed analysis?**

Thank you for your comment. To illustrate, we'll use the task **'write a python3 GUI app which allows drawing images on it'** as an example (the intermediate processes involving full team members are detailed in Appendix B). Here, we provide a qualitative analysis of each role's contribution to the final performance.

Here is the summary of the impact of removing certain agent roles:

- **Without Architect:**
  - An absence of detailed system design leads to structural and functional issues in coding, which affects the robustness of final product.
  - Task decomposition lacks sufficient granularity.
- **Without ProjectManager:**
  - Missing module dependencies result in incomplete functionality and additional bugs.
- **Without ProductManager:**
  - Lack of function design and plan results in incomplete design and missing functionality, directly impacting the total number of files and generated code.

Reviewer comment (Experiments):
Whether the framework can also solve problems other than code, such as math and QA in Autogen? Is there other challenges for these kinds of problems for the framework?

Author response:
**Q4. Whether the framework can also solve problems other than code, such as math and QA in Autogen? Is there other challenges for these kinds of problems for the framework?**

MetaGPT can address a variety of domain issues with minor implementation changes. While our current implementation (of submission) may not be ideal for specific tasks like Math and QA, we can quickly create specialized agents for these areas. Benefiting from MetaGPT's cost-effectiveness and adaptability in task modeling, we've successfully used it to replicate applications, including games like Minecraft [1] and Werewolf [2], and in roles like research assistants.

[1] Wang G, Xie Y, Jiang Y, et al. "Voyager: An open-ended embodied agent with large language models". arXiv:2305.16291, 2023.

[2] Xu Y, Wang S, Li P, et al. "Exploring large language models for communication games: An empirical study on werewolf". arXiv:2309.04658, 2023.

Reviewer comment (Novelty):
Methodological innovation: A key issue with MetaGPT is the lack of innovation in its methodology. While it effectively utilizes Large Language Models (LLMs) in multi-agent systems, this approach may not be significantly different from existing approaches. Or MetaGPT is different from other methods in a trivial way, I don't really see their differences as being significant and requiring exploration with greater depth.

Author response:
**Q1 (original W1). Methodological innovation: How does MetaGPT differ from other methods?**

> 1) **Agents in SOPs**: MetaGPT pioneers the introduction of SOPs (Standard Operating Procedures), a departure from works like CAMEL [2], which don't discuss task-specific labor division. Unlike generative agents [1] that simulate open-world human behavior freely, MetaGPT focuses more on applying valuable human practice and better organizing multiple agents.  
> 2) **Structured communication v.s. Chat-based communication**: Sec 3.2 highlights how MetaGPT, unlike other chat-based communication models such as ChatDev [3], Self-collaboration [4], and CAMEL [2], introduces structured communication. MetaGPT limits the range of message content, focusing agents on relevant information and avoiding meaningless conversational chat. Consequently, this approach reduces token usage and enhances task success rates.

[1] Park, Joon Sung, et al. "Generative agents: Interactive simulacra of human behavior." In UIST, 2023.

[2] Li, Guohao, et al. "CAMEL: Communicative Agents for" Mind" Exploration of Large Language Model Society." In NeurIPS, 2023.

[3] Qian, Chen, et al. "Communicative agents for software development." arXiv:2307.07924, 2023.

[4] Dong Y, Jiang X, Jin Z, et al. "Self-collaboration Code Generation via ChatGPT." arXiv:2304.07590, 2023.

Reviewer comment (Experiments):
Fairness of experimental comparisons: The comparison methods used to assess the effectiveness of MetaGPT do not appear to be fully equivalent to what MetaGPT offers. Such differences may lead to biased or misleading conclusions about their superiority or efficiency.

Author response:
**Q3 (Original W2, Q5, Q6). The comparison methods used to assess the effectiveness of MetaGPT do not appear to be fully equivalent to what MetaGPT offers. **(1)** "We compare recent domain-specific LLMs in the code generation field ..."; **(2)** "We modified certain role-based prompts in order to instruct the MetaGPT framework to generate individual functions instead of entire classes. Is it fair to compare with these frameworks? LangChain and AutoGPT? How can we ensure that prompt modification does not lead to unfair comparisons?"**

Thanks for pointing these out, but these are misunderstandings.
**(1) Both the baselines and MetaGPT are conducted under the same settings.** We acknowledge that comparing MetaGPT with single LLMs may seem partially unfair. However, our goal is to help readers quickly grasp the advancements in this field, where both LLMs (PaLM Code vs. GPT-4) and methodologies (Codex vs. Codex+CodeT) are essential. The most important point here is "When MetaGPT collaborates with GPT-4, it substantially improves the Pass@1 in the HumanEval benchmark, exceeding the performance of GPT-4 alone" as stated in the manuscript.
**(2) In the SoftwareDev dataset, we maintain fully equivalent settings for a fair comparison with AutoGPT, LangChain, and ChatDev.** In HumanEval and MBPP, we've slightly modified the prompts to align with response format requirements. These modifications aim to address format-specific issues (i.e., Python problems). To enhance clarity in our manuscript, we propose adding 'In HumanEval and MBPP' at the beginning of Sec 4.1.

Reviewer comment (Novelty):
My primary concern is that MetaGPT lacks methodological and theoretical innovation. To gain a more innovative edge, MetaGPT could deeply explore what deep-seated challenges in integrating newer AI techniques or unique collaboration strategies make it more clearly distinguishable from other LLM-based frameworks. (Honestly, I think MetaGPT is more suited for system demonstration conferences or tracks)

Author response:
**Q2 (Original Q1). Deep-seated challenges.**

MetaGPT also alleviates or solves these challenges with its unique designs:  
1) **Use Context Efficiently**: Two sub-challenges are present. First, unfolding short natural language descriptions accurately to eliminate ambiguity. Second, maintaining information validity in lengthy contexts, enables LLMs to concentrate on relevant data without distraction.
2) **Reduce Hallucinations**:  Using LLMs to generate entire software programs faces code hallucination problems—-including incomplete implementation of functions, missing dependencies, and potential undiscovered bugs, which may be more serious than hallucinations in natural language generation. Deep-seated challenges in task decomposition and translating from natural language to code language persist. LLMs often struggle with software generation due to vague task definitions. Focusing on granular tasks like requirement analysis and package selection offers guided thinking, which LLMs lack in broad task solving.

Reviewer comment (Writing):
In the Related Work, "In this paper, we identify the main challenges in multi-agent cooperation: maintaining consistency, avoiding unproductive cycles, and controlling beneficial interactions". There seems to be inconsistency in the description of the research challenges between the "Related Work", the "Introduction" and the rest of the paper. Furthermore, is it trivial to maintain consistency in MetaGPT? What are the professional definitions of "unproductive cycle" and "beneficial"? Are there experimental results to validate these advantages?

Author response:
**Q6 (Original Q4). Is it trivial to maintain consistency in MetaGPT? What are the professional definitions of "unproductive cycle" and "beneficial"? Are there experimental results to validate these advantages?**

* 1) **Consistency:** Maintaining consistency is not trivial and presents a challenge in many applications, as highlighted in work [1,2,3]. MetaGPT addresses this by breaking down tasks into granular sub-tasks, enabling each agent to gain a deeper understanding of their specific responsibilities. This approach is crucial, particularly in function design and coding requirements for downstream agents (e.g., Engineer), to prevent misunderstandings or error propagation during collaboration using LLMs.  
* 2) **Unproductive cycles:** In work [2], it's noted that two cooperating agents often face challenges like the 'Assistant Repeats Instruction' problem or the 'Infinite Loop of Messages'. These issues emerge from repetitive, context-less conversations leading to unproductive exchanges, such as repeatedly saying "Goodbye". Reference [4] discusses an issue where agents become stuck in a loop while addressing complex or ambiguous tasks, chaining thoughts in a behavior-like pattern. We can improve the quality of code generation by reducing unproductive cycles through SOPs, as described in Table 1.  
* 3) **Beneficial interactions:** Previous work [5] highlights the requirements engineering in software development, particularly abstraction and decomposition, significantly improving code generation. Constraints also improve LLMs' understanding of fuzzy, abstract, and multitasking semantics [6]. Thus, we introduced a structural communication protocol to assist agents in task decomposition and semantic understanding, aiming to minimize ambiguities and errors, fostering beneficial interactions.

When reproduced to other frameworks, we found that these issues occur frequently. The SOPs and communication mechanisms in MetaGPT can alleviate these issues, thus achieving better performances (see Table 1, Table 2, Table 4).

[1] Elazar Y, Kassner N, Ravfogel S, et al. "Measuring and improving consistency in pretrained language models". TACL, 2021.

[2] Li, Guohao, et al. "CAMEL: Communicative Agents for" Mind" Exploration of Large Language Model Society." In NeurIPS, 2023.

[3] Wang X, Wei J, Schuurmans D, et al. "Self-consistency improves chain of thought reasoning in language models". arXiv:2203.11171, 2022.

[4] Talebirad Y, Nadiri A. "Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents". arXiv:2306.03314, 2023.

[5] Jiang X, Dong Y, Wang L, et al. "Self-planning code generation with large language model". arXiv:2303.06689, 2023.

[6] Shin R, Lin C H, Thomson S, et al. "Constrained language models yield few-shot semantic parsers". arXiv:2104.08768, 2021.

Reviewer comment (Experiments):
In the Experiments, "We modified certain role-based prompts in order to instruct the MetaGPT framework to generate individual functions instead of entire classes" 1) Is it fair to compare with these frameworks? LangChain and AutoGPT, for example, which are not specific to the tasks involved in this commit. 2) How can we ensure that "prompt modification" does not lead to unfair comparisons?

Author response:
**Q3 (Original W2, Q5, Q6). The comparison methods used to assess the effectiveness of MetaGPT do not appear to be fully equivalent to what MetaGPT offers. **(1)** "We compare recent domain-specific LLMs in the code generation field ..."; **(2)** "We modified certain role-based prompts in order to instruct the MetaGPT framework to generate individual functions instead of entire classes. Is it fair to compare with these frameworks? LangChain and AutoGPT? How can we ensure that prompt modification does not lead to unfair comparisons?"**

Thanks for pointing these out, but these are misunderstandings.
**(1) Both the baselines and MetaGPT are conducted under the same settings.** We acknowledge that comparing MetaGPT with single LLMs may seem partially unfair. However, our goal is to help readers quickly grasp the advancements in this field, where both LLMs (PaLM Code vs. GPT-4) and methodologies (Codex vs. Codex+CodeT) are essential. The most important point here is "When MetaGPT collaborates with GPT-4, it substantially improves the Pass@1 in the HumanEval benchmark, exceeding the performance of GPT-4 alone" as stated in the manuscript.
**(2) In the SoftwareDev dataset, we maintain fully equivalent settings for a fair comparison with AutoGPT, LangChain, and ChatDev.** In HumanEval and MBPP, we've slightly modified the prompts to align with response format requirements. These modifications aim to address format-specific issues (i.e., Python problems). To enhance clarity in our manuscript, we propose adding 'In HumanEval and MBPP' at the beginning of Sec 4.1.

Reviewer comment (Evaluation):
Figure 4. What are the results of multiple attempts and are they stable? Were there any problems, like robustness?

Author response:
**Q8 (Original Q7): Figure 4. What are the results of multiple attempts and are they stable? Were there any problems, like robustness?**

1) **According to this comment, we conduct GPT-4 and GPT-3.5 experiments on HumanEval and MBPP.** For GPT-4, upon reviewing the results, we observe that MetaGPT has a lower standard deviation of 0.498% for HumanEval, while MBPP experiences a higher standard deviation of 0.862%. Most variation in MBPP results from inaccurate test case generation due to a lack of input examples, resulting in passing tests but failing actual results. MetaGPT can produce robust results for these benchmarks.

2) **Compared to GPT-4, GPT-3.5-turbo has a higher standard deviation (more unstable)** in both HumanEval and MBPP.

|||GPT-4||||GPT-3.5|||
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
||HumanEval|| MBPP-sanitized ||HumanEval|| MBPP-sanitized||
||Attempts|Pass@$1$(%)|Attempts|Pass@$1$(%)|Attempts|Pass@$1$(%)|Attempts|$Pass@$1$(%)|
||$1$|$86.0$|$1$|$87.6$|$1$|$61.6$|$1$|$75.9$|
||$2$|$84.8$|$2$|$86.7$|$2$|$64.0$|$2$|$74.5$|
||$3$|$85.4$|$3$|$85.5$|$3$|$62.8$|$3$|$73.8$|
||$\textbf{avg}$(%)|$85.4$|$\textbf{avg}$(%)|$86.6$|$\textbf{avg}$(%)|$62.8$|$\textbf{avg}$(%)|$74.7$|
||$\textbf{std}$(%)|$0.490$|$\textbf{std}$(%)|$0.860$|$\textbf{std}$(%)|$0.980$|$\textbf{std}$(%)|$0.873$|