Paper: AutoAgents: A Framework for Automatic Agent Generation (ICLR-2024)

Reviewer comment (Reproducibility):
The paper is perceived as having low readability and insufficient reproducibility. The reviewer requests a more granular description of the methodology that enables readers to implement the procedures step by step.

Author response:
Thank you for your motivating review and concrete suggestions. Detailed responses to your questions are listed as follows.

> 1. The paper is perceived as having low readability and insufficient reproducibility. The reviewer kindly requests a more granular description of the methodology that enables readers to implement the procedures step by step. For instance, while Table 1 is highly beneficial for positioning this research within the LLM-based Agent framework, in comparison to AgentVerse and SSP, it distinctly highlights the significance of Self-Refinement agents and Collaborative Refinement Action. Nevertheless, the two points mentioned above are not clearly articulated in Section 3.2, "EXECUTION STAGE." They are mentioned in the text and Figure 2, but their presentation as steps is absent, making it challenging for readers to comprehend and evaluate reproducibility.
> 2. Could you provide a more detailed, step-by-step description of the Self-Refinement agents and Collaborative Refinement Action in Section 3.2, "EXECUTION STAGE," to enhance readability and reproducibility?

In order to streamline the implementation process of self-refinement and collaborative refinement, we have introduced the concept of a 'custom agent'. The intricate design of its prompt has been meticulously detailed in Appendix D.5. For enhanced clarity and understanding, we have itemized the core design principles of this custom agent below:

1. Understanding Previous Results: Begin by analyzing the results of previous agents to grasp the context and progress of the task.

2. Task Analysis and Breakdown: Understand, analyze, and deconstruct the given task. Use available tools to assist in task completion.

3. Current Step Identification and Execution:
- 3.1 Examine completed steps and their outcomes to identify the current step that needs to be completed.
- 3.2 In the absence of completed steps, analyze the task, design a plan for the necessary steps, and accomplish the first one.
- 3.3 If steps have been completed, understand them to determine the next step to be completed.

4. Tool Selection and Action Execution:
- 4.1 Choose the appropriate tool from the given list ({tool}) to complete the current step.
- 4.2 Follow specific format guidelines when using tools like 'Write File'.
- 4.3 Once all steps are completed, use the 'Final Output' action to summarize each step's outputs, ensuring the final output is detailed, comprehensive, and solves the task.

5. Maintaining Format Consistency: Ensure that the final output adheres to the given format example, prioritizing helpfulness, relevance, accuracy, and detail.

Building upon the custom agent's framework, a critical aspect of the refinement process is the configuration of 'completed steps' in step 3.1. The specific procedural steps for self-refinement and collaborative refinement are outlined as follows:

__Self-refinement Action__: During its first execution, each custom agent omits the outlined step 3.1 and proceeds to complete the remaining steps. If the task's criteria are not met or the step limit has not been reached, the outcomes from this execution are incorporated as 'completed steps' in the prompt for the next iteration of the task. In subsequent executions, the custom agent will execute all steps, continuing this process until the task is deemed successfully completed.

__Collaborative Refinement Action__: This mirrors the self-refinement action, yet involves active collaboration between multiple agents. For instance, when agents A and B collaborate on a task, A initially bypasses step 3.1 in its first execution. Upon completion, B incorporates A's results as 'completed steps' and then executes all steps. In later cycles, A and B alternate their roles in the task, perpetuating this collaborative cycle until a joint conclusion is reached.

Reviewer comment (Reproducibility):
While Table 1 is highly beneficial for positioning this research within the LLM-based Agent framework, in comparison to AgentVerse and SSP, it highlights the significance of Self-Refinement agents and Collaborative Refinement Action. However, these two points are not clearly articulated in Section 3.2, "EXECUTION STAGE." They are mentioned in the text and Figure 2, but their presentation as steps is absent, making it challenging for readers to comprehend and evaluate reproducibility.

Author response:
Thank you for your motivating review and concrete suggestions. Detailed responses to your questions are listed as follows.

> 1. The paper is perceived as having low readability and insufficient reproducibility. The reviewer kindly requests a more granular description of the methodology that enables readers to implement the procedures step by step. For instance, while Table 1 is highly beneficial for positioning this research within the LLM-based Agent framework, in comparison to AgentVerse and SSP, it distinctly highlights the significance of Self-Refinement agents and Collaborative Refinement Action. Nevertheless, the two points mentioned above are not clearly articulated in Section 3.2, "EXECUTION STAGE." They are mentioned in the text and Figure 2, but their presentation as steps is absent, making it challenging for readers to comprehend and evaluate reproducibility.
> 2. Could you provide a more detailed, step-by-step description of the Self-Refinement agents and Collaborative Refinement Action in Section 3.2, "EXECUTION STAGE," to enhance readability and reproducibility?

In order to streamline the implementation process of self-refinement and collaborative refinement, we have introduced the concept of a 'custom agent'. The intricate design of its prompt has been meticulously detailed in Appendix D.5. For enhanced clarity and understanding, we have itemized the core design principles of this custom agent below:

1. Understanding Previous Results: Begin by analyzing the results of previous agents to grasp the context and progress of the task.

2. Task Analysis and Breakdown: Understand, analyze, and deconstruct the given task. Use available tools to assist in task completion.

3. Current Step Identification and Execution:
- 3.1 Examine completed steps and their outcomes to identify the current step that needs to be completed.
- 3.2 In the absence of completed steps, analyze the task, design a plan for the necessary steps, and accomplish the first one.
- 3.3 If steps have been completed, understand them to determine the next step to be completed.

4. Tool Selection and Action Execution:
- 4.1 Choose the appropriate tool from the given list ({tool}) to complete the current step.
- 4.2 Follow specific format guidelines when using tools like 'Write File'.
- 4.3 Once all steps are completed, use the 'Final Output' action to summarize each step's outputs, ensuring the final output is detailed, comprehensive, and solves the task.

5. Maintaining Format Consistency: Ensure that the final output adheres to the given format example, prioritizing helpfulness, relevance, accuracy, and detail.

Building upon the custom agent's framework, a critical aspect of the refinement process is the configuration of 'completed steps' in step 3.1. The specific procedural steps for self-refinement and collaborative refinement are outlined as follows:

__Self-refinement Action__: During its first execution, each custom agent omits the outlined step 3.1 and proceeds to complete the remaining steps. If the task's criteria are not met or the step limit has not been reached, the outcomes from this execution are incorporated as 'completed steps' in the prompt for the next iteration of the task. In subsequent executions, the custom agent will execute all steps, continuing this process until the task is deemed successfully completed.

__Collaborative Refinement Action__: This mirrors the self-refinement action, yet involves active collaboration between multiple agents. For instance, when agents A and B collaborate on a task, A initially bypasses step 3.1 in its first execution. Upon completion, B incorporates A's results as 'completed steps' and then executes all steps. In later cycles, A and B alternate their roles in the task, perpetuating this collaborative cycle until a joint conclusion is reached.

Reviewer comment (Evaluation):
The evaluation in the experiments lacks qualitative insights. It remains unclear how the introduction of Self-Refinement agents and Collaborative Refinement Action has led to differential outcomes compared to SSP, and what specific effects these two points have had. While accuracy has improved, it is essential to qualitatively demonstrate how these aspects have contributed to the observed results.

Author response:
> The evaluation in the experiments lacks qualitative insights. In the experiments, it remains unclear how the introduction of Self-Refinement agents and Collaborative Refinement Action has led to differential outcomes compared to SSP, and what specific effects these two points have had. While accuracy has undeniably improved, it is essential to qualitatively demonstrate how these two aspects have contributed to the observed results.

Within Appendix A of our revision, we have incorporated an ablation study specifically focused on AutoAgents within the context of trivia creative writing task. The table provided below delineates the performance fluctuations of AutoAgents when certain key components are omitted, thereby illustrating the impact of each individual element on the overall efficacy of the system.
- __AutoAgents w/o observers__: During the Drafting Stage, the Planner in AutoAgents engages in collaborative discussions with two Observers to determine the optimal list of agents and the execution plan. Table below elucidates that in the absence of observers, there is a marked 3% reduction in the overall performance of AutoAgents. This substantiates the imperative role of collaborative discussions in agent generation.
- __AutoAgents w/o self-refinement__: The performance of AutoAgents decreases by 3% in the absence of the self-refinement agent. This observation corroborates the assertion that self-refinement is instrumental in augmenting proficiency in trivia creative writing tasks. Furthermore, the enhancement of single agents via self-refinement plays a pivotal role in fortifying the integrity of the overarching multi-agent framework.
- __AutoAgents w/o collaborative refinement__: It's observable that compared to the scenario with AutoAgents, there is a decline of 2% without collaborative refinement. Since the necessity for multiple agents to collaborate on a single task is entirely dependent on the decision made by the agent in the drafting phase, not all problems necessarily involve tasks that require collaborative refinement. However, it's evident that when this principle is omitted from the prompt's design, there's a noticeable performance decrease in AutoAgents. This also corroborates the beneficial impact of collaborative refinement in tackling complex tasks.
- __AutoAgents w/o dynamic memory__: Dynamic memory predominantly addresses the requisites of specialized agents. The Action Observer amalgamates pivotal data for forthcoming tasks, utilizing the historical action records archived in long-term memory. The below table elucidates a 1% diminution in the efficacy of AutoAgents bereft of dynamic memory. Quintessential insights derived from dynamic memory are primarily assimilated into the prompt, thereby augmenting the comprehension of critical information and bolstering the operational proficiency of actions.

| Method | N (# triva questions) = 5|
| --- | --- |
| SSP | 84.4 |
| AutoAgents w/o observers | 87.0 |
| AutoAgents w/o self-refinement | 87.0 |
| AutoAgents w/o collaborative refinement | 88.0  |
| AutoAgents w/o dynamic memory | 89.0  |
| AutoAgents | 90.0 |

Reviewer comment (Reproducibility):
There are concerns regarding the reproducibility of the experiments. It is unclear whether the CASE STUDY has been practically realized or if it serves as an imagined example for application.

Author response:
> 1. There are concerns regarding the reproducibility of the experiments. It is unclear whether the CASE STUDY has been practically realized or if it serves as an imagined example for application.
> 2. Please note that further clarification on the practical realization of the CASE STUDY would be valuable to address concerns about reproducibility.

The CASE STUDY showcased within our paper are authentic execution outcomes from our study. To offer a more comprehensive understanding, we have included a video in the supplementary materials. This video illustrates the dynamic process of AutoAgents generating and executing code, thereby providing a tangible demonstration of our system's capabilities in action.

Reviewer comment (Experiments):
The paper lacks an ablation study to assess the impact of modifying or omitting certain components within the system, particularly in the Draft and Execution phases where multiple agents are involved, such as Agent Observer, Plan Observer, Researcher, Planner, Writer, Character Developer, and others. This study could help elucidate the significance of each component and its contribution to the overall system.

Author response:
> The paper lacks an ablation study to assess the impact of modifying or omitting certain components within the system, particularly in the Draft and Execution phases where multiple agents are involved, such as Agent Observer, Plan Observer, Researcher, Planner, Writer, Character Developer, and others. This study could help elucidate the significance of each component and its contribution to the overall system.

We would like to emphasize that AutoAgents are fundamentally engineered to create unique agents tailored for varied tasks. Within this design, only a select group of agents – the Planner, Agent Observer, Plan Observer, and Action Observer – are pre-established. The Planner and Action Observer, playing pivotal roles in the drafting and execution phases, respectively, are essential components of the framework and are indispensable. Consequently, our ablation study focused explicitly on examining the impact resulting from the exclusion of the Agent Observer and Plan Observer. For a more detailed exploration of these effects, we direct readers to the ablation experiment section in the section previously mentioned.

Reviewer comment (Experiments):
The absence of an ablation study regarding Short-term memory, Long-term memory, and Dynamic memory raises concerns. Investigating the effects of altering or removing these memory components could provide valuable insights into their respective roles and importance within the framework.

Author response:
> Furthermore, the absence of an ablation study regarding Short-term memory, Long-term memory, and Dynamic memory raises concerns. Investigating the effects of altering or removing these memory components could provide valuable insights into their respective roles and importance within the framework.

To further elucidate, we delineate the specific agent types that are compatible with the three modes of knowledge sharing, as depicted in Figure 7 of our revised manuscript:

- __Short-term Memory__: This is predominantly utilized by agents that emerge from self-refinement and collaborative-refinement processes. These agents leverage short-term memory to retain historical records throughout the refinement phase. Notably, every agent generated within our framework actively employs short-term memory.

- __Long-term Memory__: This component archives the execution results of each agent, primarily chronicling the cumulative progress of the execution plan. Access to long-term memory is exclusive to a pre-designated Action Observer. The Action Observer utilizes this memory to evaluate whether additional execution steps are necessary or if there's a need to generate dynamic memory content.

- __Dynamic Memory__: This memory type is essential for distilling and summarizing pivotal information from long-term memory for diverse agents. Generated by the Action Observer, dynamic memory serves to amalgamate critical aspects of the overall execution plan, thereby facilitating the agents responsible for implementing subsequent tasks. Consequently, the dynamic memory content available to each generated agent may differ, contingent upon their specific roles and tasks.

Due to the fundamental nature of Short-term Memory and Long-term Memory as indispensable components within the system, our investigation was exclusively focused on understanding and analyzing the role of dynamic memory.
In the Appendix A of the revision, we have included an ablation study of dynamic memory. The following table shows the results without the dynamic memory. It is observed that the performance of AutoAgents decreases by 1% without the summarization from dynamic memory, which also proves the importance of dynamic memory.

| Method | N (# triva questions) = 5|
| --- | --- |
| AutoAgents w/o dynamic memory | 89.0  |
| AutoAgents | 90.0 |

Reviewer comment (Novelty):
Limited novelty: according to Table 1, the main difference between the proposed framework and other existing methods like Social Simulacra, Epidemic Modeling, SSP, and AgentVerse is that this work uses self-refinement and collaborative refinement. This difference is more of a prompting technique and has already been used in many existing works like [1, 2, 3].

Author response:
Here, we further elucidate the main contributions of AutoAgents, a novel framework that addresses two salient issues: multi-agent generation and multi-agent cooperation. With respect to multi-agent generation, our method surpasses four existing methods in terms of the completeness and adaptability of agent creation, by employing Agent Generalization by Multi-Agent Discussion and Prompt Generalization. Regarding multi-agent cooperation, our framework exhibits the efficacy of Self-refinement Action and Cooperative Refinement Action. As illustrated in the table below, the distinctive features of AutoAgents encompass Agent Generalization by Multi-Agent Discussion, Prompt Generalization, Self-refinement Action, and Cooperative Refinement Action. AutoAgents is the first holistic framework that integrates multi-agent generation and collaboration. Detailed accounts of Agent Generalization by Multi-Agent Discussion and Prompt Generalization are given in below.

| Method | Task | Agent Generalization by Multi-Agent Discussion | Prompt Generalization | Self-refinement Action | Collaborative Refinement Action |
| --- | --- | --- | --- | --- | --- |
| Social Simulacra | Social Simulation | &cross; | &cross; | &cross; | &cross; |
| Epidemic Modeling | Social Simulation | &cross; | &cross; | &cross; | &cross; |
| SSP | General Autonomous Agents | &cross; | &cross; | &cross; | &cross; |
| AgentVerse | General Autonomous Agents | &cross; | &cross; | &cross; | &cross; |
| AutoAgents | General Autonomous Agents | &check; | &check; | &check; | &check; |

- __Agent Generalization by Multi-Agent Discussion__: Existing agent generation frameworks rely more on a single agent to generate the list of agents, but this often limits their adaptability to different tasks. For AutoAgents， the drafting stage is very important for the auto-generation of agents, as the list of agents to be generated is determined through cooperative discussions between the Planner agent and the two Observers. Additionally, in Appendix A, we further supplement with ablation experiments concerning the Agent Observer and Plan Observer, with the following table providing a detailed comparison. It is evident that in the absence of cooperative discussions among Observers, there is a significant decline in the overall performance of AutoAgents. This also validates the importance of generating reasonable agents for task adaptability.

| Method | N (# triva questions) = 5|
| --- | --- |
| AutoAgents w/o observers | 87.0  |
| AutoAgents | 90.0 |

__Note__: Please refer to the Appendix A of the revision for the detailed setup of the experiment.

- __Prompt Generalization__: The prompt employed by AutoAgents exhibits a more universal nature, signifying its capacity to acclimate to diverse tasks without necessitating bespoke customization. Both AgentVerse[1] and SSP[2] have implemented task-specific enhancements for varied task evaluations. In contrast, our methodology leverages a singular, unified prompt format to accommodate an array of tasks. The commendable efficacy in open-ended question answer and trivia creative writing tasks further corroborates the wide-ranging applicability and versatility of prompt design within the AutoAgents framework.

__Note__: In the revision, we have improved Table 1 and provided additional comparisons in Appendix A.

[1] https://github.com/OpenBMB/AgentVerse/tree/minecraft/agentverse/tasks

[2] https://github.com/MikeWangWZHL/Solo-Performance-Prompting/tree/main/prompts

Reviewer comment (Reproducibility):
Unclear presentation of detailed techniques: though the high-level idea is well-presented, the details of many technique are unclear. For example, how to determine when and which agents should engage in collaborative refinement? This is the main difference from other methods but there is very little detailed description.

Author response:
The implementation of collaborative refinement is entirely determined by discussions between the Planner and Observer. We have incorporated the principles of collaborative refinement into the Planner's prompt, without imposing restrictions on when to employ the collaborative refinement mode; this is left to the Planner's discretion. The specific content of the collaborative refinement prompt is as follows:
```
4.2 Each step should assign at least one expert role to carry it out. If a step involves multiple 
expert roles, you need to specify the contributions of each expert role and how they collaborate 
to produce integrated results. 
```

Reviewer comment (Experiments):
Lack of baselines: Table 1 lists 12 existing frameworks, but none is used as baseline in task 1, and only 1 is used in task 2. Comparisons with existing methods are needed to show the effectiveness of the proposed methods.

Author response:
Due to the specialized focus of Social Simulacra and Epidemic Modeling on Social Simulation tasks, a direct comparison with open-ended question-answering models is not practical. Furthermore, the operation of SSP is contingent upon an Azure GPT-4 key, which we currently do not possess. This limitation precludes us from acquiring results for task 1. As a result, we have enhanced Table 1 by including a comparative analysis featuring AgentVerse, to provide a more comprehensive overview within the scope of our available resources.

In the revised Table 2, we have enhanced the analysis by including results for AgentVerse, utilizing both FairEval and HumanEval for a comprehensive evaluation. As illustrated in the table provided below, it is apparent that AutoAgents exhibits superior performance compared to AgentVerse in addressing a majority of open-ended questions. This improved efficacy can be attributed to several key features of AutoAgents: the implementation of cooperative discussion-based agent generation, and the integration of both self-refinement and collaborative refinement mechanisms. These components collectively contribute to the more effective handling of open-ended queries by AutoAgents, as demonstrated in the comparative results.

| Method | AutoAgents vs. AgentVerse |
| --- | --- |
| FairEval | 91.3%  |
| HumanEval | 77.5% |

__Note__: The prompt configuration within AgentVerse is tailored to its [brainstorming](https://github.com/OpenBMB/AgentVerse/blob/main/agentverse/tasks/tasksolving/brainstorming/config.yaml) task, incorporating modifications to upgrade the model to GPT-4.

Reviewer comment (Experiments):
Lack of ablation study: there is no quantitative ablations on different components of the framework like self-refinement, collaborative refinement, etc.

Author response:
Within Appendix A of our revision, we have incorporated an ablation study specifically focused on AutoAgents within the context of trivia creative writing task. The table provided below delineates the performance fluctuations of AutoAgents when certain key components are omitted, thereby illustrating the impact of each individual element on the overall efficacy of the system.
- __AutoAgents w/o observers__: During the Drafting Stage, the Planner in AutoAgents engages in collaborative discussions with two Observers to determine the optimal list of agents and the execution plan. Table below elucidates that in the absence of observers, there is a marked 3% reduction in the overall performance of AutoAgents. This substantiates the imperative role of collaborative discussions in agent generation.
- __AutoAgents w/o self-refinement__: The performance of AutoAgents decreases by 3% in the absence of the self-refinement agent. This observation corroborates the assertion that self-refinement is instrumental in augmenting proficiency in trivia creative writing tasks. Furthermore, the enhancement of single agents via self-refinement plays a pivotal role in fortifying the integrity of the overarching multi-agent framework.
- __AutoAgents w/o collaborative refinement__: It's observable that compared to the scenario with AutoAgents, there is a decline of 2% without collaborative refinement. Since the necessity for multiple agents to collaborate on a single task is entirely dependent on the decision made by the agent in the drafting phase, not all problems necessarily involve tasks that require collaborative refinement. However, it's evident that when this principle is omitted from the prompt's design, there's a noticeable performance decrease in AutoAgents. This also corroborates the beneficial impact of collaborative refinement in tackling complex tasks.
- __AutoAgents w/o dynamic memory__: Dynamic memory predominantly addresses the requisites of specialized agents. The Action Observer amalgamates pivotal data for forthcoming tasks, utilizing the historical action records archived in long-term memory. The below table elucidates a 1% diminution in the efficacy of AutoAgents bereft of dynamic memory. Quintessential insights derived from dynamic memory are primarily assimilated into the prompt, thereby augmenting the comprehension of critical information and bolstering the operational proficiency of actions.

| Method | N (# triva questions) = 5|
| --- | --- |
| AutoAgents w/o observers | 87.0 |
| AutoAgents w/o self-refinement | 87.0 |
| AutoAgents w/o collaborative refinement | 88.0  |
| AutoAgents w/o dynamic memory | 89.0  |
| AutoAgents | 90.0 |

Reviewer comment (Experiments):
Unfair comparisons and potential problem in metric: in task 1, it is unfair to compare AutoAgents using GPT-4 with ChatGPT and Vicunna-13B. In task 2, the metric only considers the QA quality, how about the quality of the story around the given topic?

Author response:
Due to the specialized focus of Social Simulacra and Epidemic Modeling on Social Simulation tasks, a direct comparison with open-ended question-answering models is not practical. Furthermore, the operation of SSP is contingent upon an Azure GPT-4 key, which we currently do not possess. This limitation precludes us from acquiring results for task 1. As a result, we have enhanced Table 1 by including a comparative analysis featuring AgentVerse, to provide a more comprehensive overview within the scope of our available resources.

In the revised Table 2, we have enhanced the analysis by including results for AgentVerse, utilizing both FairEval and HumanEval for a comprehensive evaluation. As illustrated in the table provided below, it is apparent that AutoAgents exhibits superior performance compared to AgentVerse in addressing a majority of open-ended questions. This improved efficacy can be attributed to several key features of AutoAgents: the implementation of cooperative discussion-based agent generation, and the integration of both self-refinement and collaborative refinement mechanisms. These components collectively contribute to the more effective handling of open-ended queries by AutoAgents, as demonstrated in the comparative results.

| Method | AutoAgents vs. AgentVerse |
| --- | --- |
| FairEval | 91.3%  |
| HumanEval | 77.5% |

__Note__: The prompt configuration within AgentVerse is tailored to its [brainstorming](https://github.com/OpenBMB/AgentVerse/blob/main/agentverse/tasks/tasksolving/brainstorming/config.yaml) task, incorporating modifications to upgrade the model to GPT-4.

Reviewer comment (Theory):
Section 3.1 Plan Generation: why plan generation is parallel to agent generation? If the agent list has not been generated, how is it possible to "entail a clear identification of agent" for each step?

Author response:
In addressing your observation, it is important to clarify that the generation of agents and plans within our system is not conducted in parallel. The creation process of agents always precedes the development of the plan. To provide further clarity, we have added detailed explanations of the underlying principles for each prompt in the appendix, specifically outlining the process within the Planner.
```
1. Understanding and Breaking Down Tasks: The first step involves comprehensively understanding, analyzing, and deconstructing the given task or problem.

2. Utilization of Existing Expert Roles:
- Fully leverage existing expert roles suited to the problem.
- Ensure that these roles have cooperative or dependent relationships.
- Output the details of selected roles in JSON format, including their original information.

1. Creation of New Expert Roles:
- Avoid duplication of functions in new roles.
- New roles should have clear names, detailed descriptions, domain expertise, available tools, execution suggestions, and prompt templates.
- Ensure each new expert role has a distinct responsibility and domain of expertise.
- Specify the goals, constraints, and toolset for each new role.
- Provide execution suggestions and develop prompt templates for each new role.
- Output details of new roles in JSON format, following a specific structure.

1. Creation of Detailed Execution Plan:
- Develop a comprehensive plan with multiple steps addressing the problem.
- Assign at least one expert role to each step, detailing their contributions and collaborations.
- Provide detailed descriptions for each step, including expected outputs and inputs for subsequent steps.
- Include a final independent step for a language expert to provide a detailed response to the user's question.
- Present the execution plan as a numbered list, indicating the expert roles involved in each step.
```

To elaborate, the Planner's methodology is structured in a sequential manner: it begins with a thorough analysis of the problem at hand. Following this, it proceeds to generate a list of agents, tailored specifically to the identified problem. The final step involves the crafting of an execution plan. It is noteworthy, however, that agents and execution plans are inherently interdependent components. In recognition of this symbiosis, we have integrated both elements into the prompt's structure. This integrative approach facilitates the production of not only a more logical and targeted list of agents but also a more streamlined and effective execution plan.

Reviewer comment (Writing):
Section 3.2 Action Observer: how does the Action Observer interact and communicate with other agents to act as the tasks manager and how is this process determined? When will the Action Observer adapt the execution plan?

Author response:
In our manuscript, it is elucidated that the Action Observer adopts a vertical communication strategy for interfacing with agents. This entails assigning diverse tasks to different agents, guided by the specifics outlined in the execution plan, which includes both the identification of agents and their corresponding tasks. The Action Observer, in its role, meticulously reviews the results archived in long-term memory while also contemplating forthcoming execution strategies. This process is pivotal for assessing whether alterations to the plan are necessary. In instances where the extant outcomes are insufficient for the subsequent agent's requirements, the Action Observer takes the initiative to refine the execution plan, tailoring it based on these existing results. Presented below is a sample prompt that exemplifies the task modification process.

```
2.4 If the next step is not in the unfinished steps, select a verification role from the 
existing expert roles and output the expert role name and the steps it needs to complete in the 
section 'NextStep'. Please indicate the name of the expert role used at the beginning of the step. 
```

Reviewer comment (Reproducibility):
Section 3.2 Knowledge Sharing Mechanism: how to determine what knowledge is shared and who to share with?

Author response:
To further elucidate, we delineate the specific agent types that are compatible with the three modes of knowledge sharing, as depicted in Figure 7 of our revised manuscript:

- __Short-term Memory__: This is predominantly utilized by agents that emerge from self-refinement and collaborative-refinement processes. These agents leverage short-term memory to retain historical records throughout the refinement phase. Notably, every agent generated within our framework actively employs short-term memory.

- __Long-term Memory__: This component archives the execution results of each agent, primarily chronicling the cumulative progress of the execution plan. Access to long-term memory is exclusive to a pre-designated Action Observer. The Action Observer utilizes this memory to evaluate whether additional execution steps are necessary or if there's a need to generate dynamic memory content.

- __Dynamic Memory__: This memory type is essential for distilling and summarizing pivotal information from long-term memory for diverse agents. Generated by the Action Observer, dynamic memory serves to amalgamate critical aspects of the overall execution plan, thereby facilitating the agents responsible for implementing subsequent tasks. Consequently, the dynamic memory content available to each generated agent may differ, contingent upon their specific roles and tasks.

Reviewer comment (Experiments):
Compare with more baselines in Table 1 for both task 1 and task 2, especially multi-agent frameworks like Social Simulacra, Epidemic Modeling, SSP, and AgentVerse.

Author response:
Due to the specialized focus of Social Simulacra and Epidemic Modeling on Social Simulation tasks, a direct comparison with open-ended question-answering models is not practical. Furthermore, the operation of SSP is contingent upon an Azure GPT-4 key, which we currently do not possess. This limitation precludes us from acquiring results for task 1. As a result, we have enhanced Table 1 by including a comparative analysis featuring AgentVerse, to provide a more comprehensive overview within the scope of our available resources.

In the revised Table 2, we have enhanced the analysis by including results for AgentVerse, utilizing both FairEval and HumanEval for a comprehensive evaluation. As illustrated in the table provided below, it is apparent that AutoAgents exhibits superior performance compared to AgentVerse in addressing a majority of open-ended questions. This improved efficacy can be attributed to several key features of AutoAgents: the implementation of cooperative discussion-based agent generation, and the integration of both self-refinement and collaborative refinement mechanisms. These components collectively contribute to the more effective handling of open-ended queries by AutoAgents, as demonstrated in the comparative results.

| Method | AutoAgents vs. AgentVerse |
| --- | --- |
| FairEval | 91.3%  |
| HumanEval | 77.5% |

__Note__: The prompt configuration within AgentVerse is tailored to its [brainstorming](https://github.com/OpenBMB/AgentVerse/blob/main/agentverse/tasks/tasksolving/brainstorming/config.yaml) task, incorporating modifications to upgrade the model to GPT-4.

Reviewer comment (Experiments):
Unfair comparisons in Section 4.1: fair comparisons would be AutoAgents using ChatGPT v.s. ChatGPT and AutoAgents using Vicuna-13B v.s. Vicuna-13B.

Author response:
Thank you for your valuable feedback. Following your recommendation, we have revised our manuscript to exclude the comparison between ChatGPT and Vicua-13B. This decision aligns with our commitment to ensuring a fair and balanced evaluation. We will now focus our comparative analysis solely on AutoAgents, GPT-4, and AgentVerse. For a detailed overview of these changes, please refer to the updated version of our manuscript or following table. We appreciate your guidance in enhancing the quality and integrity of our work.

| Evaluator | vs. GPT-4 | vs. AgentVerse |
| --- | --- | --- |
| FairEval | 76.3% | 91.3%  |
| HumanEval | 62.5% | 77.5% |

Reviewer comment (Evaluation):
Metric problem in Section 4.2: the trivia creative writing task has two subtasks: (a) craft a coherent story around a given topic (b) answer N questions. The current metric only evaluates the result of subtask (b); there needs to be another metric for subtask (a).

Author response:
Here, we present a sample of task 2 where the topic is a noun with expansive semantic breadth. However, assessing the alignment between the narratives and these broad noun topics is less straightforward due to their general nature.
```
{
    "questions": ["Who was the man behind The Chipmunks?", "Which Lloyd Webber musical premiered in the US on 10th December 1993?", "Who was the next British Prime Minister after Arthur Balfour?", "Who had a 70s No 1 hit with Kiss You All Over?", "What claimed the life of singer Kathleen Ferrier?"], 
    "answers": [["David Seville", "david seville"], ["Sunset Blvd", "West Sunset Boulevard", "Sunset Boulevard", "Sunset Bulevard", "Sunset Blvd.", "sunset boulevard", "sunset bulevard", "west sunset boulevard", "sunset blvd"], ["Sir Henry Campbell-Bannerman", "Campbell-Bannerman", "Campbell Bannerman", "Sir Henry Campbell Bannerman", "Henry Campbell Bannerman", "Henry Campbell-Bannerman", "henry campbell bannerman", "sir henry campbell bannerman", "campbell bannerman"], ["Internal exile", "Exiles", "Transported for life", "Exile (politics and government)", "Voluntary exile", "Sent into exile", "Exile and Banishment", "Self-exile", "Forced exile", "Exile", "Exile in Greek tragedy", "Banish", "Banishment", "exiles", "voluntary exile", "forced exile", "banish", "self exile", "exile politics and government", "exile in greek tragedy", "sent into exile", "banishment", "transported for life", "exile", "internal exile", "exile and banishment"], ["Cancer pathology", "Deaths by cancer", "Anti-cancer", "Cancer (disease)", "Cancerophobia", "Malignant lesion", "Cancer medication", "Malignant tumors", "Cancer signs", "Malignant neoplasm", "Invasive (cancer)", "Malignant Neoplasms", "Malignant growth", "Sporadic cancer", "Malignant cancer", "Tumour virus", "Cancer en cuirasse", "Microtumor", "Malignant neoplasms", "Malignant tumour", "Carcinophobia", "Malignacy", "Cancer patient", "Epithelial cancers", "Solid cancer", "Cancers", "Tumor medication", "Malignant neoplastic disease", "AIDS-related cancer", "Invasive cancer", "Cancer therapy", "Cancerous tumor", "Cancer", "Financial toxicity", "Cancer diagnosis", "Cancer (medicine)", "Malignant tumor", "Cancerous", "Borderline (cancer)", "Signs of cancer", "Malignancies", "Cancer aromatase", "aids related cancer", "sporadic cancer", "cancer disease", "malignant tumors", "cancers", "carcinophobia", "cancer", "cancer diagnosis", "malignant neoplastic disease", "malignant neoplasm", "tumour virus", "cancer medicine", "deaths by cancer", "malignant tumour", "epithelial cancers", "solid cancer", "cancerous", "borderline cancer", "invasive cancer", "anti cancer", "cancer pathology", "cancer signs", "cancer aromatase", "cancer therapy", "financial toxicity", "cancerophobia", "cancer en cuirasse", "cancer patient", "cancerous tumor", "malignant cancer", "malignant neoplasms", "tumor medication", "signs of cancer", "malignacy", "malignant tumor", "cancer medication", "microtumor", "malignancies", "malignant lesion", "malignant growth"]], 
    "question_ids": ["tc_2", "tc_33", "tc_40", "tc_49", "tc_56"], 
    "topic": "Harry Potter"
}
```
To address task (a), we merged the data's answers with the topics to form new topics. We then utilize an LLM (BAAI/bge-large-en-v1.5) [1] to generate embeddings for these new topics and the stories, computing the similarity between them to gauge topic relevance. The results, as shown below, indicate that AutoAgents still exhibit superior relevance to the topic in this scenario.

| Method | N (# triva questions) = 5| N (# triva questions) = 10|
| --- | --- | --- |
| SSP | 0.643 | 0.623 |
| AutoAgents | 0.651 | 0.664 |

[1] Zhang, Peitian, et al. "Retrieve Anything To Augment Large Language Models." arXiv preprint arXiv:2310.07554 (2023).

Reviewer comment (Experiments):
Ablation study: quantitative ablation results on different components of the framework. For example, remove self-refinement, collaborative refinement, different Observers, etc.

Author response:
Within Appendix A of our revision, we have incorporated an ablation study specifically focused on AutoAgents within the context of trivia creative writing task. The table provided below delineates the performance fluctuations of AutoAgents when certain key components are omitted, thereby illustrating the impact of each individual element on the overall efficacy of the system.
- __AutoAgents w/o observers__: During the Drafting Stage, the Planner in AutoAgents engages in collaborative discussions with two Observers to determine the optimal list of agents and the execution plan. Table below elucidates that in the absence of observers, there is a marked 3% reduction in the overall performance of AutoAgents. This substantiates the imperative role of collaborative discussions in agent generation.
- __AutoAgents w/o self-refinement__: The performance of AutoAgents decreases by 3% in the absence of the self-refinement agent. This observation corroborates the assertion that self-refinement is instrumental in augmenting proficiency in trivia creative writing tasks. Furthermore, the enhancement of single agents via self-refinement plays a pivotal role in fortifying the integrity of the overarching multi-agent framework.
- __AutoAgents w/o collaborative refinement__: It's observable that compared to the scenario with AutoAgents, there is a decline of 2% without collaborative refinement. Since the necessity for multiple agents to collaborate on a single task is entirely dependent on the decision made by the agent in the drafting phase, not all problems necessarily involve tasks that require collaborative refinement. However, it's evident that when this principle is omitted from the prompt's design, there's a noticeable performance decrease in AutoAgents. This also corroborates the beneficial impact of collaborative refinement in tackling complex tasks.
- __AutoAgents w/o dynamic memory__: Dynamic memory predominantly addresses the requisites of specialized agents. The Action Observer amalgamates pivotal data for forthcoming tasks, utilizing the historical action records archived in long-term memory. The below table elucidates a 1% diminution in the efficacy of AutoAgents bereft of dynamic memory. Quintessential insights derived from dynamic memory are primarily assimilated into the prompt, thereby augmenting the comprehension of critical information and bolstering the operational proficiency of actions.

| Method | N (# triva questions) = 5|
| --- | --- |
| AutoAgents w/o observers | 87.0 |
| AutoAgents w/o self-refinement | 87.0 |
| AutoAgents w/o collaborative refinement | 88.0  |
| AutoAgents w/o dynamic memory | 89.0  |
| AutoAgents | 90.0 |

Reviewer comment (Evaluation):
How the proposed AutoAgents framework expands the scope of collaborative applications and reduces the consumption of resources should be elaborated.

Author response:
Existing agent technologies are overly dependent on pre-defined prompts. This reliance poses a substantial user barrier and limits their practicality for broader applications. AutoAgents addresses this by automating the creation of new agent roles, which streamlines both the definition of roles and the execution of tasks, thereby enhancing the system's versatility.

AutoAgents significantly extends its adaptability to diverse applications through the automatic generation of multiple intelligent agents, each tailored for specific tasks. This improvement in agent generation quality is achieved through the collaborative efforts of three predefined agents - the Planner, Agent Observer, and Plan Observer. These agents work together to discuss, refine, and finalize the list of agents and their corresponding execution plans. In Appendix A, we delve deeper into this aspect with ablation experiments focusing on the Agent Observer and Plan Observer. Our comparative analysis, as shown in the following table, reveals a marked decrease in AutoAgents' overall performance in the absence of cooperative dialogues among Observers. This outcome not only underscores the efficacy of collaborative refinement but also highlights the critical role of these discussions in enhancing task adaptability and resource optimization in the AutoAgents framework.

| Method | N (# triva questions) = 5|
| --- | --- |
| AutoAgents w/o observers | 87.0  |
| AutoAgents | 90.0 |

__Note__: Please refer to the Appendix A of the revision for the detailed setup of the experiment.

Reviewer comment (Reproducibility):
The authors do not explain how to determine the number of agents in the section of the framework for automatic agent generation.

Author response:
AutoAgents fundamentally predefines only a Planner and three types of Observers. Throughout the automatic agent generation phase, neither the number nor the types of agents to be generated are pre-specified. Instead, the decision regarding both the number and types of agents is dynamically and autonomously determined through collaborative discussions among the Planner, Agent Observer, and Plan Observer. This process ensures that the agents generated are optimally aligned with the specific requirements and nuances of each task.

Reviewer comment (Experiments):
In addition to ChatGPT, Vicuna-13B and GPT4 in Table 2, it has not enough recent models to further show the superiority of the proposed framework-AutoAgents in open-ended question answer task in the experimental part.

Author response:
In the revised Table 2, we have enhanced the analysis by including results for AgentVerse, utilizing both FairEval and HumanEval for a comprehensive evaluation. As illustrated in the table provided below, it is apparent that AutoAgents exhibits superior performance compared to AgentVerse in addressing a majority of open-ended questions. This improved efficacy can be attributed to several key features of AutoAgents: the implementation of cooperative discussion-based agent generation, and the integration of both self-refinement and collaborative refinement mechanisms. These components collectively contribute to the more effective handling of open-ended queries by AutoAgents, as demonstrated in the comparative results.

| Method | AutoAgents vs. AgentVerse |
| --- | --- |
| FairEval | 91.3%  |
| HumanEval | 77.5% |

__Note__: The prompt configuration within AgentVerse is tailored to its [brainstorming](https://github.com/OpenBMB/AgentVerse/blob/main/agentverse/tasks/tasksolving/brainstorming/config.yaml) task, incorporating modifications to upgrade the model to GPT-4.

Reviewer comment (Evaluation):
In the experimental part, the performance on N=10 is better than N=5 in trivia creative writing task, but there is no explanations.

Author response:
Table 3 clearly reveals that all evaluated methods perform more effectively at N=10 than at N=5. This enhanced performance at N=10 can be ascribed to the wider array of available resources, which considerably assists the methods in crafting narratives that are both more cohesive and realistic. In contrast, the constrained resource pool at N=5 means that the exclusion of even a single response category can result in significant fluctuations in performance. This is exemplified by a marked 20% performance drop due to the lack of one type of response. At N=10, however, such performance disparities are notably mitigated, evidenced by a more modest 10% change under similar conditions of response type absence. This observation underscores an interesting aspect for more detailed exploration in our future research.

Reviewer comment (Experiments):
The authors present results for the Open-ended Question Answer task and the Trivia Creative Writing task to evaluate the framework effectiveness. What if the Question Answer task is not open-ended? Does the proposed framework AutoAgents still work?

Author response:
The Trivia Creative Writing task serves as a rigorous test for large language models, gauging their ability to retrieve and integrate a wide range of information from their internal knowledge bases. In this task, models are required to construct a cohesive narrative around a specified topic while seamlessly incorporating answers to N trivia questions. The evaluation criteria for this task consider any match to the answer variants of a question as a correct mention.

Consequently, the Trivia Creative Writing task fits the description of a non-open-ended question-answering challenge. In this setup, each sample presents a series of either 5 or 10 common sense questions that necessitate accurate responses from diverse methodologies before the models proceed to narrative creation. The principal measure for assessing this task is the successful integration of these correct answers into the narratives. An analysis of the results in Table 3 clearly demonstrates that narratives generated by AutoAgents feature a notably higher proportion of accurate responses. This outcome robustly validates the capability of AutoAgents in the sphere of non-open-ended question-answering, underlining their adeptness at effectively amalgamating precise information within imaginative storylines.

Reviewer comment (Reproducibility):
For the agent generation, the motivation for the format of the Prompt P is unclear. Additionally, when we look at the supplementary material, the specific elements of the prompt are not explained -- are these taken from existing works?

Author response:
Existing agent technologies are overly dependent on pre-defined prompts, presenting a significant barrier to users and rendering them impractical for widespread use. By developing a system that automatically creates new agent roles, we can automate both the definition of roles and the execution of tasks, significantly increasing the technology's versatility. AutoAgents, potentially a pioneering framework, is distinctively geared towards generating universal prompts applicable to a wide range of agents. Our design not only ensures the accomplishment of each agent's predefined tasks but also, through careful consideration during the design phase, enhances the adaptability of these prompts to a variety of tasks, thus broadening their general applicability. In the updated appendix, we have incorporated a section at the outset of each prompt detailing the design principles and underlying logic.

Reviewer comment (Reproducibility):
I found details that needed to be included in a few other sections, such as the self-refinement process.

Author response:
As described in the appendix about the 'custom agent', each agent, upon receiving their assigned task, must first understand and analyze the task. They need to plan the steps required for future action and determine the immediate steps to be taken. The agent then outputs the results of completing the current step. In the next round of feedback, the history of executed steps and their outcomes (short-term memory) are fed back into the prompt under 'completed steps and responses'. This helps the agent to determine the steps needed in the future. The prompt's design rationale is additionally detailed in the appendix of the revision.

Reviewer comment (Novelty):
What motivated the design of the prompt elements for the predefined agents? Did you consider alternatives, or did existing works inspire these?

Author response:
Existing agent technologies are overly dependent on pre-defined prompts, presenting a significant barrier to users and rendering them impractical for widespread use. By developing a system that automatically creates new agent roles, we can automate both the definition of roles and the execution of tasks, significantly increasing the technology's versatility. AutoAgents, potentially a pioneering framework, is distinctively geared towards generating universal prompts applicable to a wide range of agents. Our design not only ensures the accomplishment of each agent's predefined tasks but also, through careful consideration during the design phase, enhances the adaptability of these prompts to a variety of tasks, thus broadening their general applicability. In the updated appendix, we have incorporated a section at the outset of each prompt detailing the design principles and underlying logic.

Reviewer comment (Reproducibility):
How were the roles, skills, and actions decided for the specific tasks? Were they injected in the prompt, or did the Planner agent generate them?

Author response:
The Planner agent serves as the architect in our framework, initially crafting a comprehensive array of roles specifically designed for a variety of tasks. This includes not only the creation of prompts, descriptions, and toolsets for each role but also the formulation of an intricate execution plan. This plan distinctly specifies whether self-refinement or collaborative refinement actions are required. Following this initial phase, both the list of roles and the execution plan are subject to further refinement, achieved through collaborative dialogues with Observer agents. It is imperative to note that the choice of specific tools and techniques employed by each agent to navigate the steps in the execution plan is made independently by the agents themselves. A crucial aspect of our methodology is that we do not create unique prompts for each task. Rather, the primary prompts are methodically generated by the Planner agent, ensuring a consistent and streamlined approach across various tasks.

Reviewer comment (Reproducibility):
For the self-refinement process, what was the source of the thoughts, i.e., who decided what thoughts to include and why?

Author response:
As described in the appendix about the 'custom agent', each agent, upon receiving their assigned task, must first understand and analyze the task. They need to plan the steps required for future action and determine the immediate steps to be taken. The agent then outputs the results of completing the current step. In the next round of feedback, the history of executed steps and their outcomes (short-term memory) are fed back into the prompt under 'completed steps and responses'. This helps the agent to determine the steps needed in the future. The prompt's design rationale is additionally detailed in the appendix of the revision.

Reviewer comment (Experiments):
Regarding knowledge sharing, did you experiment with each agent using different types, or were these predefined onset?

Author response:
To further elucidate, we delineate the specific agent types that are compatible with the three modes of knowledge sharing, as depicted in Figure 7 of the revision:

- __Short-term Memory__: This is predominantly utilized by agents that emerge from self-refinement and collaborative-refinement processes. These agents leverage short-term memory to retain historical records throughout the refinement phase. Notably, every agent generated within our framework actively employs short-term memory.

- __Long-term Memory__: This component archives the execution results of each agent, primarily chronicling the cumulative progress of the execution plan. Access to long-term memory is exclusive to a pre-designated Action Observer. The Action Observer utilizes this memory to evaluate whether additional execution steps are necessary or if there's a need to generate dynamic memory content.

- __Dynamic Memory__: This memory type is essential for distilling and summarizing pivotal information from long-term memory for diverse agents. Generated by the Action Observer, dynamic memory serves to amalgamate critical aspects of the overall execution plan, thereby facilitating the agents responsible for implementing subsequent tasks. Consequently, the dynamic memory content available to each generated agent may differ, contingent upon their specific roles and tasks.

Due to the fundamental nature of Short-term Memory and Long-term Memory as indispensable components within the system, our investigation was exclusively focused on understanding and analyzing the role of dynamic memory. In the Appendix A of the revision, we have included an ablation study of dynamic memory. The following table shows the results without the dynamic memory. It is observed that the performance of AutoAgents decreases by 1% without the summarization from dynamic memory, which also proves the importance of dynamic memory.

| Method | N (# triva questions) = 5|
| --- | --- |
| AutoAgents w/o dynamic memory | 89.0  |
| AutoAgents | 90.0 |

__Note__: Please refer to the Appendix A of the revision for the detailed setup of the experiment.

Reviewer comment (Experiments):
How did you decide to use the number of discussions in the two stages to 3 and 5, respectively (page 7)?

Author response:
On one hand, it's been observed through experience that a detailed plan and a complete list of agents can generally be obtained after about three rounds of collaborative planning discussions. In cases where a single agent is operating, there's a high probability that the current task can be completed within five rounds. While a greater number of execution rounds typically leads to better outcomes, on the other hand, since we rely on GPT-4, more iterations mean increased costs. This setup is designed to manage and control the costs associated with task execution.

Reviewer comment (Experiments):
In the Open-ended Q&A, the authors mention recruiting volunteers, but no more details are provided about how, whether ethics approval was sought, etc. Could the authors please provide more details on this.

Author response:
For HumanEval, we enlisted three independent volunteers to evaluate two sets of responses—one generated by AutoAgents and the other by a different model—based on criteria such as helpfulness, reliability, accuracy, and comprehensiveness. Notably, the volunteers were blinded to the identity of the model that produced each response, ensuring an unbiased assessment. We instructed the volunteers, who are responsible for assessing the quality of different feedback, to adhere to these standards.

```
We would like to request your feedback on the response to the user question
displayed above. Please rate the helpfulness, relevance, accuracy, level of 
details of their responses. 

Each response receives an overall score on a scale of 1 to 10, where a higher 
score indicates better overall performance. Please first provide a 
comprehensive explanation of your evaluation, avoiding any potential bias and 
ensuring that the order in which the responses were presented does not affect 
your judgment. 

Output with the following format:
Evaluation evidence: <your evluation explanation here>
Score: <score>
```