Experiential Co-Learning of Software-Developing Agents (2024)

Abstract: Recent advancements in large language models (LLMs) have brought significant changes to various domains, especially through LLM-driven autonomous agents. A representative scenario is in software development, where LLM agents demonstrate efficient collaboration, task division, and assurance of software quality, markedly reducing the need for manual involvement. However, these agents frequently perform a variety of tasks independently, without benefiting from past experiences, which leads to repeated mistakes and inefficient attempts in multi-step task execution. To this end, we introduce Experiential Co-Learning, a novel LLM-agent learning framework in which instructor and assistant agents gather shortcut-oriented experiences from their historical trajectories and use these past experiences for future task execution. The extensive experiments demonstrate that the framework enables agents to tackle unseen software-developing tasks more effectively. We anticipate that our insights will guide LLM agents towards enhanced autonomy and contribute to their evolutionary growth in cooperative learning. The code and data are available at https://github.com/OpenBMB/ChatDev.

## introduction in the ever-evolving field of artificial intelligence, large language models (llms) have marked a transformative shift across numerous domains @xcite @xcite . despite their impressive abilities, when dealing with complex situations that extend beyond mere chatting, these models show certain limitations inherent in their standalone capabilities @xcite . recent research in autonomous agents has significantly advanced llms † equal contribution. corresponding author. by integrating sophisticated features like contextsensitive memory @xcite , multi-step planning @xcite , and strategic tool use @xcite . this enhancement has expanded their capacity to effectively manage a broader spectrum of complex tasks, including social simulation @xcite @xcite , software development @xcite , game playing @xcite @xcite , and scientific research @xcite . in order to study the cooperative dynamics of autonomous agents more pertinently, we choose software development @xcite as an representative scenario, due to its complexity that demands a blend of natural and programming language skills @xcite , the processuality that often requires an in-depth understanding of coding and continuous alterations @xcite , and the objectivity of code that can provide quantifiable feedback @xcite . with the development of autonomous-agent technology, a successful breakthrough has been the integration of communication among multiple agents @xcite @xcite . representative methods epitomize this methodology by segmenting task execution into distinct subtasks. through engaging in cooperative communication, agents participate in instructive or responsive conversations, collaboratively contributing to the achievement of a cohesive and automated solution for task execution. for example, in chatdev @xcite , a recent agent-communication framework for software development, a reviewer agent in charge of an external compiler iteratively provides instructions for software optimization (e.g., completing unimplemented code, making functional changes, and debugging programs), to which a programmer agent then reacts to these instructions by appropriately up-dating the source code. the development of a more adaptive and proactive approach to task-solving by these agents marks a significant leap in autonomy, going beyond the typical prompt-guided dynamic in human-computer communications @xcite and substantially reducing dependence on human involvement @xcite @xcite . however, when confronted with a diverse range of task types, current multi-agent collaboration methods tend to handle each task independently, which largely stems from the absence of a methodology that can effectively incorporate the experiences accumulated from previously completed tasks @xcite @xcite . consequently, the inexperience nature frequently results in repetitive errors or unnecessary trial-and-error processes through multi-step task execution, ultimately necessitating additional human involvement especially when these methods are applied to real-world scenarios. how do we design, gather, and apply useful experiences to enhance multi-agent collaboration? in this paper, we propose experiential co-learning, a novel multi-agent learning paradigm designed to boost agents' software-developing abilities through the utilization of experiences gathered from their historical communications. the method regards agents into two functional roles-instructor and assistant, involving three core modules: 1) the co-tracking module promotes communicative rehearsals between the agents, fostering cooperative exploration and the creation of procedural trajectories for various training tasks; 2) the co-memorizing module heuristically mines "shortcuts" foot_0 from historical trajectories using external environment feedback, which are then preserved in their experience pools in an interleaved manner; 3) the co-reasoning module encourages agents to enhance their instructions and solutions by utilizing their collective experience pools when facing unseen tasks. the comprehensive assessment of collaborative processes between autonomous agents in diverse software development tasks reveals that the proposed framework significantly boosts collaborative efficiency and reduces the need for extra human involvement. in summary, our main contributions include: ## related work trained on vast datasets to comprehend and manipulate billions of parameters, llms have become pivotal in natural language processing @xcite @xcite @xcite @xcite @xcite @xcite @xcite . recent progress, particularly in the filed of autonomous agents @xcite @xcite @xcite @xcite , is largely attributed to the foundational advances in llms. these agents utilize the robust capabilities of llms, displaying remarkable skills in memory @xcite , planning @xcite @xcite and tool use @xcite @xcite @xcite , enabling them to operate independently in complex, real-world scenarios @xcite @xcite @xcite @xcite . since agents like reflexion @xcite showcase feedback-based improvement yet often lack cross-task experience, to enable an autonomous agent to improve through trial and error across various training tasks, expel @xcite innovatively defines experience as a history design a basic tetris game. add rule: bomb blocks can explode … complete the game.run( ) method. fix the bug: "numpy not imported". main.py game.py main.py game.py rule.py main.py game.py rule.py main.py game.py rule.py co-tracking co-memorizing co-reasoning develop a tetris variant where bomb blocks explode and clear nearby blocks when aligned. { : } { : } { : } { : } experience pool shortcut reasoning key and value { k : v } instructor assistant response environment file instruction task figure 1 : the framework of experiential co-learning. the co-tracking module promotes communicative rehearsals between the agents, fostering cooperative exploration and the creation of procedural trajectories for various training tasks. the co-memorizing module heuristically extracts "shortcuts" from the trajectories under external supervision, integrating these heuristic shortcuts into their collective experience pools. the co-reasoning module combines agents' collective experience pools to foster an communication of augmented instructions and solutions, improving their ability to collaboratively solve unseen tasks. parallel to these attempts, the autonomous communication among multiple agents is now a promising paradigm, signaling a shift towards multi-agent collaboration paradigm @xcite @xcite @xcite @xcite @xcite . among them, software-developing agents facilitate the breakdown of complex tasks into more manageable, finer-grained subtasks @xcite ). an instructor issues directional instructions, and an assistant provides relevant solutions, facilitating a streamlined workflow for task execution. this approach not only boosts productivity but also exhibits a degree of quality that surpasses the traditional prompt-guided paradigm in human-computer communications, reducing the need for manual involvement @xcite . ## experiential co-learning traditional multi-agent collaboration methods often neglect to accumulate experience from past tasks, leading to repetitive errors or unnecessary trial-and-error processes in similar future tasks @xcite . to remedy this, we propose experiential co-learning, illustrated in figure 1 , powered by two distinct autonomous agents and comprising three essential modules: 1) the co-tracking module establishes a rehearsal collaboration between an instructor and an assistant, focusing on tracking their cooperative "procedural trajectories" for various training tasks, showcasing clear strategies in their communicative collaboration; 2) the co-memorizing module heuristically extracts "shortcuts" from the trajectories under external supervision, integrating these heuristic shortcuts into their collective experience pools; 3) the co-reasoning module combines the two agents' collective experience pools to foster an communication of augmented instructions and solutions, improving their ability to collaboratively solve unseen tasks. ## evaluation baselines we chose different types of llmdriven software development paradigms as our baselines, which include both single-agent and multi-agent methodologies. gpt-engineer @xcite is a foundational single-agent approach in the evolving domain of llm-powered software agents; its standout feature is its exceptional proficiency in accurately comprehending input task requirements and employing one-step reasoning, which significantly enhances its efficiency in producing comprehensive software solutions at the repository level. metagpt @xcite is an innovative framework that assigns diverse roles to various llm-powered agents and incorporates standardized operating procedures to facilitate agent collaboration in software development; within each substep, solutions are generated through a single-step solution by agents with varying capabilities. chat-dev @xcite is an llm-powered agent collaborative software development framework that organizes the entire software development process into waterfall-style phases (e.g., code completion, code review, and system testing); within this framework, software agents engage in task-oriented and multi-turn communications that play a pivotal role in enhancing software development quality by iteratively providing instructions and solutions during their communications. ## conclusion recognizing the absence of a mechanism for integrating cumulative experiences from past tasks in agent collaboration, we have proposed experiential co-learning, a framework that encourages collaboration between autonomous agents. through cotracking, co-memorizing, and co-reasoning, this approach enables agents to efficiently handle unseen software development tasks by drawing on past experiences and providing mutual support. the quantitative analysis effectively showcased significant improvements in quality, leading to reduced execution times, decreased repetitive errors, and a decreased reliance on additional human intervention. we anticipate that our insights will initiate a paradigm shift in shaping the design of multi-agent collaboration, propelling agents towards achieving greater autonomy and contributing to their evolutionary growth in cooperative learning.