C hat D ev: Communicative Agents for Software Development (2024)

Abstract: Software development is a complex task that necessitates cooperation among multiple members with diverse skills. Numerous studies used deep learning to improve specific phases in a waterfall model, such as design, coding, and testing. However, the deep learning model in each phase requires unique designs, leading to technical inconsistencies across various phases, which results in a fragmented and ineffective development process. In this paper, we introduce ChatDev, a chat-powered software development framework in which specialized agents driven by large language models (LLMs) are guided in what to communicate (via chat chain) and how to communicate (via communicative dehallucination). These agents actively contribute to the design, coding, and testing phases through unified language-based communication, with solutions derived from their multi-turn dialogues. We found their utilization of natural language is advantageous for system design, and communicating in programming language proves helpful in debugging. This paradigm demonstrates how linguistic communication facilitates multi-agent collaboration, establishing language as a unifying bridge for autonomous task-solving among LLM agents. The code and data are available at https://github.com/OpenBMB/ChatDev.

## introduction large language models (llms) have led to substantial transformations due to their ability to effortlessly integrate extensive knowledge expressed in language @xcite , combined with their strong capacity for roleplaying within designated roles @xcite @xcite . this advancement eliminates the need for model-specific designs and delivers impressive performance in : corresponding author. diverse downstream applications. furthermore, autonomous agents @xcite have gained attention for enhancing the capabilities of llms with advanced features such as context-aware memory @xcite , multistep planning @xcite , and strategic tool using @xcite . software development is a complex task that necessitates cooperation among multiple members with diverse skills (e.g., architects, programmers, and testers) @xcite . this entails extensive communication among different roles to understand and analyze requirements through natural language, while also encompassing development and debugging using programming languages @xcite . numerous studies use deep learning to improve specific phases of the waterfall model in software development, such as design, coding, and testing @xcite @xcite . due to these technical inconsistencies, methods employed in different phases remain isolated until now. every phase, from data collection and labeling to model training and inference, requires its unique designs, leading to a fragmented and less efficient development process in the field @xcite @xcite . motivated by the expert-like potential of autonomous agents, we aim to establish language as a unifying bridge-utilizing multiple llm-powered agents with specialized roles for cooperative software development through language-based communication across different phases; solutions in different phases are derived from their multi-turn dialogues, whether dealing with text or code. nevertheless, due to the tendency of llm hallucinations @xcite , the strategy of generating software through communicative agents could lead to the non-trivial challenge of coding hallucinations, which involves the generation of source code that is incomplete, unexecutable, or inaccurate, ultimately failing to fulfill the intended requirements @xcite . the frequent occurrence of coding hallucination in turn reflects the constrained autonomy of agents in task completion, inevitably demanding additional manual intervention and thereby hindering the immediate usability and reliability of the generated software @xcite . in this paper, we propose chatdev (see figure 1 ), a ::: chat-powered software-::: development framework integrating multiple "software agents" for active involvement in three core phases of the software lifecycle: design, coding, and testing. technically, chatdev uses a chat chain to divide each phase into smaller subtasks further, enabling agents' multi-turn communications to cooperatively propose and develop solutions (e.g., creative ideas or source code). the chain-structured workflow guides agents on what to communicate, fostering cooperation and smoothly linking natural-and programming-language subtasks to propel problemsolving. additionally, to minimize coding hallucinations, chatdev includes an communicative dehallucination mechanism, enabling agents to actively request more specific details before giving direct responses. the communication pattern instructs agents on how to communicate, enabling precise information exchange for effective solution optimization while reducing coding hallucinations. we built a comprehensive dataset containing software requirement descriptions and conducted comprehensive analyses. the results indicate that chatdev notably improves the quality of software, leading to improved completeness, executability, and better consistency with requirements. further investigations reveal that natural-language communications contribute to comprehensive system design, while programming-language communications drive software optimization. in summary, the proposed paradigm demonstrates how linguistic communication facilitates multi-agent collaboration, establishing language as a unifying bridge for autonomous task-solving among llm agents. ## related work trained on vast datasets to comprehend and manipulate billions of parameters, llms have become pivotal in natural language processing due to their seamless integration of extensive knowledge @xcite @xcite @xcite @xcite @xcite @xcite @xcite . furthermore, llms have demonstrated strong role-playing abilities @xcite @xcite @xcite @xcite . recent progress, particularly in the field of autonomous agents @xcite @xcite @xcite @xcite , is largely attributed to the foundational advances in llms. these agents utilize the robust capabilities of llms, displaying remarkable skills in memory @xcite , planning @xcite and tool use @xcite @xcite @xcite , enabling them to reason in complex scenarios @xcite @xcite @xcite @xcite . software development is a multifaceted and intricate process that requires the cooperation of multiple experts from various fields @xcite @xcite @xcite , encompassing the requirement analysis and system design in natural languages @xcite @xcite , along with system development and debugging in programming languages @xcite @xcite . numerous studies employ the waterfall model, a particular software development life cycle, to segment the process into discrete phases (e.g., design, coding, testing) and apply deep learning to improve the effectiveness of certain phases @xcite @xcite @xcite @xcite . ## chatdev we introduce chatdev, a :::: chat-powered software-::: development framework that integrates multiple "software agents" with various social roles (e.g., requirements analysts, professional programmers and test engineers) collaborating in the core phases of the software life cycle, see figure 1 . technically, to facilitate cooperative communication, chatdev introduces chat chain to further break down each phase into smaller and manageable subtasks, which guides multi-turn communications between different roles to propose and validate solutions for each subtask. in addition, to alleviate unexpected hallucinations, a communicative pattern named communicative dehallucination is devised, wherein agents request more detailed information before responding directly and then continue the next round of communication based on these details. ## evaluation baselines we chose some representative llmbased software development methods as our baselines. gpt-engineer @xcite is a fundamental single-agent approach in llm-driven software agents with a precise understanding of task requirements and the application of one-step reasoning, which highlights its efficiency in generating detailed software solutions at the repository level. metagpt @xcite is an advanced framework that allocates specific roles to various llmdriven software agents and incorporates standardized operating procedures to enable multi-agent participation. in each step agents with specific roles generate solutions by adhering to static instructions predefined by human experts. datasets note that, as of now, there isn't a publicly accessible dataset containing textual descriptions of software requirements in the context of agent-driven software development. to this end, we are actively working towards developing a comprehensive dataset for software requirement descriptions, which we refer to as srdd (software requirement description dataset). drawing on previous work @xcite , we utilize existing software descriptions as initial examples, which are then further developed through a process that combines llm-based automatic generation with post-processing refinement guided by humans. as a result, this dataset includes important software categories from popular platforms such as ubuntu, google play, microsoft store, and apple store. it comprises 1,200 software task prompts that have been carefully categorized into 5 main areas: education, work, life, game, and creation. all these areas are further divided into 40 subcategories, and each subcategory contains 30 unique task prompts. metrics evaluating software is also a challenging task, especially when trying to assess it on a holistic level. under the current limitation of scarce benchmark resources, traditional function-oriented code generation metrics (e.g., pass@k), cannot seamlessly transfer to a comprehensive evaluation of entire software systems. the main reason for this is that it is often impractical to develop manual or automated test cases for various types of software, especially those involving complex interfaces, frequent user interactions, or non-deterministic feedback. as an initial strategy, we apply three fundamental and objective dimensions that reflect different aspects of coding hallucinations to evaluate the agent-generated software, and then integrate them to facilitate a more holistic evaluation: ## conclusion we have introduced chatdev, an innovative multiagent collaboration framework for software development that utilizes multiple llm-powered agents to integrate fragmented phases of the waterfall model into a cohesive communication system. it features chat chain organizing communication targets and dehallucination for resolving coding hallucinations. the results demonstrate its superiority and highlight the benefits of multi-turn communications in software optimization. we aim for the insights to advance llm agents towards increased autonomy and illuminate the profound effects of "language" and its empowering role across an even broader spectrum of applications.