C ode A gent: Autonomous Communicative Agents for Code Review (2024)

Abstract: Code review, which aims at ensuring the overall quality and reliability of software, is a cornerstone of software development. Unfortunately, while crucial, Code review is a labor-intensive process that the research community is looking to automate. Existing automated methods rely on single input-output generative models and thus generally struggle to emulate the collaborative nature of code review. This work introduces CodeAgent, a novel multi-agent Large Language Model (LLM) system for code review automation. CodeAgent incorporates a supervisory agent, QA-Checker, to ensure that all the agents’ contributions address the initial review question. We evaluated CodeAgent on critical code review tasks: (1) detect inconsistencies between code changes and commit messages, (2) identify vulnerability introductions, (3) validate code style adherence, and (4) suggest code revisions. The results demonstrate CodeAgent’s effectiveness, contributing to a new state-of-the-art in code review automation. Our data and code are publicly available ( https://github.com/Daniel4SE/codeagent ).

## introduction code review @xcite @xcite implements a process wherein software maintainers examine and assess code contributions to ensure quality and adherence to coding standards, and identify potential bugs or improvements. in recent literature, various approaches @xcite have been proposed to enhance the performance of code review automation. unfortunately, major approaches in the field ignore a fundamental aspect: the code review process is inherently interactive and collaborative @xcite . instead, they primarily focus on rewriting and adapting the submitted code @xcite @xcite . in this respect, an effective approach should not only address how to review the submitted code for some specific needs (e.g., vulnerability detection @xcite ). still, other non-negligible aspects of code review should also be considered, like detecting issues in code formatting or inconsistencies in code revision @xcite @xcite . however, processing multiple sub-tasks requires interactions among employees in different roles in a real code review scenario, which makes it challenging to design a model that performs code review automatically. agent-based systems are an emerging paradigm and a computational framework in which autonomous entities (aka agents) interact with each other @xcite @xcite to perform a task. agent-based approaches have been proposed to address a spectrum of software engineering tasks @xcite @xcite , moving beyond the conventional single input-output paradigm due to their exceptional ability to simulate and model complex interactions and behaviors in dynamic environments @xcite @xcite . recently, multi-agent systems have leveraged the strengths of diverse agents to simulate humanlike decision-making processes @xcite @xcite , leading to enhanced performance across various tasks @xcite @xcite . this paradigm is well-suited to the challenge of code review, where multiple reviewers, each with diverse skills and roles, collaborate to achieve a comprehensive review of the code.. this paper. drawing from the success of agentbased collaboration, we propose a multi-agentbased framework codeagent to simulate the dynamics of a collaborative team engaged in the code review process, incorporating diverse roles such as code change authors, reviewers, and decision makers. in particular, a key contribution of codeagent is that we address the challenge of prompt drifting @xcite , a common issue in multi-agent systems and chain-of-thought (cot) reasoning. this issue, characterized by conversations that stray from the main topic, highlights the need for strategies to maintain focus and coherence @xcite . this drift, often triggered by the model-inspired tangents or the randomness of large language models (llms), necessitates the integration of a supervisory agent. we employ an agent named qa-checker (for "question-answer checker") that monitors the conversation flow, ensuring that questions and responses stay relevant and aligned with the dialogue's intended objective. such an agent not only refines queries but also realigns answers to match the original intent, employing a systematic approach grounded in a mathematical framework. to evaluate the performance of codeagent, we first assess its effectiveness for typical review objectives such as detecting vulnerabilities 4.1 and validating the consistency and alignment of the code format 4.2. we then compare codeagent with state-of-the-art generic and code-specific language models like chatgpt @xcite and @xcite . finally, we assess the performance of codeagent compared to the state-of-the-art tools for code revision suggestions @xcite @xcite . since each of these related works presents a specific dataset, we also employ them toward a fair comparison. additionally, we also collect pull requests from github, featuring an extensive array of commits, messages, and comments to evaluate advanced capabilities.the experimental results reveal that codeagent significantly outperforms the state-of-the-art, achieving a 41% increase in hit rate for detecting vulnerabilities. codeagent also excels in consistency checking and format alignment, outperforming the target models. finally, codeagent showcases its robustness for code revision by presenting superior average edit progress. we summarize our contributions as follows: experimental evaluation highlights the performance of codeagent: in vulnerability detection, codeagent outperforms gpt-4 and code-bert by 3 to 7 percentage points in terms of the number of vulnerabilities detected. for format alignment, codeagent outperforms react by approximately 14% in recall for inconsistency detection. on the code revision task, codeagent surpasses the state of the art in software engineering literature, achieving an average performance improvement of about 30% in the edit progress metric @xcite . ## codeagent this section details the methodology behind our codeagent framework. we discuss tasks and definition in sec 2.1, pipeline in section 2.2, defined role cards in section 2.3, and the design of the qa-checker in sec 2.4. ## experimental setup we evaluate the performance of codeagent through various qualitative and quantitative experiments across nine programming languages, using four distinct metrics. in this section, we will discuss experimental settings, including datasets, metrics, and baselines. for more information, please see appendix c. ## experimental result analysis this section discusses the performance of codeagent in the four tasks considered for our experiments. in appendix section e, we provide further analyses: we discuss the difference in the execution time of codeagent in different languages and perform a capability analysis between codeagent and recent approaches.