CSR -Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories (2025)

Abstract: The increasing complexity of computer science research projects demands more effective tools for deploying code repositories. Large Language Models (LLMs), such as Anthropic Claude and Meta Llama, have demonstrated significant advancements across various fields of computer science research, including the automation of diverse software engineering tasks. To evaluate the effectiveness of LLMs in handling complex code development tasks of research projects, particularly for NLP/CV/AI/ML/DM topics, we introduce CSR-Bench, a benchmark for Computer Science Research projects. This benchmark assesses LLMs from various aspects including accuracy, efficiency, and deployment script quality, aiming to explore their potential in conducting computer science research autonomously. We also introduce a novel framework, CSR-Agents, that utilizes multiple LLM agents to automate the deployment of GitHub code repositories of computer science research projects. Specifically, by checking instructions from markdown files and interpreting repository structures, the model generates and iteratively improves bash commands that set up the experimental environments and deploy the code to conduct research tasks. Preliminary results from CSR-Bench indicate that LLM agents can significantly enhance the workflow of repository deployment, thereby boosting developer productivity and improving the management of developmental workflows.

## introduction with the rapid evolution of large language models (llms), it has been demonstrated that llms have increased reasoning ability over the last few years, making intelligent agents based on llm possible. current agent-related applications in computer science include code writing (codex @xcite , @xcite , etc.), code base generation (metagpt @xcite , agentless @xcite and codestar @xcite ), code correction (swebench @xcite ), and more. in computer science research projects, the associated codebases grow very rapidly, and a selfconsistent codebase typically has several parts, including the instruction file (e.g., the readme file), associated code packages, and related data. the instruction file usually contains an overview of the project, including environment setup, data preparation (e.g., data download and pre-processing, model weights download and preparation), model training process, performance evaluation, and the setup of a demo project (e.g., a chatbot project that interacts with users). in computer science, prestigious conferences, including naacl, acl, iclr, neurips, cvpr, kdd, etc., encourage researchers to release source code for reproducibility of their accepted papers, and github is the top choice for maintaining codebases for most researchers. a major step of computer science research is reproducing existing work, which is essential to gain insights and propose novel methodologies. however, even for well-documented and self-consistent projects, the setup process requires manual efforts and cannot be fully automated; many steps of setting up a code repository are rather mechanical, such as installing/updating dependency packages to configure the environment, downloading data, updating the relevant script/data directories, etc., which is tedious and often time-consuming. to tackle such challenges, we propose to use llm agents for automating the deployment of code repositories of research projects, and build a benchmark, computer science research benchmark (csr-bench) for evaluating llm agents on code repository deployment tasks. we also propose a multi-agent collaborative framework, csr-agents, to automate the deployment of code repositories by coordinating multiple llm agents with different expertise and iterative improvement with provided tools. from more than ten major nlp/cv/ai/m-l/dm conferences, we collected the 100 highly rated repositories 1 , which are carefully selected from an initial candidate set that has more than 1500 top-star repositories. our selection criteria include topic diversity and self-containment 2 so that csr-bench can provide a comprehensive evaluation of llm agents on code deployment tasks including instruction generation, command execution, and self-improvement with tools. to the best of our knowledge, csr-bench is the first benchmark for the deployment of computer science research projects, providing a reference for evaluating llm agents. we note that the success of code deployment depends not only on code generation, but also on many non-coding factors including tasks like experimental environment setup, data/model preparation, correcting bash commands, searching for solutions, and etc. our proposed multi-agent framework, csr-agents, aims to achieve automation of code deployment, which can accelerate the progress for computer science research projects. our contributions are as follows: ## csr-bench in this section, we provide the problem statement for code deployment in csr-bench, introduce the code repository collection process of computer science research projects, and show their statistics. ## csr-agents in this section, we propose csr-agents, a multiagent framework that leverages llms for different tasks and achieves effective cooperation among agents for code deployment tasks. we introduce our standardized environment for code deployment, functionalities of different agents, and their cooperation workflow. ## evaluations in this section, we evaluate csr-agents in csr-bench with a wide range of popular foundation llm families, including claude foot_0 , gpt-4 @xcite , llama-3 @xcite , and mistral foot_1 . for each llm family, we experimented with different model sizes for thorough comparisons. we show the completion rate in ## conclusion the work introduces csr-bench, a benchmark designed to evaluate the capabilities of llm agents in automating the deployment of github repositories for scientific research. our study highlights that while llms show potential in handling tasks like environment setup and data preparation, they face challenges in complex tasks such as training and inference, where success rates are notably lower. our multi-agent framework, csr-agents, exemplifies how llms can collaborate to tackle deployment challenges, offering a promising approach to improving automation in software engineering. however, the results indicate that further advancements are needed to fully realize autonomous and reliable deployment processes. overall, csr-bench serves as a crucial tool for assessing and improving llm-driven deployment workflows in scientific research, paving the way for more efficient and automated computer science projects exploration.