Agent Laboratory: Using LLM Agents as Research Assistants (2025)

Abstract: Historically, scientific discovery has been a lengthy and costly process, demanding substantial time and resources from initial conception to final results. To accelerate scientific discovery, reduce research costs, and improve research quality, we introduce Agent Laboratory, an autonomous LLM-based framework capable of completing the entire research process. This framework accepts a human-provided research idea and progresses through three stages–literature review, experimentation, and report writing–in order to produce research, including a code repository and a research report, while enabling users to provide feedback and guidance at each stage. We deploy Agent Laboratory with various state-of-the-art LLMs and invite multiple researchers to assess its quality by participating in a survey, providing human feedback to guide the research process, and then evaluate the final paper. We found that: (1) Agent Laboratory driven by o1-preview generates the best research outcomes; (2) The generated machine learning code is able to achieve state-of-the-art performance compared to existing methods; (3) Incorporating human involvement improves the overall quality of research; (4) Agent Laboratory reduces research expenses, achieving an 84% decrease compared to previous autonomous research methods. We hope Agent Laboratory enables researchers to allocate more effort toward creative ideation rather than low-level coding and writing, ultimately accelerating scientific discovery.

## introduction scientists frequently face constraints that limit the number of research ideas they can explore at any given time, resulting in ideas being prioritized based on predicted impact. while this process helps determine which concepts are worth investing time in and how best to allocate limited resources effectively, many high quality ideas remain unex-plored. if the process of exploring ideas had less limitations, researchers would be able to investigate multiple concepts simultaneously, increasing the likelihood of scientific discovery. in an effort to achieve this, recent work has explored the capability of llms to perform research ideation and automated paper generation, where llm agents perform the role of human scientists @xcite @xcite . these frameworks generate novel research ideas, write code, conduct experiments, and create scientific papers with automated peer-review systems to evaluate the work. however, while these works demonstrate that current llms can generate ideas judged to be more novel than those produced by human experts, @xcite indicates that llms still exhibit weaknesses in feasibility and implementation details, suggesting a complementary rather than replacement role for llms in research. therefore, we aim to design an autonomous agent pipeline that can assist humans toward implementing their own research ideas. in this work, we introduce agent laboratory, an autonomous pipeline for accelerating the individual's ability to perform machine learning research. unlike previous approaches, where agents participate in their own research ideation independent of human input @xcite , agent laboratory is designed to assist human scientists in executing their own research ideas using language agents. agent laboratory takes as input a human research idea and outputs a research report and code repository produced by autonomous language agents, allowing various levels of human involvement, where feedback can be provided at a frequency based on user preference. we hope that this work takes a step toward accelerating scientific discovery in machine learning, allowing researchers to allocate more effort toward creative ideation and experiment design rather than low-level coding and writing. ## background & related work contemporary agent research leverages autoregressive large language models (llms) @xcite @xcite , typically transformer-based @xcite , which learn sequence prediction (p(x t |@xmath0; θ)) from extensive text data @xcite . to improve real-world applicability, llms are structured into agents @xcite @xcite augmented with techniques like chain-of-thought prompting @xcite , iterative refinement @xcite , self-improvement @xcite , and tool integration @xcite @xcite . these agents demonstrate efficacy in complex domains including software engineering @xcite , medicine @xcite , robotics @xcite , web tasks @xcite , and game playing @xcite . while ai has historically supported scientific discovery across fields @xcite @xcite , llms now show proficiency in specific research tasks like code generation @xcite , literature search @xcite , question-answering @xcite , paper reviewing @xcite , and experiment outcome prediction @xcite . however, the effectiveness of llm-driven research ideation @xcite @xcite ) remains debated, with mixed results on novelty and creativity @xcite @xcite @xcite , suggesting potential benefits from combining human guidance with llm workflows. recent work explores end-to-end autonomous research using llm-based systems for tasks ranging from nano-body discovery @xcite and chemical experimentation (m. @xcite to full research cycle automation including ideation, experimentation, and manuscript generation @xcite @xcite . nevertheless, persistent concerns about the feasibility and detail in llm ideation @xcite underscore the potential value of human-in-the-loop systems, motivating the co-pilot approach adopted by agent laboratory. ## results in this section, we present our main findings on the efficacy of agent laboratory to produce research. we begin our results by asking how human evaluators perceive papers generated by agent laboratory running in end-to-end autonomous mode across five topics. next, we examine human evaluation when using agent laboratory in collaborative co-pilot mode from both allowing the researcher to choose any topic they want and from our set of preselected topics. we then provide a detailed runtime analysis including cost, average time, and success rate by various models. finally, we conclude with an evaluation of the mle-solver in isolation on mle-bench, a set of real-world kaggle challenges. the details of all surveys are provided in appendix e. ## discussion agent laboratory is an llm agent system designed to assist in performing end-to-end machine learning research. functioning primarily as a copilot, it aims to accelerate scientific exploration through a human-centric approach, distinct from fully automated discovery systems. the agent laboratory workflow includes three stages literature review, experimentation, and report writing. evaluations assessed the quality of outputs generated in autonomous mode using various llm backends, employing metrics such as human ratings of experimental quality and report usefulness, alongside standard academic reviewer scores. the effectiveness of agent laboratory was also examined in its co-pilot mode, comparing it to autonomous operation and integrating researcher feedback. findings revealed performance differences between llm backends; for instance, o1-preview showed higher perceived usefulness, while o1-mini demonstrated better experimental quality. although autonomous outputs were generally rated positively, human evaluations identified shortcomings in clarity and soundness relative to standards for high-quality research, highlighting that automated reviewer scores did not consistently align with human assessments. the co-pilot mode, which incorporates human feedback, generally produced higher-quality results across most evaluation metrics and received favorable utility and usability ratings. future research directions involve longitudinal user studies and the exploration of automated workflow optimization. ## limitations while our results suggest that agent laboratory demonstrates strong performance as a research tool, we now turn to a discussion of limitations that could inform future work. while some of these are also limitations of llms themselves, others are not, and we nonetheless provide a thorough and critical discussion of our work. we hope that progress in autonomous research will address these limitations. ## ethical considerations agent laboratory offers potential to accelerate the field of machine learning research by automating time-intensive tasks and enabling researchers to focus on ideation and experimental design. however, its capabilities also bring ethical challenges that require careful consideration. the ability to autonomously generate research code, reports, and experiment plans may inadvertently lower the barriers to producing substandard or misleading scientific outputs. this could overwhelm peer review systems and jeopardize the integrity of academic discourse. furthermore, the automated processes may reflect or even amplify biases inherent in the underlying datasets or algorithms, leading to skewed outcomes in research findings. transparent disclosure of ai involvement in research outputs is important in order to mitigate such risks and maintain accountability. there are additional concerns about potential misuse of agent laboratory for unethical purposes, such as developing harmful technologies or generating content that bypasses ethical oversight. for instance, the misuse of autonomous research agents in fields like cybersecurity could lead to the automated creation of malware @xcite or in environmental studies, it may generate biased analyses that downplay climate risks or overstate the benefits of certain interventions. moreover, as the platform matures, the risk of its misuse increases if safeguards are not implemented to ensure alignment with ethical research standards @xcite . thus, while agent laboratory demonstrates immense promise for accelerating scientific discovery, there is a need for robust governance mechanisms to ensure that the underlying llms produce content that aligns with ethical principles and societal values. guohao li, hasan hammoud, hani itani, dmitrii khizbullin, and bernard ghanem. 2023. camel: communicative agents for" mind" exploration of large language model society. advances in neural information processing systems, 36:51991-52008. long li, weiwen xu, jiayan guo, ruochen zhao, xingxuan li, yuqian yuan, boqiang zhang, yuming jiang, yifei xin, ronghao dang, and 1 others. 2024a. chain of ideas: revolutionizing research via novel idea development with llm agents. arxiv preprint arxiv:2410.13185. sihang li, jin huang, jiaxi zhuang, yaorui shi, xiaochen cai, mingjun xu, xiang wang, linfeng zhang, guolin ke, and hengxing cai. 2024b. scilitllm: how to adapt llms for scientific literature understanding. arxiv preprint arxiv:2408.15545. weixin liang, yuhui zhang, hancheng cao, binglu wang, daisy yi ding, xinyu yang, kailas vodrahalli, siyu he, daniel scott smith, yian yin, and 1 others. 2024. can large language models provide useful feedback on research papers? a large-scale empirical analysis. nejm ai, 1(8):aioa2400196. xinna lin, siqi ma, junjie shan, xiaojing zhang, shell xu hu, tiannan guo, stan z li, and kaicheng yu. 2024. biokgbench: a knowledge graph checking benchmark of ai agent for biomedical science. arxiv preprint arxiv:2407.00466. yiren liu, si chen, haocong cheng, mengxia yu, xiao ran, andrew mo, yiliu tang, and yun huang. 2024. how ai processing delays foster creativity: exploring research question co-creation with an llm-based agent. in proceedings of the chi conference on human factors in computing systems, pages 1-25. chris lu, cong lu, robert tjarko lange, jakob foerster, jeff clune, and david ha. 2024a. the ai scientist: towards fully automated open-ended scientific discovery. arxiv preprint arxiv:2408.06292. chris lu, cong lu, robert tjarko lange, jakob foerster, jeff clune, and david ha. 2024b. the ai scientist: towards fully automated open-ended scientific discovery. arxiv preprint arxiv:2408.06292. xiaoliang luo, akilles rechardt, guangzhi sun, kevin k nejad, felipe yáñez, bati yilmaz, kangjoo lee, alexandra o cohen, valentina borghesani, anton pashkov, and 1 others. 2024. large language models surpass human experts in predicting neuroscience results. nature human behaviour, pages 1-11. andres m. bran, sam cox, oliver schilter, carlo baldassari, andrew d white, and philippe schwaller. 2024. augmenting large language models with chemistry tools. nature machine intelligence, pages 1-11. bodhisattwa prasad majumder, harshit surana, dhruv agarwal, bhavana dalvi mishra, abhijeetsingh meena, aryan prakhar, tirth vora, tushar khot, ashish sabharwal, and peter clark. 2024. discoverybench: towards data-driven discovery with large language models. arxiv preprint arxiv:2407.01725. benjamin s manning, kehang zhu, and john j horton. 2024. automated social science: language models as scientist and subjects. technical report, national bureau of economic research. daniel mcduff, mike schaekermann, tao tu, anil palepu, amy wang, jake garrison, karan singhal, yash sharma, shekoofeh azizi, kavita kulkarni, and 1 others. 2023. towards accurate differential diagnosis with large language models. arxiv preprint arxiv:2312.00164. amil merchant, simon batzner, samuel s schoenholz, muratahan aykol, gowoon cheon, and ekin dogus cubuk. 2023. scaling deep learning for materials discovery. nature, 624(7990):80-85. erik nijkamp, bo pang, hiroaki hayashi, lifu tu, huan wang, yingbo zhou, silvio savarese, and caiming xiong. 2022. codegen: an open large language model for code with multi-turn program synthesis. arxiv preprint arxiv:2203.13474. openai. 2022. introducing chatgpt. https://openai. com/index/chatgpt/ . blog post. openai. 2024. introducing openai o1-preview. accessed: 2024-09. vishakh padmakumar and he he. 2024. does writing with language models reduce content diversity? in the twelfth international conference on learning representations. huy nhat phan, tien n nguyen, phong x nguyen, and nghi dq bui. 2024. hyperagent: generalist software engineering agents to solve coding tasks at scale. arxiv preprint arxiv:2409.16299. ori press, andreas hochlehnert, ameya prabhu, vishaal udandarao, ofir press, and matthias bethge. 2024. citeme: can language models accurately cite scientific claims? arxiv preprint arxiv:2407.12861. pranav putta, edmund mills, naman garg, sumeet motwani, chelsea finn, divyansh garg, and rafael rafailov. 2024. agent q: advanced reasoning and learning for autonomous ai agents. arxiv preprint arxiv:2408.07199. edward o pyzer-knapp, jed w pitera, peter wj staar, seiji takeda, teodoro laino, daniel p sanders, james sexton, john r smith, and alessandro curioni. 2022. accelerating materials discovery using artificial intelligence, high performance computing and robotics. npj computational materials, 8(1):84. chen qian, yufan dang, jiahao li, wei liu, zihao xie, yifei wang, weize chen, cheng yang, xin cong, xiaoyin che, and 1 others. 2023. experiential co-learning of software-developing agents. arxiv preprint arxiv:2312.17025.