C ode S cientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation (2025)

Abstract: Despite the surge of interest in autonomous scientific discovery (ASD) of software artifacts (e.g., improved ML algorithms), current ASD systems face two key limitations: (1) they largely explore variants of existing codebases or similarly constrained design spaces, and (2) they produce large volumes of research artifacts (such as automatically generated papers and code) that are typically evaluated using conference-style paper review with limited evaluation of code. In this work we introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain (like prompting a language model). We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments, with the system returning 19 discoveries, 6 of which were judged as being both at least minimally sound and incrementally novel after a multi-faceted evaluation beyond that typically conducted in prior work, including external (conference-style) review, code review, and replication attempts. Moreover, the discoveries span new tasks, agents, metrics, and data, suggesting a qualitative shift from benchmark optimization to broader discoveries.

## introduction automated scientific discovery (asd) systems have already had success in targeted domains like protein folding @xcite , antibiotic discovery @xcite , and model optimization @xcite , by using custom problem-specific systems that search (large) hand-crafted search spaces. recently, language models (lms) are fueling explorations into more problem-general discovery systems capable of the full research pipeline of ideation, planning, (codebased) experimentation, and experiment analysis, with numerous impressive systems appearing recently, including ai scientist @xcite , aigs @xcite , agentlab @xcite . impressive as these systems are, each makes simplifications to reduce complexity, such as restricting search to variants of prewritten code, using a dsl for experiments, or working on restricted problems. in this work we introduce codescientist, an asd system built with novel innovations for ideation and experiment execution -incorporating genetic search @xcite over combinations of literature and code -that we hypothesize will increase the diversity of the discoveries the system makes. we run our system at scale (hundreds of experiments) in the broad domain of agents and virtual environments, and find that of the 19 discoveries our system suggests, 6 appear to meet minimum thresholds for scientific soundness and incremental novelty after domain expert review. moreover, the discoveries our system produces qualitatively appear diverse, and span creating new tasks, agents, metrics, data, and challenging assumptions, which builds-upon (while broadening) the scope of the impressive accomplishments of existing systems that focus on improving model performance on standardized ml benchmarks (e.g. @xcite @xcite . orthogonally, asd research itself is faced with significant methodological challenges that limit progress. the first, evaluating discoveries, is complex in that scientific discovery is (by definition) at the edge of human knowledge, and as such, tasks with gold annotated outcomes (e.g., @xcite are generally unavailable. alternative assessments are needed, for example the way humans evaluate research -namely, rigorous (and expensive) manual review, or the use of proxy def prompt_llm(prompt:str, model:str, temp:float, max_tokens:int): ... metrics @xcite that are progressively improving their agreement with human ratings @xcite . the second challenge, variability, acknowledges that even when using low temperatures, workflows built upon language models rarely produce the same output -especially when each step autoregressively depends upon the output of the previous step. across successive runs, the specific ideas that an asd system generates are different, and (as we show in this work), the code it implements for those ideas (and whether it succeeds or fails) is also highly variable -making comparisons across systems (or ablations of a single system) costly, methodologically challenging, and uncommon in the literature. at the same time, with asd still in its infancy, scientifically sound and novel discoveries are relatively rare -and (like other code-generation tasks like swe-bench), systems are faced with high variability and low absolute success, but (unlike swe-bench) without the benefit of trustworthy automatic evaluation. in spite of these methodological challenges, we show that codescientist is capable of generating a number of candidate discoveries through manual (domainexpert) evaluation, and that this set of discoveries qualitatively captures a series of diverse research ideas that expand the scope of accomplishments of existing systems. ## related work ideating and executing research: the suite of recent asd systems primarily differ in their methods for ideation, experiment construction and execution, as well as their problem domain (e.g. chemistry, biology, ai) -with examples of systems in the ai domain provided in evaluating discoveries: automatic evaluation of discoveries is possible in a limited subset of domains and tasks -such as in materials science, where molecular simulators can directly evaluate the veracity of discoveries @xcite . similarly, broader science-themed games can include instrumentation for measuring discoveries @xcite , or the scope of discoveries can be reduced to some measurable quantity, such as increasing model performance on a benchmark task @xcite . manual evaluation of research artifacts (such as code) is costly to perform, and previously limited in scope (e.g., statistical analysis code in @xcite . others have worked to develop automated proxy metrics that use an llmas-a-judge paradigm to evaluate research papers on likert scales @xcite similar to conference review @xcite @xcite , though due to the difficulty of this task, these currently may have limited agreement with human judgements @xcite , may be fooled by superficial niceities (fluency, layout, etc.), and (as in the case of ai scientist), may reject all the discoveries. 2 as we show in section 4, faithfulness is a critical factor to automated evaluationwhat our system says it has discovered in its papers can strongly differ from what it actually implements in code, and we argue that conference-style likert assessments of papers (without code) may not fully assess the faithfulness of science-as-code discoveries. ## system overview an overview of the codescientist system is shown in figure 1 , with the 5 major steps of the workflow (ideation, planning, experiment building, reporting, and meta-analysis) described below. except where described otherwise, the steps of the workflow are implemented as prompts to a language model, with example prompts and additional implementation details provided in appendix f. ## discovery experiments candidate discoveries made by codescientist are described in section 4.1, with challenges and failure modes described in section 5. a description of the experiment setup is provided below. foot_4 paper and code example corpus: we assembled a corpus of 57 recent papers broadly in the area of agent architectures and virtual environments. paired with this, we assembled 10 example code snippets foot_5 for performing basic tasks in this domain, including calling language models (openai et al., 2024), building a react agent @xcite , plotting @xcite , robust inferential statistics for comparing models @xcite , using common knowledge graphs @xcite , and several benchmark environments including textworldexpress (jansen and cote, 2023), a high-performance simulator that reimplements common benchmarks. these code examples serve not only as vetted code examples of common artifacts described in the paper corpus, but also as demonstrations of the nuances required to implement code within the sandbox environment. ## discussion we outline challenges and common failure cases below, to highlight where targeted efforts might improve discovery quality. idea diversity: while the candidate discoveries demonstrate that the ideator can generate diverse ideas that span designing new agents, tasks, metrics, methods, and benchmarks, most of the ideas that are generated are highly similar, or mechanical variations of one another (e.g. apply method x to benchmark y ). @xcite observed that llmideators quickly saturate the number of unique ideas they generate. we found that using their method of uniqueness filtering (using cosine similarity over embeddings) still generated many duplicates, and had to use manual filtering to select a diverse, fairly non-overlapping subset to explore. experiment builder failures: currently, 59% of the experiments the planner designs are not able to be successfully implemented by the experiment builder. the distribution of executor errors is shown in ## conclusion we present codescientist, an end-to-end semiautomated discovery system with a novel ideator that performs genetic search jointly on combinations of literature and codeblocks, before building, running, and analyzing these software artifacts in experiments. ideating from literature and common codeblocks in the domain of agents and environments, codescientist identified 19 potential discoveries, 6 of which were judged as meeting minimum thresholds for scientific soundness and incremental novelty after both external conference-style review, and an internal code review. a qualitative review of the discoveries suggests that they broaden the scope of ideas explored by other systems, with candidate discoveries spanning novel tasks, agents, metrics, and data. we provide an analysis of common failure modes, and release our system as opensource to facilitate future research in automated scientific discovery.