SUPER : Evaluating Agents on Setting Up and Executing Tasks from Research Repositories (2024)

Abstract: Given that Large Language Models (LLMs) have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories. SUPER aims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub-problems derived from the expert set that focus on specific challenges (e.g., configuring a trainer), and 602 automatically generated problems for larger-scale development. We introduce various evaluation measures to assess both task success and progress, utilizing gold solutions when available or approximations otherwise. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of this task, and suggests that SUPER can serve as a valuable resource for the community to make and measure progress.

## introduction research and scientific discoveries often rely on the reproducibility of experiments conducted by other researchers and the ease with which scientists can build upon each other's work. in the context of empirical ml and nlp research, it is often crucial for researchers to be able to execute and reproduce diverse research experiments from 1 https://github.com/allenai/super-benchmark ... successfully installed datasets-2.19.2 transformers-4.40.2 ... 💭 next, i will edit the `data_utils.py` file to configure the loaded dataset. in practice, even when research code is available, running code from arbitrary repositories is often non-trivial and time-consuming @xcite . experimentation frequently requires substantial effort to set up and execute them: installing the environment, making non-trivial configuration changes, resolving outdated package dependencies, fixing bugs, and determining the correct execution commands, among other tasks. all of this requires a considerable understanding of the documentation and repository code, knowledge about fixing issues (e.g., cuda errors), as well as the ability to modify the code appropriately. these steps are especially timeconsuming for research repositories "in-the-wild", as support and documentation may not be available. in this work, we ask: can llms automate the set up and execution of tasks in research repositories? consider the research task in fig. 1 where the agent is asked to use a research code repository to train a model with a new optimizer, and evaluate its performance on a custom dataset. a successful agent would need to set up the experiment by installing dependencies, downloading the provided data, and making code changes to load it (first three cells in the figure), then execute the training script while responding to unexpected issues such as an incompatible dependency (fourth and fifth cell), and finally report the result metrics (last cell). while llm-based agents have recently been used to produce execution commands from popular research repositories @xcite , execute popular ml repositories @xcite , or resolve repository issues @xcite , no existing benchmark evaluates agents on the common problem faced by many researchers: both setting up and executing experiments using research repositories in-the-wild, i.e., less popular repositories that are not typically well-documented or maintained, which make experiments harder to configure and execute. as a recent study shows @xcite , both novice and advanced researchers find the challenge of "setting up the code base" to be the most difficult part of reproducing experiments. to encourage research on this problem, we introduce super (setting up and executing tasks from research repositories), a benchmark focusing on lower-profile research repositories. super consists of three distinct problem sets. the expert set contains 45 manually curated problems solved by experts. the masked set includes 152 sub-problems derived from the expert set through our proposed "code masking" mechanism, which removes parts of the expert-written code to create diverse sub-problems targeting challenges like dependency installation, data configuration, hyperparameter setup, runtime error resolution, script execution and more. finally, the auto set contains 604 automatically generated tasks with an even more diverse set of repositories, offering potential use in development, fine-tuning, or training with environment feedback. to evaluate agents on the expert and masked sets, for which we have gold solutions, we compare their answers (e.g., metrics to be reported) to the gold solutions. to allow for partial credit, we also measure the progress of the agents by checking if they reach specific 'landmark' states in their solutions, such as completing a training stage. for the automatically generated problems, for which we have no gold solutions, we simply check if a key script (e.g., the training or evaluation script) was run successfully without exceptions, which we found to be an effective approximate metric. we evaluate both proprietary and open-source llms on super as the underlying models of strong baseline agents with access to file-editing tools. we find that agents struggle to correctly solve many of the problems, with the strongest agent solving only 46.1% of the masked sub-problems. these agents are even further away from solving entire research tasks, completing correctly only 16.3% of the end-to-end expert tasks. open-source models substantially lag behind on both the sub-problems and end-to-end tasks. moreover, we find that the ranking of the agents and models on the auto set is mostly the same as it is on the curated sets, suggesting its potential usefulness for development. our analysis of model trajectories reveals that agents are better at resolving well-specified subproblems, such as solving exceptions, bugs, and other issues, than tasks requiring repository and file exploration to understand code structure. these results underscore many of the core challenges facing llm-based experiment execution systems, which our benchmark aims to help advance. ## related work coding benchmarks: while early code benchmarks @xcite @xcite mainly focused on synthesizing simple functions from descriptions, recent benchmarks have shifted to more complex competitive programming problems @xcite @xcite and evaluating proficiency with popular data science libraries @xcite . unlike these, we follow the recent trend on evaluating llms in more natural programming scenarios, such as programming with external tools and apis @xcite @xcite , code editing and debugging @xcite @xcite , resolving github issues @xcite and understanding and coding within a repository context @xcite @xcite . ## benchmark construction in this section we describe the process of building the super benchmark. the super benchmark consists of 3 sets (see environment setup. running research-oriented repositories often necessitates both being able to run system shell commands (e.g. to install dependencies and run scripts) and stateful python commands. previous work and environments typically support only one of these (e.g., only system shell commands @xcite or only python commands @xcite ). instead, we build an environment that allows running both of these commands with a jupyter notebook as engine. in this setup, each execution code is equivalent to running a notebook cell, which contains python code and/or bash commands, and where state is reserved between cell executions (e.g., each cell can use any of the previously defined python variables). the execution of each cell returns an observation string. ## conclusion we introduce super, a benchmark for evaluating llm-based agents on tasks from low-profile research repositories. our empirical results demonstrate the difficulty of the benchmark, even for leading llms like gpt-4o. super highlights key challenges in building autonomous execution agents, such as repository reasoning and code editing, and we hope it will drive measurable progress in tackling this important problem.