Paper: Benchmarking Large Language Models as AI Research Agents (ICLR-2024)

Reviewer comment (Experiments):
I understand that the motivation was: "The tasks are chosen such that they range in various difficulties and recency to test the generalizability of the research agent and avoid data contamination." But I don't see anything to constrain the actions of the agent. If the task is to improve the baseline model, can I "improve" it by sticking it as one branch of a model that uses a much larger pretrained model? Why / why not?

Author response:
Yes we do not constrain the agent from importing pre-trained models if that gives better performance. And we agree in general that the agents mainly do well on older tasks like cifar10 or spaceship-titanic which the LM has seen during pretraining but lack of lot of creativity as seen in some newer and more challenging tasks in our benchmark such as CLRS and BabyLM which are actual open research problems. These older tasks were added to show the spread in the type of tasks agents are capable of doing and some promise in improving current agents for harder novel tasks. We view our work to give people a sense of the current state of agents on ML engineering tasks which involve a lot of open-ended decision making. 

The actions were all the actions that an actual human could have taken to solve that particular problem, and hence are open-ended. The easier tasks were added to show the capabilities of current agents as opposed to having a suite a super hard tasks where all agents have a 0 performance (aren’t able to making any meaningful progress in, such as BabyLM). Thus, the 15 tasks in the benchmark are not static and can evolve very easy as agent capabilities increase. We wanted to have tasks which had a decent spread of success rates so that current agents can be compared as opposed to all very easy or very hard tasks only which doesn’t make a lot of sense for a benchmark.
Furthermore, our novel framework ensures that adding a task to the benchmark is very seamless (very disentangled from a lot of underlying agent code) which would encourage open-source contributions, and hence would make the benchmark more wholesome and varied in the types of task to be tested. We hope our benchmark can provide valuable foundation for benchmarking more complex types of research in future work. One could simply add the research problem they are working on in our benchmark in <10 mins but it is very unlikely that the agent will show a lot of creativity to make meaningful progress in it (and hence the need for a benchmark to track this).

Reviewer comment (Novelty):
Right now, the problem I see is that the work is a collection of ~15 already existing tasks, with ~6 of them having arguable relevance to contemporary ML research. There needs to be additional work on top of this to make it a generally usable benchmark, such as constraining the actions of the agent, or defining each task more precisely, or adopting a principled definition of a research workflow and only including tasks based on that.

Author response:
Yes we do not constrain the agent from importing pre-trained models if that gives better performance. And we agree in general that the agents mainly do well on older tasks like cifar10 or spaceship-titanic which the LM has seen during pretraining but lack of lot of creativity as seen in some newer and more challenging tasks in our benchmark such as CLRS and BabyLM which are actual open research problems. These older tasks were added to show the spread in the type of tasks agents are capable of doing and some promise in improving current agents for harder novel tasks. We view our work to give people a sense of the current state of agents on ML engineering tasks which involve a lot of open-ended decision making. 

The actions were all the actions that an actual human could have taken to solve that particular problem, and hence are open-ended. The easier tasks were added to show the capabilities of current agents as opposed to having a suite a super hard tasks where all agents have a 0 performance (aren’t able to making any meaningful progress in, such as BabyLM). Thus, the 15 tasks in the benchmark are not static and can evolve very easy as agent capabilities increase. We wanted to have tasks which had a decent spread of success rates so that current agents can be compared as opposed to all very easy or very hard tasks only which doesn’t make a lot of sense for a benchmark.
Furthermore, our novel framework ensures that adding a task to the benchmark is very seamless (very disentangled from a lot of underlying agent code) which would encourage open-source contributions, and hence would make the benchmark more wholesome and varied in the types of task to be tested. We hope our benchmark can provide valuable foundation for benchmarking more complex types of research in future work. One could simply add the research problem they are working on in our benchmark in <10 mins but it is very unlikely that the agent will show a lot of creativity to make meaningful progress in it (and hence the need for a benchmark to track this).

Reviewer comment (Experiments):
Though the main point of the paper is proposing a benchmark, there is relatively little information on how these tasks were selected and how the benchmark was constructed. The benchmark consists of only 15 tasks, and while there is some detail on the attributes that made the authors select these tasks, there are not enough to convince me that this is a thorough enough benchmark in its current state. Issues such as potential data contamination, formalizing what aspects of research are being tested and how these are addressed by the chosen tasks should be explained further.

Author response:
Thanks for the suggestion for clarifying how tasks are selected. As noted in section 2.4, “the tasks are chosen such that they range in various difficulties and recency to test the generalizability of the research agent and avoid data contamination”. To avoid the chance of data contamination, we selected Kaggle competitions that were launched only in the last 2-10 months at the time. We also narrowed the types of tasks to empirical machine learning tasks to specifically test the ability of AI research agents on these well-defined evaluable tasks. The tasks were hence chosen so that current agents can show some spread in performance on the benchmark.

Reviewer comment (Evaluation):
Relatedly, the framing of this as an AI benchmark is slightly misleading. It tests LLM ability to complete research in empirical machine learning tasks, but does not test their ability to do other types of research (e.g. more qualitative or descriptive research not based purely around achieving performance increases). Though some elements of the process may remain the same, 15 tasks from machine learning are not enough to test an agent’s overall research ability.

Author response:
Thank you for your response! 

We agree that research is way complex and multi-dimensional to be captured in a bunch of tasks. That is not even our goal, we do not want to say Agent A is a better “researcher” than Agent B if it does well on our benchmark. As opposed to some ultimate comparison, we would like to explore the **potential** of these agents to do research. 

The moonshot goal indeed is to have agents come up with novel ideas and performing “scientific discovery” in it’s truest sense. We started off with that goal in mind but we realised that most current LLM based agents were unable to make a lot of progress along that direction due to lack of creativity. We then realised that experimental design is something where these agents would be very helpful in a research pipeline. 
 
> If this is the goal of the benchmark, "research" needs to be rigorously formalized, even within the context of ML tasks, to avoid giving the wrong impression about what this benchmark tests (especially since "research" is a very broad concept).

Hence, to answer this, we formalize “research”, in the context of this work as designing experiments, running them, making inferences and updating the next set of experiments to perform. This is an ability which is shown by current LLM-based agents (the agent logs on cifar-10 in the supplementary capture this very well). We again agree that this does not compass a lot of “broad research”, and that is fine as agents perfecting the above definition are also an integral part of the research cycle. We would be happy to clarify this in the paper: by explicitly writing what are the different aspects of research (hypothesis generation, experimental design, testing, and inference making) and what this benchmark specifically focusses on (the latter 3 of this cycle).

> As easy as it may be to add new tasks, I don't see how the evaluation procedure can be easily extended to other types of tasks

Yes, **we agree that this benchmark only captures the ability of agents to do empirical research where the outcome can be measured by concrete performance metrics**. We would not be able to evaluate the ability of an agent to come up with a novel theoretical proof. We would clarify this in the paper as well. This is a **tradeoff we deliberately chose**: having a much more open-ended benchmark with almost no good way of evaluating or focussing on a subset of research problems where evaluations are meaningful and progress can be tracked.

Reviewer comment (Evaluation):
The success criteria for these tasks involves models improving performance by >10%. This strongly limits the types of tasks that can be tested, as discussed in the previous point. However, it also provides limited insight into the performance of the model. One of the strengths of this paper is the analysis of the types of errors models make, but this is all done through human evaluation. For tasks involving many reasoning steps where lots can go wrong, it would be more valuable to see some breakdown of which step the model failed at or some other type of automated error analysis.

Author response:
Our success criteria of improving performance by >10% indeed only shows the broad stroke of the performance of AI research agents. However, we argue that this metric still successfully sets precedence in the evaluation of AI research agents. We have also proposed process-based evaluation, but it is one of the most challenging aspects of the agent evaluation pipeline. As mentioned in the paper, we have attempted to automate this evaluation process with LLMs such as GPT3.5, but the result is too inaccurate to be used. Thus we provide our human error analysis to shed light on the importance of such process evaluation but leave its automation to future work.

Reviewer comment (Evaluation):
Did you explore how likely it is that these tasks and their solutions appeared in training data? For example, for solved tasks, it seems likely that models would have seen high performing solutions already. Did you study this at all, either by looking at the data or comparing the similarity of model output to existing solutions?

Author response:
The models we used are claude and gpt4, for which we have no information about the training data. So we try to avoid data contamination by selecting recently lauched Kaggle data (within 2-10 months) and current research benchmarks.

Reviewer comment (Experiments):
Was human annotation performed by the authors? What was the criteria for annotation? If it was not performed by the authors, how were annotators recruited/paid?

Author response:
The error analysis is performed by ourselves. The failure modes were carefully chosen by manually inspecting agent logs while also ensuring that those error modes captured some more fundamental limitations of LLM agents. For common failure modes, we outline when is an agent run classified into that mode:
Hallucination: We observe that the agent changes part of the code which degrades performance, but still claims that it performed the said task and takes the “Final Answer” action incorrectly.
Debugging: We often notice that for our engineering type of tasks, being able to debug code was very important. Very often, agents would get stuck in trying to debug a piece of code it wrote and exceed the max steps permitted.
Bad Plan: A run has a bad plan if the initial research plan of the agent itself does not help solve the goal. 
We will include these detailed descriptions in our appendix.

Reviewer comment (Evaluation):
What aspects of the tasks chosen make them suitable for testing research in general? What kind of research do you expect an agent achieving perfect performance on your benchmark to achieve?

Author response:
The tasks we chose generally involve building an ML model on a given dataset for as high performance as possible (Note that >10% is an arbitrary threshold we choose to measure the baseline agents, but this threshold could be arbitrarily high instead as a more challenging scenario). Basic improvements could be simply achieved by applying existing knowledge on hyperparameter tuning, data augmentation, architecture changes etc. This allows even naive agents to start demonstrating performance early on. If an agent achieves perfect performance on our benchmark, the agent would have to be very good at ML engineering (which requires software eng skills + knowledge of a lot of state-of-the-art ML methods) to better performance. Furthermore, CLRS and BabyLM in our benchmark are actual open research problems and having an agent solve those would indicate some potential of actually being able to do “research”. To the extreme, the perfect AI research agent should be able to achieve groundbreaking research.

Reviewer comment (Presentation):
It would be useful to have a slightly longer description of each of the MLAgentBench tasks in Table 1. At the very least, a few words in the caption explaining the definitions of the terms used in the "task" column (e.g. node classification, improve speed, implement tool) would be very useful.

Author response:
We will include clarification of each type of task in the caption, i.e. node classification task refers to classifying node in a networks to different classes; improve speed task refers to improving the run time of a given script; implement tool refers to implementing an LLM based helper script such as performing literature review. We will also add a more detailed per task description in the appendix.

Reviewer comment (Presentation):
The future work section is too thin for a benchmark paper. Can the authors provide some more suggestions for possible future lines of agent research? For this paper to be maximally impactful they should take this opportunity to provide researchers in the field with some inspiration to drive rapid progress on this benchmark!

Author response:
Good point! We will include a separate section detailing future works including: 1) improving agent architecture, such as incorporating literature review, structured memory, better hierarchical actions and even employing multiple subagents to divide up the responsibilities; 2) incorporating more tasks from different domains, such as performing experiments in biology, chemistry, and physics; 3) improving the evaluation pipeline, such as automatic evaluation of agent reasoning process and evaluating agent in more general research scenarios.