BLADE : Benchmarking Language Model Agents for Data-Driven Science (2024)

Abstract: Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents’ multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though language models possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents’ analysis approaches.

## introduction scientific data continues to accumulate rapidly, driven by advancements in scientific instrumentation and the digitization of information. however, practicing data-driven science (i.e., answering research questions from data) remains difficult, requiring rigorous methodologies, an understanding * these authors contributed equally to this work. language model (lm)-based agents @xcite @xcite , pre-trained on web-scale data and equipped with memory and tool usage capabilities @xcite , have the potential to conduct and support data-driven science. they can reason about and interact with heterogeneous data representing subjects, objects, and processes of study in the "external" world @xcite . however, to facilitate their progress, we need a reliable method to evaluate and measure their performance. recent benchmarks have enabled progress. however, they focus on either (1) data analysis execution with straightforward tasks containing a single, final, easily evaluated answer (e.g., calculate the mean and standard deviation of the "mar.2019" column @xcite @xcite ) or (2) tasks for machine learning (ml) (e.g., improve the accuracy of an ml model @xcite @xcite ). for scientific analyses, these tasks require limited integration of external knowledge, limited understanding of data semantics, and limited grounding in external scientific knowledge. in addition, these benchmarks evaluate only on single metrics, such as ml model accuracy or completion rate. however, in the process of data-driven scientific discovery, the many intermediary decisions in a multi-step analysis are themselves critical to identify, meaningfully assess, and differentiate in order to improve agent performance. evaluating agent performance on open-ended data-driven analyses, especially automatically, poses specific challenges. first, the natural flexibility in making analysis decisions @xcite @xcite makes generated analysis def transform return (df: pd.dataframe) pd.dataframe: to build blade, we collected a set of actual research questions and datasets (fig. 1 .1) from research papers, crowd-sourced analysis studies, and statistics textbooks (sec. 3). then, inspired by prior crowd-sourced analysis studies @xcite , we recruited expert data analysts and collected high-quality data analyses (fig. 1 .2) through a crowd-sourced anal-ysis for each research question (i.e., multiple analysts independently performing a single analysis). to ensure our benchmark captured a broad variety of defensible analysis approaches, we asked analysts to validate alternative decisions from their peers and lm-generated decisions seeded by analysts' own decisions. for this process, we also collected negative examples of "unjustifiable" decisions to use when testing agents' ability to discern justifiable ones. we then combined all unique decisions to form the ground truth (fig. 1 .3). next, based on studies outlining decision steps in the data analysis process @xcite @xcite , we formulated tasks. these tasks tested the discernment and formulation of analytical decisions that reflect multiple levels of abstraction, ranging from executable code implementing data transformations to higherlevel planning of conceptual variables requiring external scientific knowledge (sec. 4). finally, given our data and task, we developed representations and matching criteria for different types of analysis decisions. we also developed corresponding computational methods to enable automatic evaluation of agent responses (sec. 5). overall, blade contains 188 multiple choice and 536 ground truth analysis decisions encompassing multiple justifiable analysis approaches across 12 real-world datasets and research questions. to illustrate its utility and assess benchmark performance, we evaluate different lms and a standard react agent @xcite that interacts with a sandbox notebook environment (sec. 6). in our results (sec. 7), we find most lms are decent at discerning decisions and generating nonempty executable analyses. however, these analysis are basic and lack diversity. in particular, lm's coverage of the ground truth for forming statistical models with conceptual variables is below 13%, and for operationalizing variables, it is below 27% (fig. 4 ). the baseline react agent shows a consistent improvement in coverage, though with plenty of room for improvement. our main contributions are: (1) a rigorously expert-annotated benchmark and the first of its kind to evaluate agents' analytical decisions on open-ended scientific research questions; (2) an evaluation framework to automatically assess agent responses on fine-grained aspects of the analysis; and (3) results on various lms and a react agent indicating their current strengths and limitations. our work takes the first step in evaluating agents for open-ended data-driven scientific discovery, advancing our understanding of their capabilities to collect data, generate hypotheses, conduct analyses, and interpret results to form valid, justifiable scientific conclusions. to support further research and development, we open source our benchmark and evaluation framework 1 . ## benchmark requirements our benchmark evaluates agents on answering open-ended, data-driven scientific questions, advancing current efforts that execute analysis code based on precise single-answer instructions. existing benchmarks are limited in their ability to assess 1 https://github.com/behavioral-data/blade agent decision-making during analysis and often do not capture the full scope of their approaches. our benchmark addresses these limitations by focusing on the following key requirements ( we maintain that the ideal benchmark would evaluate an agent's abilities to (1) comprehend data semantics, understanding the semantic relationships between variables and what the data represents relative to the external world, (2) integrate domain knowledge, i.e., findings from related literature and an understanding of a "world model", (3) conduct multi-step reasoning and planning at different levels of abstraction, i.e., high level planning vs. lower level code execution, given domain knowledge, an understanding of the data, and execution outputs, and (4) differentiate justifiable decisions with firm theoretical or statistical support (simonsohn et al., 2020) from unjustifiable ones. additionally, the benchmark evaluation should be (5) automatic, requiring no human intervention, (6) decision-based, with the ground truth reflecting the intermediary decisions, and (7) flexible to decision input, being aware of multiple ways to specify the same decision. requirements 1 through 4 inform our data collection process (sec. 3) and task formulation (sec. 4). requirements 5 through 7 inform our evaluation procedure (sec. 5). ultimately, we assess whether agents can plan, develop, and execute a justifiable analysis to answer a real-world research question. ## benchmark data collection we now describe our data collection process for research questions (rqs), data, and ground-truth analyses. rqs and data. we selected scientific-grade datasets and rqs directly from scientific publications, particularly those studied in meta-analysis papers @xcite @xcite and reproduced in statistics textbooks @xcite . we chose these sources because they provide a multitude of complex analyses, and relevant properties that make analyses non-trivial and revealing of statistical knowledge. ## benchmark tasks we want the benchmark tasks to represent decisions that are vital to the analysis and to evaluate the key skills needed to conduct data-driven science (i.e., requirements 1-4 in section 2). we draw from prior studying the scientific analysis process @xcite @xcite and focus on agents' ability to discern (sec. 4.1) and make (sec. 4.2) planning decisions, i.e., those requiring a process of reasoning about and then synthesizing the data, scientific domain, and statistical knowledge. specifically we test the following decisions: ## flexible automatic evaluation to quantitatively measure the quality of agentgenerated analyses (i.e., the agent-generated artifacts) in a way that is automatic, decision-based, and flexible to decision input (requirements 5-7 in sec. 2), we need concrete representations of analysis decisions and associated matching criteria. we now discuss the representation and matching procedure for each artifact in an agent's submission. ## experiments to establish a baseline and evaluate the performance of lm-based agents on blade, we selected the following models: gpt-3.5 turbo, gpt-4o (openai, 2024), gemini 1.5 pro @xcite , and claude 3.5 sonnet @xcite to represent closed-source general-purpose lms; llama3 8b, llama3 70b (meta, 2024), and mixtral-8x22b @xcite experiment settings. for the multiple choice questions (sec. 4.1, task 1) we evaluate each lm with a temperature of 0. to generate an end-toend analysis (sec. 4.2, task 2), we evaluate lms in one turn with a one-shot example (prompt in fig. 16 ). in addition, we develop an agent (also with an example demonstration), based on the re-act framework @xcite , that interacts with a computational notebook environment containing the data, reflects on observations from executing the code, and generates next-step actions. we evaluate the react agent on mixtral-8x22b, gpt-3.5 turbo, gpt-4o, gemini 1.5 pro, and claude 3.5 sonnet. appendix a.5 contains additional details on the setup of the agent and the choice of lms. for each lm and setting (i.e., one turn vs. react agent), to encourage diversity we set the temperature to 0.8 and record a total of 40 runs for the one-turn setting and 20 runs for the agent setting to consider for computational budget. for all lms used to facilitate the evaluation (i.e., conversion and semantic matching), we use gpt-4o with a temperature of 0. appendix a.6 includes all prompts for the baselines and lm-aided evaluation. evaluation metrics. for the multiple choice tasks (task 1), we measure agents on accuracy. for the generation tasks (task 2), to measure an agent's ability to both generate justifiable analyses and capture the breadth of justifiable approaches, we calculate an adapted f1-score for each type of analysis decision (conceptual variables, transformation, and statistical model). the f1-score takes the harmonic mean of average precision across runs and coverage@k. the former quantifies how well an agent's response matched with the ground truth while the latter evaluates how comprehensive agents are in generating justifiable alternative analyses. in our experiments, we report average precision across all runs and coverage for @xmath0. appendix a.7 contains the full details of our evaluation metrics. ## results we report the performance of lms on mcqs (task 1) in figure 3 and the results of lms and our react agent for analysis generation (task 2) in in addition, we find that the best general lms (i.e., gemini-1.5 pro, mixtral-8x22b, claude-3.5-sonnet and gpt-4o) perform well on the mcqs (fig. 3 ). they can discern the obvious transformations for a given conceptual variable. in contrast, code-specific lms, like codellama and deepseek-coder, struggle to identify the correct decision. ## related work our work broadly relates to agent benchmarks for data science and lm agents in science. benchmarks for data science. many benchmarks @xcite @xcite assess agents' data science code execution but are limited in measuring complex reasoning and external knowledge integration needed for scientific analyses. other works focus on improving specific metrics in machine learning tasks @xcite without evaluat-ing intermediate decisions (examples in ## conclusion we introduce blade, a benchmark designed to advance the development of lm agents for datadriven scientific tasks. we collected a dataset of research questions and data tables, along with ground truth analyses from expert annotators. to support an automatic, decision-based, and input-flexible evaluation, we devised representations of core analysis decisions and developed corresponding matching algorithms. although current generations of lms can generate some analyses matching the ground truth sometimes, we find that these analyses are limited in complexity and lack diversity. 13944 our work is not without limitations. first, blade does not evaluate an agent's ability to interpret the results of data analyses as part of the end-to-end data analysis process. understanding and interpreting model results is vital but can be difficult to capture cleanly since it may require analysts' subjective interpretation of the problem with respect to model results. we leave this important dimension for future work. in addition, though our work elucidates the decisions an agent may make, we do not explicitly evaluate the exploratory parts of an analysis. further, we assume that the dataset is contained in a single, potentially extremely large table. this may not be common of all research datasets, but we believe this factor does not significantly reduce the scope of blade since joining tables to enable downstream analyses is a task that lms already commonly perform @xcite @xcite . finally, some components of our evaluation rely on lms (e.g., conversion of code to discrete transforms, semantically matching model, and conceptual variable), which are known to hallucinate. therefore, we made multiple efforts to validate each component and do not think that hallucination significantly impacts our ability to effectively and automatically evaluate agents. we also open source these evaluation modules so that researchers can build upon them to improve our evaluation.