Benchmarking Data Science Agents (2024)

Abstract: In the era of data-driven decision-making, the complexity of data analysis necessitates advanced expertise and tools of data science, presenting significant challenges even for specialists. Large Language Models (LLMs) have emerged as promising aids as data science agents, assisting humans in data analysis and processing. Yet their practical efficacy remains constrained by the varied demands of real-world applications and complicated analytical process. In this paper, we introduce DSEval – a novel evaluation paradigm, as well as a series of innovative benchmarks tailored for assessing the performance of these agents throughout the entire data science lifecycle. Incorporating a novel bootstrapped annotation method, we streamline dataset preparation, improve the evaluation coverage, and expand benchmarking comprehensiveness. Our findings uncover prevalent obstacles and provide critical insights to inform future advancements in the field.

## introduction data science has become significant, as it helps individuals and organizations make informed decisions, predict trends, and improve processes by analyzing large volumes of data. research on this topic continues to advance the field, driving innovations in machine learning, artificial intelligence, and big data analytics, thus enhancing its impact across various industries. however, data science requires extensive knowledge about analytical toolkits (e.g., numpy and pandas) and professional expertise to conduct analysis and correctly draw insights from data, which is challenging even for specialists. recent advancements in large language model (llm) @xcite and llm-powered agents @xcite have shown considerable potential in enhancing human table data/populations.json table https://datasets.org/populations.csv text population_description.md execution history predict the population of 2050. pop['pop2050'] = ... analyze relationship between land area and population. pop['pop2023'].corr( pop['landarea']) ## related works in this section, we provide a concise overview of the pertinent literature. please refer to appendix a for a detailed comparison table. evaluating code generation models. the field of llms @xcite has seen rapid progress, with many capable models that can generate high-quality natural language and codes for various domains and tasks @xcite . benchmarks for these models @xcite have also emerged. some of them are specifically designed for the data science domain, such as pandaseval / numpyeval @xcite , dsp @xcite and ds-1000 @xcite . however, what these benchmarks provided were pre-written prompts, mainly for a fair comparison of completion of in-filling abilities of llms themselves. they do not fully evaluate the skills of data science agents @xcite , such as handling natural language interactions, managing runtime sessions, and assembling prompts. our evaluation scope is larger, which includes the full lifecycle of the agents. ## dseval: evaluation paradigm for data science agents to comprehensively and reliably evaluate a data science agent, we must first identify the evaluation scope, i.e., the "lifecycle" of an agent ( § 3.1). then we propose a paradigm that monitors the full lifecycle for complete assessments ( § 3.2). ## benchmarks based on dseval building upon the dseval evaluation paradigm, we initiated the data collection and benchmark development process. we came to realize that tremendous efforts were still required to properly rephrase queries, configure sessions, and adapt validators for each query. simple format conversion proved insufficient due to limitations in existing data sources: some data sources lack real-world complexity (e.g., pandas-exercises (guipsamora, 2020)), while others address different-natured tasks (e.g., pandaseval @xcite ). to ensure the benchmark coverage with limited human efforts, we developed an "llmbootstrapping annotation process", leveraging llms to automatically create problemsets based on a minimal "idea", while incorporating human input. this process is facilitated by the dseal (dseval annotation language), which is designed to be compatible with the dseval framework and easily comprehensible to llms. in this section, we first introduce dseal ( § 4.1), followed by a detailed description of the annotation process, including a kaggle-inspired case study ( § 4.2). ## statistics and coverage based on dseval, we employed the annotation process to construct four benchmarks, detailed in knowledge points coverage. we use gpt-3.5 to summarize the data science knowledge points essential for solving each problem. as illustrated in the word cloud of figure 9 , the benchmarks focus on fundamental data processing concepts such as data transformation, aggregation, filtering, sorting, and grouping, as well as encompassing machine learning concepts like outlier detection, imbalanced dataset handling, and feature selection. problem dependencies. dseval-kaggle and dseval-exercise are two conversational benchmarks where there could be interdependences among problems. we define "session dependency" as a scenario where a variable from a previous problem is used in a subsequent problem, and "semantic dependency" as a situation where the comprehension of a later query relies on the context of a preceding query. we visualize dependency graphs for data transformation ## conclusion in this paper, we introduce dseval, an evaluation paradigm for data science agents. based on dseval, we created 4 benchmarks that cover different aspects of data science tasks, and existing agents were evaluated and analyzed on the benchmarks. we aim to establish a standard for evaluating data science agents and we welcome more contributions of problemsets from the community.