DA -Code: Agent Data Science Code Generation Benchmark for Large Language Models (2024)

Abstract: We introduce DA-Code, a code generation benchmark specifically designed to assess LLMs on agent-based data science tasks. This benchmark features three core elements: First, the tasks within DA-Code are inherently challenging, setting them apart from traditional code generation tasks and demanding advanced coding skills in grounding and planning. Second, examples in DA-Code are all based on real and diverse data, covering a wide range of complex data wrangling and analytics tasks. Third, to solve the tasks, the models must utilize complex data science programming languages, including Python and SQL, to perform intricate data processing and derive the answers. We set up the benchmark in a controllable and executable environment that aligns with real-world data analysis scenarios and is scalable. The annotators meticulously designed the evaluation suite to ensure the accuracy and robustness of the evaluation. We developed the DA-Agent baseline. Experiments show that although the baseline performs better than other existing frameworks, using the current best LLMs achieves only 30.5% accuracy, leaving ample room for improvement. We release our benchmark at [link](https://github.com/yiyihum/dabench)

## introduction data science is pivotal in extracting insights from data @xcite , fundamentally shaping decision-making and knowledge discovery. traditionally, this field has required high proficiency in programming and specialized knowledge, which poses significant barriers to non-experts. however, the rapid advancement of large language models (llms) @xcite @xcite has greatly enhanced their capabilities in code generation, grounding, and planning. this raises an intriguing question: can llms become 13487 autonomous data scientists, capable of independent decision-making and solving agent data science problems? we define the agent data science task as one that encompasses challenging task, diverse data source, and complex solution, as shown in figure 1 . first, a code agent task is designed to explore data and utilize programming techniques to tackle challenging objectives, rather than simply translating explicit natural language instructions into code. unlike previous benchmarks like ds-1000 @xcite and arcade @xcite , which focus on natural language grounding @xcite and transforming instructions into executable code, our approach aligns more closely with real-world programming scenarios. second, the data source is diverse, encompassing a variety of information and data from real programming scenarios-beyond just a notebook environment @xcite or code-completion tasks @xcite . it includes different data types such as databases, spreadsheets, documents, code, and more. moreover, these resources can be replete with distractions and extraneous information. third, for task solutions, data analysts must engage in a step-by-step thinking and reasoning process while programming. this involves inspecting data, writing minimal code, and continuous debugging. additionally, they utilize python, sql, and bash, which are commonly used programming languages in data science, to complete the tasks. to achieve this goal, we introduce da-code, a benchmark for evaluating llm data analysis ability, with carefully defined task scenarios. da-code contains 500 complex task examples, originating from real, challenging data analysis tasks, encompassing three main categories: data wrangling (dw), machine learning (ml) and exploratory data analysis (eda). it covers the entire data analysis pipeline. data wrangling includes a variety of tasks such as data loading, data cleaning, and data merging, specifically targeting raw data in files and databases. eda aims to gain insights and analysis using the given information and resources. it includes a wide variety of data analysis tasks using programming languages such as sql and python to get insights from data. for ml tasks, how the model governs the entire ml pipeline is a promising research direction. each example in da-code is meticulously designed according to the standards of the code agent task mentioned above, ensuring high quality and complexity. we meticu-lously design an evaluation suite for each example, and conduct red teaming experiments to validate the robustness of the evaluations. da-code is inspired by the definitions used in intercode @xcite . we create an interactive sandbox environment and implement da-code within this setting. this setup allows llms/agents to explore the environment and engage in autonomous reasoning to complete tasks. this enables researchers to conduct their explorations in this user-friendly interactive environment. based on this setting, we develop a code agent baseline framework. this is capable of writing python and sql code and can interact with command lines, databases, and other interfaces. as shown in figure 1 , for this eda task, the agent needs to explore all the resources provided to it, decide which files to use, write the code, and autonomously complete the task step by step. we evaluate multiple state-of-the-art language models on da-code, and the experimental results indicate that achieving high scores on these tasks is challenging. for llms and llm-agents, autonomously completing real and complex data science tasks continues to be a significant challenge. overall, da-code represents an agent data science benchmark with challenging real-world settings, providing valuable data resources for current llmagents aspiring to become data scientists. ## data science agent task in this section, we introduce the data science agent task and categories of tasks for da-code. ## da-code in this section, we describe the statistical information and construction pipeline of da-code. ## da-agent to effectively address the challenges of the da-code benchmark, we develop an llm-based agent, depicted in figure 1 , which operates within a versatile and robust framework designed for dynamic interaction and execution with the environment. ## experiment and analysis in this section, we present the experimental results and analysis of several llms evaluated using our da-agent baseline on da-code benchmark. ## related work code generation benchmark as models become increasingly capable, researchers start to build increasingly difficult and general code generation benchmarks. most coding benchmarks (e.g. sql-spider @xcite ; bash -nl2bash @xcite ; python -humaneval @xcite ; execution-s3eval @xcite ; competition code generation @xcite ) frame the coding problem as a sequence-to-sequence problem (from instruction to code). ds-1000 @xcite and arcade @xcite are pioneering works that collected high-quality examples from communities and proposed corresponding data science to define code generation tasks. intercode @xcite was the first to propose defining code generation tasks in an interactive environment. swe-bench @xcite proposed numerous repository-level tasks, while mlagentbench @xcite defined auto machine learning tasks in an interactive environment. some researchers have also proposed benchmarks @xcite to explore the model's multimodal capabilities in data science and engineering. ml-bench @xcite focuses on machine learning bash scripts generation. da-bench @xcite also evaluate agents on data analysis tasks, however its task setting is not fully agentic and advanced. our work focuses on data science, involving real and challenging tasks that cover the full pipeline. code agent method the value of generative code models and interactive problem solving has motivated a recent proliferation of work to augment reasoning capabilities' of existing language models @xcite @xcite @xcite . many agent methods have been proposed to solve code generation tasks. direct interaction of agents with the linux command line can cause many issues @xcite . several works @xcite @xcite have designed special actions to standardize agent operations. opendevin (open-devin team, 2024) is a new agent for solving coding tasks. it is based on codeact @xcite , a framework that consolidates llm agents' actions into a unified code action space. we introduce da-agent, a competitive agent framework designed for solving coding tasks, based on the da-code environment. ## conclusion we introduce da-code, a challenging benchmark designed for agent-based code generation tasks in data science. this benchmark comprises 500 examples characterized by diverse data sources, complex task settings, and an executable environment. we develop da-agent, a robust llm-agent baseline, to tackle this challenging benchmark. however, experiments reveal that even the most advanced llms perform poorly on da-code, achieving only about a 30.5% score. future work will focus on 1) developing a more sophisticated data agent framework, 2) training more effective agents based on open-source llms.