Execution-based Evaluation for Data Science Code Generation Models (2022)

Abstract: Code generation models can benefit data scientists’ productivity by automatically generating code from context and text descriptions. An important measure of the modeling progress is whether a model can generate code that can correctly execute to solve the task. However, due to the lack of an evaluation dataset that directly supports execution-based model evaluation, existing work relies on code surface form similarity metrics (e.g., BLEU, CodeBLEU) for model selection, which can be inaccurate. To remedy this, we introduce ExeDS, an evaluation dataset for execution evaluation for data science code generation tasks. ExeDS contains a set of 534 problems from Jupyter Notebooks, each consisting of code context, task description, reference program, and the desired execution output. With ExeDS, we evaluate the execution performance of five state-of-the-art code generation models that have achieved high surface-form evaluation scores. Our experiments show that models with high surface-form scores do not necessarily perform well on execution metrics, and execution-based metrics can better capture model code generation errors. All the code and data will be released upon acceptance.

## introduction code generation models @xcite have shown promising results to improve developer productivity by generating code from natural specifications @xcite . these promising results also bring interest to code generation for data scientists, who program data analysis scripts in interactive notebook environments like jupyter notebooks @xcite where programs are written interactively in loosely organized program cells (figure 1 (1)). this domain and style in [ ]: out [ ]: accuracy: 0.8410769068020736 d.@xmath0: input column & column to be predicted @xmath1.values[:,:-1] @xmath2.values[:,-1] # now create an estimator, train and predict from sklearn.naive_bayes import gaussiannb from sklearn.model_selection import cross_val_score from sklearn.metrics import * @xmath3.fit(x,y) @xmath4.predict(x) target: context: accuracy_score(predictions, y) accuracy_score(predictions, y) in [ ]: out [ ]: 0.8410769068020736 differences motivates new modeling resources, e.g., new datasets (e.g. juice @xcite ) and models (e.g. jupyt5 @xcite ) specific to data science tasks. however, we still lack a good methodology to evaluate data science (ds) code generation models. juice dataset uses the bleu @xcite and exact match (em), the prevailing metrics in code generation, to measure semantic similarity between the generated and reference code. however, these two surface-form metrics have limitations: the former neglects code syntactic features and the latter is too strict @xcite . executionbased metrics are another widely accepted line of metrics in general software engineering (se) domain, where the correctness of generated functions is determined by whether the outputs are consistent with oracle input-output data/unit tests. for ds problems, however, collecting an executable dataset and performing execution-based evaluation are challenging. ds notebooks usually do not come with their own set of unit tests and existing datasets like juice do not track the input data (such as tables) needed to run the notebooks. in addition, the outputs from notebook cells are often not "pure" values (e.g., numbers, strings, or lists) like the outputs of the functions in se problems. the ds notebook cell outputs are meant for human understanding and hence, may contain complex data structures (e.g., data frames, plots) accompanied with texts; thus simply checking whether outputs are the same is too strict to capture cases when the generated cell output is semantically correct but formatted differently from the reference (figure 1 2 ). in this paper, we provide a dataset for evaluating ds code generation models, dubbed exeds, which contains 534 data science problems built on juice @xcite . we collect exeds by first crawling data dependencies from original github repositories for the notebooks and filtering out notebooks with runtime errors; then, we curated 534 high-quality problems with sufficient code context and human-written natural language (nl) to describe tasks as the testset. with exeds, we can evaluate execution correctness by comparing outputs from generated code with desirable outputs. we experiment with 5 existing code generation models on exeds to identify their execution performance. experiment results show that (1) models with high/low surface-form scores do not necessarily generate execution-correct code -for example, while codex @xcite is low in bleu, it achieves high execution scores. (2) executionbased metrics can better capture code errors which might be helpful for model improvements. ## related works data science (ds) refers to the practice of analyzing data and acquiring insights with computational methods @xcite . with the goal to improve productivity, there are increasing interests in building systems to solve a variety of ds tasks, including code synthesis @xcite , code synthesis for visualization @xcite and data preparation @xcite , documentation @xcite , etc. in our work, we focus on code generation in ds, which generates code with code, nl and data context. code generation benchmarks are predominantly evaluated by matching code surface forms @xcite @xcite . these datasets evaluate explicit code generation with different input specifications, including natural language @xcite @xcite @xcite , unfinished code @xcite dataset # domain evaluation apps @xcite 10,000 se unit test mbpp @xcite 974 se unit test humaneval @xcite 164 se unit test dsp @xcite 1,139 ds unit test juice @xcite 1,981 ds surface form plotcoder @xcite ## ds surface form exeds (ours) 534 ds output match ## exeds for execution evaluation as mentioned in section 1, the lack of executable environments for notebooks is a key limiting factor of execution-based model evaluation for data science tasks. thus we first construct an evaluation dataset exeds and analyze its characteristics. then describe the methods for execution evaluation. dataset collection exeds contains 534 problems with code context, nl task description, reference code and target execution output, which is built upon juice @xcite with 659k publicly available python jupyter notebooks from github. we create exeds in the following steps. step1: crawling data context and execution. programming problems in ds often deal with data, which are often stored in files (e.g., .csv) and loaded by code. executing notebooks needs such data dependencies, which are not provided in juice. thus, we first crawl dependent data for notebooks from their github repositories. notebooks with inaccessible data or using libraries not present in python standard library and default ds environment are removed. with data dependency, we execute notebooks with a time limit of 1000 sec- function type % examples data statistic 40 avg., var., p-value, ... explore data value 19 min/max value, ... explore data property 10 dtype, shape, ... machine learning 16 loss, train, predict, ... simple math 6 arithmetic, ... data changing 5 sort, sample, ... data displaying 4 head/tail columns, ... step 2: dataset filtering and intent curation. as some cells are overly complex for code generation, for simplicity, we remove examples with more than 5 lines or using customized methods in target code cells. to keep diversity, we downsample cells with frequent outputs, e.g. df.summary(), df.info(), df.shape, etc. to ensure sufficient context is provided, we remove the target code whose variables are absent in the previous 5 cells. since some cells lack sufficient descriptions for the problems, for clarity, we recruit two university students with python and notebook experience to manually write nl descriptions for each example. after viewing the context, target code and output, they are asked to write descriptions containing information in two aspects: (1) the functions of target code; (2) the instructions to print outputs. we discard examples that annotators feel hard to describe. finally, we obtain 534 problems from 278 notebooks for exeds, each with code context, nl description, target code, and desired execution output. ## evaluating code generation on exeds based on exeds, we evaluate the models' performance on data science code generation and compare both surface-form code and execution output. ## evaluation results in this section, we show and analyze evaluation results to show the advantages of our exeds dataset. ## case study we give an example predicted by jupyt5 with a high bleu score but erroneous outputs in figure 2 , to show the advantages of execution evaluation for ds code generation. the example is a typical ds task which intends to explore the shape of a dataframe. but the model misunderstands the intents and generates code to display all dataframe information. although we can find the expected shapes from the output, i.e., 17135 entries and 17 columns, the output is not exactly correct. however, as the code is short while the variable name is long, which leads to a high overlap between prediction and ground truth, the generated code obtains above average bleu and codebleu scores. this example reveals the deficiency of surface form metrics to evaluate code correctness. ## conclusion in this paper, we propose an evaluation dataset to support execution correctness evaluation for data science code generation dubbed exeds, which consists of 534 typical data science problems from jupyter notebooks, each with code context, task description, target code, and desired execution output. by performing experiments with five strong code generation models on exeds, we find models that achieve high surface-form scores do not necessarily produce execution correct code, and execution-based metrics could capture more detailed code generation errors. we expect our efforts to attract more attention to code execution correctness and generating executable code.