NBDESCRIB : A Dataset for Text Description Generation from Tables and Code in J upyter Notebooks with Guidelines (2025)

Abstract: Generating cell-level descriptions for Jupyter Notebooks, which is a major resource consisting of codes, tables, and descriptions, has been attracting increasing research attention. However, existing methods for Jupyter Notebooks mostly focus on generating descriptions from code snippets or table outputs independently. On the other side, descriptions should be personalized as users have different purposes in different scenarios while previous work ignored this situation during description generation. In this work, we formulate a new task, personalized description generation with code, tables,and user-written guidelines in Jupyter Notebooks. To evaluate this new task, we collect and propose a benchmark, namely NBDESCRIB: , containing code, tables, and user-written guidelines as inputs and personalized descriptions as targets. Extensive experiments show that while existing models of text generation are able to generate fluent and readable descriptions, they still struggle to produce factually correct descriptions without user-written guidelines. CodeT5 achieved the highest scores in Orientation (1.27) and Correctness (-0.43) among foundation models in human evaluation, while the ground truth scored higher in Orientation (1.45) and Correctness (1.19). Common error patterns involve misalignment with guidelines, incorrect variable values, omission of im-031 portant code information, and reasoning errors.032 Moreover, ablation studies show that adding guidelines significantly enhances performance, both qualitatively and quantitatively.

## introduction in recent years, computational notebooks like jupyter have become popular among data scientists and machine learning researchers for documenting ideas, writing code, and visualizing results in a single document @xcite . description in a notebook provides a rich medium for users to record what the code does and the rea-soning behind it. description is found essential for data scientists to share or reuse code @xcite . however, research has shown that many data scientists still neglect to write appropriate descriptions for their code in notebooks, especially for code output (i.e. table), as they feel writing description will slow down their coding process @xcite . @xcite reported that among one million computational notebooks on github, 25% of them have no comment. besides, unlike other integrated development environments (ides) such as visual studio, data scientists working in computational notebooks often write concise descriptions, typically less than 100 words @xcite . this brevity poses challenges for large language models (llms), such as gpt and llama, limiting their ability to generate accurate markdown descriptions @xcite . existing literature has explored techniques to generate descriptions for code snippets or table outputs independently @xcite @xcite @xcite @xcite @xcite @xcite , achieving satisfying performance. however, in real-world applications, documentations are always related to the corresponding code and table output, at the same time. for example, ground truth descriptions require summarization from the code and table output simultaneously, as shown in section 1. moreover, while the same code-table output pair could serve different purposes under different user requirements, existing work @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite only focuses table passengerid pclass sex age fare cabin embarked y_pred y_scores 0 892 3 male 34.5 7.8292 nan q 0 0.108682 1 893 3 female 47.0 7.0000 nan s 1 0.516569 2 894 2 male 62.0 9.6875 nan q 0 0.130859 3 895 3 male 27.0 8.6625 nan s 0 0.116366 4 896 3 female 22.0 12.2875 nan s 1 0.553491 guideline description value the 'y_pred' column in the output table represents the binary prediction (0 or 1) generated by the 'model_consumer' for each passenger，such as passenger 893's prediction being 1 (indicating survival) goal builds a prediction pipeline and and applies it to a 'df_test' dataset,generating predictions and scores for further analysis association it is clear that fare has nothing to do with age in the titanic dataset outlier cabin feature has some missing values in the test set aggregation there are two classes for embarked feature reason analyze the data like passenger class, gender, age, and fare to generate predictions and scores with prediction pipeline code only table only it sets up a machine learning prediction pipeline with initial preprocessing and model consumption stages and applies it to test data from a csv file the table lists passenger data with features like class, sex, age, fare, and predictions with scores for a titanic dataset. previous work our work(guideline + code + table) on description generation for either code or table without considering various user requirements. as shown in figure 1 , under different user-written guidelines, the focus of different ground-truth descriptions could differ from each other. to address the aforementioned issues and step forward in faithful and personalized description generation, in this paper, we propose a challenging user-written guideline-based text generation task focusing on the table and code description generation (tcdg) for jupyter notebooks. given codes and output tables, the goal of tcdg is to produce a concise description under different user-written guidelines. the guidelines will be of a given category corresponding to the type of target text as shown in moreover, we construct the first benchmark (nbdescrib) for tcdg that contains 3,924 processed code-table-description pairs extracted from highly-ranked notebooks from kaggle competitions and identifies 15 guideline categories of the texts (details in section3). specifically, the raw jupyter notebook data with tables, code, and associated text from popular kaggle competitions is crawled. however, the raw data cannot be directly used due to the large amount of noise @xcite . for example, "i plan to refine the models by using more sophisticated machine learning techniques." is about personal experiences and future plans, which is not useful for text generation tasks. on the other side, the userwritten guidelines and ground-truth descriptions in the markdown cell generally contain multiple different purposes or facts on the tables. to reduce the noise in the raw data, we recruit annotators to first break down the markdown cells and make each piece of text only contain one purpose or fact. for each guideline category, we create the label as well as descriptions, and then curate the tables and filter out the noise text. finally, the text descriptions in our data are natural, faithful, and specifically targeted under different guidelines (figure 1 ). next, we evaluate the performance of existing pretrained models. the ablation study shows that guidelines significantly affect the final performance at different levels, which demonstrates the validity of our task. human evaluation further shows that these advanced models (i.e., codet5, llama, and gpt-4o) still struggle to produce faithful enough results, regardless of high-quality training data. moreover, common error patterns involve misalignment with guidelines, incorrect variable values, omission of important code information, and reasoning errors. based on these results, we integrated our approach into a user-facing application to explore human-ai collaboration in code documentation. in a follow-up study, users found that the generated documentation reminded them of documenting codes they might have overlooked and increased their satisfaction with their notebooks. the main contributions of our work are: • we formulate a novel task, tcdg, and collect a high-quality benchmark, namely nbdescrib; • experiments show that fine-tuned lms (codet5) outperform powerful pretrianed models, i.e., ## tcdg -task description in our task, the model is provided with a long text including a code cell and its table output, as well as the corresponding guideline category description. the guideline indicates the direction of the target description generation, and the specific category is described in section 4.3. the model is asked to read the input and generate reasonable descriptions based on the given guideline, code, and table. ## experiments baselines. we utilized three types of models: a fine-tuned encoder-decoder-based codet5, the popular decoder-only llms (an off-the-shelf gpt-4o, llama3, codellama3), and a fine-tuned gpt-3.0. details are shown in appendix d. ## error analysis in this section, we analyze some common error cases in this guideline-based text generation task. some examples can be found in the appendix b. the ages of passengers ranging from 27 to 62 years old with the youngest passenger being 27 years old codellama the average age of the passengers is 34.5 years old. these errors show that our task and dataset provide some challenges for existing foundation models. we firmly believe that researchers can enhance the existing foundation models in the future when they address the challenges. by building on our work and leveraging the valuable insights gained from it, they can push the boundaries even further, contributing to the continuous evolution of foundation models. ## downstream user application to demonstrate the application of our benchmark nbdescrib, we designed a jupyter notebook plugin to assist document writing in data science programming (as shown in figure 2 ). the plugin is triggered when detecting users focusing on a code cell (figure 2 .a). the plugin then reads the contents from the focused cell and its curated table output based on the selected columns and rows and sends the content to the backend. the backend server first generates documentation under different guideline categories using fine-tuned codet5-large(figure 2 .b). in addition, we used prompts to nudge users to explain an output (figure 2.c). if the user selects one of his preferred candidates, the chosen documentation will be inserted below the code cell. our plugin went through several rounds of pilot testing and iterative design. participants found that it reminded them to document code they would have ignored, reducing the time spent developing documentation while actively exploring the data science task. ## conclusion in this paper, we formulated a new task, tcdg, that aimed to automatically generate descriptive text for code and table based on the given guidelines for a computational notebook. we collected a large amount of well-documented jupyter notebooks from kaggle, resulting in a new benchmark dataset nbdescrib. from our analysis, our task imposed unique challenges to the current generation methods, including codet5 and llms. this dataset facilitated the task like creation of practical slides @xcite for jupyter notebooks and enabled evaluations on faithful, high-fidelity, and factual generation.