Solving Data-centric Tasks using Large Language Models (2024)

Abstract: Large language models are rapidly replacing help forums like StackOverflow, and are especially helpful to non-professional programmers and end users. These users are often interested in data-centric tasks , like spreadsheet manipulation and data wrangling, which are hard to solve if the intent is only communicated using a natural-language description, without including data. But how do we decide how much data and which data to include in the prompt?This paper makes two contributions towards answering this question. First, we create a dataset of real-world NL-to-code tasks manipulating tabular data, mined from StackOverflow posts. Second, we introduce a novel cluster-then-select prompting technique, which adds the most representative rows from the input data to the LLM prompt. Our experiments show that LLM performance is indeed sensitive to the amount of data passed in the prompt, and that for tasks with a lot of syntactic variation in the input table,our cluster-then-select technique outperforms a random selection baseline.

## introduction code-generating large language models (llms) promise to empower end users interested in datacentric tasks, ranging from string manipulations in spreadsheets to data cleaning and analysis in computational notebooks. for example, consider the following task on tabular data: given a column with full names, generate a new column with user names, by combining the first initial and last name, in lowercase. this task can be solved by a pandas program that: 1) splits the full name into a list of strings, 2) extracts the first and last string from the list, 3) converts both to lowercase and joins the first letter of one string to the other as shown in figure 1 . the challenge in generating this program is that input data rows have varied formats, e.g. most rows only have two names ("john smith"), but some have multiple middle names ("jake l woodhall", "jo anna emily gray"). if an llm prompt does not include any data or only includes rows with two names, the llm is more likely to generate a program that does not generalize (e.g. one that extracts the last name as the second element of the list instead of last). in this paper, we focus on solving such tasks that involve multi-step computations on the input columns to generate additional columns. towards this goal, we mine stackoverflow to construct a new dataset, dubbed sofset, of data-centric tasks, equipped with a natural-language query and a small input table. using this dataset, we conduct experiments on generating pandas programs using gpt-4 and an open-source alternative codellama, with the goal of analysizing llms' sensitivity to the amount of input data provided in the prompt. unlike input tables in stackoverflow posts, realworld data tables are often large, hence sending the entire table to the llm is likely impractical, expensive, or detrimental to performance. how do we best convey the structure of a large input table to the llm? to address this question, we propose a cluster-then-select prompting technique that clusters input rows based on their syntactic structure and then selects representative rows from each cluster; e.g. in our figure 1 example, the technique would include a row for each format of middle names. to evaluate this technique, we perform experiments on sofset augmented with larger input tables extracted from kaggle. in summary, this paper contributes: ## the sofset dataset we collect a new dataset fashioned from real-world data-centric tasks from stackoverflow (sofset). we sample tasks deterministically from the highest rated posts with the tag "excelformulas" in stackoverflow (as of march 2022). these tasks are representative of real problems spreadsheet users face frequently since they correspond to the highestrated posts. we manually check that the posts are genuine tasks and also remove post identifiers for anonymization. this gives us a total of 201 tasks. ## evaluation of data-centric tasks we perform an analysis of the role of data on model performance in data-centric tasks. we first use the original sofset dataset to examine three data regimes with increasing amounts of data: (a) nodata (b) first-row and (c) ten-rows and also the taxonomy of task classes of increasing difficulty in terms of data required: ind, dep and ext. we then use kaggle-augmented dep tasks to compare our cluster-then-select technique (which selects representative rows from the top-n most dense clusters) against a random baseline (which selects random rows from the input table). for each data setting, we construct a prompt which contains the task query and selected rows as a pandas dataframe to generate code from gpt-4 as shown in figure 1 . correctness is reported based on whether the code produces the expected output in terms of pass@k, pass@k for data-dependent tasks (with more than 2 clusters) the probability that at least one of k samples of generated code produces the correct output @xcite . we report all results using gpt-4 with a temperature of 0.5 and the generated completions are evaluated on all rows in the input table. the sofset dataset, all the evaluation results and our prototype tool can be found online. foot_0 does model performance vary with the amount of data passed for different task classes? figure 2 shows the impact of the amount of data on llm performance, first for the entire dataset and then split by task classes. we see a larger drop in performance with reduced (and no) data on dep (and ext) tasks compared to ind tasks. specifically, the performance gap (pass@5) between first-row and no-data regimes is larger for the dep and ext classes (33.8% and 83.5% resp) compared to only 7.1% for ind tasks. the fact that there is any performance drop for ind tasks indicates that having data helps the model even when the problem can be solved independently of data. in the absence of data, almost no ext task is solved (pass@1) but performance improves when a single row is passed. ## conclusion and future work our work highlights the importance of data for code generation on data-centric tasks and proposes a new dataset for evaluation of data-centric tasks. we show that providing even one data row to the model boosts performance compared to a no-data baseline. since providing the entire input table is often infeasible, we propose a cluster-then-select prompting technique that selects representative rows from the data to be added to the prompt. while randomly selecting rows also performs well, for data with a high degree of syntactic variation, it is more beneficial to add representative rows to the prompt. for future work, handling a broader problem space (e.g., multi-table inputs, hierarchical table inputs) raises interesting challenges.