An Automatic Prompt Generation System for Tabular Data Tasks (2024)

Abstract: Efficient processing of tabular data is important in various industries, especially when working with datasets containing a large number of columns. Large language models (LLMs) have demonstrated their ability on several tasks through carefully crafted prompts. However, creating effective prompts for tabular datasets is challenging due to the structured nature of the data and the need to manage numerous columns. This paper presents an innovative auto-prompt generation system suitable for multiple LLMs, with minimal training. It proposes two novel methods; 1) A Reinforcement Learning-based algorithm for identifying and sequencing task-relevant columns 2) cell-level similarity-based approach for enhancing few-shot example selection. Our approach has been extensively tested across 66 datasets, demonstrating improved performance in three downstream tasks: data imputation, error detection, and entity matching using two distinct LLMs; Google/flant-t5xxl and Mixtral 8x7B.

## introduction recent advancements in pre-training large language models have paved the way for prompt-based and in-context learning @xcite , providing an efficient approach to tackle a wide range of tasks. generating a suitable prompt is particularly important when harnessing pre-trained llms for tabular downstream tasks. unlike natural language sentences, tabular data necessitates specific formatting, column preferences, and in-context examples. in the domain of tabular data downstream tasks, prompts are frequently customized for each dataset and specific downstream tasks to ensure consistent and efficient performance. a few studies @xcite have demonstrated this approach, wherein prompts are crafted for a given dataset and task by manually selecting columns and relevant in-context (few-shot) examples. additionally, recent research {instruction} following is a serialized 'column':'value' format. the task is to predict the correct value of the specified column in the question. has introduced automated methods that target specific components of prompt such as in-context examples @xcite . this leads us to two pivotal questions; firstly, what are the components or parts of the prompt that significantly impact performance in tabular tasks? secondly, how can we devise automated methods for the components to generate prompts that can be efficient and effective for tabular data tasks? in tabular data tasks, a vanilla prompt typically includes a row of information where each column name is paired with its respective value, for example: brand: dell; price: $349.00; feature: newest dell inspiron. however, providing all column information of a row may introduce unnecessary details, redundancy, and noise, while also inefficiently using input tokens, potentially leaving inadequate space for additional crucial information such as few-shot examples. our empirical analysis reveals that carefully selecting columns to be included in the prompt improves task performance, aligning with prior research findings by @xcite . in addition to 1 191 choosing the right columns, we noticed a significant improvement in performance when we carefully arranged these column details in the prompt. this underscores the importance of both selecting the columns and arranging them in a specific order. particularly, dealing with large datasets with many columns poses a practical challenge. furthermore, our study demonstrates a performance difference between traditional few-shot example selection methods developed for natural language (nl) focused tasks and our proposed celllevel similarity few-shot (clfs) example selection approach for tabular data. the nl-based method serializes a row into a sentence and selects few-shot examples based on similarity, potentially losing relevant information by imposing a sentence structure on the tabular data during serialization. in contrast, the proposed clfs method considers each cell's information independently with the aim of selecting few-shot examples at a cell level similarity. to this end, we introduced an auto-prompt generation system designed to be compatible with various llms without the need for extensive training. this system introduces a novel approach emphasizing two essential elements: ## related work recent studies on tabular data tasks have demonstrated that llms can effectively tackle data wrangling tasks, through different strategies, including pre-training and fine-tuning @xcite @xcite @xcite , prefix-tuning @xcite and prompt learning @xcite @xcite . however, these training methodologies pose significant computational demands and exhibit high time complexity. furthermore, many of these approaches require adjustments to the model parameters, a process that proves impractical for black-box language models like chatgpt. previous research has achieved success in developing methods for generating prompts for tabular data tasks @xcite . however, a challenge exists due to the reliance on manual processes to select columns and few-shot examples. this manual approach becomes especially difficult when dealing with large datasets with many columns. the study by @xcite propose a few-shot selection method that utilizes an embedding of a row transformed into a natural language sentence, raising questions about integrating tabular structure-aware few-shot selection methods for tabular data. this paper discusses the importance of selecting and organizing columns with empirical results, followed by an overview of two auto-prompt generation systems. extensive testing was carried out across 3 different tabular data tasks; data imputation (di), error detection (ed), and entity matching (em), utilizing 66 datasets using two models: google flan-t5-xxl (11b parameters) @xcite and mixtral 8x7b (47b parameters) @xcite 1 . ## motivation tabular data, especially with wider datasets, presents a challenge when inputting all column details into llms for row-level tasks (e.g., data imputation, error detection). it becomes evident that selecting columns is crucial for optimizing the performance of llms. in addition to choosing columns, we conducted a study to explore the impact of column arrangement on downstream tasks performance. in this experiment, we manually selected specific columns for a dataset and downstream task and then created prompts using a template (see figure 1 ). figure 2 demonstrates the significant effect of different column orders on accuracy. for instance, when examining the data imputation task on the amtrak dataset, a wide range of accuracies was observed, varying from 0.06 to 0.95 for the manually selected columns but in different orders and combinations. this shows that using an optimal sequence of subset columns is crucial while generating a prompt for tabular data tasks. this can be considered as solving a sequential decision-making problem. this study uses reinforcement learning (rl) to optimize column se-1 specifically, we use the gptq @xcite quantized version of the model available at https:// huggingface.co/thebloke/mixtral-8x7b-instruct-v0 . 1-gptq. mixtral 8x7b is a sparse mixture of experts (smoe) model consisting of 8 feedforward blocks (i.e. experts) at each layer. for each token, at each layer, 2 out of 8 experts are selected for inference which results in a total of 13b active parameters. another component, namely few-shot example selection, becomes critical in light of the improved performance shown by llms when provided with a small set of illustrative examples (few-shot examples) in the prompt. selecting these examples is extensively studied in the field of natural language tasks @xcite and these methods typically retrieve similar examples from a pool using similarity metrics such as bm25 @xcite or cosine-similarity calculated over task specification embeddings obtained from encoder-only models like bert @xcite and roberta @xcite , etc. our exploration reveals that a modified approach for calculating the similarity measure leads to improved performance. to address this, our work proposes a new cell-level similarity measure for retrieving in-context examples that takes into account the semantic similarity of individual cells in a row, and is seen to outperform natural language inspired baselines. we evaluated the performance of the proposed system on 3 downstream tasks data imputation (di), error detection (ed) and entity matching (em). more details on the downstream as provided in appendix a.4. ## method and implementation in this section, we explain the methodology of developing our auto-prompt generation system. we begin by describing the overall architecture of the system, followed by description of individual components. ## datasets we gathered datasets from various sources like kaggle 2 and open ml 3 , ensuring datasets containing numerous columns (upto 120 columns) across different domains. all the datasets gathered are in the format comma seperated values (csv) files. for the data imputation task, specific columns were chosen for imputing values across all rows within those columns. real-world databases commonly have both syntactic and semantic errors @xcite @xcite . in the error detection task, selected columns in the datasets were introduced with errors: around 25% of cell values were replaced with out-of-domain strings for semantic errors (e.g., 'stationer' in the 'county name' column), while approximately 25% of cell values within a specific column had random letter additions introduced as syntactic errors. for entity matching tasks, we used datasets previously studied in literature @xcite . ## experimental results this section describes the experiments carried out in our study. we compared 5 different conditions including our proposed system to highlight the importance of each of the components. these experiments included 12 datasets across 3 tasks using google flan-t5-xxl as shown in ## conclusion and future work the results indicate that our proposed system significantly outperforms the baseline as well as methods based on combinations of manual column selection, random few-shot selection and natural language based few-shot selection. our research underscores the efficacy of an auto-prompt generation system in enhancing tabular data tasks across various datasets and tasks using large language models. manual column selection and sequencing, found to be a cumbersome process, necessitates an automatic method, a gap which is suitable filled by our rlbased algorithm. our proposed cell-level similarity measure exhibits improved performance compared to the nl-based few-shot selection method. overall, the auto-prompt generation system showcases versatility and generalizability across diverse tasks and datasets, providing a streamlined solution for efficient tabular data tasks. as part of future work, the following directions of study may be worthwhile: