TABGEN - ICL : Residual-Aware In-Context Example Selection for Tabular Data Generation (2025)

Abstract: Large Language models (LLMs) have achieved encouraging results in tabular data generation. However, existing approaches require fine-tuning, which is computationally expensive. This paper explores an alternative: prompting a fixed LLM with in-context examples. We observe that using randomly selected in-context examples hampers the LLM’s performance, resulting in sub-optimal generation quality. To address this, we propose a novel in-context learning framework: TabGen-ICL, to enhance the in-context learning ability of LLMs for tabular data generation. TabGen-ICL operates iteratively, retrieving a subset of real samples that represent the residual between currently generated samples and true data distributions. This approach serves two purposes: locally, it provides more effective in-context learning examples for the LLM in each iteration; globally, it progressively narrows the gap between generated and real data. Extensive experiments on five real-world tabular datasets demonstrate that TabGen-ICL significantly outperforms the random selection strategy. Specifically, it reduces the error rate by a margin of up to 42.2% on the fidelity metric. We demonstrate for the first time that prompting a fixed LLM can yield high-quality synthetic tabular data.

## introduction tabular data, despite being one of the most prevalent data modalities in real-world applications @xcite , often encounters several issues in practical use. these include imbalanced data categories @xcite , privacy concerns @xcite (as many tabular datasets contain sensitive personal information that cannot be directly shared), insufficient data quality @xcite , and high data collection costs @xcite . tabular generation is an important means to address these problems. classic tabular generation methods such as gans @xcite , vaes @xcite , and diffusion models @xcite @xcite have two main limitations. first, they require large amounts of tabular data for training, which leads to a noticeable decline in performance in low-resource scenarios. this is particularly problematic considering that most real-world situations requiring tabular generation lack abundant data. second, they need special preprocessing to handle heterogeneous data types, making them less flexible. the rapid development of large language models (llms) brings new possibilities for solving table data generation problems with their powerful semantic understanding, reasoning, and generation capabilities. llms can understand and process various data types and structures without complicated data preprocessing, offering more flexible and principled solutions. moreover, llms' few-shot learning ability may alleviate data scarcity issues, enabling excellent performance in low-resource scenarios. previous works @xcite @xcite @xcite @xcite resort to fine-tuning general-purpose llms on target tables. while effective, fine-tuning requires substantial computational resources, making it inapplicable in resourcescarce scenarios. in-context learning effectively solves such problems. by adding examples to the context, distribution characteristics can be provided to llms, guiding them to generate data that conforms to the target distribution without specific fine-tuning @xcite . however, simple in-context learning strategies still face challenges. figure 1(a) shows that even without in-context examples (see the full prompt at appendix a.2), llms can generate reasonable distributions, reflecting the influence of the llm's pre-training distribution. figure 1( @xcite demonstrates the strategy proposed by @xcite , which involves random sampling from ground truth as in-context examples. although the generated results are closer to the ground truth shown in figure 1 (d) compared to figure 1 (a), they are still mainly influenced by the llm's original distribution and struggle to fit the ground truth. this phenomenon reveals the importance of choosing in-context examples. in this work, we propose tabgen-icl, a dynamic in-context example selection method. inspired by the observation in figure 1 (c), we found that using fixed range incontext examples leads to generated distributions closely mimicking those examples, significantly differing from the llm's original distribution. this indicates in-context learning's ability to simulate distributions. by carefully selecting in-context examples, we can more effectively guide llms to generate distributions closer to the ground truth. central to our framework is the design of an auto-mated strategy for selecting effective in-context examples while ensuring global consistency with the real data distribution. our key idea is to utilize simple, discernible patterns in subsets of real samples, which can effectively guide llms in generating realistic tabular data. specifically, tabgen-icl identifies subsets of real samples that exhibit simple patterns and closely match the residual between the current generated data distribution and the real data distribution. this idea can be categorized as a novel residual-aware rag technique, where we retrieve in-context examples based on the residual between the generated and real data distributions. the residual-aware sampling measures the discrepancy between the generated and real data distributions, focusing on areas where the model needs improvement. this approach enables tabgen-icl to progressively narrow the distribution gap while maintaining the use of easily learnable patterns in the in-context examples. our sampling technique offers two key advantages: flexibility in selecting simple patterns for effective learning, and consistent generation through progressive distribution alignment. the contributions of this paper are as follows: ## related works deep generative models for synthetic tabular data generation generative models for tabular data have become increasingly important and have widespread applications @xcite @xcite . for example, ctgan and tave @xcite deal with mixed-type tabular data generation using the basic gan @xcite and vae (kingma and welling, 2013) framework. goggle @xcite incorporates graph attention networks in a vae framework such that the correlation between different data columns can be explicitly learned. recently, inspired by the success of diffusion models in image generation, a lot of diffusion-based methods have been proposed, such as tabddpm @xcite , stasy @xcite , codi @xcite , and tabsyn @xcite . llms for synthetic tabular data generation. collecting high-quality training data for advanced deep learning models is often costly and timeconsuming. pre-trained large language models (llms), been exposed to vast corpora, have emerged as a promising direction for synthetic data generation. however, while llms have demonstrated strong capabilities in generating highquality natural language, their ability to accurately replicate tabular data distributions at scale remains uncertain @xcite . to address this challenge, several works have explored fine-tuning llms such as gpt-2 to generate synthetic tabular data more effectively @xcite @xcite . another line of work leverages the in-context learning abilities of llms @xcite , which is closely related to our method. @xcite adopts a straightforward strategy to uniformly sample in-context examples; we identify the root of inefficiency of this strategy and instead propose a residual-aware in-context example selection strategy, which is shown to improve the generation quality. ## preliminaries notation. tabular dataset refers to data organized in a tabular format with n row and d columns, where each row denotes a data record or sample, and each column denotes an attribute or feature. each attribute can be either discrete (e.g. categorical) or continuous (e.g. real number r). we use p(x) to denote the probability distribution of x. data setup. we have access to a training dataset of n samples: d @xmath0, each sample x i is i.i.d. drawn from an unknown distribution p(x). objective. the goal is to generate a new dataset d @xmath1.i.d. sampled from p(x). direct copy of training data is not allowed. serialization. as llms primarily process text input, it is necessary to convert tabular data into a suitable textual format. there are many serialization formats for tabular data, such as json @xcite , markdown @xcite , sentences @xcite , etc. notably, the json format is widely supported by llms, with models like gpt-4o capable of generating structured outputs in json format through constrained decoding @xcite . therefore, in this study, we adopt a json format to serialize tabular data. for instance, a row from a table containing three columns-name (categorical), age (numerical), and city (categorical)-is transformed into a json object: {name:'alice', age:25, city:'new york'}. for a table comprising n rows, the serialized data becomes a list of n json objects. see appendix a.3 for the implementation of the json schema. during each prompting iteration, tabgen-icl retrieves a subset of these json objects to serve as in-context examples. this process will be elaborated upon in subsequent sections. ## tabgen-icl this section presents tabgen-icl framework for tabular data generation. tabgen-icl retrieves a subset of samples from the training dataset that satisfy two properties: 1) local: at each prompting iteration, the llms can effectively extract patterns from the in-context examples; 2) global: after enough iterations, the overall generated samples mimic the distribution of the real samples. in the following, we will introduce each component of tabgen-icl in detail. ## experiments we validate the performance of tabgen-icl through extensive experiments. in particular, we investigate the following questions: ## conclusion this work introduces tabgen-icl, an in-context learning (icl) framework for tabular data generation using large language models (llms). by iteratively identifying and sampling from underrepresented regions in the data distribution, tabgen-icl effectively steers the llm's prior toward the real data distribution. this residual-aware generation process enables more diverse and faithful synthetic samples. extensive experiments validate the effectiveness of our approach across a variety of domains. a key strength of tabgen-icl lies in its ability to leverage the rich prior knowledge embedded in pre-trained llms, making it particularly valuable in practical scenarios where collecting large-scale datasets is costly or infeasible.