Towards Universal Debiasing for Language Models-based Tabular Data Generation (2025)

Abstract: Large language models (LLMs) have achieved promising results in tabular data generation. However, inherent historical biases in tabular datasets often cause LLMs to exacerbate fairness issues, particularly when multiple advantaged and protected features are involved. In this work, we introduce a universal debiasing framework that minimizes group-level dependencies by simultaneously reducing the mutual information between advantaged and protected attributes. By leveraging the autoregressive structure and analytic sampling distributions of LLM-based tabular data generators, our approach efficiently computes mutual information, reducing the need for cumbersome numerical estimations. Building on this foundation, we propose two complementary methods: a direct preference optimization (DPO)-based strategy, namely UDF-DPO, that integrates seamlessly with existing models, and a targeted debiasing technique, namely UDF-MIX, that achieves debiasing without tuning the parameters of LLMs. Extensive experiments demonstrate that our framework effectively balances fairness and utility, offering a scalable and practical solution for debiasing in high-stakes applications.

## introduction large language models (llms) @xcite @xcite demonstrate extraordinary ability to understand @xcite , reason @xcite , and generate text @xcite . these advancements have pushed new boundaries across a wide range of domains @xcite . as one of the most common data forms @xcite , there has been a growing trend to leverage llms for tabular data tasks, such as understanding @xcite , prediction @xcite , and generation @xcite @xcite . despite their powerful capabilities, llms suffer from fairness issues when acting on tabular data, i.e., advantaged features (e.g., income) are often correlated with protected attributes (e.g., gender). such biases widely exist in tabular data due to historical reasons @xcite . consequently, when llms are trained on this data, they will inherit existing biases @xcite . moreover, because the generated data is often used to train downstream prediction tasks for high-stakes domains such as job applications, the inherited bias raises fairness concerns for the downstream models as well @xcite . to address fairness concerns in llms, one approach is to adapt debiasing methods from non-llm tabular data generators to ensure fairness in llm-based generation. however, these existing methods only target bias between one advantage feature-protected attribute pair (e.g., incomegender pair) that adheres to only one downstream task @xcite @xcite @xcite . that is to say, only the bias in gender when predicting income is guaranteed eliminated when using the generated dataset. when users want to work on other downstream tasks such as predicting education level, fairness guarantee requires retraining the data generator again. yet, tabular datasets typically contain multiple advantaged features (e.g., income, education, occupation) and protected attributes (e.g., age, gender, race), making retraining for every possible pair computationally prohibitive. another approach is to adapt for debiasing llms in text generation. most existing methods focus on debiasing a single protected attribute @xcite @xcite . therefore, these methods still cannot address settings with multiple protected attributes. rather than relying on pairwise debiasing methods, we propose a group-wise debiasing approach that eliminates all dependencies between advantaged features and protected attributes. thus, our formulation partitions features into advantaged features (e.g., income, education, occupation), protected attributes (e.g., race, gender), and remaining features, and minimizes the group-level mutual information (mi) between the advantaged and protected features. notably, pairwise debiasing is a special case of this broader framework, where the protected attribute and advantaged feature groups each contain only one feature. additionally, the minimization of group-level mi also aligns with the principle of intersectional fairness @xcite in the sense that, when mutual information is zero, every advantaged feature is guaranteed to be independent of any combination of protected features. while the minimization of mi provides fairness guarantees, breaking these dependencies inherently alters the learned distribution, potentially causing the generated data to deviate from the original. to prevent excessive distortion while still reducing bias, we impose an additional constraint that balances fairness against data utility during optimization. this universal debiasing framework for tabular data generation is our first key contribution. however, mi lacks a closed-form expression, making its computation challenging, let alone minimization for debiasing. this difficulty is exacerbated in high-dimensional spaces, where tabular data often lie in complex manifold @xcite . while this challenge cannot be solved in general, the unique auto-regressive nature of llmbased tabular data generators allows us to derive efficient solutions for them. specifically, llms generate different features of a tabular-typed sample one by one in a sequential manner, and each feature is drawn from an analytic-form distribution. taking advantage of these analytic sampling distributions that are accessible, we propose a fine-tuning based solution for debiasing that eliminates the need for numerical estimation of mi. this solution can be readily implemented with direct preference optimization (dpo) @xcite , making our debiasing task no more difficult than standard finetuning. in addition, the debiased model maintains all applicability of the base llm and can seamlessly replace the latter in all cases -notably, the fairness guarantee generalizes to diverse scenarios beyond data generation, such as data imputation. this strong one-for-all guarantee makes our solution highly valuable. we refer to this dpo-based debiasing method as udf-dpo. built upon udf-dpo, we derive udf-mix, a more efficient debiasing solution specialized for data generation. udf-mix not only leverages the analytic sampling distribution, but also exploits the sequential nature of the generation process. specifically, udf-mix identifies a few generation steps that cause the bias, and precisely alters these steps without changing others. this design leads to two remarkable efficiency improvements. first, as udf-mix only needs to debias a few generation steps, it relies on far fewer parameters, thereby achieving much better parameter efficiency. second, through an innovative parametrization, we incorporate the fairness and utility balancing factor, which is usually treated as a hyper-parameter to tune, directly into udf-mix training. consequently, udf-mix by design can handle the balance of conflicting fairness and utility without retraining, thereby substantially reducing the human burden and computation costs for tuning hyperparameters for different tasks. these two effective and efficient debiasing methods are also key contributions of our work. our paper is organized as follows. sec 2 and 3 introduce preliminary and limitations of current methods. sec 4 details our new universal debiasing framework and two effective solutions. sec 5 presents extensive experiments to demonstrate the effectiveness of our methods. in the remaining part of this paper, we review related works in sec 6, and conclude the paper in sec 7. ## preliminary tabular data. tabular data is structured in a table format, where each row corresponds to a sample and each column represents a feature, which can be of mixed types @xcite . mathematically, a tabular dataset can be expressed as @xmath0, where each sample ## experiments we evaluate our methods on two practical use cases with generated data from three diverse tabular datasets, each featuring different protected attributes and advantaged features. our methods achieve debiasing between multiple potential target variables and protected attributes while preserving high data utility. ## conclusion we propose a universal debiasing framework for llm-based tabular data that balances the fairnessutility trade-off for multiple advantaged features and protected attributes. our dpo-based method, udf-dpo, and the efficient adaptive approach, udf-mix, mitigate bias while preserving high data quality. mathematical insights and experiments confirm that our approach outperforms existing pairwise methods, offering robust and scalable debiasing for high-stakes applications.