Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models (2025)

Abstract: Large Language Models (LLMs) have shown strong potential for tabular data generation by modeling textualized feature-value pairs. However, tabular data inherently exhibits sparse feature-level dependencies, where many feature interactions are structurally insignificant. This creates a fundamental mismatch as LLMs’ self-attention mechanism inevitably distributes focus across all pairs, diluting attention on critical relationships, particularly in datasets with complex dependencies or semantically ambiguous features. To address this limitation, we propose GraDe (Graph-Guided Dependency Learning), a novel method that explicitly integrates sparse dependency graphs into LLMs’ attention mechanism. GraDe employs a lightweight dynamic graph learning module guided by externally extracted functional dependencies, prioritizing key feature interactions while suppressing irrelevant ones. Our experiments across diverse real-world datasets demonstrate that GraDe outperforms existing LLM-based approaches by up to 12% on complex datasets while achieving competitive results with state-of-the-art approaches in synthetic data quality. Our method is minimally intrusive yet effective, offering a practical solution for structure-aware tabular data modeling with LLMs.

## introduction tabular data forms the foundations of countless real-world applications across healthcare analytics @xcite , financial modeling @xcite , demographic research @xcite , and scientific experimentation @xcite . generating high-quality synthetic tabular data has become increasingly important for data augmentation, privacy preservation, and model testing @xcite . however, this task presents 1 our code is available at https://github.com/ oooranz/grade . education is bsc education-num is 13 age is 35 gender is male income @xmath0. unlike images or text, tabular data exhibits distinct patterns of relationships between features. some features have strong correlations, while others remain entirely independent from one another @xcite . more critically, certain features may uniquely determine the values of others through logical or business rules. database theory formalizes these patterns as functional dependencies (fds), where knowing the value of one feature (or a set of features) allows us to determine another's with certainty @xcite . for example, postal codes uniquely determine cities and states, while having no connection to personal attributes like age or income. without preserving these dependencies, synthetic data suffers from logical inconsistencies such as customer ids linked to multiple conflicting personal details, rendering the generated data unrealistic for practical applications. recent years have seen rapid progress in tabular data generation @xcite @xcite . among these, large language models (llms) offer a compelling solution due to their strong generalization capabilities from pre-training on vast corpora and understanding of complex real-world relationships @xcite . @xcite first applied llms to this task by con-verting data into text sequences (e.g., "age is 39, income @xmath1, allowing the model to generate data conditioned on any subset of features. @xcite later showed that this random shuffling disrupts the model's ability to capture dependencies between features, proposing a fixed topological ordering approach instead. this reveals a fundamental design challenge: how to maintain both flexible conditioning and accurate modeling of feature relationships in llm-based approaches. the root cause of this challenge is a structural mismatch between llms and tabular data. as shown in figure 1 , llms use attention mechanisms where every token potentially relates to every other token @xcite . in contrast, tabular data has sparse, non-sequential dependencies where most features are conditionally independent. this fundamental disconnect creates a dilemma: random feature ordering offers flexibility but weakens dependency modeling, while fixed ordering better captures relationships but sacrifices flexibility. when feature-value pairs are linearized as text, llms face the inherently difficult task of correctly interpreting each pair while preventing interference from unrelated features @xcite . to fully leverage llms for tabular data, we need an approach that maintains flexible ordering while explicitly modeling the sparse dependency structure inherent in tabular data. to bridge this structural gap, we propose grade (graph-guided dependency learning). unlike previous approaches that rely on implicit learning through feature ordering, grade explicitly enhances llms by incorporating learnable sparse dependency graphs into the attention mechanism ( §4.2). our method dynamically models tokenlevel relationships while leveraging externally extracted functional dependencies as guidance ( §4.3), allowing focused attention on structurally important connections. the model is trained with a multiobjective loss balancing language fluency, structural sparsity, and functional dependency alignment ( §4.4). for parameter efficiency, we also introduce grade-light, a variant that only updates our attention modules while maintaining competitive performance. overall, our contributions are as follows: deep generative models for tabular modeling early research on tabular data generation adapted deep generative models such as gans, vaes, and diffusion models (shwartz-ziv and armon, 2022). ctgan (xu et al., 2019) and ctab-gan (zhao et al., 2021) tackled mixed-type data through conditional sampling and specialized normalization strategies (grinsztajn et al., 2022), while tvae (xu et al., 2019) employed a variational framework for more stable training. more recent diffusionbased methods like tabddpm (kotelnikov et al., 2023) and tabsyn (zhang et al., 2024) improved sample quality through iterative denoising of multimodal distributions. notably, goggle (liu et al., 2023b) introduced explicit graph-based modeling of feature dependencies. despite these advances, most generative approaches still struggle to flexibly capture the sparse dependency structures inherent in tabular data. language models for tabular modeling another line of research models tabular data by serializing rows into textual sequences and applying large language models for generation. great (borisov et al., 2023) pioneered this direction by serializing feature-value pairs as text and employing random feature permutations to enhance flexibility. however, this randomization weakens the model's ability to learn structural dependencies (müller et al., 2024). later works such as tabllm (hegselmann et al., 2023) and taptap (zhang et al., 2023a) proposed refined serialization formats and large-scale tabular pretraining to better preserve relational patterns. recent work such as p-ta (yang et al., 2024) further explored enhancing llm-based tabular generation through policy optimization, while spada @xcite uses llms to build sparse feature dependencies in tabular data. these efforts highlight the need for stronger structural guidance beyond text serialization alone. ## problem formulation modeling tabular data as language generative modeling for tabular data aims to learn a probability distribution p x over tabular samples @xmath2. given a training dataset d consisting of n i.i.d. samples x ∼ p x , the objective is to approximate p x by learning parameters θ of a generative distribution p θ . while traditional approaches model this distribution directly in the feature space, language model-based methods typically transform tabular data into textual representations to model the joint distribution as: attention as graph structure in transformerbased language models, the self-attention mecha-nism can be treated as a fully connected graph over input tokens @xcite . for a sequence of tokens {t 1 , t 2 , ..., t n }, the attention weight a ij between tokens t i and t j is computed as: relational structure of tabular data in contrast, tabular data typically exhibits sparse relational structures, where each feature depends only on a small subset of other features @xcite . such relationships can be represented as a sparse graph formally, the generative process of each feature value v i can be described as: structure mismatch problem when applying language models to tabular data generation, a critical mismatch arises between the dense attention structure g lm and the sparse relational structure g tab . this mismatch leads to two key problems: (1) the attention mechanism assigns nonnegligible weights across all token pairs, diluting its ability to capture the few critical dependencies as the number of features increases; (2) language models do not encode any inductive bias about tabular feature dependencies, and must learn such structural patterns purely from data, which becomes increasingly difficult as table complexity grows. motivated by these limitations, we propose to explicitly incorporate functional dependency structures into language models, as detailed in the following sections. ## graph-guided dependency learning we propose grade, a novel method that enhances language models to better capture the sparse dependency structures inherent in tabular data. building on the observation that tabular data contains natural sparsity in feature relationships, grade integrates this structural information into the attention mechanism of language models. ## experimental setup models we use gpt-2 @xcite with 124 million parameters as our backbone model for all main experiments. we evaluate both the full grade and grade-light variants on this model. additional experiments with gpt-2 medium with 355 million parameters are reported in appendix a.3. we use nucleus sampling @xcite ) for text generation. datasets for our main experiments, we use five real-world datasets from diverse domains with complex dependencies: bird (ecology), sick and diabetes (medical), income (social), and housing (real estate). dataset sizes range from 700 to 40,000 samples with 5 to 30 features, including continuous, categorical, and integer types. for our scaling experiments ( §a.1), we use the loan dataset, a large-scale financial dataset with approximately 250,000 samples. detailed descriptions and the dataset statistics are provided in appendix b.3. column-wise correlation error the preservation of feature relationships is crucial for generating realistic tabular data. following gulati and roysdon (2023), we assess fidelity by measuring the absolute difference between pearson correlation coefficients in real versus synthetic data. for categorical features, we first apply one-hot encoding before calculating correlations. figure 3 shows the correlation error histograms comparing grade, grade-light, and great across different datasets. we focus this comparison on llm-based approaches since prior work has shown that autoregressive models typically struggle with capturing joint distributions @xcite , making this a particularly challenging aspect for llm-based tabular data generation. our results demonstrate that grade's graph-guided approach provides a clear advantage by explicitly modeling functional dependencies and directing attention to structurally relevant connections. distance to the closest record (dcr) privacy preservation is a key application for synthetic tabular data, requiring generated samples to resemble but not duplicate training examples. following zhang et al. (2024), we use the l1 norm to measure the minimum distance between each synthetic sample and real training examples. for numerical features, differences are normalized by the observed feature range; for categorical features, the distance is defined as 0 if the values match, and 1 otherwise. the final dcr score is computed as the average of the minimum distances across all synthetic samples. ## conclusion in this work, we address the structural mismatch between llms' dense attention patterns and tabular data's sparse dependencies. our solution, grade, guides token interactions through learnable dependency graphs that integrate functional relationships into attention mechanisms. this approach helps models focus on meaningful connections without altering the base architecture. experiments across diverse datasets show consistent gains in utility, fidelity, and privacy. grade outperforms prior llmbased methods by up to 12% on complex medical datasets and remains effective even with only 250 training samples. the lightweight grade-light variant achieves similar results while reducing trainable parameters by nearly 100 million. these findings confirm that structural inductive bias significantly enhances llms for tabular modeling. by bridging the gap between powerful semantic representations and domain-specific structural awareness, grade offers a practical and effective solution for generating high-quality synthetic tabular data.