T ab2 T ext - A framework for deep learning with tabular data (2024)

Abstract: Tabular data, from public opinion surveys to records of interactions with social services, is foundational to the social sciences. One application of such data is to fit supervised learning models in order to predict consequential outcomes, for example: whether a family is likely to be evicted, whether a student will graduate from high school or is at risk of dropping out, and whether a voter will turn out in an upcoming election. While supervised learning has seen drastic improvements in performance with advancements in deep learning technology, these gains are largely lost on tabular data which poses unique difficulties for deep learning frameworks. We propose a technique for transforming tabular data to text data and demonstrate the extent to which this technique can improve the performance of deep learning models for tabular data. Overall, we find modest gains (1.5% on average). Interestingly, we find that these gains do not depend on using large language models to generate text.

## introduction tabular data is abundant in the social sciences and for social good applications @xcite @xcite . one source of tabular data is public opinion surveys, which are used to monitor public opinion on everything from public health to the economy. another common source of tabular data is interactions with public services, for example, when a family receives assistance with food or paying rent, this interaction can populate a table. this data is also used to train supervised learning models which have been designed and deployed in the service of the public good @xcite @xcite . for example, supervised learning has been deployed to detect at-risk high-school students @xcite and prevent eviction-caused homelessness * * denotes equal contribution @xcite . however, while deep learning models have often yielded transformative results when applied to supervised learning problems with homogeneous data sources (e.g., in the text domain), such models generally offer sub-optimal results applied to tabular data @xcite . in part, this performance is due to tabular data often being composed of mixed feature types from categorical features to numerical. furthermore, we generally see deep learning models suffer from overparametrization and sensitivity to particular hyperparameter combinations on tabular data @xcite @xcite . here, we propose to close this gap by introducing a new method to adapt tabular data for large language models (llms) by transforming tabular data into text data. this paper makes three contributions: (1) we introduce tab2text, a deep learning framework for tabular data that exploits both tabular and text representations, enhancing learning capability through llm-generated narratives foot_0 ; (2) we show that including text representations consistently outperforms the baselines which use tabular data only; and (3) models leveraging only text representations underperform compared to those using only tabular data. ## problem statement consider a dataset with d columns and n rows: y ∼ {c1 : x1, c2 : x2, . . . , c d : generator pq: taylor is female / taylor is 18-25 years old. write a social media post which explains taylor's thoughts on the easy access to birth control and their political party. g"&&")hijq: ## our approach -tab2text consider the setting of predicting one's income. given access to someone's social media posts, this may be relatively easy. a person of very high income may use language related to luxury goods and services, while a lower-income person may employ vocabulary that reflects different socioeconomic experiences. similarly, consider the simple task of predicting whether a flower is a rose or an iris. the text descriptions of each flower would differ greatly, given a dataset of descriptions of roses and descriptions of irises, a llm model would likely learn to differentiate them easily. without access to a text dataset for a target population, our intuition is that llms already contain enough general knowledge such that they can generate related text given limited information. for example, given a person's occupation, an llm may be able to generate an example social media post which could be indicative of income; given a person's age and race, a llm can hypothesize about how they may feel about politically relevant issues-hypotheses which could help with the political inference tasks; or, given the size and color of a flower, an llm should be able to generate a description. our approach (tab2text) generates versions of tabular data as text which allow us to learn more sophisticated predictive functions and leverage textual embedding spaces. additionally, we propose inference architectures which combine textual with tabular data. combining textual representations from llms with tabular data should enable models to leverage the knowledge capabilities of llms while grounding them in the specific task distribution, potentially leading to better performance than using either representation alone. this complete framework is shown in figure 1 . ## empirical evaluation we evaluate tab2text with two common social science datasets. additionally, we compare both the descriptive and narrative approaches to generating text. finally, we analyze both the influence of specific llms and inference architectures. data we explore two datasets: the american national election studies (anes) (american national election studies, 2021) and income @xcite . anes consists of responses to surveys about political beliefs and behaviors, as well as demographic information. we use a version with 3,905 respondents from 2020. income consists of records from the 1994 u.s. census bureau database and contains demographic information about individuals. this dataset has been used for implementation details several narrative encoder models were tested: distilbert @xcite , roberta @xcite , and polibert @xcite . polibert is a robertabased model that was fine-tuned on political twitter data, and which likely has context-specific information for these social science data. additionally, we employed automm and tabularpredictor models from the autogluon framework @xcite . automm utilizes a ft-transformer architecture @xcite for encoding tabular inputs, while leveraging the electra-base discriminator model (clark, 2020) for text encoding. we used the following narrative generator formulation to produce narrative text. narrative generator formulation the narrative generator consist of two main components: tabular data (both column names and values) and the narrative queries that solicits information about the relevant outcome. we describe these queries in we introduce the gender-neutral name "taylor" and use third-person narration to present the demographics, shifting llm's perspective from expressing its own views to considering those of another. for anes, we also request that the llms provide a confidence level score for their assessments. in addition, we adopt @xcite method by adding a directive at the end of each narrative queries, instructing the llms to start their reply with "absolutely! i think. . . ". additionally, to avoid redundant information, we ask the llms to limit their responses to 40 words. example narrative queries for anes based on the demographic feature "gender" with the narrative query "birth control" is shown as follows: taylor is female. write a social media post which explains taylor's thoughts on the easy access to birth control and their political party. could you provide a confidence level for your assessment, expressed as a number between 0 and 1? you have to start your reply with 'absolutely! i think '. limit your response within 40 words. an example narrative query for income based on the demographic feature "education" with the narrative query "vacation" is shown as follows: taylor's education level is doctorate. what do you think taylor's income is and given this income, describe taylor's dream vacation? limit your answer to 20 words. for anes, an adjusted macro f1 score was used as our evaluation metric, where we took the average of the f1 measure for the democrat and republican class. 2 for income, the macro f1 score was used as the evaluation metric. the datasets were partitioned into 6-folds, holding out the 6th fold as the test set. the experiments were ran across 3 seeds (1,2, and 3) and the results were averaged across those seeds. a mlp classification head consisting of one hidden layer, gelu activation function, and 0.5 dropout was used (see appendix). we explore a number of narrative queries for each dataset. for anes, each narrative is a social media post describing a person's stance on one from a set of issues (e.g., abortion or climate change). for income, the narratives are about a dream vacation. ## discussion and conclusion our method exploits a joint representation of tabular and text data. here, we explore straightforward joint representations, although the general principle can be applied to learning more complex spaces. we also generate rich text from tabular data, e.g. narratives around opinions and beliefs. this is in contrast to existing work which explores serialization approaches similar to our descriptive text @xcite @xcite , work which employs chatgpt as a few-shot classifier @xcite and work which generates synthetic tabular data by fine-tuned llms @xcite . while others have focused on table understanding tasks, such as question answering and table reasoning @xcite @xcite , our contribution is geared towards inference. by posing appropriate and contextually relevant questions to llms, we generate responses that integrate additional insights drawn from the extensive pre-training of these models. this enriched textual data provides supplementary context and information, potentially enhancing the performance of downstream predictive tasks. overall, representing the data as text tends to improve predictive performance over the tabular data itself, which is notable given that neither dataset provides text data. however, an interesting result of this work is that the descriptive text often outperforms the narrative text. this finding suggests that the improvements we see arise chiefly from the general language understandability of the encoder llms rather than the generative llms. we hope this work provides inspiration for future work towards the end of improving predictive performance for tabular data.