Generative Table Pre-training Empowers Models for Tabular Prediction (2023)

Abstract: Recently, the topic of table pre-training has attracted considerable research interest. However, how to employ table pre-training to boost the performance of tabular prediction remains an open challenge. In this paper, we propose TapTap, the first attempt that leverages table pre-training to empower models for tabular prediction. After pre-training on a large corpus of real-world tabular data, TapTap can generate high-quality synthetic tables to support various applications on tabular data, including privacy protection, low resource regime, missing value imputation, and imbalanced classification. Extensive experiments on 12 datasets demonstrate that TapTap outperforms a total of 16 baselines in different scenarios. Meanwhile, it can be easily combined with various backbone models, including LightGBM, Multilayer Perceptron (MLP) and Transformer. Moreover, with the aid of table pre-training, models trained using synthetic data generated by TapTap can even compete with models using the original dataset on half of the experimental datasets, marking a milestone in the development of synthetic tabular data generation. The code and datasets are available at https://github.com/ZhangTP1996/TapTap.

## introduction recently, pre-trained language models (lms) have attracted a lot of research interest in different domains, especially in the area of natural language processing. after pre-training on a large-scale unstructured text corpus with a self-supervised training objective, e.g., masked language modeling (mlm) proposed by bert @xcite , lms can significantly benefit downstream tasks. furthermore, recent progress on generative lms @xcite @xcite suggests that it is possible to unify different tasks via one lm. the remarkable success table table question answering @xcite question: how many people with a bachelor's degree earn less than 50k? table fact verification @xcite hypothesis: more than 20% has a hs-grad or higher education. table-related tasks with representative table pre-training models, including tapas @xcite , tabert @xcite , intermediate @xcite , tapex @xcite and our taptap. ## conclusion & future work in this paper, we propose taptap, a table pretraining method to empower models for tabular prediction. it can be combined with various backbone models and boost them via synthesizing highquality tabular data. a large-scale empirical study demonstrates that taptap can benefit different sota backbone models on four tabular prediction scenarios. in the future, we plan to extend taptap to process tables with a large number of features.