T ab P rompt: Graph-based Pre-training and Prompting for Few-shot Table Understanding (2023)

Abstract: Table Understanding (TU) is a crucial aspect of information extraction that enables machines to comprehend the semantics behind tabular data. However, existing methods of TU cannot deal with the scarcity of labeled tabular data. In addition, these methods primarily focus on the textual content within the table, disregarding the inherent topological information of the table. This can lead to a misunderstanding of the tabular semantics. In this paper, we propose TabPrompt, a new framework to tackle the above challenges. Prompt-based learning has gained popularity due to its exceptional performance in few-shot learning. Thus, we introduce prompt-based learning to handle few-shot TU. Furthermore, Graph Contrastive Learning (Graph CL) demonstrates remarkable capabilities in capturing topological information, making Graph Neural Networks an ideal method for encoding tables. Hence, we develop a novel Graph CL method tailored to tabular data. This method serves as the pretext task during the pre-training phase, allowing the generation of vector representations that incorporate the table’s topological information. The experimental results of outperforming all strong baselines demonstrate the strength of our method in few-shot table understanding tasks.

## introduction abundant tabular data resources are readily available nowadays @xcite . however, the extensive knowledge within these corpora remains largely untapped because most tables are designed to be "human-friendly" rather than "machine-friendly" @xcite . consequently, to extract knowledge stored within these vast tables, it is crucial to enable machines to comprehend the semantics of tabular data, referred to as table understanding @xcite . furthermore, tu is fundamental in numerous subsequent tasks, including knowledge base augmentation @xcite and table qa @xcite . early methods involved manual feature construction designed for specific table datasets @xcite or utilizing cnn or lstm architectures @xcite . these methods demonstrate good performance only when there is an ample amount of manually labeled training data available. however, labeling tabular data requires high labor costs due to the inherent characteristics of tables, such as their two-dimensionality and flexible layouts. consequently, these methods struggle to cope with the challenge posed by the scarcity of labeled tabular data. more recently, researchers have explored the application of pre-trained language models to the task of tu @xcite @xcite . during the pre-training phase, the model learns optimal initialization parameters through self-supervised training on many unlabeled tabular data, enabling it to achieve excellent performance after fine-tuning downstream tasks. although self-supervised pre-training reduces the burden of manual annotation to some extent, it still requires a substantial amount of labeled table data during the fine-tuning process. thus, the existing "pre-training, fine-tuning" methods also remain insufficient in addressing the scarcity of labeled tabular data when performing tu tasks. in other words, existing methods struggle to perform well in few-shot tu. on the other hand, tables are structured data that encompass not only textual content but also topological information related to their layout. however, the existing methods sometimes misunderstand the tabular semantics resulting from disregarding the topological information within tables. for example, many pre-training models @xcite flatten a table and concatenate its contents row by row into one-dimensional plain texts as the input of the encoder. these methods only focusing on textual content overlook the topological information of the table, as cells in the same column typically have semantic commonalities. as depicted in fig. 1 (a), cells in the second column belong to the language category. to address the above two challenges, we propose a new framework called tabprompt. first, we incorporate soft prompts @xcite into the framework and devise a prompt-based learning method suitable for tabular data. recently, the new paradigm of "prompt-based learning" has attracted extensive attention due to its remarkable performance in few-shot scenarios @xcite . prompt-based learning enables more effective knowledge transfer by making downstream tu tasks more compatible with the pre-training task. by leveraging it, tabprompt is capable of effectively tackling the challenges posed by few-shot tu. second, we employ a novel graph contrastive learning (graph cl) to encode tabular data during pre-training. gnns are well-suited for processing data with topology @xcite , making them an ideal choice for capturing the topological information of tabular data. in addition, to encode the intrinsic structure knowledge, cl is one the most effective and popular pretext tasks @xcite . in this regard, tabprompt introduces a graph constructing method that fully considers the internal relationship between cells, which enables tabprompt to better learn the topological structure information of the table during pretraining. to evaluate tabprompt, we conduct extensive experiments in few-shot scenarios on three public datasets by comparing tabprompt against several strong baselines. the results of outperforming all the baselines demonstrate the effectiveness of tabprompt in the few-shot scenarios. the contributions of this paper are summarized in the following points: ## related works graph neural network (gnn). gnn architectures, such as gcn @xcite and gin @xcite , have gathered substantial interest among researchers for their remarkable capability to handle real-world data containing inherent topological structure. recognizing that tables inherently embody topological information, several studies @xcite have explored the application of gnns in tablerelated research fields. however, their methods of constructing the tabular graph fail to fully consider the topological relationships between different cell types, resulting in the misunderstanding of tabular semantics. pre-training and fine-tuning. since the introduction of bert, the "pre-training and fine-tuning" paradigm has gained significant attention. researchers have adapted this paradigm to tu by designing customized pretext tasks tailored to tabular data @xcite @xcite . while these methods effectively leverage unlabeled data during pre-training, they still rely on a substantial amount of labeled data in the fine-tuning stage to achieve optimal performance. as a result, these methods struggle to handle fewshot tu scenarios where only a limited amount of labeled data is available. additionally, these methods often flatten the table into a sequential input, disregarding the inherent topology of the table. this can lead to the loss of crucial topological information during the modeling process. other deep learning-based tu. in the early stages of table understanding (tu) research, the emphasis was primarily on cell entity linking @xcite @xcite . these methods often relied on pre-defined ontologies and external knowledge bases, which limited their versatility. however, more recent work has shifted towards addressing broader tu tasks such as table cell classification @xcite and table type classification @xcite . these methods typically utilize manual features or early neural architectures like lstm, which need a large quantity of manually labeled data for training. prompt-based learning. the main idea of the new paradigm "pre-train, prompt, and predict" @xcite is to reformalize the target task to look more like the pretext task to use better what the model has already learned. an increasing number of novel prompt-based learning methods have been proposed, encompassing various forms of prompts, such as soft prompts (li and liang, 2021). however, it is worth noting that there is currently no research available on the development of prompts specifically designed for tabular data. ## proposed method in this section, we first introduce the preliminaries of tu sub-tasks relevant to our work and then describe tabprompt in detail. in this section, we begin by introducing the preliminaries of tu subtasks relevant to our work. subsequently, we present a comprehensive description of tabprompt. ## experiments in this section, we conduct experiments on public datasets to evaluate the performance of tabprompt in both the ctc and ttc tasks. ## conclusion in this paper, we proposed a new framework tabprompt. to tackle the scarcity of labeled tabular data and capture the untapped topological information, we resorted to prompt-based learning and graph cl, respectively. the experimental results of outperforming all baselines demonstrate the effectiveness of tabprompt in few-shot tu. in future work, we aim to extend our method to handle domain-specific tables and more complex table structures, further enhancing its capabilities. in addition, we introduce a multimodal technique that leverages both text and image data within the table to enhance the understanding of its semantics.