Few-Shot Table Understanding: A Benchmark Dataset and Pre-Training Baseline (2022)

Abstract: Few-shot table understanding is a critical and challenging problem in real-world scenario as annotations over large amount of tables are usually costly. Pre-trained language models (PLMs), which have recently flourished on tabular data, have demonstrated their effectiveness for table understanding tasks. However, few-shot table understanding is rarely explored due to the deficiency of public table pre-training corpus and well-defined downstream benchmark tasks, especially in Chinese. In this paper, we establish a benchmark dataset, FewTUD, which consists of 5 different tasks with human annotations to systematically explore the few-shot table understanding in depth. Since there is no large number of public Chinese tables, we also collect a large-scale, multi-domain tabular corpus to facilitate future Chinese table pre-training, which includes one million tables and related natural language text with auxiliary supervised interaction signals. Finally, we present FewTPT, a novel table PLM with rich interactions over tabular data, and evaluate its performance comprehensively on the benchmark. Our dataset and model will be released to the public soon.

## introduction relational tables, as a typical form of structured data on the web, store a vast amount of knowledge. table understanding @xcite aims to understand the semantics of tabular data as well as the associated text jointly, which further improves the evaluation results of several tasks, including table question answering @xcite @xcite , table retrieval @xcite , and table fact verification @xcite . recently, inspired by the huge success of pre-trained language models (plms) in understanding freeform natural language (nl) sentences, researchers * both authors contributed equally to this work. in real-world scenario, table understanding usually faces more challenging situations, in which tables are from different domains and each table contains very limited annotations. despite its importance, few-shot @xcite table understanding is rarely explored in previous works due to the following three obstacles. first, the deficiency of public well-designed benchmark datasets for few-shot table understanding makes the model evaluation inconvenient. second, the lack of public large-scale high-quality table pre-training corpora blocks the exploration of table plms. lastly, a table pre-training baseline tailored to the few-shot table understanding is also needed for better performance comparison. especially, the lack of benchmark datasets and pre-training corpora in chinese hinders the research on table understanding. to fill the above gaps, in this paper, we focus on the dataset construction for table-understanding related tasks in chinese. we first establish a few-shot table understanding benchmark dataset, fewtud, with five table related tasks, by which research and exploration can be carried out extensively. different from existing table understanding tasks, which concentrate on the information inside tables, our tasks, based on real-world scenarios, lay emphasis on the interaction between tables and the corresponding nl text @xcite , and focus on the whole content of tables (e.g., table classification). we collect tables with meta information and the corresponding nl text from the web, then manually annotate the dataset for table processing with column identification and cell value splitting, (3) table-text alignment and relevance score estimation. second, we collect a large-scale chinese table pre-training corpus, tpc-1m, with more than 1 million tables and their associated nl text. how to obtain semantically relevant nl text is critical for constructing the table pre-training corpus. previous works either obtain the associated nl text by synthesising pseudo nl text on available tables @xcite @xcite or crawling surrounding nl text of tables simply based on position information @xcite . differently, we locate the associated nl text via semantic matching approach. we also provide the relevance score between the nl text and each row/column of the table as auxiliary supervised interaction signal, which may further facilitate the table pre-training. lastly, we propose a novel table pre-training model, fewtpt, to serve as the baseline for the few-shot table understanding benchmark. different from previous table pre-training models @xcite , we focus on catching the interactions between tables and nl text. specifically, we devise two novel spatial-aware pretraining tasks, i.e., column relevance prediction (crp) and row relevance prediction (rrp), which predict the relevant column from table schema and the relevant row from table content respectively based on the supervised interaction signals. we also employ variants of masked language modeling (mlm) objective to predict the tokens in nl text, column names and cell values. finally, we conduct extensive experiments to show the advantage of fewtpt against several strong baselines. to summarise, our main contributions are threefold. (1) we establish a chinese benchmark dataset for few-shot table understanding, which includes five downstream tasks: table fact verification, table question answering (table qa), table selection, schema detection, and table classification. we hope the benchmark can be a testbed for future few-shot table understanding research in chinese. (2) we contribute a large-scale high-quality table pre-training corpus in chinese, which covers 1 million tables across 13 domains and their semantically-associated nl text. (3) we propose a novel table pre-training model to serve as the baseline for the benchmark, and experimental results show the competitiveness of our model. ## construction of tpc-1m and fewtud in this section, we introduce the construction of table pre-training corpus, tpc-1m, and benchmark dataset for few-shot table understanding, fewtud. ## experiments here, we first introduce the experimental setup of pre-training and the comparable baselines, then we analyze the main results and also conduct ablation study to show the contribution of different modules. ## conclusion in this paper, we focus on the few-shot table understanding problem and establish a benchmark dataset with five downstream tasks including table fact verification, table qa, table selection , schema detection, and table classification. we also contribute a large-scale chinese tabular corpus which covers 1 million tables across 13 domains and the semantically-associated nl text. finally, we provide a table pre-training method and conduct extensive experiments on the few-shot table understanding benchmark to set up the baselines. experimental results demonstrate that catching the interactions between text and tables can improve the downstream tasks significantly. we hope the benchmark, tabular corpus, and the baselines can facilitate the future research on this field. in the future, we will explore more table structure friendly objectives to improve the pre-training.