T able GPT : Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching (2020)

Abstract: Although neural table-to-text models have achieved remarkable progress with the help of large-scale datasets, they suffer insufficient learning problem with limited training data. Recently, pre-trained language models show potential in few-shot learning with linguistic knowledge learnt from pretraining on large-scale corpus. However, benefiting table-to-text generation in few-shot setting with the powerful pretrained language model faces three challenges, including (1) the gap between the task’s structured input and the natural language input for pretraining language model. (2) The lack of modeling for table structure and (3) improving text fidelity with less incorrect expressions that are contradicting to the table. To address aforementioned problems, we propose TableGPT for table-to-text generation. At first, we utilize table transformation module with template to rewrite structured table in natural language as input for GPT-2. In addition, we exploit multi-task learning with two auxiliary tasks that preserve table’s structural information by reconstructing the structure from GPT-2’s representation and improving the text’s fidelity with content matching task aligning the table and information in the generated text. By experimenting on Humans, Songs and Books, three few-shot table-to-text datasets in different domains, our model outperforms existing systems on most few-shot settings.

## introduction table-to-text generation, aiming at generating descriptive text about important information in structured data, has well application prospect in communicating with human in a comprehensible and natural way, such as financial report @xcite , medical report @xcite generation, etc. in recent years, data-driven models have shown impressive capability to produce informative and fluent text with the help of large-scale datasets, such as wikibio @xcite and e2e @xcite . however, it is not always feasible to collect large-scale labeled dataset for various domains in the real world, resulting in unsatisfying performance due to the insufficient training. such few-shot learning setting for table-to-text generation is not well-explored, and in this paper, we focus on exploring how to efficiently model for few-shot table-to-text generation with limited training pairs. recently, pre-trained language models have shown promising progress in various natural language processing tasks @xcite @xcite . they can capture linguistic knowledge by pretraining on large-scale unlabeled dataset and generalize to downstream tasks with little labeled data in target domain, effectively modeling for few-shot setting @xcite . however, efforts to benefit table-to-text generation from the powerful pre-trained language model, especially in few-shot setting, are non-trivial due to three challenges. (1) there is a gap between the structured data input for table-to-text generation and natural language input that is used for pretraining gpt-2. (2) also, it lacks modeling of the table's structure which contains rich information to understand the input before generating text. (3) additionally, it doesn't address how to maintain text's fidelity for table-to-text gener-attribute value name james beattie fullname james scott beattie birth 27 february 1978 lancaster , england position striker currentclub swansea city ( assistant first team coach ) nationalteam england u21 england article james beattie ( footballer ) ... ... reference: james scott beattie ( born 27 february 1978 ) is an english former professional footballer who played as a striker . in order to alleviate aforementioned problems, we propose tablegpt that focus on generating highfidelity text for table-to-text generation with limited training pairs. addressing the gap between structured table input and natural language input that gpt-2 processes during pretraining, we utilize a table transformation module that employs template to naturally transform structured table into natural language. in addition, we utilize two auxiliary tasks under the framework of multi-task learning, table structure construction and content matching, targeting pretrained gpt-2's lack of modeling for table structure and text's fidelity. in detail, the table structure reconstruction task is proposed for gpt-2 which force it to embed table structure into its representation when modeling structured table. besides, we utilize content matching task that help model correctly describe important information from table via optimal-transport @xcite technique, which measures the distance between the information in generated text and information in table and use the distance as penalty for text with incorrect information. we conducted experiments on three data-to-text datasets on different domains @xcite ): humans, books, songs in various settings. both automatic evaluation and human evaluation results show that our model can achieve new state-of-the-art performance for table-to-text generation in terms of generating fluent and high-fidelity text in most few-shot settings. ## approach in this section, we propose to address the three incompatibility between pretrained language model and table-to-text generation illustrated in section 1. figure 1 presents the overall multi-task training framework of our model. we first utilize table transformation module to reasonably transform structured table into text sequence and aggregate it with reference text, resulting in a suitable training sequence for gpt-2 model. then, two auxiliary tasks: table structure reconstruction and content matching with optimal transport @xcite are performed on top of gpt-2's representation of the training sequence. those two auxiliary tasks' training objectives along with gpt-2's language model training objective are jointly finetuned based on the pre-trained gpt-2 under the multi-task training framework. the overall objective here is to produce high-fidelity text while maintaining its fluency. ## related work in recent years, neural models for generating texts directly from preprocessed data @xcite @xcite @xcite , have become mainstream for table-to-text generation and achieved impressive performance with the help of largescale dataset. @xcite proposes a pre-selector on encoder-aligner-decoder model for generation, which strengthens model's content selection ability and obtains considerable improvement over standard seq2seq model. @xcite proposes a hybrid attention mechanism for modeling the order of content when generating texts. @xcite presents a field-gating encoder focusing on modeling table structure and dual attention mechanism to utilize the structure information when decoding. in addition, @xcite develops a table-aware sequence-to-sequence model on this task. however, @xcite shows that the well-performed seq2seq model trained on large-scale dataset suffer from limited training data in few-shot setting. recently, gpt-2 has been successfully adapted to dialogue generation in few-shot setting @xcite , showing potential to address insufficient training data problem for few-shot learning with the help of learnt knowledge from pretraining on vast corpus. as for table-to-text generation, @xcite propose a switch model that use gpt-2 to generate template-like functional words while generating factual expressions via copying records' values from table in few-shot scenario. different from this work, we model the table and generate text within a gpt-2 model in a unified way and we show that our proposed tablegpt can perform well in the few-shot scenario. in addition, different from both works mentioned above, we enhance gpt-2's ability to model table structure and to improve text fidelity. another closely related paper is @xcite , which predicts whether a statement align with records in the table. since the nature of classification task makes it possible to model table records bidirectionally, it use bert with templates to transform and model the table. meanwhile, @xcite explores coarse-to-fine table-to-text generation with standard gpt-2 model. different from above two works, we adapted gpt-2 in a text generation scenario for structured data input and more importantly address table structure modeling and improve text fidelity. in addition, one of the auxiliary task: content matching is inspired by ideas in machine translation @xcite and seq2seq @xcite model. the closest paper on data-to-text generation is @xcite . they assume that the expected generated text should cover all information in the table. but in a more realistic scenario, like the task we explore in this paper, the table consists of redundant information and only the important ones should be used to constraint model to generate high-fidelity text. therefore, we propose to match important information only in the table and information in text as an auxiliary task during training. ## conclusion in this work, we present tablegpt, which enhances gpt-2 for table-to-text generation with two auxiliary tasks, table structure reconstruction and content matching, for improving text fidelity while exploiting gpt-2's learnt linguistic knowledge from pretraining on large-scale corpus. in detail, we use table transformation to bridge the gap between structured table and natural language input for gpt-2 and further enhance gpt-2 with following two auxiliary tasks for table-to-text generation. the table structure reconstruction task help model preserve the structural information of input while representing table with powerful pretrained gpt-2. in addition, the content matching task guides model to generate high-fidelity task with less incorrect expressions that are contradicting to the table via measuring distance between table and information in generated text. experiments are conducted on three datasets, humans, books and songs, in different domains. both automatic evaluation and human evaluation results show that our model achieves new state-of-the-art performance in most few-shot setting.