Few-shot Table-to-text Generation with Prefix-Controlled Generator (2022)

Abstract: Neural table-to-text generation approaches are data-hungry, limiting their adaption for low-resource real-world applications. Previous works mostly resort to Pre-trained Language Models (PLMs) to generate fluent summaries of a table. However, they often contain hallucinated contents due to the uncontrolled nature of PLMs. Moreover, the topological differences between tables and sequences are rarely studied. Last but not least, fine-tuning on PLMs with a handful of instances may lead to over-fitting and catastrophic forgetting. To alleviate these problems, we propose a prompt-based approach, Prefix-Controlled Generator (i.e., PCG), for few-shot table-to-text generation. We prepend a task-specific prefix for a PLM to make the table structure better fit the pre-trained input. In addition, we generate an input-specific prefix to control the factual contents and word order of the generated text. Both automatic and human evaluations on different domains (humans, books and songs) of the Wikibio dataset prove the effectiveness of our approach.

## introduction table -to-text generation is a significant branch of natural language generation (nlg), aiming at generating descriptive text given an input table. there is a wide range of application scenarios for automatic table-to-text generation, such as sport news generation @xcite , story generation @xcite , weather forecasting report @xcite , and open-domain question answering @xcite . recent years have witnessed the great development of pre-trained language models (plms) @xcite @xcite , which achieve state-of-the-art performance on many text generation tasks, such as neural machine translation, document summarization, etc. unlike these tasks, table-to-text generation faces the lack of labeled data. due to the development of data science, many statistical tables are generated in our daily life, but they scarcely have corresponding natural language descriptions, which limits the real-world application of data-hungry pre-trained models. to address this problem, researchers investigate workarounds in the few-shot setting. @xcite , @xcite and @xcite leverage pre-trained linguistic knowledge of neural language models, then fine-tune them in target domains with limited labeled data. this "pre-train and fine-tune" paradigm performs relatively well in generating descriptive text from tables. recently, another paradigm named "pre-train and prompt" has been proposed in order to adapt plms to downstream tasks without fine-tuning, which is more suitable for few-shot and zero-shot scenarios. li and @xcite prepends prompt tokens to adapt table-to-text generation to sequential generation task, and freezes plms' weights to fully leverage their prior knowledge learned in the pre-training stage. despite their contributions, however, two main challenges for table-to-text generation remain to be explored, namely (1) the topological structure difference between tables and sequential inputs and (2) model's ability to select and rearrange factual content from tables. in order to address the aforementioned problems, we follow the "pre-train and prompt" paradigm and propose prefix-controlled generator (i.e., pcg), an end-to-end generation framework along with two kinds of prefix tokens. specifically, we prepend a task-specific (i.e., static) prompt and an input-specific (i.e., dynamic) prompt to the tabular input. the task-specific prompt aims to bridge the topological structure gap between a table and a word sequence, while the input-specific prompt aims to plan the factual content and the slot order of a table. both prefixes are optimized during attribute(k) value(v) name edinho júnior fullname edon júnior viegas amaral birth_date 7 march 1994 birth_place faro , portugal currentclub farense clubnumber 21 position forward gold: edon júnior viegas amaral , known as edinho júnior ( born 7 march 1994 ) is a portuguese footballer who plays as a forward for farense . prefix-tuning: edon júnior ( born 7 march 1994 in faro , portugal ) is a portuguese football midfielder . bart: edino júnior ( born 7 march 1994 ) is a portuguese footballer who plays for farense as a forward . switch+plm: edon júnior viegas amaral or edinho júnior ( born 7 march 1994 in faro ) is a portugal football player and he plays for farense . ours: edinho júnior viegas amaral ( born 7 march 1994 ) is a portuguese footballer who plays as a forward for farense . ). switch+plm represents chen et al. (2020)'s approach with bart-large. the training phase with the plm remaining frozen, making our approach parameter-efficient -we only save one copy of the plm while training in three different domains. we basically follow the idea of prefix-tuning (li and @xcite to design the task-specific prefix, except for some modifications. firstly, due to the importance of a proper initialization of prefix tokens, we use task-relevant words (e.g., "summarize the following table:", or "tl;dr:") as the initial prefix to better linearize the tabular input. secondly, @xcite proves that the length of prefix tokens and the design of adding additional parameters solely on the attention module are two bottlenecks of prefix-tuning. inspired by their work, we add scaled parallel adapters @xcite in parallel with both the attention layer and the feed-forward layer to improve the bottleneck of prefix-tuning. for the input-specific prefix, we expect it can hint to the model which key-value pairs should be selected and in what order they should be arranged. therefore, we propose a content planner to select the keys that appear in the gold summary and sort them according to the order of occurrence in the summary. for example, given a table in fig. 1 , we expect the content planner to generate a word sequence "fullname name birth_date birth_place position currentclub" that indicates all the keys and their occurrence order whose values appear in the gold summary. the word sequence will be used as hard prompts to feed into the plm. we evaluate our model on multi-domain table-totext dataset @xcite . we show that our model outperforms previous state-of-the-art methods on both automatic evaluation metrics ( §5.4) and human evaluation metrics ( §5.5). we also con-duct ablation studies to verify the effectiveness of the two kinds of prefixes ( §5.6). in a nutshell, our contributions are as follows: ## problem definition given a table t with n key-value pairs {k i : the key and the value of the i-th table slot respectively, we aim to generate a fluent and faithful natural language description of the table in a low-resource constraint. note that k i and v i represent sequences of m and m ′ words respectively. ## methodology we first provide intuition of using a task-specific prefix and an input-specific prefix for few-shot table-to-text generation ( §4.1). fig. 2 depicts the overall architecture of our method. as shown in the figure, given the input table, the content planner selects the factual contents and reorders them to form a dynamic prompt ( §4.2). after that, a static prompt is designed and fed to the generative plm along with the dynamic prompt ( §4.3). ## conclusion in this paper, we propose prompt-controlled generator, using two kinds of prompts to address current challenges in few-shot table-to-text generation. the task-specific prefix aims to bridge the topological structure gap between tables and sequences, which is learned via freezing the plm and tuning the continuous prompt vectors. the input-specific prefix is designed to guide the generation process in terms of factual content and word order. we propose content planner to generate the input-specific prefix. experiments on wiki-humans, wiki-books and wiki-songs datasets prove the effectiveness of our method from the aspects of generation fluency and text fidelity to the table. table content generated description name: cody zeller article_title: cody zeller image: cody zeller iu hoosiers standout in chicago img 5907.jpg position: power forward/center height_ft: 7 height_in: 9 weight_lb: 240 league: nba team: charlotte bobcats/hornets number: 40 nationality: american birth_date: 5 october 1992 birth_place: washington, indiana high_school: washington, indiana college: indiana (2011 -2013) draft_year: 2013 draft_round: 1 draft_pick: 4 draft_team: charlotte bobcats career_start: 2013 years: 2013 start -present gold: cody allen zeller (born october 5, 1992) is an american professional basketball player who currently plays for the charlotte hornets of the national basketball association (nba). plan: birth_date nationality draft_team league switch+bart: cody zeller (born 5 october 1992 in washington, indiana) is an american basketball player for the charlotte hornets in the nba as a power forward. pcg: cody zeller (born october 5, 1992) is an american professional basketball player who currently plays for the charlotte hornets of the national basketball association (nba). name: brandon pieters fullname: brandon paul pieters birth_date: 22 april 1976 birth_place: germiston , south africa height: 1.93 0 weight: 106 kg lb st on residence: benoni , south africa yearpro: 1994 tour: sunshine tour prowins: 4 sunwins: 4 article_title: brandon pieters gold: brandon paul pieters (born 22 april 1976) is a south african professional golfer. plan: full_name birth_date birth_place switch+bart: brandon paul pieters (born 22 april 1976) is a south african cricketer who plays for the central bank of southern africa. pcg: brandon paul pieters (born 22 april 1976) is a south african wrestler. name: david highbaugh smith image: david-highbaugh-smith.jpg alt: a man with dark hair and a mustache wearing a dark coat and white shirt state: kentucky district: 4th predecessor: john w. lewis successor: ben johnson birth_date: 19 december 1854 birth_place: hart county , kentucky death_date: 17 december 1928 death_place: hodgenville , kentucky restingplace: red hill cemetery party: democrat profession: lawyer article_title: david highbaugh smith gold: david highbaugh smith (december 19, 1854 -december 17, 1928) was a u.s. representative from kentucky. plan: name birth_date death_date state switch+bart: davic highbaugh smith (1854 -1928) was a u.s. lawyer from kentucky. pcg: david highbaugh smith (december 19, 1854 -december 17, 1928) was an american representative from kentucky. table content (3) image_size: 225px position: left wing shoots: left height_ft: 5 height_in: 7 weight_lb: 159 status: retired former_teams: modo hockey , skellefteå aik birth_date: 7 october 1975 birth_place: örnsköldsvik , swe draft: 234th overall draft_year: 1996 draft_team: boston bruins career_start: 1993 career_end: 2012 article_title: anders söderberg generated description anders söderberg (born october 7, 1975) is an american retired ice hockey player who played in the modo hockey league and the boston bruins of the nhl.