TWT : Table with Written Text for Controlled Data-to-Text Generation (2021)

Abstract: Large pre-trained neural models have recently shown remarkable progress in text generation. In this paper, we propose to generate text conditioned on the structured data (table) and a prefix (the written text) by leveraging the pre-trained models. We present a new data-to-text dataset, Table with Written Text (TWT), by repurposing two existing datasets: ToTTo and TabFact. TWT contains both factual and logical statements that are faithful to the structured data, aiming to serve as a useful benchmark for controlled text generation. Compared with existing data-to-text task settings, TWT is more intuitive, the prefix (usually provided by the user) controls the topic of the generated text. Existing methods usually output hallucinated text that is not faithful on TWT. Therefore, we design a novel approach with table-aware attention visibility and copy mechanism over the table. Experimental results show that our approach outperforms state-of-the-art methods under both automatic and human evaluation metrics.

## introduction data-to-text refers to the task of generating a target textual description conditioned on the structured source data such as tables, graphs, and meaning representations. @xcite suggest that a natural language generation (nlg) system consists of content planning (what to say) and surface realization (how to say it). recent deep neural network-based approaches do not explicitly model these stages and are trained in an end-to-end fashion using the popular encoder-decoder architecture @xcite with the attention mechanism @xcite . they achieved promising results on existing data-to-text datasets, such as webnlg @xcite , e2enlg @xcite , wikibio @xcite , rotowire @xcite , totto @xcite , and logicnlg @xcite . it should be noted that content planning is the key factor for data-to-text generation @xcite . different users might interpret different parts of the structured data. this issue may not be severe for datasets (e.g. webnlg @xcite ) that require the generated text to cover all records. however, when the golden sentence only covers part of the records (e.g. wik-ibio @xcite ), end-to-end methods that do not explicitly address content planning may output open-ended targets, which leads to unreliable generated results, and places challenges in evaluation. in nlg, one way to provide signals on what to generate is to add constraints to the model output, which falls in the task of controlled text generation (ctg). most ctg tasks are conditioned on several key-value pairs of control factors such as tone, tense, length, and sentiment @xcite @xcite . in data-to-text, @xcite propose the dataset totto to address content planning by highlighting some cells in the table, the highlighted cells provide strong guidance on what to generate. however, totto lacks practical use, it would be difficult to have tables with highlighted cells or ask the users to highlight the cells in the real application. one important application of nlg is to provide writing assistance such as next word prediction or text auto-completion. in this scenario, a natural content planning signal will be the written text provided by the user, which could be a word, a phrase, or an incomplete sentence. for the example shown in figure 1 , given the table, users might interpret different parts of the data with different prefixes. text generation under this scenario requires inferring the user's intention on content planning based to encourage the research in controlled data-totext generation, we present a new dataset, table with written text (twt), by repurposing two existing datasets: totto @xcite and tab-fact @xcite . see section 3 for details about the dataset construction. twt contains both factual and logical statements that are faithful to the structured data. compared with other datasets, twt is of practical use. the prefix controls the topic of the generated text, and the output model could assist in writing with structured data. note that twt differs from those datasets that provide only one golden sentence with no content planning signals. to generate text faithful to the data, we design a novel approach that leverages large pre-trained models @xcite with table-aware attention visibility (based on the written text) and copy mechanism @xcite over the table. experimental results show that our approach outperforms state-of-the-art methods under both automatic and human evaluation metrics, particularly in terms of faithfulness to the structured data. these results suggest that twt could be a useful controllable data-to-text benchmark, and may help innovate models to provide intelligent assistance for writing with structured data. ## related work data-to-text aims to generate natural language from structured data, which has been widely studied recently. most prior works focus on surfacelevel text generation in a specific domain or schema, such as robocup @xcite , weathergov @xcite , e2enlg @xcite , and webnlg @xcite . these datasets expect the generated text to describe all the records from the data. wikibio @xcite requires the target text to cover salient records with no explicit guidance on the generated topic. totto @xcite guide the topic of the generated target with a set of highlighted table cells. logic-nlg @xcite and logic2text @xcite address logical inference/generation in data-to-text. rotowire @xcite and totto @xcite also contain data that requires reasoning. many existing works tend to train neural models in an end-to-end fashion @xcite @xcite . recently, large pre-trained models @xcite @xcite have also achieved new state-of-the-art results on data-to-text tasks. @xcite suggest that an nlg system consists of content planning and surface realization. @xcite propose totto to control the topics of generated text with highlighted cells. @xcite brings the sense of numerical value comparison into content planning. @xcite propose to generate templates and then fill the slots, while @xcite incorporate writers' information to generate text step-by-step. @xcite utilize hierarchical encoders with dual attention to consider both the table structure and history information. in nlg, controlled text generation is also a hot research topic. it considers controlling attributes, such as identity of the speaker @xcite , sentiment @xcite , tense @xcite , politeness @xcite ) and text length @xcite . our work could be considered as a middle-ground between data-to-text and controlled text generation and has more practical usage. ## methodology with transformer-based structures, finetuning taskspecific models with pre-trained parameters has achieved state-of-the-art results in text generation, achieving an astonishing level of fluency and coherence. pre-trained models with a encoderdecoder structure such as bart @xcite , bert2bert @xcite , and t5 @xcite can be easily applied to data-to-text tasks. for example, on totto, feeding the highlighted cells with row and column header as input and finetuned with bert2bert or t5 achieves relatively high performance @xcite . figure 4 presents an overview of our model. we use a transformer-based encoder with additional positional (row/column) embeddings to encode table structure. we introduce structured encoder-decoder attention visibility based on the prefix to attend to the prefix-relevant sub-structure of the original table. for the decoder, we employ bi-directional attention for the prefix and uni-directional attention for the generated target as the decoder selfattention visibility. we also introduce the copy mechanism over the table data to assure the faithfulness of the generated target. note that our model is based on the transformer encoder-decoder architecture @xcite , both the encoder and the decoder are initialized with pre-trained parameters. ## experiments 1 following @xcite on selecting the baselines on totto, we exam the following state-ofthe-art text generation approaches on twt. note that for baseline models, the input is the metadata concatenated with the table flattened row by row, with no additional table-aware embeddings introduced in section 5.1. ## conclusion in this paper, we propose table with written text (twt), a new controlled data-to-text generation dataset. for this task, we design a novel approach with table-aware attention visibility and copy mechanism over the table. experimental results show that our approach could generate faithful text over state-of-the-art pre-trained models under both automatic and human evaluation. for future work, we will focus on generating text with logical inference on twt.