ToTTo : A Controlled Table-To-Text Generation Dataset (2020)

Abstract: We present ToTTo, an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. To obtain generated targets that are natural but also faithful to the source table, we introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. We present systematic analyses of our dataset and annotation process as well as results achieved by several state-of-the-art baselines. While usually fluent, existing methods often hallucinate phrases that are not supported by the table, suggesting that this dataset can serve as a useful research benchmark for high-precision conditional text generation.

## introduction data-to-text generation @xcite is the task of generating a target textual description y conditioned on source content x in the form of structured data such as a table. examples include generating sentences given biographical data @xcite , textual descriptions of restaurants given meaning representations @xcite , basketball game summaries given boxscore statistics @xcite , and generating fun facts from superlative tables in wikipedia @xcite . existing data-to-text tasks have provided an important test-bed for neural generation models @xcite . neural models are known to be prone to hallucination, i.e., generating text that is fluent but not faithful to the source @xcite @xcite and it is often easier to assess faithfulness of the generated text when the source content is structured @xcite . moreover, structured data can also test a model's ability for reasoning and numerical inference @xcite and for building representations of structured objects @xcite , providing an interesting complement to tasks that test these aspects in the nlu setting @xcite @xcite . however, constructing a data-to-text dataset can be challenging on two axes: task design and annotation process. first, tasks with open-ended output like summarization @xcite @xcite lack explicit signals for models on what to generate, which can lead to subjective content and evaluation challenges @xcite . on the other hand, data-to-text tasks that are limited to verbalizing a fully specified meaning representation @xcite do not test a model's ability to perform inference and thus remove a considerable amount of challenge from the task. secondly, designing an annotation process to obtain natural but also clean targets is a significant challenge. one strategy employed by many datasets is to have annotators write targets from scratch @xcite @xcite which can often lack variety in terms of structure and style @xcite ). an alternative is to pair naturally occurring text with tables @xcite . while more diverse, naturally occurring targets are often noisy and contain information that cannot be inferred from the source. this can make it problematic to disentangle modeling weaknesses from data noise. world championships gothenburg, sweden 3rd 4x100 m relay 43.01 original text: after winning the german under-23 100 m title, she was selected to run at the 1995 world championships in athletics both individually and in the relay. text after deletion: she at the 1995 world championships in both individually and in the relay. text after decontextualization: gabriele becker competed at the 1995 world championships in both individually and in the relay. final text: gabriele becker competed at the 1995 world championships both individually and in the relay. for dataset construction, to ensure that targets are natural but also faithful to the source table, we request annotators to revise existing wikipedia candidate sentences into target sentences, instead of asking them to write new target sentences @xcite . due to the varied nature of wikipedia tables, totto covers a significant variety of domains while containing targets that are completely faithful to the source (see task design most existing table-to-text datasets are restricted in topic and schema such as weath-ergov @xcite , robocup @xcite , rotowire @xcite , e2e @xcite , kbgen @xcite , and wikibio @xcite . in contrast, totto contains tables with various schema spanning various topical categories all over wikipedia. moreover, totto takes a different view of content selection compared to ## preliminaries our tables come from english wikipedia articles and thus may not be regular grids. 2 for simplicity, we define a table t as a set of cells @xmath0. each cell contains: (1) a string value, (2) whether or not it is a row or column header, (3) the row and column position of this cell in the table, (4) the number of rows and columns this cell spans. let @xmath1, m section-title , m section-text ) indicate table metadata, i.e, the page title, section title, and up to the first 2 sentences of the section text (if present) respectively. these fields can help provide context to the table's contents. let @xmath2, ..., s η ) be a sentence of length η. we define an annotation example 3 @xmath3, m, s) a tuple of table, table metadata, and sentence. here, @xmath4. ## dataset collection we first describe how to obtain annotation examples d for subsequent annotation. to prevent any overlap with the wikibio dataset @xcite , we do not use infobox tables. we employed three heuristics to collect tables and sentences: number matching we search for tables and sentences on the same wikipedia page that overlap with a non-date number of at least 3 non-zero digits. this approach captures most of the table-sentence pairs that describe statistics (e.g., sports, election, census, science, weather). cell matching we extract a sentence if it has tokens matching at least 3 distinct cell contents from the same row in the table. the intuition is that most tables are structured, and a row is usually used to describe a complete event. hyperlinks the above heuristics only consider sentences and tables on the same page. we also find examples where a sentence s contains a hyperlink to a page with a title that starts with list (these pages typically only consist of a large table). if the table t on that page also has a hyperlink to the page containing s, then we consider this to be an annotation example. such examples typically result in more diverse examples than the other two heuristics, but also add more noise, since the sentence may only be distantly related to the table. using the above heuristics we obtain a set of examples d. we then sample a random subset of tables for annotation, excluding tables with formatting issues: 191,693 examples for training, 11,406 examples for development, and 11,406 examples for test. among these examples, 35.8% were derived from number matching, 29.4% from cell matching, and 34.7% from hyperlinks. ## data annotation process the collected annotation examples are noisy since a sentence s may only be partially supported by the table t. we thus define an annotation process that guides annotators through incremental changes to the original sentence. this allows us to measure annotator agreement at every step of the process, which is atypical in existing generation datasets. the primary annotation task consists of the following steps: (1) table readability, (2) cell high-lighting, (3) phrase deletion, (4) decontextualization. after these steps we employ a final secondary annotation task for grammar correction. each of these are described below and more examples are provided in the table readability if a table is not readable, then the following steps will not need to be completed. this step is only intended to remove fringe cases where the table is poorly formatted or otherwise not understandable (e.g., in a different language). 99.5% of tables are determined to be readable. cell highlighting an annotator is instructed to highlight cells that support the sentence. a phrase is supported by the table if it is either directly stated in the cell contents or meta-data, or can be logically inferred by them. row and column headers do not need to be highlighted. if the table does not support any part of the sentence, then no cell is marked and no other step needs to be completed. 69.7% of examples are supported by the table. for instance, in phrase deletion this step removes phrases in the sentence unsupported by the selected table cells. annotators are restricted such that they are only able to delete phrases, transforming the original sentence: @xmath6. in on average, s deletion is different from s for 85.3% of examples and while s has an average length of 26.6 tokens, this is reduced to 15.9 for s deletion . we found that the phrases annotators often disagreed on corresponded to verbs purportedly supported by the table. decontextualization a given sentence s may contain pronominal references or other phrases that depend on context. we thus instruct annotators to identify the main topic of the sentence; if it is a pronoun or other ambiguous phrase, we ask them to replace it with a named entity from the table or metadata. to discourage excessive modification, they are instructed to make at most one replacement. 4 this transforms the sentence yet again: ## dataset analysis basic statistics of totto are described in ## machine learning task construction in this work, we focus on the following task: given a table t, related metadata m (page title, section title, table section text) and a set of highlighted cells t highlight , produce the final sentence s final . mathematically this can be described as learning a function f : @xmath7, m, t highlight ) and @xmath8. this task is different from what the annotators perform, since they are provided a starting sentence requiring revision. therefore, the task is more challenging, as the model must generate a new sentence instead of revising an existing one. ## experiments we present baseline results on totto by examining three existing state-of-the-art approaches (note that since our tables do not have a fixed schema it is difficult to design a template baseline). details about hyperparameter settings are provided in the appendix. moreover, we explore different strategies of representing the source content that resemble standard linearization approaches in the literature @xcite • full table the simplest approach is simply to use the entire table as the source, adding special tokens to mark which cells have been highlighted. however, many tables can be very large and this strategy performs poorly. overall though, we find this representation leads to higher performance. in all cases, the cells are linearized with row and column separator tokens. we also experiment with prepending the table metadata to the source table . foot_3 evaluation metrics the model output is evaluated using two automatic metrics: bleu @xcite and parent @xcite . parent is a metric recently proposed specifically for data-to-text evaluation that takes the table into account. we modify it to make it suitable for our dataset, described in the appendix. human evaluation is described in § 8.2. ## model errors and challenges ## conclusion we presented totto, a table-to-text dataset that presents a controlled generation task and a data annotation process based on iterative sentence revision. we also provided several state-of-the-art baselines, and demonstrated totto could serve as a useful research benchmark for model and metric development. totto is available at https://github.com/ google-research-datasets/totto . target sentence: extreme temperatures of montpellier have ranged from -17.8 °c recorded in february and up to 37.5 °c (99.5 °f) in july. table title: montpellier section title: climate table description: none