Text-to-Text Pre-Training for Data-to-Text Tasks (2020)

Abstract: We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5 (Raffel et al., 2019), enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternatives such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-ofdomain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

## introduction natural language generation from structured data, or data-to-text @xcite , is the task of generating natural language text conditioned on source content provided in the form of structured data such as a table, graph etc. some example applications include task oriented dialog @xcite , summarizing weather forecasts @xcite , etc. in this work we study the applicability of large scale text-to-text transfer learning learning for this task. in particular, we focus on pre-training in the form of the "text-to-text transfer transformer" (t5) models released by @xcite . finetuning t5 achieves state-of-the-art results on diverse benchmarks spanning task oriented dialogue (multiwoz), tables-to-text (totto) and graph-totext (webnlg). empirical results further demonstrate the following: our approach is simple, only scratching the surface of what is possible. there is much to be explored in the space of leveraging unlabelled data, developing unsupervised objectives etc. that are more tailored for generating text from structured data. we hope our work serves as a useful baseline for future research, as pre-training becomes ever more prevalent for this task. we rely on the t5 pre-trained models released by @xcite . they consist of a transformer based encoder-decoder architecture. these models were pre-trained in a multitask fashion with an unsupervised "span masking" objective on common crawl data as well as supervised translation, summarization, classification, and question answering tasks. note that none of the supervised tasks include language generation from structured data. to study the impact of model capacity, we experiment with different t5 variants -small (60 million parameters), base (220 million), large (770 million) and 3b (3 billion). ## fine-tuning our modeling approach is simple. the data-totext task is cast in the text-to-text framework by representing the structured data as a flat string (linearization). figure 1 shows examples of the input representation for each dataset. we then fine-tune t5 on the data-to-text corpus for a small number of steps. following @xcite , models are finetuned with a constant learning rate of 0.001. we use a batch size of 131,072 tokens, and a maximum input length of 512 tokens. the maximum training steps is set to 5k for webnlg, while the larger totto dataset is trained for 10k steps. the t5 vocabulary consists of 32,000 sentencepieces. all the model parameters are updated in the fine-tuning process. the best checkpoint is chosen based on the bleu @xcite score on the development set. decoding is done via greedy search. in the final evaluation, for each dataset we rely on metrics used by prior work. ## datasets we conduct experiments on 3 english datasets spanning a variety of domains. each dataset uses a different kind of structured data (tables, meaning representations and graph/triples). ## conclusion in this study we evaluated pre-training in the form of t5 for the data-to-text task. we found that it leads to state-of-the-art results, while greatly improving robustness to out-of-domain inputs. in the future, we hope to design unsupervised pre-training objectives that are specifically tailored for the datato-text task. we also hope to extend this work to multiple languages, especially low resource ones.