Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity (2020)

Abstract: End-to-end neural data-to-text (D2T) generation has recently emerged as an alternative to pipeline-based architectures. However, it has faced challenges generalizing to new domains and generating semantically consistent text. In this work, we present DataTuner, a neural, end-to-end data-to-text generation system that makes minimal assumptions about the data representation and target domain. We take a two-stage generation-reranking approach, combining a fine-tuned language model with a semantic fidelity classifier. Each component is learnt end-toe-nd without needing dataset-specific heuristics, entity delexicalization, or post-processing. We show that DataTuner achieves state of the art results on automated metrics across four major D2T datasets (LDC2017T10, WebNLG, ViGGO, and Cleaned E2E), with fluency assessed by human annotators as nearing or exceeding the human-written reference texts. Our generated text has better semantic fidelity than the state of the art on these datasets. We further demonstrate that our model-based semantic fidelity scorer is a better assessment tool compared to traditional heuristic-based measures of semantic accuracy.

## introduction data-to-text generation (d2t) is defined as automatically generating natural language texts from nonlinguistic inputs @xcite . interest in this task has been driven by its applicability to specialized domains. for instance, d2t has been applied to generating weather reports @xcite , restaurant descriptions @xcite , and video game dialogues @xcite . recently, researchers have investigated d2t with more diverse domains to arrive at more generalizable text generation (such as works on ldc2017t10 @xcite and webnlg @xcite . traditional approaches to d2t follow a pipeline-based methodology, dividing the problem into several sub-problems @xcite . these include content selection (which information to include in the text), text structuring (the order in which to present the data), sentence aggregation (which information goes in individual sentences), lexicalization (finding the right words and phrases to express the data), referring expression generation (selecting the words and phrases to identify domain objects), and linguistic realization (combining all the generated words and phrases into wellformed sentences). in recent years, there has been a growing interest in going beyond pipeline-based approaches towards end-to-end (e-to-e) methods driven by recent advancements in deep learning @xcite @xcite . such methods can be trained with @xcite tuples that can be efficiently collected at scale. in contrast, each step in pipeline-based approaches requires its own setup and training data, such as semantic alignments between sections of text and components of the meaning representation (mr). this makes them more costly and complex to develop and more prone to error propagation. to date, end-to-end d2t has faced two main challenges: (1) generalization to unseen domains and (2) maintaining semantic fidelity to accurately convey the source data. in a recent comparative @xcite found that, compared to the best pipeline-based system, e-to-e approaches based on gru and transformer architectures scored more than 35 bleu points lower on unseen domains from the webnlg dataset, and scored worst for semantic accuracy. to address these challenges, we introduce datatuner, an e-to-e, domain-independent d2t system that makes no assumptions about the generated text's domain or the mr's structure. datatuner leverages a pretrained language model and fine-grained state embeddings to achieve strong generalization. it also employs a weakly-supervised semantic fidelity classifier (sfc) to detect and avoid generation errors (such as hallucination and omission). we also leverage this classifier to assess outputs from any d2t system, overcoming the limitations of existing heuristic methods for detecting semantic errors. in this work, we deliver four main contributions across four major d2t datasets from various domains and mrs. we open-source the datatuner code at https://github.com/amazon-research . ## related work pipeline vs. end-to-end approaches: within the pipeline-based paradigm, several studies have illustrated that breaking the d2t problem into sub-problems improves overall performance. @xcite showed that separating planning from realization helps achieve better semantic faithfulness compared to an e-to-e neural approach on the webnlg dataset. castro @xcite conducted a comparative study across a variety of e-to-e and pipeline approaches with webnlg, concluding that the latter are significantly better at generalizing to unseen domains. however, so far the e-to-e approaches in these studies have been trained from scratch on the task dataset. our work investigates whether using a pretrained model with strong language generation capabilities raises the performance of e-to-e models. ## problem description the d2t task is formally defined as generating text t from data d that is encoded via a meaning representation mr. we assume that content selection is done prior to the d2t task, an assumption also made in the datasets we use. therefore, the text t should have semantic fidelity by conveying all the input data, and only the input data. ## datatuner architecture we designed datatuner to be highly generic in order to tackle diverse meaning representations and allow d2t generators to be built for new datasets with minimal work beyond data preprocessing. at a high-level, our text generation system takes a 2-stage approach: generation and reranking. first, we fine-tune a pretrained language model on the d2t task using the task's training data. next, we build a specialized semantic fidelity classifier trained on an automatically-generated task-specific corpus. using these models, we construct a customized beam-search decoder that ranks candidates based on the probabilities from the language model, and, at its final stage, reranks them based on the classifier's labels. ## experiments for each dataset, we generate outputs from three versions of datatuner for our ablation studies. datatuner no fc/fs simply relies on the d2t-lm, with no sfc-based reranking and a coarsegrained version of the state embeddings that contains @xmath0. datatuner no fc adds the fine-grained state embeddings described in section 4.1 to datatuner no fc/fs. datatuner fc adds the sfc-based reranking. for the sfc, we train the model using the roberta-large model (355m parameters) on lower-cased text. on the synthetic test set generated, the classifier has a macro-averaged f1-score (across 5 classes) of 97%, 97%, 98%, and 98% for the ldc2017t10, webnlg, cleaned e2e, and viggo datasets respectively. we use the models bundled within the huggingface transformers library @xcite . the d2t-lm uses the gpt-2-medium model (with 345m-parameters) as its base model. the beam search width during decoding is 5. training was performed on a single machine (amazon aws p3.8xlarge). during inference, text from datatuner is generated at an average rate of 11.8 tokens per second on nvidia tesla k80 gpus. we evaluate each variant's outputs with automated metrics and crowdsourced fluency and fidelity evaluation. we also quantify the efficacy of our semantic fidelity classifier with expert-annotations. we compare against the state of the art systems on each dataset, selected based on bleu scores. these are a graph-optimized transformer seq2seq model by @xcite for ldc2017t10, transformerbased models (pipeline and e2e) by castro @xcite for webnlg, an lstm seq2seq model with a rule-based reranker by @xcite for cleaned e2e, and a transformer seq2seq model by @xcite for viggo. the supplementary material contains the outputs from our system variants and the main training hyperparameters (obtained via manual tuning). ## conclusion we presented datatuner, an end-to-end data-to-text generation system equipped with a semantic fidelity classifier. datatuner records new state of the art results on four different datasets, with significant margins on automated metrics. we also show that our system has a clear fluency advantage over all the previous state of the art models. we further illustrate datatuner's strengths for delivering semantically accurate outputs.