Faithful Low-Resource Data-to-Text Generation through Cycle Training (2023)

Abstract: Methods to generate text from structured data have advanced significantly in recent years, primarily due to fine-tuning of pre-trained language models on large datasets. However, such models can fail to produce output faithful to the input data, particularly on out-of-domain data. Sufficient annotated data is often not available for specific domains, leading us to seek an unsupervised approach to improve the faithfulness of output text. Since the problem is fundamentally one of consistency between the representations of the structured data and text, we evaluate the effectiveness of cycle training in this work. Cycle training uses two models which are inverses of each other: one that generates text from structured data, and one which generates the structured data from natural language text. We show that cycle training, when initialized with a small amount of supervised data (100 samples in our case), achieves nearly the same performance as fully supervised approaches for the data-to-text generation task on the WebNLG, E2E, WTQ, and WSQL datasets. We perform extensive empirical analysis with automated evaluation metrics and a newly designed human evaluation schema to reveal different cycle training strategies’ effectiveness of reducing various types of generation errors. Our code is publicly available at https://github.com/Edillower/CycleNLG .

## introduction a wealth of information exists in the form of structured knowledge, such as movie information databases or product catalogs, which we may want to verbalize for a variety of purposes, such as comparing two items, or presenting detailed descriptions in a natural language form suitable for conversational assistants. recent work has tackled this data-to-text generation task using freely available † the research was done during an internship at amazon. * these two authors contributed equally to this work. public datasets, most notably webnlg @xcite and totto @xcite . however, there remain two major challenges. first, the volume of training data required for good performance, especially if it is not in a domain represented by one of the existing corpora, is very large. second, multiple recent papers @xcite , inter alia, point out that neural natural language generation (nlg) from structured data tends to produce multiple kinds of errors which limit the utility of these models in customer-facing applications. hallucinations occur when nlg models inject nonsensical words or information not related to the input structured data, into the generated output text. for instance, an nlg model may claim a shirt's color is "three". simple factual errors occur when an nlg model produces coherent but factually wrong output. there are two threads of research to consider as we attempt to tackle these problems in the datato-text setting. the first is designing models that directly produce output more faithful to the input data. the second is designing models to detect and correct factual errors or hallucinations after the output text is generated. in both cases, prior research has generally assumed sufficient pairs of structured data and text as training data to achieve human-level performance on the task. while fact verification models can achieve very high performance, they generally do so when trained on large corpora of 100,000 examples or more. since performance appears to degrade when evaluated on outof-domain data @xcite , this presents a significant limitation of fact-verification models. similarly, corpora like webnlg contain about 20,000 examples; this is probably too small to achieve human performance even under full supervision @xcite but is large enough to make it prohibitive to generate domain-specific corpora of the size of webnlg. in spite of the above mentioned limitations, very few of the models developed for data-to-text and table -to-text tasks take advantage of the fact that the task of faithful text generation is fundamentally one of consistency between the data and the corresponding text. in fact, despite the webnlg 2020 challenge being explicitly bi-directional, only three models competing in the challenge leveraged this idea of consistency. to overcome the aforementioned limitations related to the lack of training data (especially out-of-domain data) and the consistency between structured data and text, we adopt a cycle training @xcite approach. we assume unpaired data d, in the form of subject-predicateobject triples, and text t , which may or may not be from the same domain. we also make use of a small (100 samples) set of paired data and text, d pr , t pr . cycle training makes use of two iteratively trained models, a forward model f : @xmath0: @xmath1. training is unsupervised, namely, we freeze one model and use it to transform one set of inputs, and train the other by using it to predict the original input from the output of the first model. concretely, in one cycle, we freeze f, and train r by reconstructing the input d as r(f(d)). after one training epoch, we reverse the roles of the two models. remarkably, even though the models are initially quite poor, this can converge to models with near-supervised performance, as we will show. moreover, we show that this process ensures the faithfulness of the output text with respect to the input data, and vice versa, even with very little or no paired data. we note that a previous data-to-text system, cy-clegt, has used cycle training @xcite ). we will discuss in detail the differences between cyclegt and our proposed approach in section 2. moreover, we examine in detail the conditions under which cycle training works well, with an emphasis on domains and the nature of the training text and structured data. we find that unsupervised cycle training outperforms low-resource fine-tuned models and can achieve near fully-supervised performance when initialized and post-tuned with a small amount of annotated data. we detail the results and findings in section 5. thus, to build on past research in self-consistent data-to-text generation, we make these novel contributions: ## related work multiple data-to-text and table-to-text tasks have been presented in the literature, such as webnlg @xcite @xcite , dart @xcite , totto @xcite , and wikitablet @xcite , which primarily consist of data from general-purpose sources like wikipedia. several large language models @xcite @xcite have been trained on large scale table-to-text corpora @xcite to perform fact verification. however, these models may not perform well on specific domains they have not been trained on, such as ecommerce @xcite . therefore, we must either find a way to easily generate new data to train large data-to-text models, or use unsupervised methods. recently, @xcite attempted to augment training data using gpt-3 @xcite , and @xcite employed an information retrieval system to build prototypes for the generation. our work makes orthogonal contributions to these studies, as we directly utilize the underlying unpaired data and text of a target corpus without recourse to any additional information retrieval or generation systems. further, the above-mentioned data-to-text tasks have been evaluated primarily on automatic word-or ngram-level metrics such as bleu @xcite or meteor @xcite , with minimal (and mostly subjective) evaluation of faithfulness. in this work, we design a novel annotation schema to perform a more comprehensive evaluation of the faithfulness of the generated text to the input data. cycle training @xcite relies on two models which are essentially inverse transforms of each other that are used to create "cycles", which should return identical output to the input given. there are two distinct forms of cycle training. the first form @xcite aims to learn to transform from one input form to another, e.g., to learn rotations of a car in one image to another. the second is the use of a "cycle consistency loss" as an auxiliary loss to some other task, e.g., in generative adversarial networks performing style transfer on images @xcite . nlg typically relies on models which are autoregressive and non-differentiable. this precludes the direct use of cycle consistency losses @xcite @xcite . nonetheless, we can still use cycle training via an alternating training strategy where we freeze one model and train the other, and vice versa @xcite . in this work, we train solely using cycle consistency. cycle training has been recently applied to language processing tasks. in one text-to-text application, @xcite use a similar unsupervised methodology to perform bidirectional text transformations for converting keyword search queries to natural language questions, and vice versa. it has also been used for named entity recognition in the absence of large annotated text @xcite . in this case, one model extracts entities, and the inverse model creates text from those entities. the approach is limited by the fact that there are many ways to realize sentences with the same entities. put differently, there is no strong requirement of cycle consistency, and this will become even more apparent as we analyze the conditions under which cycle training works well in data-to-text tasks. to the best of our knowledge, the only work to explicitly call out the self-consistency requirement of data-to-text generation tasks is the cyclegt model @xcite developed for data-totext generation on the webnlg dataset. one key advantage of cycle training is that it need not rely on any supervision, and instead relies primarily or solely on the self-consistency of inputs and outputs. however, cyclegt relies on a pre-existing ner model to extract entities from the output text. the authors then train an inverse model to predict the links between entities and predicates. should the entities not be recognized by their ner system, the model will fail overall; this is not an uncommon situation in applications such as online shopping @xcite , where entities are complex or change frequently @xcite . in principle, a separate ner model could be built using cycle training, as in cyclener @xcite , but the cyclegt authors did not do so. in this work, we design a simple approach using pre-trained language generation models, fine-tuned for both data-to-text and text-to-data generation cycles. ## conclusions in this work, we demonstrated the application of cycle training for data-to-text generation. we systematically investigated the effectiveness of cycle training across different domains, and the application of pre-cycle fine-tuning in low-resource settings. we showed that our approach substantially improved data-to-text generation performance in low-resource settings, achieved competitive performance compared to fully-supervised models, and also improved the faithfulness of the generated text through a reduction in factual errors, hallucinations and information misses, even when compared to fully supervised approaches. we also designed a schema for effective human evaluation of data-totext generation, that improves upon prior work and encourages more objective and consistent reviews of faithfulness.