R 2 D 2: Robust Data-to-Text with Replacement Detection (2022)

Abstract: Unfaithful text generation is a common problem for text generation systems. In the case of Data-to-Text (D2T) systems, the factuality of the generated text is particularly crucial for any real-world applications. We introduce R2D2, a training framework that addresses unfaithful Data-to-Text generation by training a system both as a generator and a faithfulness discriminator with additional replacement detection and unlikelihood learning tasks. To facilitate such training, we propose two methods for sampling unfaithful sentences. We argue that the poor entity retrieval capability of D2T systems is one of the primary sources of unfaithfulness, so in addition to the existing metrics, we further propose named entity based metrics to evaluate the fidelity of D2T generations. Our experimental results show that R2D2 systems could effectively mitigate the unfaithful text generation, and they achieve new state-of-theart results on FeTaQA, LogicNLG, and ToTTo, all with significant improvements.

## introduction data-to-text generation is the task of generating a text sequence that describes some salient information of a knowledge source. unlike text-to-text generation whose input source is a text sequence containing knowledge that is not extracted and represented in the canonical structured format, we assume that the input of a data-to-text system is represented in some structured format, e.g., rdf @xcite , relational or entity tables @xcite . the data-to-text task can be divided into two distinct components as in many other text generation tasks @xcite . the first component involves selecting salient information from the structured knowledge either based on natural language query or other indication of saliency, and the second component comprises organizing and planning of the previous selections to allow realization of the surface text. although this task has been studied comprehensively in many works, from task design, modeling techniques, to application in different domains @xcite @xcite @xcite , existing data-to-text (d2t) systems exhibit a shortcoming that cannot be neglected: they fail to reliably generate sentences that are faithful given the salient content of the input table @xcite @xcite . this limitation prevents the application of d2t systems in real world scenarios. we therefore need to investigate possible remedies. we introduce a framework that prevents unfaithful data-to-text generation by training a data-to-text system both as a generator as well as a faithfulness discriminator. for faithfulness discrimination, we adopt the replaced token detection objective, which was first proposed in electra @xcite . it was applied to the pre-training stage of the large-scale language models for more sample-efficient training of contextualized representations of sentences. electra is tasked to discriminate between original natural sentences and token-replaced sentences by locating the positions of replacement. the replaced tokens are sampled from a proposal distribution using a generator such as a masked language model to fill some masked tokens. in our work, we perturbed the entailed reference sentences with two different methods, a knowledgebased one and a model-based one, to obtain unfaithful sentences whose surface forms are close to those of original sentences (therefore having similar sequence likelihoods), but contradict to the input table. then we investigated ways of incorporating the discrimination task into the existing maximum likelihood learning. specifically, we explored the settings of learning the sentence-level detection and generation in tandem, and the token-level detection and generation in tandem. in addition, we also experiment with incorporating the unlikelihood training objective @xcite on these unfaithful sentences to test its utility. we conduct experiments on three data-to-text datasets to test the general applicability of our approach: fetaqa @xcite , logicnlg @xcite , and totto @xcite . each dataset presents distinct challenges while faithful generation is a common problem. we find that adding the faithfulness discrimination task mitigates the unfaithful data-to-text generation, supported by our results on multiple datasets, on all of which we are able to achieve new stateof-the-art results with evident improvements. we compare and analyze the performance of our system and existing state-of-the-art systems. to ensure the validity of the comparison, we also evaluate various metrics for their aptness of faithfulness evaluation. we released our model and code at https: //github.com/yale-lily/r2d2 . ## experiments we first introduce the datasets that we experiment with in section 3.1, and the metrics we adopted for evaluation in section 3.2. then we report the baseline models we are comparing with, and the implementation and training details in section 3.3. in section 3.4 and 3.5, we report and analyze both the automatic evaluation results and the human evaluation results. ## conclusion in this work, we introduced r2d2, a training framework that mitigates the unfaithful text generation problem for the d2t task. training with the regular maximum likelihood loss can lead to generation of sentences that are similar to the references but are unfaithful to the input. we therefore propose to add a discrimination task and an unlikelihood training to encourage the model to generate separable representations of these critical sentences. we proposed two methods of sampling these unfaithful sentences: the knowledge-based method exploits the structure of the input knowledge, and the model-based method samples the d2t model's own mistakes. we proposed ne-based metrics that assess the entity retrieval capability of the data-to-text systems, as we argued the incompetence of which is one of the leading causes of unfaithfulness. we experimented on multiple data-to-text datasets of different task constructs, and achieved noticeable improvements over the state-of-the-art performance.