u FACT : Unfaithful Alien-Corpora Training for Semantically Consistent Data-to-Text Generation (2022)

Abstract: We propose uFACT (Un-Faithful Alien Corpora Training), a training corpus construction method for data-to-text (d2t) generation models. We show that d2t models trained on uFACT datasets generate utterances which represent the semantic content of the data sources more accurately compared to models trained on the target corpus alone. Our approach is to augment the training set of a given target corpus with alien corpora which have different semantic representations. We show that while it is important to have faithful data from the target corpus, the faithfulness of additional corpora only plays a minor role. Consequently, uFACT datasets can be constructed with large quantities of unfaithful data. We show how uFACT can be leveraged to obtain state-of-the-art results on the WebNLG benchmark using METEOR as our performance metric. Furthermore, we investigate the sensitivity of the generation faithfulness to the training corpus structure using the PARENT metric, and provide a baseline for this metric on the WebNLG (Gardent et al., 2017) benchmark to facilitate comparisons with future work.

## introduction data-to-text (d2t) generation is the task of generating fluent text t given a set of information units, linearised into data source string d ( ## related work early approaches @xcite formalize d2t generation as three subtasks: content determination, structuring/grouping of information, and surface realisation. a handcrafted system is designed to solve each task. recently, the focus has shifted towards end-to-end neural approaches, incorporating each of the subtasks into one system @xcite @xcite . a number of end-to-end approaches to increasing faithfulness in d2t generation are curative, i.e. address generation quality post-hoc. for instance, @xcite and @xcite produce candidate generations first, and then judge faithfulness with a separate model, by checking entailment between d and t g . another approach to enhance faithfulness is to alter the generation model. @xcite propose a generation model comprised of a copy-generate gate within an lstm positional encoder. the gate acts as a soft switch between a copy-from-data mode and a language-generation mode. kale (2020) utilise transfer learning to enhance their generation model, through pre-training on a large unsupervised, taskagnostic corpus. a different line of research focuses on preventative approaches, where the typical aim is to obtain a better model by improving the training data quality. @xcite apply a unigram-based dataset selection process, by removing examples for which t r is not sufficiently related to d. parikh et al. (2020) also investigate this approach, releasing the noise-free totto dataset, to ensure the training data does not encourage unfaithful generation. filippova (2020) look for hallucinative examples in their dataset, either considering word-overlap, or comparing how strongly a language model vs. a conditional language model anticipates subsequent text. @xcite develop the parent metric, a faithfulness-quantifying f-score that takes into account the data source in addition to the potentially divergent reference, providing a more robust assessment of the d2t mapping. in their work on model-agnostic meta-learning, @xcite note that training on different instances of a required task (e.g., training on different corpora) can facilitate learning a particular task. inspired by this approach, we add other corpora with different semantic representations to the training dataset. we find not only that adding corpora boosts the semantic faithfulness of the d2t generator, but also that said corpora need not necessarily satisfy stringent faithfulness requirements, unlike the target corpus. ## constructing a ufact dataset typically, a d2t generation model is obtained by task-specific fine-tuning, where a large-scale pretrained model such as t5 @xcite is fine-tuned on a small corpus. ufact however, as an instance of mixed-corpus training, takes a different approach: examples from multiple corpora which do not share semantic representations, are linearised and tagged to form a large training corpus. a ufact dataset is comprised of a target dataset for which we desire to maximise d2t generation fidelity and alien corpora. the latter are d2t corpora that may differ thematically and structurally from the target corpus and whose role is to improve generation fidelity on the target corpus. ## conclusion we have presented the ufact-method, which boosts the faithfulness of data-to-text generation models by appropriately constructing the training corpus. training t5 on a mixture of d2t corpora results in strong semantic accuracy increase, as long as and the target corpus remains faithful. ufact's lax constraints on the majority of the training set mitigates the scarcity problem in finding faithful d2t corpora, thus making faithful d2t generation more practically feasible. the new state-of-the-art meteor score proves that language models alone, if trained with a carefully constructed dataset, can be highly effective data-to-text generators.