PARENT ing via Model-Agnostic Reinforcement Learning to Correct Pathological Behaviors in Data-to-Text Generation (2020)

Abstract: In language generation models conditioned by structured data, the classical training via maximum likelihood almost always leads models to pick up on dataset divergence (i.e., hallucinations or omissions), and to incorporate them erroneously in their own generations at inference. In this work, we build on top of previous Reinforcement Learning based approaches and show that a model-agnostic framework relying on the recently introduced PARENT metric is efficient at reducing both hallucinations and omissions. Evaluations on the widely used WikiBIO and WebNLG benchmarks demonstrate the effectiveness of this framework compared to state-of-the-art models.

## introduction data-to-text aims at generating natural language descriptions from structured data @xcite ; fostered by recent advances on neural approaches and the emergence of large scale datasets made of (structured-data, reference text) pairs @xcite @xcite . figure 1 illustrates an example from the wikibio dataset @xcite . these datasets are either hand-crafted via crowdworkers or automatically built by aligning sources found on the internet. as such, reference texts might include divergences of two types, limiting the ability of generation models to produce realistic descriptions. first, reference texts might contain information not grounded in the source data; especially for automatically constructed datasets, where references were not written with the sourcedata description task in mind. for instance, the phrase "who served as lieutenant [...]" in figure 1 has no basis in the associated infobox. second, reference texts do not always cover the entirety of the table (items battles/wars in figure 1 ). in most settings, this second point is referred to as content selection and is inherent of most data-totext tasks. however, some hand-crafted datasets are designed where annotators are asked to transcribe every fields, with models also expected to do the same. in this case, incomplete references (i.e. where some part of the source data is missing from the realization) can lead to models failing to learn to transcribe all information, and only partially cover data-sources at inference. divergence in training examples leads to hallucinated/omitted content in model output; which is a well-known problem in neural approaches for text generation @xcite . this problem arises both from the training procedure (training via maximum likelihood leads to language models strongly mimicking human behaviors), and from the testing protocols. indeed, current standard metrics only measure similarity (such as bleu @xcite , rouge @xcite , meteor @xcite ) to ground truth reference texts and do not fully capture relevance to the source data. thus, there is no distinction between a mismatch caused by a paraphrase, poor lexicalization of content, or made-up/incorrect statement, leading to imperfect model selection. while a number of work argue for the need for novel automatic evaluation method @xcite @xcite , to the best of our knowledge only @xcite and @xcite propose metrics based on both the reference and the source data. recently, different regularization methods have also been proposed to mitigate the negative influence of divergences in reference texts. these approaches can be either at the dataset level @xcite , where authors propose techniques to clean/standardize instances; or at the training level @xcite , where authors propose novel neural modules designed to limit hallucinations/omissions. however, these approaches are severely limited: e.g., they require significant annotation labor, model-specific tricks and/or manual tuning. furthermore, virtually all proposed neural approaches still suffer from 1) exposure bias and 2) inconsistency between train/test measurement. indeed, current neural models are trained via a mechanism called teacher forcing @xcite , where the decoder is fed the previous correct token, no matter its actual prediction (1), in order to maximize the log-likelihood of the target sentence (including divergent phrases), but are evaluated through the previously discussed n-gram metrics (2). see section 3.3 for a more detailed discussion about this subject. to the best of our knowledge, there have been few approaches @xcite focused on the training procedure. @xcite train a hierarchical encoder-decoder on three auxiliary tasks (namely sequence labeling, text auto-encoder and multi-labeling classification) which are meant to guide the decoding process. closest to our work, @xcite propose a novel neural module for constrained attention, along with a reinforcement learning (rl) training procedure based on bleu and tfidf. in our work, to remedy the above shortcomings and building upon the work of @xcite , we show that no novel neural module is necessary to handle hallucinations and omissions. we propose a model-agnostic rl framework, called parenting, where pretrained models are further trained with a self-critical policy gradient algorithm @xcite to limit the impact of divergences in training examples on text generation. specifically, we use the parent metric @xcite which exhibits a strong correlation with human evaluation, while being easier to use out of the box. we provide extensive automatic evaluations on two data-to-text model families (lstms and transformers) on two widely used benchmarks (wikibio and webnlg), as well as a more focused human evaluation on wikibio. we report new state of the art parent scores on both datasets while bleu scores are on par with previous sota approaches, which shows that our framework efficiently reduces pathological behaviors while keeping generation fluent. ## model-agnostic reinforcement learning for reducing divergences we propose parenting, a model-agnostic rl framework for data-to-text aiming at reducing divergences. it is based on the self-critical policy gradient algorithm @xcite and leverages the parent metric @xcite . ## discussion and conclusion in this work, we have proposed a model-agnostic reinforcement learning framework for data-to-text aimed at reducing hallucinations and improving recall/coverage of relevant information. we shaped the reward based on parent @xcite , which is a recently proposed metric with a high correlation with human judgement. this allows for a more flexible training, where the model learns to depend less on the reference and more on the source data. framework effectiveness is assessed via thorough experiments on two model family (rnns and transformers) and two benchmarks (wikibio and webnlg). furthermore, quantitative and qualitative evaluations show that our parenting framework obtains better results than a dedicated attention module or a less source-relying reward. however, this approach relies on the metric employed and crafting an effective metric is still an open problem. in particular, parent is designed for single-entity datasets, like wikibio and webnlg, which is not reliable for more complex datasets containing multiple entities (i.g., the ro-towire dataset @xcite ). in this setting, the sentence "james harden scored 20 points." could achieve a high parent score if any player had scored 20 points in the game. an interesting future work would be the design of an evaluation metric more robust to dataset peculiarities.