Guided Beam Search to Improve Generalization in Low-Resource Data-to-Text Generation (2023)

Abstract: In this paper, we introduce a new beam search algorithm that improves the generalization of neural generators to unseen examples, especially in low-resource data-to-text settings. Our algorithm aims to reduce the number of omissions and hallucinations during the decoding process. For this purpose, it relies on two regression models to explicitly characterize factual errors. We explain how to create a new dataset to train these models given an original training set of less than a thousand data points. We apply our approach in the low-resource, legal setting using the French Plum2Text dataset, as well as in English using WebNLG. We observe in our experiment that this combination improves the faithfulness of pre-trained neural text generators using both human and automatic evaluation. Moreover, our approach offers a level of interpretability by predicting the number of omissions and hallucinations present in a given generation with respect to the input data. Finally, we visualize our algorithm’s exploration of the hypothesis space at different steps during the decoding process.

## introduction data-to-text generation is commonly referred to as the task of verbalizing a structured input also known as a table of values. the table may contain several types of values such as text, numbers, categories, etc. in our study, we are specifically interested in improving the faithfulness of neural data-to-text generators. the relevance of their generations can be evaluated with respect to the coverage of the input table, i.e. to what extent the model omits values from the table. moreover, neural text generators unfortunately have the tendency to hallucinate facts from the training set. hence, generations can also be evaluated based on the number of hallucinated facts produced by the model @xcite . the tendency of neural data-to-text generators to omit values and/or hallucinate facts can be exacerbated in low-resource settings. the models may overfit the training set, and bring generalization to unseen data points at stake. in this paper, we propose to improve the faithfulness of data-to-text neural generators by reducing the number of hallucinations and omissions during the generation process, without having to re-train the generation models. this perspective has many incentives, especially since models are becoming larger and larger, thus harder to train @xcite . to this end, we propose a modified version of the beam search algorithm specifically for the data-to-text setting. @xcite studied the behavior of the beam search algorithm under a regularized framework, showing that beam search enforces uniform information density. that is, "it produces text with evenly distributed surprisal, a feature that human readers tend to prefer". inspired by this regularization framework, we introduce two characterization models that will guide the decoding algorithm by promoting generated beams containing fewer hallucinations and omissions. the characterization of omissions and hallucinations is crucial in the legal setting. hence, we apply our new decoding algorithm and analyze its benefits on the task of verbalizing criminal docket files using the plum2text dataset @xcite . using automatic and manual evaluation, we show that our algorithm improves generalization in a low-resource setting, especially on unseen data points. we also show that our approach generalizes to other datasets, such as webnlg @xcite . in the next section, we introduce related work regarding the mitigation of omissions and hallucinations for neural text generators. we then introduce the main contribution of this paper in section 3, a new decoding algorithm for the data-to-text setting. we present the experiments and analysis in section 4. we assess the generalization of our approach in sec-tion 5 by applying it on webnlg @xcite . we conclude with our observations in section 6. the models, datasets, generations and human evaluations are made publicly available 1 . ## related work in this section, we study current mitigation techniques of omissions and hallucinations to improve neural generators' performance, limiting ourselves to the data-to-text setting 2 . these techniques may require changing the architecture and are enforced either during training or during inference. we classify mitigation techniques as being "invasive" or "non-invasive" to the generator. invasive techniques require fine-tuning, adding a new objective function, or modifying the inner architecture of the generator. in this paper, we focus on non-invasive techniques, which consider the generator as a black box and act either on the input or during the decoding process. these techniques are appealing for the fact that they do not require re-training the generator on the original training dataset. for example, @xcite proposed autoprompt, a model that learns how to create prompts for various sets of tasks. they basically search for "trigger" tokens using the gradient from the downstream task. according to their results, autoprompt outperformed fine-tuning methods in cases where the training dataset is small (i.e. 100-1000 samples). this method, however, does not necessarily mitigate the omission and hallucinations in the data-to-text setting. similarly, prefix-tuning (li and liang, 2021) proposed a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen, but optimizes a small continuous task-specific vector, called the prefix. then again, their method does not specifically mitigate omissions and hallucinations. @xcite proposed hafez, a method weighing the current beam state based on a set of feature functions that take as input a target word, and sometimes the beam state (e.g. to check for repetitions). these feature functions, in our case, could be used to force the generation of proper charges, decisions, and pleading, for example. they added two terms to the standard beam 1 https://drive.google.com/drive/folders/ 1gdgwxr7gxoww2dc83xch3txgwyb3ov4w?@xmath0. guerreiro et al. (2023) introduced dehallu-cinator, a model that flags hypotheses once they are fully generated so that they can be overwritten. our model differs from their approach since we are guiding the exploration of the tree during decoding. finally, @xcite introduced diverse beam search, an algorithm that promotes diverse generations amongst groups of beams but does not strictly reward or penalize beams for specific properties. to the best of our knowledge, no method in the literature proposes a way that can be adapted without major changes to handle both omissions and hallucinations at the semantic level during the decoding step. moreover, none of the methods can explicitly estimate the number of hallucinations and omissions in the hypotheses. we thus wish to fill this gap by proposing a guided beam search algorithm to create more faithful neural data-to-text generations. ## guided decoding by predicting omissions and hallucinations in this section, we introduce a new decoding algorithm that is designed to mitigate and explicitly characterize omissions and hallucinations for data-to-text generation. to this end, we create two predictive models: one predicting the number of omitted values from the table, m o , and the other predicting the number of hallucinations, m h . these models will thus weigh the current beam score to promote generated sequences with few, or hopefully no omissions or hallucinations, enforcing semantically accurate generations. ## experiments in our experiment, we use criminelbart, a generative model introduced by @xcite . we only analyze the vanilla and guided versions of criminelbart since other methods proposed in the literature do not explicitly mitigate omissions and hallucinations. we trained criminelbart on the train set of plum2text, and we begin by automatically evaluating different versions of the weighted beam search using a grid search over the hyperparameters previously introduced. we then manually evaluate the performance of our new algorithm in section 4.2. to assess the generalization performance of our algorithm, we added examples with 37 new provisions from the criminal code of canada having no or very few occurrences in the original training set. furthermore, we qualitatively analyze the behavior of our algorithm in section 4.3. ## generalization of the approach we illustrate the generalization of our proposed approach to improve the beam search algorithm to other data-to-text settings by using webnlg, a well-known dataset in the nlp community. to this end, we used the same methodology described in section 3.3: the omission dataset o of webnlg consists of 20,448 examples resulting in train and test sets of 16,358 and 4,090 examples respectively using an 80%-20% split. the hallucination dataset h consists of 20,600 examples resulting in train and test sets of 16,480 and 4,120 examples respectively also using an 80%-20% split. similar to the plum2text setting, we trained the english version of bert @xcite available in the huggingface library using the same hyperparameters previously selected. for the actual data-to-text generation task we trained bart @xcite on the training set of webnlg and evaluated it on the test set after performing a grid-search over the guided beam search hyperparameters. we used the same automatic evaluation metrics as with plum2text. we can see from ## conclusion in this paper, we introduced a new guiding mechanism for the beam search algorithm in the datato-text generation setting. we presented how to train two models, one to predict the number of omissions, and the other for the number of hallucinations. these predictors are used not only to mitigate the number of omissions and hallucinations