Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data (2020)

Abstract: Neural text generation (data- or text-to-text) demonstrates remarkable performance when training data is abundant which for many applications is not the case. To collect a large corpus of parallel data, heuristic rules are often used but they inevitably let noise into the data, such as phrases in the output which cannot be explained by the input. Consequently, models pick up on the noise and may hallucinate–generate fluent but unsupported text. Our contribution is a simple but powerful technique to treat such hallucinations as a controllable aspect of the generated text, without dismissing any input and without modifying the model architecture. On the WikiBio corpus (Lebret et al., 2016), a particularly noisy dataset, we demonstrate the efficacy of the technique both in an automatic and in a human evaluation.

## introduction deep neural network-based (dnn) models have demonstrated remarkable performance on a multitude of text-to-text @xcite @xcite as well as data-to-text generation tasks @xcite . to reach high performance, dnn models require a large training corpus which is normally not readily available. indeed, it is rare to have a sufficiently large human-curated corpus of parallel data @xcite , and researchers have come up with heuristic rules to mine input-output pairs on a large scale @xcite @xcite . no matter how powerful, dnn models are known to be sensitive to data artifacts @xcite and pick on the noise in the training data. while hallucinations have not been defined formally, the term is standardly used to refer to the generated content which is either unfaithful to the figure 1 : two wikibio sources and targets with example attributes: tense and length can be read-off the target directly. when added to the input, the model gets a knob to control for length and tense. we propose to estimate the noise degree by comparing the source with the target thus obtaining a hallucination knob. we consider a particularly noisy dataset, wik-ibio @xcite , which has been found to have extra information in 62% of the references @xcite and where 1:1 correspondence between the input and the output never holds perez-beltrachini and gardent (2017). our models demonstrate superior performance to the model of @xcite which reports sota bleu results on wikibio. in sum, our contributions are (1) a novel idea of controlling hallucinations which requires no modification to the model, (2) a dataand task-independent technique of implementing this idea and (3) three-way evaluation with human raters which confirms that faithfulness does not need to be traded for coverage. ## controlling hallucinations controlled language generation is used when when one wants the output to exhibit a certain attribute. for example, in sentence compression @xcite one may wish to control the length of the output to fit a length budget or fairly compare different models. this can be achieved by reading the length off the training data and using it as an additional input during training so that during inference one obtains a "length knob" @xcite , fig. 1 ). apart from length, many other attributes like sentiment, style or theme can be controlled for, becoming an additional input for the encoder or the decoder @xcite . controlled generation is a powerful technique which has recently been shown to work in a multi-task setting when the task itself becomes an attribute @xcite . the attribute that we are interested in controlling for is the amount of hallucinations or noise. we define a special vocabulary of hallucination degrees and add such a degree as a prefix to the input for every datapoint. figure 2 shows the same input prepended with three different degrees and the three corresponding outputs from our controlled model trained on wikibio. while it is straightforward to measure output length or detect sentiment, it is less obvious how to estimate the amount of noise in a given example. in what follows, we use the words noise and hallucinations interchangeably. ## detecting hallucinations in the training data to detect hallucinations in the training data targets, we consider (3.1) an overlap-based technique, which has a clear foundation but cannot be applied to any seq2seq task, and (3.2) a simple procedure applicable in any setting. both methods give us a hallucination score @xmath0, 1] for every sourcetarget pair. the scores are converted into categorical values with quantiles: five intervals, each covering 20% of the full range, are introduced and a special tag is used for every interval. during training, the data2text model learns an embedding for each of the five tags and during inference the tag with the lowest hallucination value, hal 0, is used (fig. 2 ). ## experiments the primary goal of the experiments is to verify whether hallucinations can indeed be controlled for: we compare a seq2seq model trained on the wikibio data as is with the same model trained with the noise attribute annotated (by the word overlap and lm-based methods). we also evaluate the model of @xcite , which reported sota bleu results, and the model of @xcite , which was designed to generate hallucination-free output. in our automatic evaluation, we measure bleu @xcite as well as the recently introduced parent metric designed specifically for data2text tasks and verified on wikibio @xcite . unlike bleu, it compares the output not only with the reference but also measures how much of it is entailed by the input table. while parent is much more appropriate than bleu for data2text evaluation, in its standard implementation it may miss a paraphrase of a table field in the target sentence (e.g., spouse hardly ever occurs on the target side). it may also assign points for a match with the reference which is unsupported by the table. thus, it can give a wrong estimate of both precision and recall and should be complemented with a human evaluation if two similar performing models are compared. to this end, in our experiments with human raters we measure fluency and faithfulness of generated sentences as well as coverage: we need all three as we do not want to favor models which generate fluent and faithful but short sentences because fluency and faithfulness can be trivially achieved with a handful of templates. fluent sentences are natural and grammatically correct (fluent, mostly fluent and not fluent). we report the percentage of fluent sentences. faithful sentences express information supported by the table or by non-expert background knowledge (faithful, mostly faithful and not faithful). since there is a grey area of what can be inferred from the table without expert knowledge 1 , we report the percentage of faithful and mostly faithful sentences to the total. coverage counts table cells with the information expressed in the generated sentence. faithfulness and coverage can be seen as precision and recall metrics, respectively. we randomly selected 200 examples from the test set and collected three ratings for every input table and a generated output. ## discussion comparing the two methods of estimating the amount of hallucinations in a target, for applications where the input and the output use the same vocabulary with a comparable term distribution the overlap method may be better as it has a clear foundation. the lm-based method that we proposed has an important advantage that it makes no assumptions about the data. in our wikibio experiment it also produced better results in the human evaluation, presumably because it allowed for paraphrasing and straightforward inferences. for example, the target ozren nedoklan was a yugoslav footballer and manager. has a high hal w o score because the source table has no occupation field and does not mention yugoslav. the hal lm score of that example is zero because footballer and manager can be inferred from the names of the clubs and the manageryears fields in the source. possible extensions it should be emphasized that alternative methods of detecting noise can be explored and may perform better in the controlledhallucination framework. for example, it is possible to measuring target-source similarity in an embedded space or use word alignment tools to find unsupported information. while here we have focused on eliminating hallucinations, one can think of applications where one is interested in generating adversarial sentences which sound fluent but are guaranteed to include unsupported information. figure 2 shows how the amount of hallucinations in the output increases following the value of the hallucination knob. why is blue so different? it is striking that while all the models tested outperform @xcite in terms of parent and human evaluation scores, none could approach its bleu performance. we do not have an explanation of why this is so but note that our results are in line with the review by @xcite who concludes that bleu is an inappropriate metric for generation tasks other than mt. can we measure length instead of noise? one may wonder whether an even simpler approach of controlling for length would deliver a similar reduction in hallucinations. indeed, hallucinations and length are expected to correlate, and shorter length should result in fewer hallucinations. however, as pointed out in sec. 4, drastically reducing hallucinations may be possible without any control mechanism and can be achieved, at least on wikibio, with templates. the main challenge lies in doing so without a big drop in informativeness, that is, in coverage of input fields. comparing the outputs of hal lm with those of hal w o , and both with those of @xcite , we note that the ranking in terms of average sentence length (in sentencepiece tokens) coincides with the ranking in terms of coverage ( ## conclusions we presented a simple but powerful idea of controlling hallucinations which are caused by the noise in the training data and proposed two ways of detecting such noise. we demonstrated that it is possible to reduce the amount of hallucinations at no coverage cost by informing the model about how noisy every source-target example is and without changing the model architecture. importantly, this was done without making any assumptions about the data. in an evaluation with humans we showed that the faithfulness of generated sentences can be significantly improved at no loss in fluency or coverage. the results we reported on the noisy wikibio dataset improve upon the prior work.