Elastic Weight Removal for Faithful and Abstractive Dialogue Generation (2024)

Abstract: Generating factual responses is a crucial requirement for dialogue systems. To promotemore factual responses, a common strategyis to ground their responses in relevant documents that inform response generation. However, common dialogue models still often hallucinate information that was not containedin these documents and is therefore unfaithful. In this work, we propose to alleviate suchhallucinations by ‘subtracting’ the parametersof a model trained to hallucinate from a dialogue response generation model in order to‘negate’ the contribution of such hallucinatedexamples from it. Extensive automatic and human evaluation shows favourable results whencompared to state-of-the-art methods that combine the distributions of multiple models, suchas DExperts (Liu et al., 2021), and others thatchange the training procedure, such as Quark(Lu et al., 2022a). Finally, we show how wecan not only reduce hallucinations but also discourage extractive responses, which are oftena consequence of reducing hallucinations byencouraging copy-pasting of document spans.We publicly release our code for reproducibilityand facilitating further research.

## introduction current-day large language models (llms) impressively generate coherent, grammatical, and seemingly meaningful text, but are prone to hallucinating incorrect information. while grounding them in relevant documents can alleviate this @xcite , models still tend to generate information that conflicts these documents, which would again be classified as hallucination @xcite . this raises major safety concerns. such hallucinations could impair student learning, or proliferate convincing-but-inaccurate news articles. therefore, ensuring trustworthiness is crucial for k: the flash first appeared in "showcase" #4 (october 1956) [...] u t : what comic series is he from? u t +1 f a he first appeared in "showcase" #4 @xcite ). ## background the goal of dialogue response generation is to continue a dialogue u t 1 := (u 1 , . . . , u t ) of t turns by generating a new turn u t +1 . here, each turn u t is just a sequence of n t tokens [u t ] nt 1 ∈ v nt from the model vocabulary v. in document-grounded response generation, u t +1 is grounded in one or more documents k ⊆ k from a document knowledge base k, meaning that k informs the information content of u t +1 . therefore, u t +1 should also faithfully reflect it. this means that neither contradicting nor unverifiable information should be added. in this work, we assume that k is given. a common strategy for generating u t +1 is using language generators that model the distribution ## elastic weight removal in our proposed method, elastic weight removal (ewr), we use the fisher information matrix (or fisher) to combine models with importanceweighted scaling factors for each parameter. thereby, we aim to preserve positive behaviour in the model fine-tuned for dialogue response generation while removing the most important parameters in the anti-expert task vector, which lead to hallucinated generations. we take inspiration from prior works that successfully use the fisher for similar parameter-specific scaling, for example, against catastrophic forgetting @xcite , for merging checkpoints of the same model trained independently on different tasks (matena and raffel, 2022), or preconditioning updates in stochastic optimization @xcite . we refer the reader to prior works @xcite @xcite for more information about theoretical properties of the fisher. of practical importance is that the fisher has size d 2 for a neural network model with d parameters. therefore, it is commonly approximated by its diagonal (matena and raffel, 2022, inter alia). the diagonal can be estimated efficiently by summing or averaging the squared gradients of the model over the training data. here, the label is sampled from the model at each step instead of taking the annotated token (cf. @xcite ). for a model p θ (y | x) this means calculating: we start by taking equation (2) and setting λ 0 , which scales pre-trained parameters θ 0 , to λ 0 • f θ (note that λ 0 is equal to 1 in equation (5) for task arithmetic). similarly, for each task vector τ i , we replace the scalar factor λ i with λ i • f τ i . this way, we can still control the influence of each model with a scalar hyper-parameter, while the diagonal fisher estimate controls individual parameters. since the entries in f can have different magnitudes than the entries in θ, we use a scaling constant z. then, our parameter combination is defined as: @xcite . then, using only a hallucination anti-expert θ ae , we can rewrite the update as: therefore, f θ 0 and f τ ae determine how much each parameter should be changed-parameters with large f θ 0 are preserved and parameters with large f τ 1 are changed more due to their contribution to negative behaviour. when an expert model is added, as well, it is only possible to obtain a similar rewrite when the sign of the corresponding α i is flipped in the denominator, i.e. z := we have found this to be more stable empirically. however, it can introduce divisions by 0 which can be avoided by adding a small constant. finally, we have found calculating the fisher at τ to perform well empirically, even though calculating it at θ ae or θ e , respectively, is theoretically better grounded. next, we describe how we train the expert and anti-expert models. pseudocode for ewr is shown in appendix a.1. ## experiments we experiment on multiple datasets outlined in section 4.1. we compare ewr to cape and task arithmetic, as well as a set of other unlearning methods, which we apply for faithful dialogue generation for the first time. furthermore, we compare to stateof-the-art methods for faithful dialogue generation. we list these baselines in section 4.2. crucially, parameter combination can be added independently on top of many of the other baselines. all experiments are implemented using huggingface transformers @xcite and models are initialised from publicly available flan-t5 checkpoints @xcite , which we have found to perform substantially better than previously introduced encoder-decoder models like bart @xcite or t5 @xcite . we organise our experiments using sisyphus @xcite and release configuration files to reproduce our results. further experimental details, such as learning rate or number of epochs, are given in appendix b.1. we use beam search with a beam size of 10 for decoding. ## results we first introduce our main results on wow and dstc9 in section 5.1. then, we characterise tradeoffs between faithfulness and abstractiveness in section 5.2 before discussing the controllability of model interpolation in section 5.3. finally, we discuss ablations on various datasets in section 5.4, provide a qualitative analysis in section 5.5, and report human evaluation results in section 6. ## human evaluation in addition to the automatic evaluation, we conduct a human evaluation on wow and dstc9 with the help of three expert annotators foot_2 , using the attributable to identified source (ais) framework @xcite . first, we ask them to score responses as attributable (a) only if all their content can be attributed to the knowledge that grounds the dialogue response. furthermore, we ask annotators to rate cooperativeness (c), i.e. the ability of the model to connect with and follow up on user turns on a 3-point likert scale. here, 1 indicates a response that does not cooperate with the dialogue, 2 a response that brings the dialogue forward, and our results also emphasize the trade-off between faithfulness and both paraphrasing (which reflects abstractiveness) and cooperativeness. increased attributability often leads to a decrease in both other criteria. nevertheless, ewr with a faithfulness anti-expert and an abstraction expert, labelled ewr abs , improves both paraphrasing and attributability on wow and attributability on both datasets compared to vanilla flan-t5. while ewr abs does not outperform this baseline in paraphrasing on dstc9, we believe that this stems from the way the expert dataset d e is constructed, related to the comparatively less strong performance of quark and ctrl. as the groundtruth responses in dstc9 contain longer follow-up questions, it is likely that density-based binning does not pick up nuances, such as the difference between non-paraphrased responses and follow-up questions independent from the knowledge. ## related work hallucination in lms the impressive abilities of lms are offset by the potential for generating hallucinated text @xcite @xcite which sparked an increasing interest in tackling this problem in the context of grounded language generation @xcite , encompassing several tasks such as data-to-text generation @xcite , machine translation @xcite , summarisation @xcite , generative question answering @xcite , and dialogue generation @xcite @xcite @xcite . different studies aim to address the issue of hallucination by either developing automatic metrics to detect it @xcite , or by identifying potential causes, such as out-of-domain generalisation, noisy training data, and exposure bias @xcite @xcite . for neural dialogue models it has been shown that retrieving relevant knowledge can reduce -but not completely eliminate-hallucinations @xcite . therefore, different methods have been proposed to tackle it, such as token-level critics @xcite , or control token- @xcite and reranking-based methods @xcite . lastly, as hallucinations in training data can greatly exacerbate those in models @xcite , a hallucination-free dialogue benchmark has been proposed @xcite . controllable text generation different works steer model behaviour by controlled generation, for example by combining models at decoding time @xcite or in parameter space @xcite , conditioning on reward tokens assigned to model generations in training @xcite or the initial training data @xcite . finally, different methods constrain text generation with logical constraints @xcite or by forcing specific words to appear @xcite . ## conclusion & future work we introduce elastic weight removal (ewr), a novel method for steering the behaviour of language generation models by combining their pa-rameters with those of (anti-)experts, weighted by fisher information. we show how ewr can be used to reduce hallucinations in documentgrounded dialogue response generation across different settings. we compare it to other state-of-theart methods, many of which we adapt to faithful response generation for the first time. automated metrics and human evaluation show that ewr improves faithfulness over multiple baselines, and can furthermore provide complementary improvements with them. moreover, we show that faithfulness comes at the expense of abstraction. therefore, we combine an abstraction expert with the hallucination anti-expert to promote responses that are both more faithful and abstractive than the baseline. the main contribution of this work is that it outlines an unexplored way of promoting faithfulness in document-grounded dialogue by using experts and anti-experts not at inference time-and thereby incurring significant overhead-but rather to navigate the parameter space towards an improved set of parameters without altering the model architecture. this opens up many potential areas for future work, such as controlling for further dimensions, or developing more sophisticated data augmentation techniques to create data for (anti-)experts.