Dial B e I nfo for Faithfulness: Improving Factuality of Information-Seeking Dialogue via Behavioural Fine-Tuning (2024)

Abstract: Factual faithfulness is a crucial requirement in information-seeking dialogue: the system should respond to the user queries so that the responses are meaningful and aligned with the knowledge provided to the system. However, most modern large language models (LLMs) suffer from hallucinations, that is, they generate responses not supported by or even contradicting the knowledge source. To mitigate the issue and increase faithfulness of information-seeking dialogue systems supported by the LLMs, we introduce BeInfo, a simple yet effective method that applies ‘behavioural tuning’ on the LLMs to aid information-seeking dialogue. Relying on three standard information seeking dialogue datasets, we show that models tuned with BeInfo become considerably more faithful to the knowledge source both for datasets and domains seen during BeInfo-tuning, as well as on unseen domains, when applied in a zero-shot manner. In addition, we present a ‘real-life’ case study on conversations with real users, showcasing that the models with 3B parameters (e.g., Flan-T5) tuned with BeInfo demonstrate strong performance on data from real ‘production’ conversations: when tuned on a limited amount of such realistic in-domain dialogues, they surpass much larger LLMs used ‘off-the-shelf’, both on automatic and human evaluation metrics.

## introduction pretrained large language models (llms), being able to generate natural and grammatical text and respond coherently to user queries, are the mainstay of modern nlp @xcite . they have demonstrated their capabilities in a plethora of tasks where the general world knowledge, which can be learnt via pretraining directly from the data, is required @xcite , figure 1 : an example of an information-seeking dialogue based on the doqa dataset @xcite . potential responses r1, r2, r3 at the bottom illustrate different issues with two crucial aspects of factual faithfulness: selectivity and response adequacy. a widely used method to ground and control the content of the output of an llm is retrievalaugmented generation (rag; @xcite , where the input to the model is complemented with a retrieved external knowledge source relevant to the user's query. however, even with the use of rag, the model's output can be unpredictable and not fully controllable: they still sometimes do not adhere to the knowledge source and hallucinate @xcite , which can decrease their applicability in user-facing scenarios, as well as raise concerns of their safety @xcite . the problem of adherence to the knowledge sources is especially important in the context of information-seeking dialogue @xcite . the core of this task is to maintain a conversation with the user and respond to their queries based on the provided knowledge source. figure 1 presents an example of information-seeking dialogue between the user and the system and potential responses of the system. orthogonally to improving retrieval systems themselves @xcite , prior work has attempted to combat hallucinations with task arithmetic @xcite , conditioning generation on special control tokens @xcite , and by incorporating a token-level critic which judges the faithfulness of the generated response @xcite . however, the proposed approaches requires either training an additional model or using complex inference processes such as context-aware decoding @xcite . in this work, we propose beinfo, a simple yet effective method that applies 'behavioural finetuning' of llms to increase faithfulness of the generated responses for information-seeking dialogue supported by the llms. the model is tuned on a reasonably sized collection of publicly available dialogue data with the true knowledge source(s) extended with randomly sampled facts from a large knowledge base. intuitively, this should teach the model to become more selective in the information it uses to generate the response and 'prepare' its expected behaviour (hence the term 'behavioural tuning') for the intended task of knowledge-grounded dialogue. the tuned model can either be used 'as is' or as a starting point to fine-tune it further to a specific domain. first, we assess the effectiveness of beinfo on three standard datasets for information-seeking dialogue: faithdial, topiocqa and doqa. our results demonstrate that beinfo leads to consistent improvements in factual faithfulness across several standard evaluation metrics, also with on par or larger lexical overlap between the generated and golden responses. the improvements are especially pronounced when models tuned with be-info are applied in a zero-shot manner to unseen datasets and domains, indicating the usefulness of behavioural tuning for the task. we then present a case study focused on conversations with real users: the main result demonstrates that combining be-info with a small number of in-domain dialogues can substantially increase dialogue factuality even in specialized dialogue domains. behavioural tuning for faithfulness. an effective model for faithful information-seeking dialogue needs to perform two actions correctly: 1) select the correct part of information provided in k (termed selectivity) to base the generated response on and 2) provide the response, with the requirement to (i) inform the user when k contains no information relevant to u, or (ii) ask for clarification (termed (response) adequacy); see figure 1 again. put simply, in our setup response adequacy discerns between 1) the case when the model does have the correct information in the knowledge source and should provide it versus 2) the case when the model is certain that it cannot provide a correct answer to the user query or it does not even understand the query and requires further clarification to be able to react in the next turn. foot_0 beinfo aims to improve on both desiderata via behavioural fine-tuning @xcite of any instruction-tuned llm. to instill the capability for information-seeking dialogue into the model, we perform behavioural tuning on the combination of (i) conversational qa and (ii) information-seeking dialogue datasets. in both tasks, the response has to be generated based on some knowledge source k, making them suitable for faithful response generation. further, beyond tuning on related tasks, we propose to augment the datasets to steer the model towards the selectivity and adequacy behaviour, as follows. for selectivity, ground truth k provided in the dataset is extended with additional knowledge sources k ′ which are irrelevant to user query u, serving as negative examples or distractors. intuitively, distractors mimic the presence of information irrelevant to u in k ′ , this way promoting the model's selectivity. we augment ground truth knowledge source k with n distractors; they are randomly sampled from the knowledge base of the corresponding dataset. 2 for response adequacy, we augment the finetuning datasets with dialogues without any relevant k provided, making them unanswerable for the system. to construct such dialogs, for a dialogue history h and a corresponding user query u we randomly sample unrelated knowledge sources k ′ . during fine-tuning, the response r is substituted with a special response signifying that the combination of h and u cannot be answered based on provided k ′ . in our experiments, we augment the original dataset with 10% unanswerable dialogues. further task-specific fine-tuning. the output of the 'general' behavioural fine-tuning step is a 'behaviour-specialised' llm for factually faithful information seeking dialogue. it can be used directly 'as is', or as a starting point for further taskspecific tuning, as illustrated in figure 2 . ## experimental setup training setup. in order to leverage inductive biases of instruction-tuned models, the input for beinfo includes the following: (i) instructions to respond as factually accurately as possible, (ii) augmented knowledge source which includes: ground truth k and @xmath0, and 5 randomly sampled k ′ -s for unanswerable dialogues and (iii) dialogue history which combines all the previous turns (the set h) and the current user query u. an example input and instruction text are shown in appendix a. the models are then trained in a standard sequence-to-sequence fashion with cross-entropy loss. the output is either ground truth responses for answerable dialogues, where knowledge source k contains the information to address user's query, or a predefined response 'could you please clarify or rephrase the query?' if the dialogue is unanswerable. training the models using beinfo proceeds at turn level: dialogue history at every turn is used as input. datasets. to perform behavioural fine-tuning, we use a standard dataset for information seeking dialogue, faithdial @xcite , and an established conversational qa dataset, topiocqa @xcite . generalisation capabilities of the models after the beinfo tuning are evaluated on another domain and dataset (i.e., this could be seen as 'zero-shot' from the domain adaptation perspective). for this, unless explicitly stated otherwise, we rely on a multi-domain conversational qa dataset, doqa @xcite . the key statistics of the datasets are in ## results and discussion faithfulness on unseen data. one of the main aims of behavioural fine-tuning with beinfo is to increase the factual faithfulness of responses in zero-shot domain transfer, on unseen data in any domain. therefore, we start by presenting the results of the variant tuned with beinfo on faithdial plus topiocqa, where inference is run on the dataset unseen during beinfo tuning: doqa (i.e., general beinfo from figure 2 ). the results are presented in using a smaller dataset for beinfo tuning. the previous results from results in beinfo with task-specific fine-tuning. we have demonstrated that the models tuned with be-info largely improve factual faithfulness on unseen datasets and domains (i.e., the general be-info setup). here, we study whether these models can serve as an effective starting point for continued task-specific fine-tuning. to this end, we first tune the models with beinfo on the combination of faithdial and topiocqa as before, and then continue fine-tuning/specialising the model on a single dataset (e.g., faithdial or topiocqa): the full setup from figure 2 . 4 figure 3 demonstrates that already task-only be-info yields strong performance, while models with beinfo perform on par or better on average than the models which were tuned to a specific dataset both on semantic similarity of generated responses and factual faithfulness. while prior work @xcite typically optimised one aspect (e.g., semantic similarity) at the expense of the other (faithfulness), and vice versa, here we show that through the use of knowledge distractors beinfo achieves competitive performance on both aspects and retains the cross-dataset generalisation ability. beinfo versus catastrophic forgetting. further, one issue which might arise from further specialising a model to a given task/dataset is a wellknown phenomenon of catastrophic forgetting: pretrained language models are prone to forgetting previously learnt knowledge or skills when tuned on new data @xcite . to evaluate whether the models would retain their ability to respond faithfully to examples considerably different from the ones seen during finetuning, we evaluate the models tuned on faithdial on topiocqa. 5 the scores in ## evaluating beinfo on real conversations experimental setup. to probe the potential of beinfo for boosting real user-facing production systems, we rely on a small internal dataset of 200 fully anonymised dialogs with real users in the hotel reservation domain (termed hotel-200 henceforth); the dialogues concern hotel bookings and faq-s about its various facilities. it is crucial to evaluate the models on examples also collected from real user-system communication, as the language use is considerably different to some established datasets such as doqa or faithdial compiled via crowdsourcing work. for instance, the average length of the user query in hotel-200 is only 6.35 tokens, while it is 17.25 in faithdial or 13 in doqa (cf., as the data comes from real conversations, there are no gold responses which could be used for automated evaluation. thus, we resort to evaluation of correctness/factual faithfulness with an llm: here, we use gpt4 (termed gpt4-eval henceforth) as its judgements were shown to be most correlated with human judgements @xcite ). 6 for gpt4-eval we prompt gpt4 to act as the evaluator providing it with natural language instructions, knowledge source k, conversation history h with user query u and the system-generated response. in the instructions we request the model to rate generated responses on a 7-point likert-scale for 6 as running evaluation with large models such as gpt4 behind proprietary apis incurs large costs @xcite , we only evaluate the outputs for a smaller dataset where other means of evaluation cannot be used. to obtain data for the task-specific tuning stage, we collect 2,000 examples from the same conversational system, then generate 'silver' responses via gpt4 and treat the silver responses as true outputs for task-specific fine-tuning. 8 results and discussion. the main results are reported in the 40b falcon model. further, the scores indicate the importance of being able to fine-tune smaller models with in-domain data: the 3b model tuned with the full beinfo even outperforms gpt4 on gpt4-eval, and it also obtains strong performance with task-only beinfo. these results further support our hypothesis that beinfo actually 'behaviourally prepares' the models to respond to user's queries in a factually faithful manner and tuning on further task-specific data only amplifies its impact as it gets further adapted to the domain. put simply, behavioural fine-tuning via beinfo performs structural (or behavioural) adaptation, while further task-specific fine-tuning combines the behavioural adaptation with (semantic) domain adaptation. ablation: distributions of scores. we further study the actual distributions of gpt4-eval scores for the four models variants of flan-t5 xl and compare it against the distribution obtained by gpt-4. the distributions are shown in figure 4 . as only a small fraction of responses is labelled with intermediate scores (1,2,3,5), the core differences lie in relative distribution of perfect, poor and 'not great, not terrible' responses (scores 6,0 and 4, respectively). 9 the model tuned with task-only beinfo rarely provides wrong facts but mostly responds with not great, not terrible responses which do not mislead the user but might not be helpful. on the other hand, the model tuned with the general-only beinfo and gpt-4 both yield responses that typically fall into the extreme categories. in other words, the responses are either perfect (score 6) or will provide the user with wrong information (score 0), which is not desirable for a user-facing system. the model tuned with the full beinfo combines the benefits of behavioural tuning with the use of indomain data: the model produces the least factually unfaithful responses (score 0) while maintaining the ability to respond with information relevant to the user's query (a large number of responses with scores 6). in sum, 'pre-tuning' the model with the general-only beinfo stages raises faithfulness of the model by extracting relevant information from the knowledge source k while further tuning on task-specific data further helps avoid providing misleading or irrelevant information to the user. ## previous work mitigating hallucinations in information-seeking dialogue has achieved increased interest recently with the omnipresence of large language models @xcite @xcite @xcite . previous methods can be largely divided into those which increase factuality of pretrained models via further training or modification of the generation procedure. the former includes, e.g., tuning the models with contrastive learning @xcite or a special focus learning loss which reduces hallucinations on token level @xcite . the latter includes, e.g., conditioning generation process on special control tokens @xcite , task arithmetic @xcite or training a critic network which can detect problematic tokens and replace them @xcite . other approaches have been developed to specifically improve faithfulness with respect to retrieved knowledge source in decoding. one proposed option is to do context-aware decoding (cad; @xcite where generative probabilities are contrasted between those based only on user query and those based on the user query and the knowledge source. the aim is to force llms to rely more on the knowledge source than the model's internal knowledge from pretraining. in contrast to cad, @xcite propose to contrast generation probabilities from different layers of llms to promote factual knowledge in the resulting output probabilities. improving faithfulness via supervised tuning. task-specific supervised fine-tuning could be seen as an option to improve faithfulness of the model's responses @xcite . prior work @xcite has demonstrated that fine-tuning on higher-quality data improves the model's factuality on benchmarks such as truth-fulqa @xcite . in contrast, supervised fine-tuning on the data which includes numerous irrelevant or factually inconsistent responses can lead the model to amplifying the noise in the training data. a recent analysis from @xcite has shown that over 60% of responses in three standard datasets for information-seeking dialogue (wow, @xcite @xcite contain hallucinations, making them unsuitable for supervised fine-tuning aimed at improving factuality. to resolve this, @xcite released a corrected version of wow where the responses were fixed to be factually consistent with the knowledge source. as behavioural fine-tuning heavily relies on the quality of the underlying data, we have carefully selected and resorted to faithdial and topiocqa in the first stage of beinfo with highest factual faithfulness of their ground truth responses (see appendix b for further details). ## conclusion and future work we presented beinfo, a simple yet effective method that applies behavioural fine-tuning of large language models underlying information-seeking dialogue systems, with the goal of improving factuality of system responses. instruction-tuned models are fine-tuned on a collection of publicly available dialogue data for two related tasks, conversational question answering and information-seeking dialogue, where the model must use the correct knowledge source among several 'knowledge distractors' and provide a factually correct and adequate response. the main results indicated the effectiveness of beinfo both in in-and cross-dataset setups. in addition, we demonstrated that further tuning on task-specific data might yield further gains in terms of faithfulness as well as reducing extractiveness, also in experiments with real conversations from a production-ready dialogue system. this work leads up to several potential directions of future work. firstly, beinfo is orthogonal to other existing approaches to improving faithfulness. for instance, a combination of cad @xcite and beinfo could further improve factuality of responses. secondly, beinfo was evaluated on information-seeking dialogue. another interesting direction could be to applying it to other language generation tasks where faithfulness to the knowledge sources is crucial, such as summarisation. furthermore, the effectiveness of the approach can be also tested on other instructiontuned models (e.g., t0, @xcite and models of larger sizes, e.g., flan-ul2 and beyond. 10 we hope that our work will inspire the research community to build stronger models for factually faithful information-seeking dialogue.