Towards Faithful Dialogues via Focus Learning (2023)

Abstract: Maintaining faithfulness between responses and knowledge is an important research topic for building reliable knowledge-grounded dialogue systems. Existing models heavily rely on elaborate data engineering or increasing the model’s parameters ignoring to track the tokens that significantly influence losses, which is decisive for the optimization direction of the model in each iteration. To address this issue, we propose Focus Learning (FocusL), a novel learning approach that adjusts the contribution of each token to the optimization direction by directly scaling the corresponding objective loss. Specifically, we first introduce a positioning method by utilizing similarity distributions between knowledge and each response token to locate knowledge-aware tokens. Then, we further design a similarity-to-weight transformation to provide dynamic token-level weights for the cross-entropy loss. Finally, we use the weighted loss to encourage the model to pay special attention to the knowledge utilization. Experimental results demonstrate that our method achieves the new state-of-the-art results and generates more reliable responses while maintaining training stability.

## introduction although open-domain conversation systems can generate smooth and fluent responses with the help of large-scale pre-trained models @xcite , vacuous responses @xcite continue to be prevalent. to enrich the content of responses, an effective way is to introduce external knowledge @xcite . the knowledge-grounded model, however, frequently generates responses that appear knowledgeable but are not derived from the given knowledge. this means that the correctness of the knowledge used in responses cannot be guaranteed. as shown in figure 1 , the "oklahoma" * corresponding author figure 1 : the different learning focus (i.e., the tokens that corresponding losses significantly influence the total objective loss) between general learning and focus learning. original learning focus without guidance in general learning are fragmented with no rules to follow. our methods make the model focus on the knowledge-aware tokens (i.e., tokens that have high semantic relevance to knowledge) to alleviate the hallucinations . in the response is not present in the given knowledge and relevant knowledge is unverifiable. this phenomenon is known as the hallucination @xcite problem. due to the inability to verify knowledge, hallucinations can mislead users and reduce the model's credibility. numerous methods have been developed to tackle the hallucination problem by knowledge graph @xcite , contrastive learning @xcite or control code @xcite . these models enhance the model's attention to knowledge by increasing parameters and elaborate data engineering. an im-figure 2 : distributions of the loss and the relevance between knowledge and response tokens. we select a response as an example and visualize the loss, semantic relevance to knowledge, and the adjusted loss (focusl) at the beginning of training. in the original loss, the model is less sensitive to optimization of knowledgeaware tokens. in contrast, the loss of knowledge-aware tokens in focusl are larger than the others, and the knowledge-irrelevant tokens' loss are scaled down. to address above issues, we propose a novel learning approach, focus learning (focusl). instead of impacting knowledge utilization implicitly, we directly scale the corresponding objective loss to adjust the contribution of each token to the optimization direction. specifically, for the first challenge, we first define the desired learning focus in knowledge-grounded dialogue task as knowledge-aware tokens. then we devise a positioning method to get the relevance score distribution between knowledge and each response token. for the second challenge, we explore a relevanceto-weight transformation method to provide dynamic token-level weights for the cross-entropy loss. finally, we use the corrected learning focus to guide the model training. as we can see in figure 2 , the losses of knowledge-aware tokens do not gain a high proportion of the original loss distribution. in contrast, our approach can expand the gap between knowledge-aware tokens and the others, which increases the impact of the change of knowledge-aware tokens' loss on the final loss, thus affecting the optimization direction and guiding the model to pay more attention to knowledge utilization. our main contributions are summarized as below: hallucinations in text generation generating responses that are unfaithful to the provided knowledge, known as the hallucination, is a tricky problem in knowledge-grounded dialogue systems. recently, the hallucination problem has attracted increasing attention because the generated text appears smooth and fluent but usually contains false knowledge, which significantly threatens the model's credibility. some studies reduce hallucinations by introducing knowledge graph @xcite , controllable generation @xcite , and contrastive learning @xcite . in a recent study, @xcite analyze the source of hallucination in detail and find that the most knowledge-grounded conversation datasets @xcite inherently contain hallucinations, and models trained on such dataset further amplify hallucinations, which demonstrate that the pattern of hallucination responses is more likely to be learned by the model. to address this problem, @xcite further propose faithdial, a new dataset that removes hallucinations in the wizard of wikipedia @xcite . different from these studies about models and datasets, we find that the training method with unexpected learning focus also plays a vital role in the hallucination problem and then present a method to adjust the original focus. ## experiments to evaluate the effectiveness of our method, we conduct experiments following the settings in @xcite . we use pre-trained t5 @xcite 1 from the huggingface library @xcite as our base language model and train 10 epochs via accumulating gradients for 4 steps. we utilize a learning rate of 6.25e-5, and adamw @xcite for optimization. we set the warmup ratio to 4% followed by a linear decay. the max length of the input and output is 256 and 128 respectively. we set the batch size to 8. for adjust weights, we set the smoothing factor λ to 0.01 and the threshold value θ to 0.5. as for decoding, we use nucleus sampling with @xmath0.6. we train our model on a single nvidia tesla v100 gpu with 32gb memory. each epoch takes about 130 minutes for wow and 35 minutes for faithdial. our code is available at https: //github.com/mute-zen/agilelightning . ## results the results on faithdial and wow are shown in ## conclusion in this paper, we propose a novel learning approach with more direct guidance on the training process to improve the faithfulness of knowledge-grounded dialogue systems, referred to as focusl. by leveraging semantic relevance between the response and knowledge, focusl correct the model's learning focus, leading to more consistent and fluent response generation. we empirically show that our approach has the best performance with a stable training process and is robust to data size and out-of-domain knowledge. focusl is simple yet effective and can achieve state-of-the-art results in two knowledgegrounded datasets.