B o K : Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation (2024)

Abstract: The standard language modeling (LM) loss by itself has been shown to be inadequate for effective dialogue modeling. As a result, various training approaches, such as auxiliary loss functions and leveraging human feedback, are being adopted to enrich open-domain dialogue systems. One such auxiliary loss function is Bag-of-Words (BoW) loss, defined as the cross-entropy loss for predicting all the words/tokens of the next utterance. In this work, we propose a novel auxiliary loss named Bag-of-Keywords (BoK) loss to capture the central thought of the response through keyword prediction and leverage it to enhance the generation of meaningful and interpretable responses in open-domain dialogue systems. BoK loss upgrades the BoW loss by predicting only the keywords or critical words/tokens of the next utterance, intending to estimate the core idea rather than the entire response. We incorporate BoK loss in both encoder-decoder (T5) and decoder-only (DialoGPT) architecture and train the models to minimize the weighted sum of BoK and LM (BoK-LM) loss. We perform our experiments on two popular open-domain dialogue datasets, DailyDialog and Persona-Chat. We show that the inclusion of BoK loss improves the dialogue generation of backbone models while also enabling post-hoc interpretability. We also study the effectiveness of BoK-LM loss as a reference-free metric and observe comparable performance to the state-of-the-art metrics on various dialogue evaluation datasets.

## introduction open-domain dialogue generation is a dynamic area of research, aiming to generate contextually relevant and meaningful responses given a dialogue context. as deep learning models continue to thrive in the field of natural language processing (nlp), a widely adopted strategy to solve any natural language generation (nlg) task involves pre-training and fine-tuning large language models (llms). perhaps you'd be interested in red roses. what would the roses cost me ? dialogue context response bag-of-keywords (bok) roses cost i need some flowers for my wife. the llms are predominantly trained with language modeling (lm) loss, which essentially corresponds to cross-entropy loss for predicting the next word or token. while lm loss remains effective in training nlg models for diverse tasks, including dialogue generation @xcite @xcite , it may not be the optimal choice for training models specifically tailored for dialogue generation. it is well-established that perplexity, a measure associated with lm loss, primarily gauges fluency and weakly correlates with human dialogue evaluation @xcite @xcite . consequently, relying solely on lm loss may not guarantee generations with desirable conversational qualities. therefore, exploring alternative loss functions and training methods is crucial to advance the development of generative open-domain dialogue models. in order to mitigate the exclusive dependence on lm loss in the training of open-domain dialogue models, various approaches have been explored in the existing literature. these techniques can be broadly categorized into two classes -a) auxiliary loss and b) human feedback. the first approach combines one or more auxiliary losses with lm loss to train the dialogue models. various types of auxiliary losses have been explored in the context of open-domain dialogue learning. for instance, bag-of-words (bow) loss computes the cross-entropy loss to predict words/tokens of the next utterance from the given dialogue context @xcite @xcite . some methodologies involve predicting the sentence-level encoding of the next utterance and determining the loss through l1/l2 norms and kl divergence @xcite @xcite . few approaches incorporate a next-utterance classification loss @xcite , wherein the auxiliary loss is computed for a classification or ranking task to predict the true utterance from a set of candidate responses. on the other hand, the second approach is based on refining the pre-trained dialogue model through human feedback. these methods mostly follow the training principle of reinforcement learning from human feedback (rlhf), where the model is fine-tuned to maximize the reward associated with the generated response using reinforcement learning. rlhf has gained significant interest recently, particularly with the popularity of models like chat-gpt @xcite . however, acquiring quality human feedback data is challenging and expensive @xcite . furthermore, relying on automated dialogue evaluation metrics as a substitute for human feedback can pose challenges, as they may not strongly correlate with human judgments @xcite . in this work, our objective is to propose a novel auxiliary loss for open-domain dialogue systems. specifically, we address the limitation of bow loss by introducing bag-of-keywords (bok) loss, which is defined as the cross-entropy loss to predict the keywords of the next utterance. while training, we extract the keywords of the ground-truth response using yake! @xcite , an unsupervised feature-based keyword extractor. the keywords can be seen as a proxy for the core idea of the response. in a conversation, a reply can be generated in multiple ways. as a result, bow loss can induce training data bias since it considers all the words/tokens of the ground-truth response for prediction. in contrast, bok loss focuses on the core idea (as shown in fig. 1 ) that alleviates the problem of generalization. the main contributions of this work are summarized as follows foot_0 : ## background and related works open-domain dialogue generation is a challenging nlg task. let @xmath1, u 2 , ...u t-1 } be a multi-turn conversation where u j represents the utterance at turn j. let c t be the condition (like persona, document, etc.) other than dialogue history for generating u t . the task of open-domain dialogue generation is to generate u t given @xmath2. like any nlg task, it is modeled using language models and generally trained using the next word/token prediction task. the corresponding language modeling (lm) loss is defined as, as discussed earlier, a dialogue context can have many relevant responses. hence, the task of predicting all the words/tokens can induce training data bias. in this work, we aim to address this limitation of bow loss. ## methodology in this section, we describe bag-of-keywords loss followed by its application in open-domain dialogue systems. ## discussions 6.1 qualitative analysis and interpretability in this section, we perform a qualitative analysis of the models trained with bok-lm loss. in both tables, we observe an overlapping of tokens between the generated response and the predicted tokens. for example, in turn 4 of turn utterance (ground-truth) top-8 predicted tokens model generated response 1 i need to buy some flowers for my wife . kind, ġfresh, ġred, ġroses, ros, es, ġkind, l we have lots. what kind of flowers do you want? 2 perhaps you 'd be interested in red roses . ros, ġred, es, ġsounds, good, <nok>, expensive, ide no, i don't like red roses at all. 3 what would the roses cost me ? dozen, ġprice, ġdollars, ars, oll, es, ten, ġred @xmath0 20 a dozen . ġsounds, dozen, bad, expensive, ġgood, price, deal, reasonable that sounds like a good deal. i'll take a dozen. ## conclusion this paper proposes bag-of-keywords (bok) loss, a novel auxiliary loss for training open-domain dialogue systems. the main idea of bok loss is to improve the generalizability of bag-of-words (bow) loss by predicting only the keywords or the core idea of the next response. we show that bok loss enhances the generative performance of the vanilla t5 and dialogpt models on the dailydialog and