LSTD ial: Enhancing Dialogue Generation via Long- and Short-Term Measurement Feedback (2024)

Abstract: Generating high-quality responses is a key challenge for any open domain dialogue systems. However, even though there exist a variety of quality dimensions especially designed for dialogue evaluation (e.g., coherence and diversity scores), current dialogue systems rarely utilize them to guide the response generation during training. To alleviate this issue, we propose LSTDial (Long- and Short-Term Dialogue), a novel two-stage framework which generates and utilizes conversation evaluation as explicit feedback during training. Specifically, we fine-tune pre-trained dialogue systems through using turn-level quality feedback in the first stage and further train ever-improving dialogue agents through using dialogue-level quality feedback in the second stage. By using our approach on dialogue systems, capable of enabling dialogue generation with both short-term capabilities (generating more fluent, relevant and varied responses at the turn-level) and long-term capabilities (generating more coherent, engaging and informative responses at the dialogue-level). We implement LSTDial on four strong baseline models and experiment with two open-domain dialogue datasets. Experimental results show that LSTDial achieves significant improvement, enabling to generate better dialogue responses in terms of both human and automatic evaluation.

## introduction generating high-quality responses is a key challenge for any open domain dialogue systems @xcite @xcite @xcite . current state-of-the-art systems do this by training agents with supervised learning on large amounts of labeled text data. however, these labeled data are usually reference responses or pre-defined task goals instead of multi-faceted quality dimensions designed for dialogue evaluation (e.g., relevance, coherence, and engagement, etc.), which leads to a great gap between dialogue generation and evaluation. response generation systems are typically trained by optimizing the average likelihood of the training data (maximum likelihood estimation, mle), without a clear signal on the progression of the ongoing conversation @xcite . a possible solution to the above issue is to introduce quality annotations of the ongoing conversation during training. however, selecting relevant and meaningful quality dimensions and generating corresponding measurement annotations are crucial. in the dialogue evaluation task, an evaluation metric can be evaluated at the turn and dialogue level @xcite . in a turn-level evaluation setting, the goal is, given prior dialogue history (frequently denoted as context) c and a response r, the metric will assign a score s of r conditioned on c while assuming a measured quality dimension q. examples of turnlevel quality dimensions include fluency, relevance, diversity, appropriateness, and safety, etc @xcite . conversely, in a dialogue-level evaluation setting, the goal is to evaluate the performance throughout the full dialogue. examples of dialogue-level quality dimensions include coherence, engagement, topic depth, informativeness, and consistence, etc @xcite . because human evaluation is both expensive and timeconsuming, automatic evaluation metric models that strongly correlate with human judgments have been proposed. intuitively, if the mentioned-above quality annotations can be automatically generated and recast into explicit feedback, it is promising to study ever-improving dialogue agents that learn through interaction. feedback-based reinforcement learning (rl) has gained increasing attention in dialogue generation @xcite @xcite . because human-generated feedback is expensive to obtain, researchers have devised learned feedback generators while assuming one can train downstream models to utilize generated feedback @xcite . motivated by this, we design feedback functions which depict multi-sided qualities of conversation and utilize generated feedback to guide response generation in the rl-based setting. considering the balance between efficiency and performance, we fine-tune pre-trained dialogue systems through using turn-level quality feedback (called short-term feedback in our method) while train ever-improving dialogue agents via continual learning through using dialogue-level quality feedback (called long-term feedback in our method). a motivation example is shown in figure 1 . to this end, we propose the lstdial (longand short-term dialogue) framework. there are two stages involved in the learning process. in the first stage (stdial), we first extract single-turn dialogues from the training set of the experimental dataset and then calculate their turn-level evaluation scores. after this, we fine-tune dialogue systems via a multi-objective loss function including mle and short-term feedback. in the second stage (ltdial), we further train the first-stage dialogue system but in the rl-based mode. specifically, we first generate a response r via the training system conditioned on current history c. then, we calculate dialogue-level evaluation scores of the @xmath0, @xmath1. particularly, to estimate future reward of r, we generate a future response f via user simulator @xcite while assuming the conversation is ongoing. next, we calculate turn-level evaluation scores @xmath2, @xmath3. finally, these two rewards are integrated and adapted in policy updating for rl. a novel feature of our methodology is generating and utilizing feedback that accesses relevant and meaningful conversation evaluation for better response generation. the contributions of this paper are three folds: ## methodology the approach proposed in this paper belongs to the augmentation task of dialogue generation, and the specific objectives and motivations have been elaborated in figure 1 . figure 2 shows the architecture of our method lstdial. first, we crafted two sets of conversation evaluators to generate long-and short-term feedback (described in §3.1 and §3.2, respectively). then, the two-stage training of lst-dial will be presented in §3.3 and §3.4. ## conclusion in this study, we introduced lstdial, a novel framework that utilizes turn-and dialogue-level conversation evaluation as explicit feedback during training. by bridging the gap between dialogue generation and evaluation, lstdial enables dialogue systems to exhibit both short-term capabilities at the turn-level and long-term capabilities at the dialogue-level. experimental results on two popular open-domain dialogue datasets demonstrate the significant improvement achieved by lst-dial in terms of both human and automatic evaluation metrics. these findings highlight the importance of incorporating conversation evaluation into dialogue system training for generating highquality responses. future research will explore more feedback mechanisms to better utilize feedback to enhance dialogue generation.