Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning (2023)

Abstract: Maintaining a consistent persona is a key quality for any open domain dialogue system. Current state-of-the-art systems do this by training agents with supervised learning or online reinforcement learning (RL). However, systems trained with supervised learning often lack consistency as they are never punished for uttering contradictions. Additional training with RL can alleviate some of these issues, however the training process is expensive. Instead, we propose an offline RL framework to improve the persona consistency of dialogue systems. Our framework allows us to combine the advantages of previous methods as we can inexpensively train our model on existing data as in supervised learning, while punishing and rewarding specific utterances as in RL. We also introduce a simple importance sampling method to reduce the variance of importance weights in offline RL training which we call Variance-Reducing MLE-Initialized (VaRMI) importance sampling. Our automatic and human evaluations show that our framework improves both the persona consistency and dialogue quality of a state-of-the-art social chatbot.

## introduction the rapid advancements of large language models in recent years has enabled the development of dialogue agents that can generate remarkably fluent and natural responses @xcite . these dialogue systems are typically trained on large amounts of unlabeled text data with some additional fine-tuning on dialogue tasks. while this does allow models to effectively learn many of the patterns and syntax of natural language, dialogue agents still suffer from many problems including a lack of consistency @xcite @xcite . to resolve consistency issues in the context of social dialogue, prior work has proposed conditioning dialogue generation on a persona describing the agent @xcite . this persona consists of descriptions such as "i enjoy skiing" or "i have blonde hair" (figure 2 ). given the advantages of persona grounded dialogue, previous research has been focused making dialogue agents more persona consistent @xcite . existing methods to improve persona consistency are typically centered around the use of supervised learning or online rl @xcite . these methods have been somewhat successful, but still face many problems. supervised learning methods only focus on encouraging persona entailing examples without properly punishing contradictions. this results in dialogue systems that are insensitive to contradictory utterances, leading to inconsistency @xcite . some work has attempted resolve the problems with supervised learning through the use of online rl @xcite . however, the training process for rl is quite expensive since the dialogue model must continuously generate new training samples. furthermore, online rl methods require the use of accurate critics to evaluate the generated bot utterances. these critics must incentivize persona consistency while also enforcing strong constraints on dialogue fluency, as without them the model will degenerate @xcite . this requires training multiple, separate critic models or using human critics during training which is also expensive. given these challenges, we propose an offline rl framework to improve the persona consistency of open domain dialogue systems (figure 1 ). offline rl has several advantages over existing training methods. unlike supervised learning, offline rl explicitly punishes contradictory utterances during training. this further improves persona consistency by making the bot more sensitive to contradictions. unlike online rl, offline rl does not require our dialogue model to generate new samples during training. instead, we can inexpensively train our figure 1 : an overview of our offline rl training framework. our setup is similar to that of supervised learning. the key difference being that our offline samples can have either a positive or negative reward associated with them, as determined by our critic. the policy gradient is obtained by weighting our loss gradient by this reward along with importance sampling. model using large existing datasets that have been collected/synthesized for supervised learning. we exploit this pre-existing data to train our model on human annotated reward labels instead of classifier based rewards which are common in online rl. training on human-annotated rewards also reduces the chance of training failures due to policy divergence. this can arise in settings where value function approximation is needed to determine q-values and may require the use of behavior regularization @xcite . despite the advantages of offline rl, offline rl training can suffer from high variance due to the need for importance sampling. to alleviate this, we introduce an importance sampling method called varmi to reduce the variance of importance weights. this method can be applied beyond our task to other settings where policy-gradient offline rl training is used. prior work has explored the application of offline rl on task-oriented dialogue @xcite @xcite . task oriented dialogue is a natural extension of offline rl as crafting a reward function is straightforward. applying offline rl to social dialogue is less clear as there is no obvious reward to use for our policy. we exploit the fact that persona consistency is a key component of open domain dialogue. intuitively, this makes sense as humans naturally speak with a persona during a conversation. prior studies have shown that improving persona consistency also improves the quality of social dialogue @xcite . our contributions can be summarized as follows: ## related work persona consistent dialogue in recent years, persona-based dialogue generation has typically been centered around the personachat dataset @xcite . one easy method to achieve persona consistent dialogue is to simply fine-tune a model on this dataset using supervised learning @xcite @xcite . however agents trained in this manner still suffer from consistency issues for reasons discussed previously. given this, prior work has been centered around improving persona consistency by more explicitly encouraging entailing utterances and discouraging contradictory ones. many of these methods involve the use of online rl such as @xcite which uses a natural language inference (nli) classifier and naturalness module as a critic or liu et al., 2020 which uses mutual persona perception. some other approaches attempt to improve consistency without any additional training of the dialogue policy. these methods use frameworks such as multistage re-writing @xcite or bayesian rational speech acts @xcite . multistage re-writing is limited by its inability to handle multi-turn persona consistency. on the other hand the bayesian rsa has an increased computational cost during inference time due to the modified decoding scheme. this results longer response times from the bot as well as the need for greedy decoding, which reduces response diversity and dialogue quality. some methods also propose the use of unlikelihood training as a method to improve persona consistency @xcite . however, unlikelihood training suffers from the fact that it does not explicitly reward entailing utterances and instead treats entailing and neural utterances as equally good @xcite . furthermore, unlikelihood training punishes contradictory utterances at a token level which can lead to incoherent responses and uninterpretable behaviors @xcite . our offline rl method can instead distill utterance level information about contradictions and entailment to improve training and maintain coherence. offline rl offline rl applications to dialogue tasks are somewhat limited, with the ones that have been proposed focused on task oriented dialogue. this includes tasks such as price negotiation @xcite or task oriented dialogue benchmarks such as multiwoz @xcite . furthermore many previous studies choose to use offline rl methods centered around q-learning @xcite . while these methods can be effective for dialogue tasks, they require training additional models to steer the dialogue policy towards optimal actions. this adds to both the complexity and resources needed to train and deploy dialogue models for real world applications. we introduce a policy-gradient based offline rl framework with fixed rewards which eliminates the need to use any additional models during training or deployment. instead our training method can be set up as a supervised learning task with modified loss, which gives it the advantage of being much more simple and efficient to train and deploy. despite these advantages, policy-gradient offline rl has seen limited use in practice due to the high variance that arises as a result of importance sampling. variance reduction for importance weights emerged in off-policy learning @xcite and has been widely studied in the context of offline rl (pang and he, 2021; @xcite . given this, we introduce varmi to reduce importance weight variance and improve offline rl training. ## method in this section, we discuss our offline rl framework to improve persona consistency as well as our novel method of importance sampling. section 3.1 gives an overview of how offline rl training is performed. section 3.2 details our varmi importance sampling method. lastly, section 3.3 outlines our framework and section 3.4 discusses how we implement our framework on a dialogue model. ## experiments we test the effectiveness of our offline rl framework for persona consistency using both automatic and human evaluations. our results show that both importance sampling methods are able to improve the persona consistency of bb3. human evaluations also show that our varmi importance sampling method improves the overall dialogue quality of the model. ## conclusion and future work in this paper, we demonstrated that offline rl can be effectively used to improve the quality and utility of open-domain dialogue systems. to do this, we applied offline rl to a persona consistency task and demonstrated its ability to improve persona consistency and dialogue quality over a system trained with only imitation learning. we developed a persona consistency critic that uses human anno-tated labels for persona consistency as well as a novel importance sampling method called varmi. our automatic and human evaluations show that our framework is able to successfully improve the persona consistency of bb3 as well as the overall dialogue quality of the system. a promising direction of future work is to extend our framework to improve other aspects of open domain dialogue such as reducing hallucinations and offensive language. given the ability of llms to generate quality synthetic data, this can be done more easily without having to collect human conversations. it is also worth exploring how well varmi can generalize to other tasks. offline policy gradient methods have seen somewhat limited use due to their high variance so it is worth testing if varmi can reduce these issues more broadly.