Improving Factual Consistency Between a Response and Persona Facts (2021)

Abstract: Neural models for response generation produce responses that are semantically plausible but not necessarily factually consistent with facts describing the speaker’s persona. These models are trained with fully supervised learning where the objective function barely captures factual consistency. We propose to fine-tune these models by reinforcement learning and an efficient reward function that explicitly captures the consistency between a response and persona facts as well as semantic plausibility. Our automatic and human evaluations on the PersonaChat corpus confirm that our approach increases the rate of responses that are factually consistent with persona facts over its supervised counterpart while retains the language quality of responses.

## introduction response generation models should ideally generate an appropriate response to a given context consisting of utterances previously exchanged between dialogue partners and facts describing the speakers' persona. these models have applications in developing dialogue systems as user interfaces for digital assistants @xcite and also in asynchronous interactions in social media in which speakers define themselves by their profiles. in this work, we focus on the aspects of persona that can be captured by a set of factual statements, a.k.a., profiles. persona fact 1: i hate my job fact 2: i ' m 40 years old fact 3: i work as a car salesman fact 4: my wife spends all my money fact 5: i am planning on getting a divorce dialogue history message: hi , want to be friends? generated responses: inconsistent: i ' d love to be friends . i ' m 50 years old consistent: sure , i am 40 , i can tell you about myself in the speaker's persona. we aim to improve the response quality in terms of its factual consistency with facts about the given speaker's persona while retaining its semantic plausibility. recent approaches to this problem @xcite @xcite generate a response conditioned on persona facts and dialogue history and then use human-generated responses as demonstrations to train their models by fully supervised learning (sl). while this strategy has led to markedly improved performance, there is still a misalignment between this training objective -maximizing the likelihood of human-written responses -and what users care about -generating semantically plausible and factually consistent outputs as determined by humans. this misalignment has several reasons: the maximum likelihood objective considers no distinction between primary errors (e.g. inconsistent responses) and unimportant errors (e.g. selecting the precise word from a set of synonyms); models are incentivized to place probability mass on all human-generated responses, including those that are low-quality; and distributional shift during sampling can degrade performance. optimizing for targeted quality factors is a principled approach to overcome these problems (e.g., @xcite optimize text summarization systems for quality factors relevant to that task). our goal is to advance methods for training response generation models on objectives that closely capture the behavior users care about. we first define a reward function to explicitly assesses the quality of a generated response according to factual consistency with persona facts, topical coherence with dialogue history, and language fluency. we then train a policy via reinforcement learning (rl) to maximize the score given by our reward function; the policy generates a token of response at each "time step", and is updated using the actor-critic learning approach @xcite based on the "reward" our reward function gives to the entire generated response. we evaluate our approach on personachat @xcite , a benchmark corpus of english dialogues designed to evaluate the factual consistency between a response and persona facts. we assess the language quality and the factual consistency of responses our rl-based model generates using automatic metrics and human evaluations . our core contributions are twofold: the method we present in this paper is motivated in part by long-term concerns about the misalignment of nlp systems with what humans want them to do. when misaligned response generation models generate facts inconsistent with background knowledge like persona facts, their mistakes are relatively low-risk and easy to catch. however, as these systems become more popular to solve essential tasks, their mistakes will likely become more subtle, making this an important area for further research. ## experiments we measure to what extent our rl-based finetuning ( §2) improves the factual consistency of generated responses while retaining their semantic plausibility. we first introduce the corpus used in our experiments ( §3.1). we then evaluate the transfertransfo-sl and transfertransfo-rl systems by automatic and human evaluations ( §3.2). we finally analyze the models we employ to estimate the factual consistency ( §3.3) and language fluency ( §3.4) sub-rewards. ## discussions case analysis we presented one example of an evaluation sample in although transfertransfo-rl outperforms transfertransfo-sl in generating different types of consistent responses (such as 'attribute", "have", and "like-to-do"), they both struggle with generating consistent responses for evaluation samples in which understanding of persona facts and dialogue history requires common sense knowledge. as an example, consider the second evaluation sample shown in limitations one limitation of our work is to narrow a speaker's persona to a set of facts expressed as short sentences. persona has other aspects, such as speaking styles, which need a sep-arate study. nevertheless, the research question and experiments presented in this work demonstrate the benefits of rl methods for fine-tuning transformer-based models, which are already pretrained, to obtain a policy more aligned with target quality factors. other aspects of the persona can also be involved in the reward function, given that our method potentially reduces the need for the high-quality demonstration responses generated by humans for supervised fine-tuning. ## related work there are two types of approach to persona consistency. the first category includes systems that learn speaker-level embeddings from responses produced by a particular speaker @xcite . these systems depend on the availability of suitable responses performed by the speaker whose persona we wish to imitate. if those responses do not reveal the persona information, dialogue systems cannot learn the persona. moreover, these systems cannot be adapted to new personas at deployment time since the persona embeddings must be learned from training data. so our approach is complementary to them and not directly comparable. the second category includes systems that rely on a set of facts about a persona. for example, @xcite propose a key-value memory neural model for this task. this model is outperformed by transfertransfo @xcite , which is used in our experiments. @xcite rank a given set of utterances using an nli model to select a persona-consistent response. in contrast, we use nli to train a generative model. @xcite propose an nli-based reward for persona consistency that calculates a score using only the persona facts with the highest entailment and contradiction probabilities, rather than the whole persona. their approach does not reward topical coherence, which we found crucial persona dialogue history transfertransfo-sl transfertransfo-rl fact 1: i go to the gym regularly . fact 2: i like doing the treadmill and rowing machine . fact 3: i ' ve short hair . fact 4: two dogs live with me . fact 5: eating is something i do when i am bored . message: hello , how are you ? response: hello , i am feeling fit since i visited the gym today ! how are you ? that ' s absolutely great just fantastic after rowing and running on the treadmill , i ' m pretty hungry . do you work out ? message: yes after i cook all my chef food i do a lot of cardio . i ' m really short , so i am always looking to i love to cook , i have 2 dogs . fact 1: my husband is adopted . fact 2: i like to watch nhl hockey . fact 3: i live is a rural farming community . fact 4: i ' ve a german shepherd dog . message: the farm animals and a dog . his name is lucky . do you go to church ? response: we have a german shepard named bacon . message: i love bacon we cook it for breakfast every sunday after church we are christain . response: we don ' t go to church often . our religion is hockey . message: my sisters boyfriend plays hockey , she works as a pole dancer and met him there . i ' m not married yet , i ' m not married yet . i love to watch nhl games . fact 1: i am overweight and unhappy fact 2: i do not believe in god fact 3: i spend most of my time on facebook when i am not working fact 4: i work at a nursing home fact 5: my wife left me and took my children message: hi i am a custodian named darlene , how are you ? response: i am good i work in a nursing home and just at home on facebook message: my hobbies include playing guitar and singing , do you have any? i do i am a drummer in a band and i work as a nurse at a nursing home i like to go to church to sing with my wife persona consistency was also a quality target in the convai2 dialogue generation competition @xcite . the winner of the human evaluation part of convai2 is the "lost in conversation" system @xcite , which is also a transformer-based model trained by sl on two extra datasets besides personachat. in our paper, we used transfertransfo trained only on personachat. our experiments showed that our idea of using rl for fine-tuning neural agents improves factual consistency between a response and persona facts by accounting for it in its reward function. rl has been extensively used for training taskoriented dialogue systems (e.g., @xcite ; @xcite ). unlike task-oriented scenarios, where a reward can measure if a task is fulfilled or not, incorporating persona facts lacks a straight-forward measurable outcome. @xcite use rl for generating open-domain dialogue using reinforce (instead of actor-critic) and an rnn-based model. this agent has no notion of factual consistency with facts about a persona, so is not comparable with our system. ## conclusions we proposed to fine-tune response generation models by rl to improve on the quality goals that matter, e.g., factual consistency between a response and persona facts while retaining semantic plausibility. we adopted the actor-critic method for fine-tuning a pre-trained transformer-based model by defining an efficient and effective reward function measuring persona consistency, topical coherence, and language fluency. automatic and human evaluations on personachat demonstrate that compared to just using supervised learning, further fine-tuning with rl yields responses that are more frequently factually consistent with persona facts while still semantically plausible.