S im OAP : Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluation (2023)

Abstract: Language models trained on large-scale corpora can generate remarkably fluent results in open-domain dialogue. However, for the persona-based dialogue generation task, consistency and coherence are also key factors, which are great challenges for language models. Existing works mainly focus on valuable data filtering, model structure modifying, or objective function designing, while their improvements are limited and hard to generalize to all types of pre-trained language models. However, we find that language models can produce consistent and coherent responses if we consider enough generations. Thus, the problems lay in large-scale response generation and target response selection. In this work, a simple but effective two-stage SimOAP strategy is proposed, i.e., over-sampling and post-evaluation. The over-sampling stage takes large-scale responses from existing trained models efficiently via off-the-shelf distilling and compressing methods, and the post-evaluation stage selects a good response based on multiple well-designed evaluation metrics from large-scale candidates. Experimental results show that the proposed plug-in SimOAP strategy improves the backbone models and outperforms the baseline strategies in both automatic and human evaluations.

## introduction open-domain dialogue systems need to give appropriate responses based on history utterances. an ideal open-domain dialogue system should generate consistent, coherent, and diverse responses. part of the existing open-domain dialogue generation work focuses on improving the diversity of responses @xcite , while avoiding generating generic responses and achieving good results. how to improve the consistency of dialogue generation is also an urgent problem to be solved @xcite . in addition, there is still the figure 1 : we use a dialogue model to generate 10,000 responses each for 100 utterances in personachat and use perplexity (ppl) to rerank them. the response is good when the tf-idf similarity between the response and ground truth is above 0.25 and the result of the natural language inference model between the response and persona information is entailment. ppl in brackets is the average value of all responses in each rank. to improve the consistency and coherence of dialogue generation, the existing works mainly improve from three aspects: valuable data construction and filtering @xcite , model structure modifying @xcite @xcite and objective function designing @xcite . however, the problem of poor consistency and coherence is still a tough challenge, especially in persona-based dialogue generation @xcite . because multiple constraints need to be satisfied simultaneously, part of which cannot be directly optimized, and part of the constraints conflict with each other, such as the conflict between the fluency of responses and the consistency of persona information. in addition, the above methods need to retrain the model and can only adapt to the part of the existing dialogue models. for example, @xcite carefully design the objective function and scale model sizes from 117m to 8.3b parameters, which brings a lot of training costs. fortunately, we find that the existing dialogue models actually have strong capabilities that can generate consistent and coherent responses, and we just need to find ways to release their capabilities. first, we take a deep look at the characteristics of dialogue models, which believe that the response with the highest probability is the best. however, we wonder whether the high-probability responses generated by dialogue models are necessarily better than the low-probability responses. based on the statistics in figure 1 , when the generation probability of responses decreases, the ratio of good responses increases first and then decreases. it shows that the ratio of good responses among lowprobability responses is higher than that of highprobability responses, which is counter-intuitive. this is most likely because dialogue models use ppl as an optimization goal, but it is inconsistent with the requirements of coherence and consistency. to verify whether the good response with high tf-idf similarity and high probability of entailment foot_0 is indeed superior to the response directly generated by the model, we use the human evaluation for experimental validation. as shown in inspired by the aforementioned motivations, we propose a simple two-stage method: over-sampling and post-evaluation (simoap) to improve the coherence and consistency in persona-based dialogue. there are two challenges in our work. the largescale sampling will bring additional time cost, how to accelerate the model is a challenge. large-scale sampling can produce good responses, how to pick good responses from them is another challenge. we address the above two challenges using oversampling and post-evaluation, respectively. in the over-sampling stage, simoap uses existing dialogue models for large-scale sampling, and the distilled or compressed models are used to reduce the to verify the effectiveness of our method, we conduct experiments on a persona-based dialogue dataset personachat @xcite . automatic evaluations and human evaluations show that our method effectively boosts the performance of dialogue models and outperforms two baselines @xcite on most metrics. in addition, applying our method to small models can also achieve a better performance than using large models directly. our contributions in this paper are three folds: ## related work dialogue generation has made remarkable progress in recent years. many pre-trained dialogue models have been proposed @xcite @xcite . to improve the consistency of dialogue generation and make dialogue models applicable to various scenarios, zhang et al. (2018) propose a persona-based dialogue dataset personachat. persona-based dialogue generation is limited by the scale of data and expensive annotation costs. song et al. (2019) generate persona-based dialogue by using additional natural language inference data. @xcite use data augmentation to extend data and use data distillation to make it easier to fit. however, labeling data for persona-based dialogue takes a high cost, and data from other domains is difficult to apply to persona-based dialogue fully. part of the work modifies the model structure for persona-based dialogue. @xcite propose a pre-trained model, which uses personabased sparse data for pre-training. @xcite design a three-stage framework of generating, deleting, and rewriting. @xcite learn the persona features by designing a response generation decoder and a consistency understanding decoder. however, there are multiple constraints that need to be satisfied simultaneously, some of which cannot be directly optimized. the above works also bring a huge training cost. part of the work designs the related objective function. @xcite modify the unlikelihood loss to improve the consistency of dialogue. @xcite use the previous dialogue content of users to control the dialogue of specific personas. however, it is difficult to design the objective function. we found a simple strategy without filtering valuable data, modifying the model structure, or designing objective functions, but only needs to use existing models for large-scale sampling and post-evaluation to improve the performance. @xcite use dual systems to improve the coherence and consistency of neural sequence models. this work uses a neural system to generate the candidate and a logic system to evaluate it. the candidate is generated and evaluated one by one until it meets the criteria. however, the small number of candidates limits the effectiveness of dialogue generation. in addition, the logic system evaluates the candidate by tracking common sense information. it is difficult to apply to dialogue generation. in dialogue generation, maximum mutual information (mmi) @xcite uses the mutual information between history utterances and responses to evaluate responses. mmi can reduce the generation of generic responses but brings the large-scale time cost. to eliminate the influence of response length on likelihood, @xcite use length-normalized loglikelihood score (lls) to evaluate candidate responses. however, it is verified that using large-scale sampling for lls performs worse than fewer candidate responses. it shows that lls cannot release the ability of models by over-sampling. furthermore, simple evaluation methods for the above two methods are difficult to work well in complex persona-based dialogue. ## our approach persona-based dialogue consists of persona information sentences @xmath0, p 2 , ..., p |p | }, history utterances @xmath1, h 2 , ..., h |h| }, and a gold response g. dialogue models need to generate a response r, which is coherent with history utterances h and consistent with persona sentences p . the framework of simoap is shown in figure 2 . simoap consists of two stages: over-sampling and post-evaluation. in the over-sampling stage, simoap uses existing dialogue models for largescale sampling, and accelerates the model to reduce the extra time cost. in the post-evaluation stage, the tf-idf algorithm @xcite and natural language inference are used for coherence and consistency evaluation, respectively. ## conclusion in this work, we propose a simple but effective two-stage strategy to improve the coherence and consistency in persona-based dialogue generation. in the over-sampling stage, we use dialogue models for large-scale sampling, and compressed or distilled models are used to accelerate. in the postevaluation stage, multiple well-designed evaluation metrics select the final response from large-scale candidates. experimental results show that our method improves the backbone models and outperforms the baseline strategies. for reproducibility, we publish the source code foot_4 . in future work, we will consider further acceleration of our method.