Dialogue Response Ranking Training with Large-Scale Human Feedback Data (2020)

Abstract: Existing open-domain dialog models are generally trained to minimize the perplexity of target human responses. However, some human replies are more engaging than others, spawning more followup interactions. Current conversational models are increasingly capable of producing turns that are context-relevant, but in order to produce compelling agents, these models need to be able to predict and optimize for turns that are genuinely engaging. We leverage social media feedback data (number of replies and upvotes) to build a large-scale training dataset for feedback prediction. To alleviate possible distortion between the feedback and engagingness, we convert the ranking problem to a comparison of response pairs which involve few confounding factors. We trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data and the resulting ranker outperformed several baselines. Particularly, our ranker outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback. We finally combine the feedback prediction models and a human-like scoring model to rank the machine-generated dialog responses. Crowd-sourced human evaluation shows that our ranking method correlates better with real human preferences than baseline models.

## introduction conversing freely in natural language is one of the greatest challenges of artificial intelligence. endto-end open-domain dialog systems have become increasingly powerful, with advanced model architectures and large-scale training @xcite @xcite . in some settings, human annotators figure 1 : for many online communities, posts and comments have a tree structure and user can upvote or downvote each node individually. this allows us to define measures (e.g. width, depth, and updown) of human feedback and build a large-scale training dataset for response quality prediction. however, a meaningful evaluation of response generation must take into account more than whether a generated turn is relevant in context, or whether it "sounds human." conventional neural conversation models often generate trivial or bland responses @xcite that are relevant to context but are not engaging. even human responses can vary dramatically in terms of tonal appropriateness and whether they are interesting enough to prompt a rich listener reaction. a successful dialog turn must be proactive, engaging, and consistent with social norms @xcite . in this work, we move beyond simple prediction of response relevance, augmenting this with a prediction of how likely a response is to elicit a positive reaction from an interlocutor. by incorporating a measure of engagingness into the response generation ranking algorithm, we hope to improve the overall behavior of data-driven conversational agents. existing methods are suboptimal for this ranking task. conventional perplexity based ranking methods @xcite focus only on context-hypothesis relevancy. online conversational systems such as xiaoice @xcite employ a manually-designed set of features to rank hypotheses, but the design of these rankers is not directly based on real-world human preferences or feedback in an end-to-end fashion. large-scale training data is necessary because of the one-to-many nature of dialog and the scope and complexity of human conversation. however, labeling conversations at scale is too expensive and time-consuming for this purpose. labeling the "engagingness" of a response is not something a single annotator can do; the task requires something more like a large-scale, collective vote. and yet there is no obvious automated substitute for this kind of human labeling. conventional quality measurements such as reference-based similarity @xcite or lexical diversity @xcite capture only limited aspects of response quality, and are not strongly predictive of human reactions: simply because a response is different from others does not necesarily mean that it will be perceived as "bad". our solution involves leveraging existing human feedback data (e.g., number of replies and likes) from online social communities. while there is work in the field of social media on feedback prediction (sparling and @xcite @xcite , it has not previously been applied to dialog systems and response generation. as illustrated in figure 1 , each comment has its own number of replies and upvotes (termed as "likes" in some communities). these can be used as engagingness labels after careful normalization and formulation. there exist billions of online threads available and the number is growing fast, thus making it possible to build a large-scale training dataset. however, the relation between feedback and quality may be distorted due to social influence and other confounding factors @xcite . in order to ameliorate this problem, we propose a contrastive formulation, shifting from ranking to pairwise classification. using a dataset of 133m pairs of human comments and their associated number of replies or up-/downvotes, we train a set of large-scale transformer-based feedback ranking models which outperform several baselines. in particular, dialog perplexity shows little predictive power of human feedback. we also show that a classifier trained on human-vs-artificial data can achieve good zero-shot relevancy prediction accuracy. finally, we describe an ensemble model that is capable of merging the predictive powers of all these models, tuned using human calibration. human evaluation shows that our ranking method outperforms the baselines in terms of correlation with actual human preferences. ## human feedback many social media platforms, such as reddit, twitter, and facebook allow users to reply or upvote contents, leveraging that feedback to make decisions about what content to display, highlight, and hide. these collective ratings are treated as a proxy for content engagingness. in this section we discuss a few metrics of user vote data, along with some of the issues posed by its use. ## the dialogrpt method in this section we introduce dialog ranking pretrained transformers (dialogrpt). ## conclusion we leverage reddit human feedback data to build and release a large-scale training dataset for feedback prediction.we trained gpt-2 based models on 133m pairs of human feedback data and demonstrate that these models outperform several standard baselines. in particular, the conventional dialog perplexity baseline shows little predictive power on reddit human feedback data. we ensemble the feedback prediction models and a humanlike scoring model to rank the machine generated dialog responses. human evaluation shows that human preference is improved with our ranking method. for the future work, we suggest to integrate the ranking models and generation model, e.g., in beam search stage or reinforcement learning using ranking score as reward signal.