Leveraging Implicit Feedback from Deployment Data in Dialogue (2024)

Abstract: We study improving social conversational agents by learning from natural dialogue between users and a deployed model, without extra annotations. To implicitly measure the quality of a machine-generated utterance, we leverage signals like user response length, sentiment and reaction of the future human utterances in the collected dialogue episodes. Our experiments use the publicly released deployment data from BlenderBot (Xu et al., 2023). Human evaluation indicates improvements in our new models over baseline responses; however, we find that some proxy signals can lead to more generations with undesirable properties as well. For example, optimizing for conversation length can lead to more controversial or unfriendly generations compared to the baseline, whereas optimizing for positive sentiment or reaction can decrease these behaviors.

## introduction a core strategy to improve social conversation models is through human feedback. there has been remarkable progress in learning from feedback, including reinforcement learning with human feedback @xcite , where a large number of human annotations are needed to ensure a good reward function. for social conversation models, the feedback usually involves binary ratings @xcite , numerical scores @xcite , ranking @xcite , or natural language comments of a dialogue turn or episode @xcite . these signals are most often collected explicitly using crowdworkers, as organic users may not want to be burdened with providing explicit signals, or else may provide unreliable information @xcite . in this work, we consider the setting where we have a large number of dialogue episodes of the chicago symphony orchestra -that sounds wonderful! what's special about the conductor? muti is a great music director! he has outstanding interpretation for the pieces, and he has also contributed a lot to the city. i really like noodles with cheese and butter sauce. what do you like? okk pizza vs. [conversation history omitted] implicit training signals chatbot test-time because human response is long & joyful maximize chance of joyful response or maximize chance of long response [conversation history omitted] figure 1 : overview of the approach. implicit signals are extracted from conversations, such as whether future human turns are long or short, or joyful or not. for example, the bot turn in the top-left is labeled as "good" and the bottom-left is labeled as "bad" according to both of these signals. we train a binary classifier to predict whether the bot turn is "good" given the conversation history and the bot turn, and we leverage the classifier at the bot's test time. we study various kinds of implicit signals in this work ( §3). ## related work researchers and practitioners have strived to build better neural open-domain dialogue models for years @xcite @xcite . dialogpt @xcite and blenderbot @xcite have released the models as well as the training pipelines which have enabled follow-up dialogue projects from the community @xcite @xcite . in particular, for blenderbot, dialogue interaction data has been released @xcite , which makes the study of implicit signals in our work possible. the class of training strategies that are most relevant to this work -decoding utterances for future success -is discussed in @xcite , in which they interpolate the mle-trained token-level conditional probability with a value function that is trained to predict the property of a completed sequence (e.g., length, bleu/rouge against the reference) given a partial sequence. this overall idea is extended in @xcite where a chatbot learns to generate utterances that have the maximal information gain about the human in the future, as well as @xcite that propose to generate the current bot utterance that leads to the most probable sequence of future utterances. @xcite use conversation engagement metrics (e.g., approximated by retry rate, manually-annotated engagement metrics) to optimize for engaging bot responses; in contrast, our work highlights both the strengths as well as the challenges of using implicit feedback, and in particular that conversation engagement metrics have negative consequences that can be alleviated through other choices of implicit signal. ## conclusion in summary, we find that optimizing for certain implicit feedback signals from human responses is effective, providing improved models over the baseline. however, the choice of implicit signal to extract has important behavioral consequences. conversation length-based signals tend to increase controversial and unfriendly messages, while sentiment or reaction-based signals tend to do the opposite, decreasing the frequency of this behavior compared to the baseline. we note, however, that if we discount generations that are off-topic, controversial, unfriendly, or insincere, and only evaluate on the rest of the examples, then the human annotation would prefer our implicit feedback models over the baseline even more (see the end of §a.3.2). hence, future work could try to extract signals towards that goal (of optimizing toward implicit signals while reducing the amount of undesirable generations), or consider additional safeguards or mitigations while optimizing toward implicit signals.