When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2024)

Abstract: Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.

## introduction existing dialogue models are primarily trained on human-human conversations @xcite @xcite . as dialogue agents become increasingly powerful and carry substantial conversations with humans @xcite , it becomes pressing to have the models learn from dialogue successes and failures in the wild, and hence improve after deployment. prior work has studied how to collect and learn from feedback in human-model dialogues @xcite @xcite . but most existing methods were proposed under settings where either feedback can be obtained whenever needed or all turns are annotated with human feedback. for instance, @xcite introduced a dataset with all turns annotated by crowdworkers with three types of feedback in this work, we introduce juicer, a framework to "squeeze the juice" out of the sparse human feedback in human-model conversations to improve the dialogue models after deployment. juicer consists of four steps: (1) we first train a binary satisfaction classifier and a reply corrector on existing binary feedback and gold corrections; (2) we then use the satisfaction classifier to label all the bot responses that are missing human labels; (3) next we use the reply corrector to correct bad bot responses (lemons ) into good ones, conditioning on human textual feedback; (4) finally we augment the training data with the new good responses (cherryade ) and re-train our final dialogue models. to evaluate juicer on state-of-the-art chatbots in such a setting, we thus construct a new sparse sampled version of the existing fits dataset from @xcite , which consists of fully annotated human-model conversations between users and existing state-of-the-art internet-augmented models such as blenderbot 2 @xcite and seeker @xcite . we explore a variety of methods to take advantage of limited human feedback at each step of the juicer framework. our main results are: many recent works have studied how to align language models with human feedback @xcite @xcite @xcite . for instance, instructgpt @xcite was finetuned using feedback from labelers who ranked model outputs. @xcite fine-tuned gpt-3 and instructgpt on 100 examples of freeform textual feedback from humans to improve summarization tasks and found that only the larger models such as gpt-3 (175b) @xcite can generate accurate refinements using feedback. @xcite fine-tuned large language models to generate self-critiques for summarization tasks to assist human annotators, and continued to refine the models on feedback. in this work, we focus on improving dialogue agents given various human feedback signals (binary, free-form natural language and gold corrections) and compare our methods to @xcite . existing works have also studied how to correct language model output. for instance, @xcite proposed a model to understand natural language feedback and produce a series of edits to correct a text-to-sql semantic parser. @xcite trained a memory-augmented corrector to convert feedback to edits and fix model outputs for a script generation task. some recent large language model research can also repair generations given human feedback @xcite . past research has also explored how to integrate feedback into dialogue agents @xcite @xcite . @xcite investigated how to improve the chatbot's question-answering ability with general textual feedback in a reinforcement learning setting. @xcite developed a self-feeding chatbot that can construct new examples from existing human-bot conversations and ask for feedback when necessary to improve itself. @xcite proposed a dataset with internetaugmented dialogues, where each turn is annotated with human feedback, and they found that continuously retraining the model on binary feedback after deployment is helpful. our work focuses on converting bad responses into good ones to augment the data and learn from feedback about failures. ## human feedback setting as illustrated in the dialogue example in figure 1 , we consider a deployed system where one can collect three types of feedback: in a deployment setting, it is unnatural to ask users to always click the thumbs up and down and write gold corrections whenever the bot makes a mistake. instead, users tend to provide free-form textual feedback on what was wrong in their dialogue response to express dissatisfaction when the bot makes errors @xcite . therefore many responses may be missing binary feedback @xcite . in this paper, we consider a sparse thumbs up/down signal and sparse gold correction signal setting, but a dense free-form textual feedback signal (i.e., mistakes are followed by textual feedback). after collecting conversations with these feedback signals, we can consider methods to utilize them to improve the dialogue model. in juicer, (1) we first train a satisfaction classifier and a reply corrector on existing feedback, (2) we then use the satisfaction classifier to predict binary satisfaction labels for the un-annotated turns, (3) next we use the reply corrector to convert the bad replies to good ones, (4) finally we collect the good and bad replies including corrections and re-train the final dialogue model to improve it with human feedback. ## the juicer method figure 1 shows the overview of our framework juicer to incorporate limited human feedback to improve the deployed dialogue model. the framework consists of training a satisfaction classifier, a reply corrector, and eventually the final dialogue model itself. we define the notation here. for a given bot reply: (1) denotes un-annotated turns; (2) and : annotated as good or bad responses by users, as defined before; (3) : predicted as bad by the satisfaction classifier; (4) : predicted as good by the satisfaction classifier. juicer involves four steps, summarized here: 1. step 1. train two supervised models: a satisfaction classifier to detect good and bad replies, and a reply corrector to correct the bad replies to good ones. 2. step 2. apply the satisfaction classifier to predict binary labels for all replies missing binary feedback ( → or ). after this step, each bot reply has a label. 3. step 3. use the reply corrector to convert the bad replies that are either disliked by human users or are predicted as bad by the satisfaction classifier in step 2 to good replies ( → , → ). 4. step 4. re-train the final dialogue model by augmenting the training data with the good ( + ) and bad ( + ) replies derived from human feedback and the predictions from the previous steps. now we describe each step in more details. ## experimental setup in our experiments, we used the 3b parameter blenderbot2 (bb2 3b) @xcite as the base dialogue model and try to improve it with human feedback from deployment. ## results we first evaluate the satisfaction classifier ( ## conclusion deployed dialogue agents should continuously improve by using human feedback gathered during interactions. unfortunately, feedback collected in the wild can be limited. in this paper, we proposed juicer, a framework to efficiently use limited organic feedback signals (binary labels and gold corrections) if free-form textual feedback is provided. juicer works by correcting bad responses into good ones to augment the training data for the final dialogue model. experiments show that augmenting with such predictions can integrate human feedback and improve overall performance.