Learning Improvised Chatbots from Adversarial Modifications of Natural Language Feedback (2020)

Abstract: The ubiquitous nature of dialogue systems and their interaction with users generate an enormous amount of data. Can we improve chatbots using this data? A self-feeding chatbot improves itself by asking natural language feedback when a user is dissatisfied with its response and uses this feedback as an additional training sample. However, user feedback in most cases contains extraneous sequences hindering their usefulness as a training sample. In this work, we propose a generative adversarial model that converts noisy feedback into a plausible natural response in a conversation. The generator’s goal is to convert the feedback into a response that answers the user’s previous utterance and to fool the discriminator which distinguishes feedback from natural responses. We show that augmenting original training data with these modified feedback responses improves the original chatbot performance from 69.94%to 75.96% in ranking correct responses on the PERSONACHATdataset, a large improvement given that the original model is already trained on 131k samples.

## introduction enabling chatbots to indulge in engaging conversations requires massive datasets of human-human conversations @xcite @xcite @xcite . training such dialog agents requires substantial time and effort expended in the collection of adequate number of high quality conversation samples. @xcite alleviate this problem by introducing a self-feeding chatbot which can directly learn from user interactions. this chatbot requests users to provide natural language feedback when the users are dissatisfied with its re1 our code is released at https://github.com/ ekunnii/adversarial-feedback-chatbot/ you could have talked about your age. sponse. @xcite this feedback as a gold response to the wrong turn and use it as an additional training sample to improve the chatbot. although natural language feedback is cheap to collect from a chatbot's end-users, most often, feedback cannot be used directly as a training sample since feedback is usually not the answer itself, but simply contains hints to the answer. in this work, we propose a generative adversarial setup for converting such noisy feedback instances you could say hey, i'm 30. how old are you? yes, i play battlefield would be a great answer tell me what your favorite breakfast food is answer the question about having children! our main contributions are the following: bart discriminator bart feedback text style loss cycle loss self loss conversation history + feedback text response style label feedback style label response bart feedback text we pose the problem of converting feedback to resemble natural response as a text style transfer problem. we observe that feedback is more instructional and judgemental, whereas natural response is direct (answering questions) and engaging (asking questions, contains humor). we naturalize the feedback to a response and use it as an additional training sample to improve the chatbot. a fully supervised approach to convert feedback to natural response is infeasible as we do not have paired (feedback ↔ response) examples and thus we adopt an adversarial setup. we utilize a gan @xcite formulation where the generator modifies the feedback's style to make it seem part of a natural conversation, and in turn fool the discriminator which knows how to distinguish natural responses and feedback. our model, feed2resp, is shown in figure 2 . ## experimental setup in feed2resp, the optimizer for both the generator and discriminator is adamw. the learning rate of generator is 5e-6 while the learning rate of discriminator is 1e-4. the discriminator uses 4 stacked transformer layers and 4 attention heads. the token embedding size, style embedding size, positional embedding size and hidden size are all 256. for the bart @xcite generator, we use the implementation from huggingface @xcite and initialize the model with pretrained weights from the cnn/daily mail summarization task. due to the characteristics of human response(refer appendix a), we limit the length of text generation to a maximum of 50 words and impose a repetition penalty of 2.0 to improve diversity of output. while evaluating the effectiveness of the modified feedback responses, we use two implementations of dialog agents provided by parlai @xcite , biencoder and polyencoder. biencoder has two transformer layers and 2 attention heads. the optimizer is adamax with learning rate of 0.0025. polyencoder uses 12 transformer layers and 12 attentions heads. the optimizer is adamax with learning rate of 5e-05. the hyperparmeters for the best performing model are arrived at by random sampling and subsequently verifying the outputs using human evaluation to rate the outputs from the style transfer task. the entire list of hyper-parameters is listed in the ## experiments our goal is to test whether feedback helps improve the chatbot. to do this, we compare models trained on conversational data with and without feedback data. below we describe the chatbot evaluation setting, our datasets, the main models and different settings of these models with and without feedback. ## results and discussion the experimental details of the model variants are described in section 3. coming to polyencoder, it is a much stronger chatbot than biencoder. we see that naive use of feedback or heuristic deteriorates the performance of polyencoder while feed2resp emerges a clear winner with +6.0 point improvement on the test set over nofeedback. ## conclusion in this work, we show that while chatbots can be improved using natural language feedback, converting feedback to natural responses that fit in the conversation outperform the naive usage of feedback. we presented feed2resp, a generative adversarial model, that converts feedback to natural responses without requiring manually annotated parallel data. our results show that feed2resp results in a 6 point improvement for the polyen-coder chatbot, an already powerful dialog ranking agent. this is a strong result as hits@1/20 is a tough metric to improve upon @xcite . our work joins the class of models that use natural language feedback to improve different tasks, e.g., image captioning @xcite , classification @xcite @xcite . while these methods use feedback for reward shaping or feature extraction, we use feedback to produce correct response using adversarial learning. we pose this problem as a style transfer problem inspired from the style transfer literature @xcite @xcite @xcite . while these focus on studying the stylistic attributes of sentences, e.g, sentiment, we explore this problem in the context of improving chatbots.