Put Chatbot into Its Interlocutor’s Shoes: New Framework to Learn Chatbot Responding with Intention (2021)

Abstract: Most chatbot literature that focuses on improving the fluency and coherence of a chatbot, is dedicated to making chatbots more human-like. However, very little work delves into what really separates humans from chatbots – humans intrinsically understand the effect their responses have on the interlocutor and often respond with an intention such as proposing an optimistic view to make the interlocutor feel better. This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework includes a guiding chatbot and an interlocutor model that plays the role of humans. The guiding chatbot is assigned an intention and learns to induce the interlocutor to reply with responses matching the intention, for example, long responses, joyful responses, responses with specific words, etc. We examined our framework using three experimental setups and evaluated the guiding chatbot with four different metrics to demonstrate flexibility and performance advantages. Additionally, we performed trials with human interlocutors to substantiate the guiding chatbot’s effectiveness in influencing the responses of humans to a certain extent. Code will be made available to the public.

## introduction humans have evolved to become sensitive to their social interactions. the more they interact, the more they generally learn what to say and what not to say to light up people's mood or to avoid upsetting others. in this paper, we aimed to train a chatbot to emulate these human-like qualities by making it learn from interactive conversation. a chatbot that understands the effect its utterances have on the interlocutor could be a significant step towards achieving human-level chatbots. a chatbot that understands the effect of its utterances on the interlocutor is also critical in realworld applications. for instance, as shown in figure 1 , given a context from the interlocutor, both figure 1 : an example of the dialogue to show that how the chatbot interacts with the interlocutor, and how our chatbot affects the interlocutor's response when assigned the intention, making people respond joyful. much literature combine reinforcement learning(rl) @xcite with transformerbased @xcite models to control the chatbot's output. @xcite proposed models to concentrate on crucial keyphrases presented in the context. their models tended to generate outputs that were more coherent and specific to the conditionals, which leaded to more non-generic words. by training with a combination of the above criteria, their approach leaded to more diverse and interesting responses. however, these previous works focused on controlling the chatbot's responses and completely neglected the interlocutor in their training. in this paper, we made extensive use of the interlocutor's responses as interactive experiences to train our guiding chatbot to influence the interlocutor with intentions. we introduce a novel train-ing framework, in which there were two conversational models that simulated chatbot-interlocutor interaction. one model acted as the interlocutor model, while the other was the guiding chatbot to be trained. the interlocutor took the guiding chatbot's output as its input and generated corresponding responses. the learning framework was given a controllable factor, which represented the intention it had. we defined reward functions according to three different controllable factors, sentence length, emotion, and specific words, to make the guiding chatbot learn to induce the interlocutor model to generate desired responses using rl. to evaluate our guiding chatbot, we designed several experiments to examine the rewards corresponding to three controllable factors, and empirical results demonstrate that our guiding chatbot can influence humans' responses. moreover, we found that training with more interlocutor models together improved the guiding chatbot's performance on the human evaluation experiment. furthermore, we analyzed recent off-the-shelf chatbots based on experimental results, aiming to find hidden tendencies these chatbot models had, such as cursing more or being more irritative. ## related work the most common chatbot model is sequence-tosequence based @xcite . recently, numerous researchers applied transformer to build coherent chatbots by retrieval-based @xcite @xcite @xcite and generative-based @xcite @xcite approaches. despite the decent fluency and coherence these chatbots achieved, they still hardly converse like a human. the reason might be that they are essentially devoid of emotions. furthermore, some used rl to improve their chatbot's performance @xcite and others combined rl with gpt-2 @xcite models to control the sentiment of their chatbot's response to make it more user-friendly @xcite . beyond the viewpoint of sentiment, the empathet-icdialogues (ed) dataset was collected @xcite to train a chatbot that could recognize the feeling of the interlocutor and know how to reply accordingly @xcite . however, these researchers neglected what really separated humans from chatbots -humans understand the impact their responses have on the interlocutor and often responded with intentions and expectations. note that it is not just about being empathetic as human's intentions could vary widely. one previous work also considered interlocutor responses @xcite . it used a sentiment predictor to predict the interlocutor's sentiment given the chatbot's response, and also trained the chatbot with rl. unlike this previous work, our proposed framework explicitly modeled the possible responses of interlocutors. explicitly modeling interlocutor responses give the proposed framework more flexibility. for example, in addition to steering the interlocutor's sentiment as in this paper, the framework could be used to build a chatbot that induce the interlocutor to become more talkative by setting its learning target to be making the interlocutor generate longer sentences. moreover, we also developed techniques to preserve the dialogue's coherence, so our chatbot could still generate fluent and appropriate responses in addition to having a particular intention. apart from influencing the interlocutor, the proposed framework also served as a way to analyze the underlying inclination of the off-the-shelf chatbots playing the role of interlocutor. through the interaction, we could know what factors are apt to influence these off-the-shelf chatbots. @xcite claimed that the appealing performance of recent robust and sota models belied a potential problem of black-box models: these models lacked an explicit declarative knowledge representation. hence, calling for a transparent representation, they dug into explaining trained models. in contrast to the previous contributions, we tried to explain the implied tendency of a chatbot, which was not obvious to recognize. according to the experiments, we were capable of telling whether the off-the-shelf black box chatbot possessed certain predispositions, such as tending to swear more or having a short temper. ## controllable factors below are the three types of controllable factors studied in this paper. r i in section 3.3 could be either r l for sentence length, r e for emotion, or r w for specific words, introduced below. sentence length a chatbot that could inspire the interlocutor to become more talkative is desirable in many real world applications. we aimed to observe whether our chatbot was able to make the interlocutor more talkative, and extend the length of conversations. hence, we counted the sentence length of interlocutor models' responses as r l . by optimizing this reward, we anticipated that the guiding chatbot might extend sentence length from the interlocutor. emotion we studied whether our chatbot was capable of inducing the interlocutor to respond with different emotions. we selected eight emotions, including anger, anxiety, contentment, disgust, hope, joy, sadness, surprise. we selected the eight emotions such that two emotions are located in each of the four different quadrants of the valence-arousal coordinate @xcite . to ascertain the emotion of sentences, we established an emotion detector. the emotion detector was an emotion classifier used to classify emotion given an input sentence. we trained the emotion detector on the enpatheticdialogue (ed) dataset @xcite . for each sentence, the emotion detector will employ a valence-arousal (va) projection grounded on the valence-arousal coordinate @xcite . given an input sequence, the emotion detector would output a two-dimensional vector representing sequence's emotion, defined as emotional valence 1 . more details related to the valence-arousal coordinate will be discussed in section 5.2. we utilized the bert @xcite architecture, a pretrained contextualized embedding model, to improve language understanding. next, we fine-tuned the bert model on an emotional classification task to enhance the model's capability of categorizing each emotion. the accuracy of our emotion detector was up to 82%, and, therefore, we could obtain a detected emotion and its emotional valence given an input sentence. the emotion detector takes i(c(x)) as input and predicted its emotional valence according to the va coordinate. therefore, we could calculate the mean square error (mse) between the emotional valence of the interlocutor models' responses and the target emotion's emotional valence as the reward r e . ## conclusion this paper introduced a novel framework that aims to train a guiding chatbot to influence the interlocutor. we designed three different controllable factors for the guiding chatbot to induce the interlocutor to reply with responses matching the intention. we managed to prolong the length of the interlocutor's responses, influence the interlocutor to reflect with a particular emotion, and induce the interlocutor to use some specific words more frequently. furthermore, we further enhanced the performance of the guiding chatbot by training it with more interlocutor models. experiment results show that our proposed framework can successfully train chatbot with intentions.