M ovie C hats: Chat like Humans in a Closed Domain (2020)

Abstract: Being able to perform in-depth chat with humans in a closed domain is a precondition before an open-domain chatbot can be ever claimed. In this work, we take a close look at the movie domain and present a large-scale high-quality corpus with fine-grained annotations in hope of pushing the limit of movie-domain chatbots. We propose a unified, readily scalable neural approach which reconciles all subtasks like intent prediction and knowledge retrieval. The model is first pretrained on the huge general-domain data, then finetuned on our corpus. We show this simple neural approach trained on high-quality data is able to outperform commercial systems replying on complex rules. On both the static and interactive tests, we find responses generated by our system exhibits remarkably good engagement and sensibleness close to human-written ones. We further analyze the limits of our work and point out potential directions for future work

## introduction being able to converse like humans in a closed domain is a precondition before an intelligent opendomain chatbot, which further requires transiting among various domains, can be designed @xcite . nonetheless, even if constrained in a specific domain, current chatbots are still far from satisfactory. unlike task-oriented systems that can be relatively well-resolved with handcrafted templates, human conversations feature a complex mixture of qa, chitchat, recommendation, etc. without pre-specified goals or conversational patterns @xcite @xcite . selecting proper domain knowledge to support response generation at all the different situations is challenging @xcite @xcite . in this work, we direct our focus to the movie domain and present a large-scale, crowdsourced chinese dataset with fine-grained annotations in hope of boosting the study towards a human-like closed-domain chatbot. a variety of dialogue datasets with grounded domain knowledge have already been proposed. however, they are collected either through (1) online forum crawling @xcite @xcite @xcite , which are noisy, multi-party, mostly contain only single-exchange qa, or (2) crowdsourced @xcite @xcite , which are small-scale and often created in an overconstrained setting like teacher-student @xcite . even for datasets crowd-sourced in unconstrained scenarios, suggestive domain knowledge is provided for humans before an utterance is provided. this would inevitably prompt humans to utilize these knowledge deliberately, yielding unnatural conversations simply connecting the knowledge @xcite . we show examples from other datasets in appendix different from previous crowd-sourced works, our annotation process is conducted posteriori so that it will not interfere with human conversations, e.g., prompt them to overuse suggested knowledge. built upon our dataset, we propose a simple unified language model approach to push the limits of movie-domain chatbots. the model is first pretrained on 2.2b words collected from various general-domain conversational resources, then finetuned on the movie dataset with additional knowledge and dialogue acts incorporated. we pool all components like intent prediction and knowledge retrieval into a sequence prediction task and solve them with a unified language model architecture. it avoids designing complex systems for individual components separately and all subtasks can be easily trained simultaneously @xcite . we show our simple unified approach outperforms strong baselines for each separate subtask. knowledge retrieval, dialogue acts prediction and general-domain pretrain benefit from each other and altogether bring improvement to the generation quality. in the online interactive test, our best model succeeds at chatting with humans for 11.4 turns without being detected to be a machine, outperforming even commercial chatbots mitsuku foot_1 and microsoft xiaoice foot_2 which further rely on complex rules. by analyzing the limitations of our model, we find it especially has difficulty at dealing with in-depth discussions over long turns. future research can consider employing larger knowledge base or explicit state tracking. in summary, our main contributions are (1) presenting a high-quality, large-scale chinese conversational corpus with fine-grained annotations in the movie domain to benefit future study, (2) showing that a simple unified neural model trained on the high-quality dataset can approach human performance and even outperform commercial systems replying on complex rules, and (3) studying the shortcomings of current techniques, providing suggestive directions for future research. ## dataset construction the dataset construction consist of (1) crowdsourcing the dialogues, (2) annotating dialog acts and entities and (3) linking utterances into grounded knowledge. we explain these three steps in order and present the dataset statistics in the end. ## model architecture language models have demonstrated impressive performance as a universal learner across nlp tasks @xcite @xcite . inspired by this, our dialogue generation model is implemented as a transformer-based language model like gpt2 @xcite . it contains a pipeline process of movie tracker, intent prediction, knowledge retrieval and text gener- comment [comment] movie comment(s) recommend [recommend] movie name(s) track [tracker] [inherit] or a new movie name intent [intent] da sequence retrieve [retrieve] knowledge response [response] response ## experiment setting we tokenize the text in the unit of chinese characters and keep all unique non-chinese unique tokens appearing for more than 5 times. the whole vocabulary contains 13,317 words. we train our model on 24 nvidia v100 gpus (32gb) with three different model sizes as shown in pytorch @xcite . ## results and analysis automatic evaluation in in the static evaluation, we sample 300 responses for each model from the test set (mixing seen and unseen). the responses can come from any turn in a conversation. we show the results in figure 2 . our largest model with 762m is clearly preferred by human evaluators on almost all metrics and approaches human performance. by training a larger model and increasing the training size, the gap might be further closed. in the interactive evaluation, humans can chat with any topic but restricted in the movie domain. we conduct an online turing test where one side is always a human participant not aware whom he is talking with. the other side could be either mitsuku, xiaoice 13 , our model (762m with pretraining) or a real human. mitsuku interacts in english, so we hire only english native speakers for the experiment. we collect 100 conversations for all models. humans can stop interacting once they (1) find the other side is a machine or (2) reaches the maximum turn of 20. responses from all models are later passed to the third party to judge the scores. the results are shown on the right of figure 2 . our model outperforms mitsuku and xiaoice by a large margin. as mitsuku and xiaoice are designed to be open-domain chatbots, restricting to be on the movie domain will give our model some natural advantage. we can also notice that mitsuku and xiaoice almost never produce fake facts. the cost is the extremely low ratio of informative responses since they tend to behave over-safely and will only answer it when they are 100% sure. our model is closer to humans in that sense. it will converse actively at some risk of containing fact errors. 2 4 6 8 10 12 14 16 18 20 0 0.2 0.4 0.6 0.8 turn number sea 2 4 6 8 10 12 14 16 18 20 0.5 0.55 0.6 0.65 turn number fia mitsuku xiaoice our human figure 3: change of sea and fia as the turn proceeds. in figure 4, we further show the "dying distribution" of our model, namely, in which da our model fails to pass the turing test and thereby "dies". unsurprisingly, we can see the system fails mostly when informing facts or feelings. only a small portion are from non-grounded chitchats (other). this suggests the most crucial bottleneck lies in the interaction with movie-specific knowledge and seamlessly incorporating it into the response generation. we show some snippets of interactions with our model in ## conclusion we present moviechats: a movie-domain chatbot built upon a large-scale, high-quality conversational corpus with fine-grained annotations. the model can be trained end-to-end with a simple unified language model architecture. we show that our model, powered by well-defined knowledge grounding, is able to approach human performance in some perspective, though still lagged behind when it comes to dealing with detailed knowledge or long-turn consistency. online forum @xcite a: 求推荐励志电影，谢谢大家 (any inspirational movies? thanks everyone). b: 《当幸福来敲门》 (the pursuit of happyness). crowd-sourced (constrained) @xcite a: hey have you seen the inception? b: no, i have not but have heard of it. what is it about ? a: it's about extractors that perform experiments using. . . crowd-sourced (unconstrained) @xcite a: 知道重庆森林这个电影吗 (do you know the movie chungking express)? b: 知道啊，是王家卫导演的 (yes, it's directed by wong kar-wai). a: 具体是哪年上映的你还记得吗？ (remember which year it was on)? b: 记得，是在1994年07月14日 (yes, july 14th, 1994). . .