Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models (2021)

Abstract: Existing dialog system models require extensive human annotations and are difficult to generalize to different tasks. The recent success of large pre-trained language models such as BERT and GPT-2 (Devlin et al., 2019; Radford et al., 2019) have suggested the effectiveness of incorporating language priors in down-stream NLP tasks. However, how much pre-trained language models can help dialog response generation is still under exploration. In this paper, we propose a simple, general, and effective framework: Alternating Recurrent Dialog Model (ARDM). ARDM models each speaker separately and takes advantage of the large pre-trained language model. It requires no supervision from human annotations such as belief states or dialog acts to achieve effective conversations. ARDM outperforms or is on par with state-of-the-art methods on two popular task-oriented dialog datasets: CamRest676 and MultiWOZ. Moreover, we can generalize ARDM to more challenging, non-collaborative tasks such as persuasion. In persuasion tasks, ARDM is capable of generating human-like responses to persuade people to donate to a charity.

## introduction it has been a long-standing ambition for artificial intelligence researchers to create an intelligent conversational agent that can generate human-like responses. recently, data-driven dialog models are more and more popular. however, most current state-of-the-art approaches still heavily rely on extensive human annotations such as belief states and dialog acts @xcite . however, dialog content can vary considerably in different dialog tasks. having a different intent or dialog act annotation scheme for each task is costly and even impossible for tasks such as open-domain social chat. thus, it is difficult to utilize these methods on challenging dialog tasks where dialog states and acts are difficult to annotate such as persuasion and negotiation. @xcite proposed a simple sequence-to-sequence architecture that requires no explicit annotations. the model learns to extract information from dialog history with attention and copy mechanism. however, due to the limited language modeling capability in the previous model, sequicity @xcite , which uses belief states as inputs for supervision, outperforms @xcite 's method significantly in the recent dialog datasets. but with the success of large pre-trained language models such as bert @xcite and gpt-2 @xcite , we investigate how large-scale pre-trained language models can help dialog tasks. previous sequence-to-sequence models are used to tackle documents with only one narrator. however, in dialogs, two speakers have different roles; therefore, their language model distributions are very different from each other. to address this issue, we propose ardm, a dialog model that encodes and decodes different speaker utterances in alternating order. this structure makes the model more flexible and efficient than traditional sequence-to-sequence models in processing various dialogs. we evaluate our model on three different task-oriented dialog datasets: camres676, multiwoz, and persuasionforgood. the first two datasets are traditional information request dialog datasets with well-defined automatic evaluation metrics on task completion. by contrast, persua-sionforgood is a new dataset that focuses on persuading people to donate to a charity. due to the complexity of dialog content, there is no explicit dialog state defined in this task. we observe that ardm is capable of improving task-oriented dialog tasks performance over the previous state-of-the-art methods without in-corporating any explicit supervision from belief states or dialog acts. also, because of ardm's simplicity and generality, one can rapidly build a dialog prototype on different types of applications using only conversations without additional human annotations. we also found that ardm works well on complex dialogs, such as persuasion. the model generates dialog responses that successfully persuade people to donate to a charity, suggesting the potential of ardm being used in wide-scale real-world settings. ## related work traditional dialog systems consist of a dialog manager to maintain dialog states and control the conversation flow. however, a dialog manager requires extensive manual annotations for training the sub-modules such as dialog state tracker or policy decision-maker. an alternative is to model dialog without explicitly modeling belief states. specifically, @xcite proposed a sequence-to-sequence model that utilizes copymechanism to copy history information directly from raw dialog history. this method achieved the state-of-the-art results on dstc2 @xcite , which is a simple dialog restaurant booking task with abundant data. however, such method did not perform well on more complex dialog task data sets camres676 @xcite and kvret @xcite . sequicity @xcite attributed the bad performance of @xcite 's method to the omission of belief tracker. they introduced the concept of belief span and added belief tracker back to the model and achieved state-of-the-art performance. compared to sequicity, eric and manning (2017)'s method provides a more general framework that reduces manual dialog state, user intent, and dialog act labeling by bypassing any symbolic annotations. such a model can apply to datasets with no or partial annotations of belief states. in a real-world setting, if the dialog task introduces new slot values in belief states (i.e. a new type of food), sequicity will suffer from the belief span decoder error in response generation. thus, @xcite 's method may be potentially more robust than sequicity in this situation. besides, if the task requires belief states for database search, we can treat belief tracking as a separate task. we can train a good belief tracking with only a small amount of annotated data, which reduces the an-notation required and is easier to fix errors. also, since belief states are a set of important entities condensed from dialog history (i.e., often exact words from utterances), they do not introduce extra information to the model. therefore, a dialog model with powerful representation learning should learn a form of belief states information automatically without human annotations as the scaffold. recent success of bert @xcite and gpt2 @xcite suggests the possibility of applying large pre-trained language models to dialog systems. there are some studies of applying large pre-trained language model to dialog generation. transfertransfo @xcite fine-tuned the pre-trained language model gpt @xcite on persona-chat dataset @xcite and obtained significant improvements on chitchat response generation, suggesting the potential of fine-tuning large pre-trained language models on other dialog response generation tasks. a more recent work @xcite adopted the framework of trans-fertransfo. they made the first attempt to leverage large pre-trained language models gpt and gpt-2 on task-oriented dialog generation, but it included belief states modeling as the input and did not achieve better results than the baseline. we propose to model dialogs without any annotation but rely on pre-trained large-scale language models that alternate. ## alternating recurrent dialog model we propose alternating recurrent dialog model (ardm) by compositing two separate pre-trained language model in alternate order to learn the user and system utterance distribution through memory recurrence. figure 1 shows an overview of ardm. ## experiments and results data scarcity is one of the biggest challenges in dialog research. it is costly to collect human-human conversations under a specific setting. it is even more time-consuming to annotate the corresponding belief states and dialog acts. with the success of transfer learning in nlp, we aim to mitigate the low-resource problem with the large pre-trained language model. we validate our proposed ardm on three task-oriented dialog datasets, camrest676, mulitwoz, and persuasionforgood. ## error analysis since camrest676 is similar to multiwoz in terms of task content and dialog structure, we only describe the errors in multiwoz for simplicity. we randomly selected 30 generated error responses from our model with zero inform and success score. to our surprise, we observed that nearly 63.3% of errors are not really mistakes. it is mainly due to the limitation of the automatic evaluator. for example, at turn one, the user asks about a restaurant, and the ground truth system response is "the [restaurant name] is located at . . . ", but the generated system response is "what food preference do you have?". our generated response is correct with respect to the dialog context. it is narrowing down the restaurant choices before providing a restaurant recommendation. however, the evaluator sticks to the only possible response it has. unless the user can dynamically interact with the system, there is no good way to change such mistakes in the automatic evaluator. we find that another 20% errors our model makes are when the system asks information the user already provided. this type of errors calls for a better history representation. another 10% errors are due to ignoring the user's request for information, such as phone number. however, when we look at the ground truth responses, some crowd workers also made such errors. so resolving these errors requires a cleaner training dataset. finally, the rest of 6.7% errors are about incorrect dialog domain understanding. for example, the user is asking for a hotel, but we present a restaurant recommendation. this is because of the data noise during the delexicalization process in which some domain labels are wrong. the donation persuasion system trained with transfertransfo and our model has some common problems, such as inconsistency, lack of logic, and hallucination. for example, if the persuader provides the information about "save the children", then the persuadee asks "can you tell me more about it?". the system ends up providing the same information as before. it also sometimes makes up facts that have never happened, such as "save the children has an operation about a hurricane in hawaii". all those errors would prevent users from trusting the bot, and therefore resulting in less donation. however, we also observe that users have a higher tolerance for errors in the persuasion setting than the customer service setting. overall, our model performs better on persua-sionforgood by having longer and diversified utterances. this suggests our model which utilizes alternating parameters is more effective than trans-fertransfo. ## discussions and ethical consideration ardm models speakers separately on top of a large pre-trained language model. such simple adaptation demonstrates substantial performance gain. we suspect it is because the interleaved structure of two language models provides a collabora-role utterance sys hi, how are you doing today? usr i am good! sys have you ever heard of the charity save the children? usr no, i have not heard about it. would you like to talk about it? ## conclusions we propose to build alternating recurrent dialog model (ardm), a simple, general, and effective dialog method that models user and system separately with large-scale pre-trained language models. since ardm does not require any annotations, it generalizes to different dialog applications. experimental results on camrest676 and multiwoz suggest that ardm outperforms or is on-par with the current state-of-the-art methods that use manual annotation information, such as belief states and dialog acts. furthermore, we find our model's excellent performance generalizes to more complex non-collaborative dialog settings. it can generate high-quality responses to persuade people to donate to charity. however, the easiness of training ardm raises concerns about the misuse of the