TOD - BERT : Pre-trained Natural Language Understanding for Task-Oriented Dialogue (2020)

Abstract: The underlying difference of linguistic patterns between general text and task-oriented dialogue makes existing pre-trained language models less useful in practice. In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling. We propose a contrastive objective function to simulate the response selection task. Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task-oriented dialogue applications, including intention recognition, dialogue state tracking, dialogue act prediction, and response selection. We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue.

## introduction pre-trained models with self-attention encoder architectures @xcite have been commonly used in many nlp applications. such models are self-supervised based on a massive scale of general text corpora, such as english wikipedia or books @xcite . by further fine-tuning these representations, breakthroughs have been continuously reported for various downstream tasks, especially natural language understanding. however, previous work @xcite shows that there are some deficiencies in the performance to apply fine-tuning on conversational corpora directly. one possible reason could be the intrinsic difference of linguistic patterns between human conversations and writing text, resulting in a large gap of data distributions @xcite . therefore, pre-training dialogue language models using chit-chat corpora from social media, such as twitter or reddit, has been recently investigated, especially for dialogue response generation @xcite and retrieval @xcite . although these opendomain dialogues are diverse and easy-to-get, they are usually short, noisy, and without specific chatting goals. on the other hand, a task-oriented dialogue has explicit goals (e.g. restaurant reservation or ticket booking) and many conversational interactions. but each dataset is usually small and scattered because obtaining and labeling such data is time-consuming. moreover, a task-oriented dialogue has explicit user and system behaviors where a user has his/her goal, and a system has its belief and database information, which makes the language understanding component and dialogue policy learning more important than those chit-chat scenarios. this paper aims to prove this hypothesis: selfsupervised language model pre-training using taskoriented corpora can learn better representations than existing pre-trained models for task-oriented downstream tasks. we emphasize that what we care about the most is not whether our pre-trained model can achieve state-of-the-art results on each downstream task since most of the current best models are built on top of pre-trained models, and ours can easily replace them. we avoid adding too many additional components on top of the pre-training architecture when fine-tuning in our experiments. we collect and combine nine human-human and multi-turn task-oriented dialogue corpora to train a task-oriented dialogue bert (tod-bert). in total, there are around 100k dialogues with 1.4m utterances across over 60 different domains. like bert @xcite , tod-bert is formulated as a masked language model and uses a deep bidirectional transformer @xcite encoder as its model architecture. unlike bert, tod-bert incorporates two special tokens for user and system to model the corresponding dialogue behavior. a contrastive objective function of response selection task is combined during pretraining stage to capture response similarity. we select bert because it is the most widely used model in nlp research recently, and our unified datasets can be easily applied to pre-train any existing language models. we test tod-bert on task-oriented dialogue systems on four core downstream tasks, including intention recognition, dialogue state tracking, dialogue act prediction, and response selection. what we observe is: tod-bert outperforms bert and other strong baselines such as gpt-2 @xcite and dialogpt @xcite ) on all the selected downstream tasks, which further confirms its effectiveness for improving dialogue language understanding. we find that response contrastive learning is beneficial, but it is currently overlooked not well-investigated in dialogue pretraining research. more importantly, tod-bert has a stronger few-shot ability than bert on each task, suggesting that it can reduce the need for expensive human-annotated labels. tod-bert can be easily leveraged and adapted to a new taskoriented dialogue dataset. our source code and data processing are released to facilitate future research on pre-training and fine-tuning of task-oriented dialogue 1 . some language models can support both unidirectional and bi-directional attention, such as unilm @xcite . conditional language model pre-training is also proposed. for example, ctrl @xcite is a conditional transformer model, trained to condition on control codes that govern style, content, and task-specific behavior. recently, multi-task language model pretraining with unified sequence-to-sequence generation is proposed. text-to-text transformer (t5) @xcite unifies multiple text modeling tasks and achieves the promising results in various nlp benchmarks. dialogue pre-trained language models are mostly trained on open-domain conversational data from reddit or twitter for dialogue response generation. transfertransfo @xcite achieves good performance on convai-2 dialogue competition using gpt-2. dialogpt @xcite is an extension of gpt-2 that is pre-trained on reddit data for open-domain response generation. con-vert @xcite pre-trained a dual transformer encoder for response selection task on large-scale reddit (input, response) pairs. plato @xcite uses both twitter and reddit data to pre-trained a dialogue generation model with discrete latent variables. all of them are designed to cope with the response generation task for opendomain chatbots. pretraining for task-oriented dialogues, on the other hand, has few related works. @xcite first apply the gpt-2 model to train on response generation task, which takes system belief, database result, and last dialogue turn as input to predict next system responses. it only uses one dataset to train its model because few public datasets have database information available. @xcite pre-trained a response selection model for task-oriented dialogues. they first pre-train on reddit corpora and then fine-tune on target dialogue domains, but their training and fine-tuning code is not released. @xcite focus on the natural language generation (nlg) task, which assumes dialogue acts and slot-tagging results are given to generate a natural language response. pre-training on a set of annotated nlg corpora can improve conditional generation quality using a gpt-2 model. ## method this section discusses each dataset used in our taskoriented pre-training and how we process the data. then we introduce the selected pre-training base model and its objective functions. ## downstream tasks we care the most in this paper whether tod-bert, a pre-trained language model using aggregated taskoriented corpora, can show any advantage over bert. therefore, we avoid adding too many additional components on top of its architecture when fine-tuning on each downstream task. also, we use the same architecture with a similar number of parameters for a fair comparison. all the model parameters are updated with a gradient clipping to 1.0 using the same hyper-parameters during finetuning. we select four crucial task-oriented downstream tasks to evaluate: intent recognition, dialogue state tracking, dialogue act prediction, and response selection. all of them are core components in modularized task-oriented systems @xcite . intent recognition task is a multi-class classification problem, where we input a sentence u and models predict one single intent class over i possible intents. the model is trained with cross-entropy loss between the predicted distributions p int and the true intent labels. dialogue state tracking can be treated as a multi-class classification problem using a predefined ontology. unlike intent, we use dialogue history x (a sequence of utterances) as input and a model predicts slot values for each (domain, slot) pair at each dialogue turn. each corresponding value v j i , the i-th value for the j-th (domain, slot) pair, is passed into a pre-trained model and fixed its representation during training. response selection is a ranking problem, aiming to retrieve the most relative system response from a candidate pool. we use a dual-encoder strategy @xcite and compute similarity scores between source x and target y , ## evaluation datasets we pick up several datasets, oos, dstc2, gsim, and mwoz, for downstream evaluation. the first three corpora are not included in the pre-trained task-oriented datasets. for mwoz, to be fair, we do not include its test set dialogues during the pretraining stage. details of each evaluation dataset are discussed in the following: ## results for each downstream task, we first conduct the experiments using the whole dataset, and then we simulate the few-shot setting to show the strength of our tod-bert. we run at least three times with different random seeds for each few-shot experiment to reduce data sampling variance, and we report its mean and standard deviation for these limited data scenarios. we investigate two versions of tod-bert; one is tod-bert-mlm that only uses mlm loss during pre-training, and the other is tod-bert-jnt, which is jointly trained with the mlm and rcl objectives. we compare tod-bert with bert and other baselines, including two other strong pre-training models gpt-2 @xcite and dialogpt @xcite . for a gpt-based model, we use mean pooling of its hidden states as its output representation, which we found it is better than using only the last token. ## visualization in figure 2 , we visualize the embeddings of bert, tod-bert-mlm, and tod-bert-jnt given the same input from the mwoz test set. each sample point is a system response representation, which is passed through a pre-trained model and reduced its high-dimension features to a two-dimension point using the t-distributed stochastic neighbor embedding (tsne) for dimension reduction. since we know the true domain and dialogue act labels for each utterance, we use different colors to represent different domains and dialogue acts. as one can observe, tod-bert-jnt has more clear group boundaries than tod-bert-mlm, and two of them are better than bert. to analyze the results quantitatively, we run kmeans, a common unsupervised clustering algorithms, on top of the output embeddings of bert and tod-bert. we set k for k-means equal to 10 and 20. after the clustering, we can assign each utterance in the mwoz test set to a predicted class. we then compute the normalized mutual information (nmi) between the clustering result and the actual domain label for each utterance. here is what we observe: tod-bert consistently achieves higher nmi scores than bert. for @xmath0, tod-bert has a 0.143 nmi score, and bert only has 0.094. for @xmath1, tod-bert achieves a 0.213 nmi score, while bert has 0.109. ## conclusion we propose task-oriented dialogue bert (tod-bert) trained on nine human-human and multiturn task-oriented datasets across over 60 domains. tod-bert outperforms bert on four dialogue downstream tasks, including intention classification, dialogue state tracking, dialogue act prediction, and response selection. it also has a clear advantage in the few-shot experiments when only limited labeled data is available. tod-bert is easy-to-deploy and will be open-sourced, allowing the nlp research community to apply or fine-tune any task-oriented conversational problem.