F uture TOD : Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue (2023)

Abstract: Pre-trained language models based on general text enable huge success in the NLP scenario. But the intrinsical difference of linguistic patterns between general text and task-oriented dialogues makes existing pre-trained language models less useful in practice. Current dialogue pre-training methods rely on a contrastive framework and face the challenges of both selecting true positives and hard negatives. In this paper, we propose a novel dialogue pre-training model, FutureTOD, which distills future knowledge to the representation of the previous dialogue context using a self-training framework. Our intuition is that a good dialogue representation both learns local context information and predicts future information. Extensive experiments on diverse downstream dialogue tasks demonstrate the effectiveness of our model, especially the generalization, robustness, and learning discriminative dialogue representations capabilities.

## introduction pre-trained language models @xcite based on a massive scale of general text corpora @xcite have been commonly used in many nlp applications. finetuning models on these plms significantly improves the performance of various downstream tasks, especially natural language understanding. despite their success, directly applying them to conversational corpora is proved to be suboptimal due to the large linguistic gap between conversations and plain text @xcite . therefore, it's vital to explore dialogue-specific pre-trained models for solving various downstream dialogue tasks. early pre-trained dialogue language models use chit-chat corpora from social media, such as twitter or reddit, aiming at retrieval (henderson et al., ⇤ the first two authors contribute equally. weiran xu is the corresponding author. update distillation loss mlm loss ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... teacher ... ... ... ... ... [cls] [usr] is fee [sys] no unpaid 1 layer l-1 layer l layer 1 layer l-1 layer l layer student context future legal [cls] is [mask] [usr] context + = ... ... loss gradient fee stop gradient dialogue domain because of ignoring the intrinsic properties of dialogue data @xcite . tod-bert @xcite takes the dialogue context 2 and next response as a positive pair thus achieving promising performance on the response selection task. however, there is a large discrepancy in both semantics and data statistics between each response and its context 3 , which reduces its generalization ability to other dialogue tasks. further, dse @xcite learns from dialogues by taking consecutive utterances of the same dialogue as positive pairs. but the assumption that consecutive utterances represent similar semantics fails sometimes when answers are general and ubiquitous. along with the issues of choosing positive pairs, these models regard other instances in the same batch as negative samples, which also induces potential noise to contrastive learning @xcite , such as false negatives @xcite and relying on a large batch size @xcite . overall, these contrastive methods face the challenges of both selecting true positive pairs and negative pairs that we aim to solve using a new non-contrastive pre-training framework. in this paper, we propose a novel dialogue pretraining model, futuretod, which distills future knowledge to the representation of the previous dialogue context using future utterances based on a standard transformer architecture bert @xcite . we argue that a good dialogue representation both learns local context information and predicts future knowledge. instead of existing contrastive works, we employ a self-training framework and dismiss the requirements of con-2 throughout this paper, we denote a system turn including all the system sentences as the response (utterance), and all the history turns as the dialogue context. 3 in the implementation of tod-bert, the context is often the concatenation of 5 to 15 utterances but the response is only a single utterance. our contributions are: (1) we propose a novel tod dialogue pre-training model, futuretod, which distills future knowledge to dialogue representations. to the best of our knowledge, we are the first to use a non-contrastive self-training framework and knowledge distillation for dialogue pre-training. (2) our model achieves consistent improvements on diverse downstream dialogue tasks over strong baselines. extensive analyses prove the generalization, robustness, and learning discriminative dialogue representations capabilities. ## conclusion we propose a novel dialogue pre-training model, futuretod, which distills future knowledge to dialogue representations. instead of existing contrastive works, we employ a simple self-training framework to learn from each other and dismiss the requirements of contrastive pairs. we perform comprehensive experiments on various task-oriented dialogue tasks, including intent classification, out-ofdomain detection, dialogue state tracking, dialogue act prediction, and response selection. futuretod significantly outperforms tod-bert, dse, and other strong baselines in all the scenarios. fu-turetod is of excellent performance and easy-todeploy without modifying any model architecture. yukun zhu, ryan kiros, richard s. zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. 2015 ieee international conference on computer vision (iccv), pages 19-27.