SPACE -2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding (2022)

Abstract: Pre-training methods with contrastive learning objectives have shown remarkable success in dialog understanding tasks. However, current contrastive learning solely considers the self-augmented dialog samples as positive samples and treats all other dialog samples as negative ones, which enforces dissimilar representations even for dialogs that are semantically related. In this paper, we propose SPACE-2, a tree-structured pre-trained conversation model, which learns dialog representations from limited labeled dialogs and large-scale unlabeled dialog corpora via semi-supervised contrastive pre-training. Concretely, we first define a general semantic tree structure (STS) to unify the inconsistent annotation schema across different dialog datasets, so that the rich structural information stored in all labeled data can be exploited. Then we propose a novel multi-view score function to increase the relevance of all possible dialogs that share similar STSs and only push away other completely different dialogs during supervised contrastive pre-training. To fully exploit unlabeled dialogs, a basic self-supervised contrastive loss is also added to refine the learned representations. Experiments show that our method can achieve new state-of-the-art results on the DialoGLUE benchmark consisting of seven datasets and four popular dialog understanding tasks.

## introduction task-oriented dialog (tod) systems aim to help users to accomplish specific tasks through natural language interactions, e.g., restaurant booking, hotel reserving, and movie searching @xcite . in order to fulfill a user goal, the dialog systems must be capable of extracting structured semantics from the dialog utterances, which is often referred to as dialog understanding. common tasks of dialog understanding include: 1) intent prediction, where the system classifies the user intention given the current utterance, 2) slot filling, where the system extracts the value spans for predefined slots from user utterances, and 3) dialog state tracking, where slot-value pairs are predicted and updated according to multi-turn dialog history. recent methods address the dialog understanding problems mainly by task-adaptive training with pre-trained language models (plms) @xcite @xcite . by fine-tuning the entire parameters of plms on dialog corpora, the versatile knowledge acquired from large-scale corpora can be adapted to handle various dialog understanding tasks. for example, tod-bert @xcite continually pre-trained a bert on large-scale task-oriented dialog corpora with similar objectives to grasp more knowledge in conversations and achieved good results on a wide range of tasks. to further enhance the representations of dialogs, contrastive learning is widely used for dialog pre-training via pulling semantically similar sentences together and pushing apart dis-similar ones, which has shown strong ability in few-shot dialog understanding @xcite . although contrastive learning (cl) is effective in pre-training dialog models, it still suffers from several problems. first, the current practice of cl often neglects the semantic structures of dialog examples and only utilizes shallow self-supervised information. figure 1 illustrates the semantic frames of two dialog examples, which indicate the hierarchical structures with semantic meanings in terms of intentions and slots. the semantic structure is often summarized as the turn-level structural annotation and is prominent in multi-domain taskoriented dialogs. exploiting such structure during ## pre-training data in this section, we first describe the dialog datasets @xcite used for pre-training, including a labeled dialog dataset (anpredial) and a largescale unlabeled dialog corpus (unpredial). then, we elaborate on the semantic tree structure (sts) for the task-oriented dialog data. ## methods in this section, we first introduce the model architecture. then we expound on the pre-training procedure with the proposed tree-structured semisupervised contrastive learning (cl) paradigm. ## conclusion in this work, we propose a new pre-trained conversation model named space-2, which learns dialog representations from both labeled and unlabeled corpora via tree-structured semi-supervised contrastive learning (cl). to be specific, we utilize a semantic tree structure (sts) to unify the inconsistent annotation schema for all datasets. then, we propose a novel multi-view scoring function to compute the semantic similarity based on sts's sub-structures for labeled dialogs. different from traditional cl, our method increases the relevance of representations for all similarly labeled dialogs and only pushes away completely different dialogs. a self-supervised cl loss is also integrated to exploit unlabeled dialogs. extensive experiments on the dialoglue benchmark with seven datasets demonstrate that our model achieves the best results on all dialog understanding tasks, under the full-data and the few-shot settings.