Learning Dialogue Representations from Consecutive Utterances (2022)

Abstract: Learning high-quality dialogue representations is essential for solving a variety of dialogue-oriented tasks, especially considering that dialogue systems often suffer from data scarcity. In this paper, we introduce Dialogue Sentence Embedding (DSE), a self-supervised contrastive learning method that learns effective dialogue representations suitable for a wide range of dialogue tasks. DSE learns from dialogues by taking consecutive utterances of the same dialogue as positive pairs for contrastive learning. Despite its simplicity, DSE achieves significantly better representation capability than other dialogue representation and universal sentence representation models. We evaluate DSE on five downstream dialogue tasks that examine dialogue representation at different semantic granularities. Experiments in few-shot and zero-shot settings show that DSE outperforms baselines by a large margin, for example, it achieves 13% average performance improvement over the strongest unsupervised baseline in 1-shot intent classification on 6 datasets. We also provide analyses on the benefits and limitations of our model.

## introduction due to the variety of domains and the high cost of data annotation, labeled data for task-oriented dialogue systems is often scarce or even unavailable. therefore, learning universal dialogue representations that effectively capture dialogue semantics at different granularities @xcite @xcite provides a good foundation for solving various downstream tasks @xcite . contrastive learning @xcite has achieved widespread success in representations learning in both the image domain @xcite @xcite and the text domain @xcite @xcite . contrastive learning aims to reduce the distance between semantically similar (positive) pairs and increase the distance between semantically dissimilar (negative) pairs. these positive pairs can be either human-annotated or obtained through various data augmentations, while negative pairs are often collected through negative sampling in the mini-batch. in the supervised learning regime, @xcite ; @xcite demonstrate the effectiveness of leveraging the natural language inference (nli) datasets @xcite to support contrastive learning. inspired by their success, a natural choice of dialogue representation learning is utilizing the find me some restaurants. korean food, please. ## why contrastive learning on consecutive utterances? when performing contrastive learning on consecutive utterances, we encourage the model to treat an utterance as similar to its adjacent utterances and dissimilar to utterances that are not consecutive to it or that belong to other dialogues. on the one hand, this training process directly increases an utterance's similarity with its true response and decreases its similarities with other randomly sampled utterances. the ability to identify the appropriate response from many similar utterances is beneficial for dialogue ranking tasks (e.g., response selection). on the other hand, consecutive utterances also contain implicit categorical information, which benefits dialogue classification tasks (e.g., intent classification and out-of-scope detection). consider pairs 1 and 4 in figure 2 : we implicitly learn similar representations of i am looking for restaurants and find me some restaurants, since they are both consecutive with what type of food do you like?. in contrast, simcse does not enjoy these benefits by simply using dropout as data augmentation. although tod-bert also leverages the intrinsic dialogue semantics by combining an utterance with its dialogue context as positive pair, the context is often the concatenation of 5 to 15 utterances. due to the large discrepancy in both semantics and data statistics between each utterance and its context, simply optimizing the similarity between them leads to less satisfying representations on many dialogue tasks. as shown in section 4, tod-bert can even lead to degenerated representations on some downstream tasks when compared to the original bert model. ## experiments we run experiments with five different backbones: bert base , bert large @xcite , roberta base , roberta large @xcite , distilbert base @xcite . due to the space limit, we only present the results on bert base in the main text. the results of other models are summarized in appendix d. we use the same training data as tod-bert for a fair comparison. we summarize the implementation details and data statistics of both pre-training and evaluation in appendices a and b, respectively. ## related work positive pair construction popular supervised sentence representation learning often takes advantage of the human-annotated natural language inference (nli) datasets @xcite for contrastive learning @xcite @xcite . these sentence pairs either entail or contradict each other, making them the great choice for constructing positive and negative training pairs. unsupervised sentence representation learning often relies on variant data augmentation strategies. logeswaran and @xcite and @xcite propose using sentences and their surrounding context as positive pairs. other works resort to popular nlp augmentation methods such as word permutation @xcite and back-translation @xcite . recently, @xcite demonstrates the superiority of using dropout over other data augmentations that directly operate on the discrete texts. contrastive learning methods contrastive learning is key to recent advances in learning sentence embeddings. many contrastive learning approaches utilize memory-based methods, which draw negative samples from a memory bank of embeddings @xcite @xcite . on the other hand, chen et al. (2020) introduces a memory-free contrastive framework, simclr, that takes advantage of negative sampling within large mini-batches. promising results were also reported in the nlp domain. to name a few, @xcite leverages both within batch negatives and the 'contradiction' annotations in nli; and @xcite propose an unsupervised hard-negative sampling strategy. dialogue language model learning dialoguespecific language models has attracted a lot of attention. along this line, @xcite adapts the pre-trained gpt-2 model @xcite on reddit data to perform open-domain dialogue response generation. @xcite evaluates multiple dialogue generation tasks after training on twitter and reddit data @xcite . for dialogue understanding, @xcite propose a response selection approach using a dual-encoder model. they pretrain the response selection model on reddit and then fine-tune it for different response selection tasks. following this, @xcite introduces a more efficient conversational model that is pre-trained with a response selection target on the reddit corpus. however, they did not release code or pre-trained models for comparison. @xcite combines nine dialogue datasets to obtain a large and high-quality task-oriented dialogue corpus. they introduce the tod-bert model by further pre-training bert on this corpus with both the masked language modeling loss and the contrastive response selection loss. ## conclusion in this paper, we introduce a simple contrastive learning method dse that learns dialogue representations by leveraging consecutive utterances in dialogues as positive pairs. we conduct extensive experiments on five dialogue tasks to show that the proposed method greatly outperforms other stateof-the-art dialogue representation models and universal sentence representation methods. we provide ablation study and analysis on our proposed data construction from different perspectives, investigate the trade-off between different data construction variants, and discuss the potential limitation to motivate further exploration in representation learning on unlabeled dialogues. we believe dse can serve as a drop-in replacement of the dialogue representation model (e.g., the text encoder) for a wide range of dialogue systems.