Dial2vec: Self-Guided Contrastive Learning of Unsupervised Dialogue Embeddings (2022)

Abstract: In this paper, we introduce the task of learning unsupervised dialogue embeddings.Trivial approaches such as combining pre-trained word or sentence embeddings and encoding through pre-trained language models (PLMs) have been shown to be feasible for this task.However, these approaches typically ignore the conversational interactions between interlocutors, resulting in poor performance.To address this issue, we proposed a self-guided contrastive learning approach named dial2vec.Dial2vec considers a dialogue as an information exchange process.It captures the interaction patterns between interlocutors and leverages them to guide the learning of the embeddings corresponding to each interlocutor.Then the dialogue embedding is obtained by an aggregation of the embeddings from all interlocutors.To verify our approach, we establish a comprehensive benchmark consisting of six widely-used dialogue datasets.We consider three evaluation tasks: domain categorization, semantic relatedness, and dialogue retrieval.Dial2vec achieves on average 8.7, 9.0, and 13.8 points absolute improvements in terms of purity, Spearman’s correlation, and mean average precision (MAP) over the strongest baseline on the three tasks respectively.Further analysis shows that dial2vec obtains informative and discriminative embeddings for both interlocutors under the guidance of the conversational interactions and achieves the best performance when aggregating them through the interlocutor-level pooling strategy.All codes and data are publicly available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/dial2vec.

## introduction dialogue embedding, as a critical prerequisite of semantically understanding a dialogue, has been * corresponding author. a central issue in dialogue-related research such as dialogue clustering @xcite , conversational sentiment analysis @xcite , context-dependent text-to-sql @xcite , and dialogue summarization @xcite . trivial unsupervised approaches generally encode dialogues by combining their pretrained word or sentence embeddings @xcite or using plms @xcite @xcite . however, such methods are not specifically designed for dialogues and thus fail to adequately capture the key conversational information. in this paper, we formally introduce the task of learning unsupervised dialogue embeddings, which aims to learn dialogue embeddings that can well reflect conversational semantics without any additional manual annotations. previous studies have extensively demonstrated the importance of encoding token-level interactions for learning semantic textual embeddings. however, for dialogue embedding, encoding interlocutor-level interactions is also essential but is overlooked in trivial approaches. figure 1 shows an example. we highlight the significant interaction patterns between the interlocutors with red color. as we can see, although these patterns only appear in three utterances, they highly represent the key conversational semantics (e.g., topics) and are more important than the other parts (e.g., greetings and chit-chats). we hold that capturing and leveraging them is one of the keys to learning high-quality unsupervised dialogue embeddings. in this work, we propose dial2vec, a self-guided contrastive learning approach to solve the proposed task. dial2vec considers a dialogue as an information exchange process between the two interlocutors and learns embeddings for both interlocutors with the help of each other. specifically, dial2vec firstly encodes a dialogue through a plm and assigns each interlocutor a self-representation by masking the non-corresponding positions in the encoding outputs. then it calculates a matching matrix via the token-level dot-product operation between the two self-representations, obtaining a cross-representation for each interlocutor. finally, the two cross-representations are leveraged as guidance to help the two self-representations gradually learn the interlocutor-level interaction-aware information and eliminate the interaction-free information during the training procedure. to verify our model, we build a comprehensive benchmark comprising a total of 98,879 dialogues by introducing six widely-used dialogue datasets, including bitod @xcite , doc2dial @xcite , metalwoz @xcite , multi-woz @xcite , self-dialogue @xcite , and sgd @xcite . each dataset consists of thousands of dialogues, where each dialogue is provided with a domain label (e.g., hotel booking and movie). we leverage these labels and design three evaluation tasks: domain categorization, semantic relatedness, and dialogue retrieval. we categorize them into intrinsic and extrinsic tasks according to their different focus. experimental results on this benchmark show that dial2vec outperforms the baselines by a substantial margin. compared with the strongest baseline, dial2vec achieves on average 8.7, 9.0, and 13.8 points absolute improvements in terms of purity, spearman's correlation, and mean average precision (map) on the three tasks respectively. we also conduct experiments with the single interlocutor's embeddings, their aggregation strategies, and the overall dialogue embedding distributions to study how dial2vec achieves such advanced performance. the results demonstrate that dial2vec learns both informative and discriminative embeddings for the two interlocutors and achieves the best performance when combining them through the proposed interlocutor-level pooling aggregation strategy. ## proposed approach in this section, we take a two-party dialogue as an example to describe how dial2vec works. it is worth mentioning that dial2vec can be extended to the multi-party version through the ovr (one vs. the rest) scheme with no modification of the architecture. ## conclusion in this paper, we formally introduce the task of learning unsupervised dialogue embeddings and propose dial2vec to solve this task. we introduce a self-guided mechanism that leverages the conversational interactions to guide the learning of the embeddings for both interlocutors and propose the interlocutor-level strategy to aggregate them. we further release a benchmark consisting of six widely-used dialogue datasets and three tasks designed based on the domain labels. our model achieves superior performances on all datasets across the three tasks, and further analysis shows that the dialogue embeddings learned by our model are more informative and discriminative than the baselines. we believe there is still much room for improvement to generate satisfactory dialogue embedding.