Learning sentence embeddings from dialogues has drawn increasing attention due to its low annotation cost and high domain adaptability . Conventional approaches employ the siamese-network for this task , which obtains the sentence embeddings through modeling the context-response semantic relevance by applying a feed-forward network on top of the sentence encoders . However , as the semantic textual similarity is commonly measured through the element-wise distance metrics ( e.g. cosine and L2 distance ) , such architecture yields a large gap between training and evaluating . In this paper , we propose DialogueCSE , a dialogue-based contrastive learning approach to tackle this issue . DialogueCSE first introduces a novel matching-guided embedding ( MGE ) mechanism , which generates a contextaware embedding for each candidate response embedding ( i.e. the context-free embedding ) according to the guidance of the multi-turn context-response matching matrices . Then it pairs each context-aware embedding with its corresponding context-free embedding and finally minimizes the contrastive loss across all pairs . We evaluate our model on three multi-turn dialogue datasets : the Microsoft Dialogue Corpus , the Jing Dong Dialogue Corpus , and the E-commerce Dialogue Corpus . Evaluation results show that our approach significantly outperforms the baselines across all three datasets in terms of MAP and Spearman 's correlation measures , demonstrating its effectiveness . Further quantitative experiments show that our approach achieves better performance when leveraging more dialogue context and remains robust when less training data is provided . Sentence embeddings are used with success for a variety of NLP applications ( Cer et al . , 2018 ) and many prior methods have been proposed with different learning schemes . Kiros et al . ( 2015 ) ; Logeswaran and Lee ( 2018 ) ; Hill et al . ( 2016 ) train sentence encoders in a self-supervised manner with web pages and books . Conneau et al . ( 2017 ) ; Cer et al . ( 2018 ) ; Reimers and Gurevych ( 2019 ) propose to learn sentence embeddings on the supervised datasets such as SNLI ( Bowman et al . , 2015 ) and MNLI ( Williams et al . , 2018 ) . Although the supervised-learning approaches achieve better performance , they suffer from high cost of annotation in building the training dataset , which makes them hard to adapt to other domains or languages . Recently , learning sentence embeddings from dialogues has begun to attract increasing attention . Dialogues provide strong semantic relationships among conversational utterances and are usually easy to collect in large amounts . Such advantages make the dialogue-based self-supervised learning methods promising to achieve competitive or even superior performance against the supervised-learning methods , especially under the low-resource conditions . While promising , the issue of how to effectively exploit the dialogues for this task has not been sufficiently explored . Yang et al . ( 2018 ) propose to train an input-response prediction model on Reddit dataset ( Al-Rfou et al . , 2016 ) . Since they build their architecture based on the single-turn dialogue , the multi-turn dialogue history is not fully exploited . Henderson et al . ( 2020 ) demonstrate that introducing the multi-turn dialogue context can improve the sentence embedding performance . However , they concatenate the multi-turn dialogue context into a long token sequence , failing to model intersentence semantic relationships among the utterances . Recently , more advanced methods such as ( Reimers and Gurevych , 2019 ) achieve better performance by employing BERT ( Devlin et al . , 2019 ) as the sentence encoder . These works have in common that they employ a feed-forward network with a non-linear activation on top of the sentence en-coders to model the context-response semantic relevance , thereby learning the sentence embeddings . However , such architecture presents two limitations : ( 1 ) It yields a large gap between training and evaluating , since the semantic textual similarity is commonly measured by the element-wise distance metrics such as cosine and L2 distance . ( 2 ) Concatenating all the utterances in the dialogue context inevitably introduces the noise as well as the redundant information , resulting in a poor result . In this paper , we propose DialogueCSE , a dialogue-based contrastive learning approach to tackle these issues . We hold that the semantic matching relationships between the context and the response can be implicitly modeled through contrastive learning , thus making it possible to eliminate the gap between training and evaluating . To this end , we introduce a novel matching-guided embedding ( MGE ) mechanism . Specifically , MGE first pairs each utterance in the context with the response and performs a token-level dot-product operation across all the utterance-response pairs to obtain the multi-turn matching matrices . Then the multi-turn matching matrices are used as guidance to generate a context-aware embedding for the response embedding ( i.e. the context-free embedding ) . Finally , the context-aware embedding and the context-free embedding are paired as a training sample , whose label is determined by whether the context and the response are originally from the same dialogue . Our motivation is that once the context semantically matches the response , it has the ability to distill the context-aware information from the context-free embedding , which is exactly the learning objective of the sentence encoder that aims to produce context-aware sentence embeddings . We train our model on three multi-turn dialogue datasets : the Microsoft Dialogue Corpus ( MDC ) ( Li et al . , 2018 ) , the Jing Dong Dialogue Corpus ( JDDC ) ( Chen et al . , 2020 ) , and the E-commerce Dialogue Corpus ( ECD ) ( Zhang et al . , 2018 ) . To evaluate our model , we introduce two types of tasks : the semantic retrieval ( SR ) task and the dialogue-based semantic textual similarity ( D-STS ) task . Here we do not adopt the standard semantic textual similarity ( STS ) task ( Cer et al . , 2017 ) for two reasons : ( 1 ) As revealed in ( Zhang et al . , 2020 ) , the sentence embedding performance varies greatly as the domain of the training data changes . As a dialogue dataset is always about several certain domains , evaluating on the STS benchmark may mis-lead the evaluation of the model . ( 2 ) The dialoguebased sentence embeddings focus on context-aware rather than context-free semantic meanings , which may not be suitable to be evaluated through the context-free benchmarks . Since previous dialoguebased works have not set up a uniform benchmark , we construct two evaluation datasets for each dialogue corpus . A total of 18,964 retrieval samples and 4,000 sentence pairs are annotated by seven native speakers through the crowd-sourcing platform1 . The evaluation results indicate that DialogueCSE significantly outperforms the baselines on the three datasets in terms of both MAP and Spearman 's correlation metrics , demonstrating its effectiveness . Further quantitative experiments show that Dia-logueCSE achieves better performance when leveraging more dialogue context and remains robust when less training data is provided . To sum up , our contributions are threefold : • We propose DialogueCSE , a dialogue-based contrastive learning approach with MGE mechanism for learning sentence embeddings from dialogues . As far as we know , this is the first attempt to apply contrastive learning in this area . • We construct the dialogue-based sentence embedding evaluation benchmarks for three dialogue corpus . All of the datasets will be released to facilitate the follow-up researches . • Extensive experiments show that Dia-logueCSE significantly outperforms the baselines , establishing the state-of-the-art results . 2 Related Work In this work , we propose DialogueCSE , a dialoguebased contrastive learning approach to learn sentence embeddings from dialogues . We also propose uniform evaluation benchmarks for evaluating the quality of the dialogue-based sentence embeddings . Evaluation results show that DialogueCSE achieves the best result over the baselines while adding no additional parameters . In the next step , we will study how to introduce more interaction information to learn the sentence embeddings and try to incorporate the contrast learning method into the pre-training stage .

Challenge: Existing methods to learn representations from dialogues have a similarity-measurement gap between training and evaluation time and do not exploit the multi-turn structure of data.
Approach: They propose a dialogue-based contrastive learning approach to learn sentence embeddings from dialogues by modelling semantic matching relationships between the context and response implicitly.
Outcome: The proposed approach outperforms baseline methods on two newly introduced tasks coupled with three multi-turn dialogue datasets in terms of MAP and Spearman's correlation measures.