Dial- MAE : C on T extual Masked Auto-Encoder for Retrieval-based Dialogue Systems (2024)

Abstract: Dialogue response selection aims to select an appropriate response from several candidates based on a given user and system utterance history. Most existing works primarily focus on post-training and fine-tuning tailored for cross-encoders. However, there are no post-training methods tailored for dense encoders in dialogue response selection. We argue that when the current language model, based on dense dialogue systems (such as BERT), is employed as a dense encoder, it separately encodes dialogue context and response, leading to a struggle to achieve the alignment of both representations. Thus, we propose Dial-MAE (Dialogue Contextual Masking Auto-Encoder), a straightforward yet effective post-training technique tailored for dense encoders in dialogue response selection. Dial-MAE uses an asymmetric encoder-decoder architecture to compress the dialogue semantics into dense vectors, which achieves better alignment between the features of the dialogue context and response. Our experiments have demonstrated that Dial-MAE is highly effective, achieving state-of-the-art performance on two commonly evaluated benchmarks.

## introduction the retrieval-based dialogue system is a popular research topic. pre-trained language models (plms), especially deep bidirectional transformer language models (lms) like bert encoder @xcite , have been widely used in dialogue response. common uses of deep lm are cross-encoder and bi-encoder @xcite . recent works @xcite @xcite @xcite on dialogue response retrieval systems are mostly based on cross-encoders, which feed both the dialogue context and response directly into lm and use attention over all tokens to * these authors contributed equally to this work. ## related work in this section, we first discuss traditional retrieval dialogue systems based on neural networks, and then we discuss current dialogue systems based on pre-trained language models. ## methodology this section first introduces masked language model pre-training as preliminary knowledge. then we introduce detailed post-training, including the construction of data and the auxiliary task. finally, we introduce the details of fine-tuning. ## experiment in this section, we first introduce our experimental details, including datasets, evaluation metrics, posttraining, and fine-tuning. then we introduce the experimental results. ## conclusion in this paper, we propose a post-training method tailored for dialogue response, considering the semantics of dialogue context and its corresponding responses. precisely, we leverage a shallow decoder to force the encoder output dialogue embeddings to be more expressive. experimental results show that our post-training method leads to considerable improvements, achieving state-of-the-art on two benchmark datasets. we also demonstrate the effectiveness of dial-mae through ablation experiments. specifically, both contrastive learning and our post-training method are effective in achieving dialogue context and response feature alignment, and their effects can be additive.