Dialog-Post: Multi-Level Self-Supervised Objectives and Hierarchical Model for Dialogue Post-Training (2023)

Abstract: Dialogue representation and understanding aim to convert conversational inputs into embeddings and fulfill discriminative tasks. Compared with free-form text, dialogue has two important characteristics, hierarchical semantic structure and multi-facet attributes. Therefore, directly applying the pretrained language models (PLMs) might result in unsatisfactory performance. Recently, several work focused on the dialogue-adaptive post-training (DialPost) that further trains PLMs to fit dialogues. To model dialogues more comprehensively, we propose a DialPost method, Dialog-Post, with multi-level self-supervised objectives and a hierarchical model. These objectives leverage dialogue-specific attributes and use self-supervised signals to fully facilitate the representation and understanding of dialogues. The novel model is a hierarchical segment-wise self-attention network, which contains inner-segment and inter-segment self-attention sub-layers followed by an aggregation and updating module. To evaluate the effectiveness of our methods, we first apply two public datasets for the verification of representation ability. Then we conduct experiments on a newly-labelled dataset that is annotated with 4 dialogue understanding tasks. Experimental results show that our method outperforms existing SOTA models and achieves a 3.3% improvement on average.

## introduction as an indispensable way of communication, dialogue is related to many research and application scenarios in academia and industry. better dialogue representation and understanding serve for several tasks, including intent classification, emotion recognition, and response selection, thus how to represent and model dialogues is an essential topic. compared with free-form text, dialogue modeling has to pay more attention to the following * corresponding author. to better utilize plms for dialogue representation and understanding, researchers use data samples from dialogue corpora to conduct a second phase pre-training of plms, i.e, dialogue-adaptive post-training (dialpost). at first, the training objectives were just those for general language modeling (masked language modeling and next sentence prediction) @xcite @xcite . after that, researchers tried to design some novel objectives that fit dialogue characteristics more. for example, @xcite utilized span boundary objective and perturbation masking objective in post-training to capture the dialogue semantics in span and token levels. @xcite and @xcite constructed positive and negative samples for context-response pairs, and continued training plms with contrastive learning to better maintain the dialogue coherence. existing dialpost methods either focus on tokenlevel or utterance-level semantics, which only consider a limited subset of dialogue attributes, e.g., speaker-shift @xcite , coherence @xcite , and response-similarity @xcite . however, the comprehensive modeling of multi-facet attributes with multi-level training objectives is not well explored. moreover, previous dialpost methods handle the whole dialogue as a linear sequence of successive tokens and feed it to plms that obtain the token representations indiscriminately with flat self-attention mechanisms. such a way of modeling is sub-optimal to capture the hierarchical semantic relations of dialogues @xcite . to evaluate the proposed method, we conduct experiments in two aspects, i.e., dialogue representation and understanding. we first verify the representation ability of dialog-post with dialoguebased semantic textual similarity (d-sts) and semantic retrieval (sr) tasks on two public datasets, jddc and ecd. dialog-post outperforms baselines by 1.7% in d-sts task of jddc. then, we annotate a dataset with four dialogue understanding tasks and conduct experiments on them. experimental results show that our method consistently outperforms baselines and achieves a 87.5% average score (+3.3%) for dialogue understanding. our contribution can be summarized as follows: (1) we propose a post-training method (dialog-post) for dialogue representation and understanding, which consists of five multi-level ssos and a hierarchical model. (2) we conduct extensive experiments to evaluate dialog-post with two public and one newly-labelled dataset. (3) we analyse the effectiveness of each component of dialog-post, and conduct ablation study to demonstrate the necessity of objectives in different levels. ## approach in this section, we introduce the multi-level selfsupervised objectives (ssos) and hssa model. ## experiments to verify the effectiveness of dialog-post, we conduct extensive experiments on both dialogue representation and understanding tasks. we first introduce the experimental setup, then elaborate implementation details, and finally illustrate the main experimental results. ## discussion in this section, we conduct further in-depth discussions to analyse the hssa model, the contribution of each sso, and visualize the training process of cmtl. due to space limitation, we report the results of jddc/sr, jddc/d-sts, ic and senti. ## related work dialogue encoding networks. to handle the particularity of dialogue structure, previous works have proposed several typical networks for dialogue encoding, including hierarchical attentionbased models @xcite @xcite , recurrence-based models @xcite , and long conversation oriented models @xcite . for example, jiao et al. (2019) and zhu et al. (2020) encode each utterance at first, and then leverage lstms and transformers to aggregate the utterances, while shen et al. (2021b) use memory caches to encode utterances sequentially. huang et al. (2021) integrate sparse attention to encode long dialogue sequences (e.g., with 5000 words). in this work, we propose a novel dialogue encoding network hssa to capture the semantic structure of dialogues. it takes the dialogue as input and leverages inner-segment and inter-segment selfattention to capture the hierarchical dependencies. finally, we devise an updater to obtain the contextual encoding of dialogues by aggregating the inner-and inter-segment representations. dialogue post-training. with the booming of plms @xcite @xcite , researchers try to apply plms in the field of dialogues. an intuitive idea is to conduct a second-stage pre-training with massive dialogue corpora, but without changing the training objectives @xcite . recently, some works @xcite @xcite are proposed to design several new objectives for dialogue-adaptive post-training and achieve astonishing performance on downstream tasks of dialogue understanding. platos @xcite leverage the large-size unified language model @xcite to fulfill context encoding and response generation tasks with curriculum learning. response selection is widely used as a self-supervised post-training task due to the convenience of constructing training data @xcite @xcite . @xcite propose span boundary objective and perturbation masking objective to capture the dialogue semantics in span and token levels. above works either focus on token-level or utterance-level semantics, and only consider a small set of dialogue attributes. differently, we propose five self-supervised objectives in token, utterance and dialogue levels, aiming to modeling multi-facet dialogue attributes, including fact-awareness, speaker-shift, coherence, and content-relatedness of both utterance-response pairs and context-context pairs. ## conclusion and future work in this paper, we propose a novel dialogue-adaptive post-training method, dialog-post, by devising five multi-level training objectives and a hierarchical dialogue encoder network. these training objectives capture the multi-facet attributes of dialogues by leveraging token-level, utterance-level, and dialogue-level self-supervised signals. the dialogue encoder learns the hierarchical semantic structure of dialogues. to validate the effectiveness of our method, extensive experiments on dialogue representation and understanding tasks are conducted. experimental results demonstrate the competitiveness of our method against strong baselines of both tasks. in the future, we will explore more efficient model architectures and try to pretrain the dialogue-oriented plms from scratch.