Structural Pre-training for Dialogue Comprehension (2021)

Abstract: Pre-trained language models (PrLMs) have demonstrated superior performance due to their strong ability to learn universal language representations from self-supervised pre-training. However, even with the help of the powerful PrLMs, it is still challenging to effectively capture task-related knowledge from dialogue texts which are enriched by correlations among speaker-aware utterances. In this work, we present SPIDER, Structural Pre-traIned DialoguE Reader, to capture dialogue exclusive features. To simulate the dialogue-like features, we propose two training objectives in addition to the original LM objectives: 1) utterance order restoration, which predicts the order of the permuted utterances in dialogue context; 2) sentence backbone regularization, which regularizes the model to improve the factual correctness of summarized subject-verb-object triplets. Experimental results on widely used dialogue benchmarks verify the effectiveness of the newly introduced self-supervised tasks.

## introduction recent advances in large-scale pre-training language models (prlms) have achieved remarkable successes in a variety of natural language processing (nlp) tasks @xcite @xcite @xcite . providing fine-grained contextualized embedding, these pre-trained models are widely employed as encoders for various downstream nlp tasks. although the prlms demonstrate superior perfor-u 1 : how can i connect to url streaming server ? u 2 : i think mplayer does it u 3 : i use kino , could be ? u 4 : kino is editing software . i don't think it supports media . u 5 : sorry emoji , i mean movie player u 6 : not sure about it . if it can't try vlc and mplayer too . u 7 : is mplayer aptable ? u 8 : yes but i 'm not sure if it is in the main repo . u 9 : have you ever updated your sources.list ? u 10 : i have no idea..i use adept on kubuntu . mance due to their strong representation ability from self-supervised pre-training, it is still challenging to effectively adapt task-related knowledge during the detailed task-specific training which is usually in a way of fine-tuning @xcite . generally, those prlms handle the whole input text as a linear sequence of successive tokens and implicitly capture the contextualized representations of those tokens through self-attention. such fine-tuning paradigm of exploiting prlms would be suboptimal to model dialogue task which holds exclusive text features that plain text for prlm training may hardly embody. therefore, we explore a fundamental way to alleviate this difficulty by improving the training of prlm. this work devotes itself to designing the natural way of adapting the language modeling to the dialogue scenario motivated by the natural characteristics of dialogue contexts. as an active research topic in the nlp field, multi-turn dialogue modeling has attracted great interest. the typical task is response selection @xcite @xcite that aims to select the appropriate response according to a given dialogue context containing a number of utterances, which is the focus in this work. how-ever, selecting a coherent and informative response for a given dialogue context remains a challenge. the multi-turn dialogue typically involves two or more speakers that engage in various conversation topics, intentions, thus the utterances are rich in interactions, e.g., with criss-cross discourse structures @xcite @xcite . a critical challenge is the learning of rich and robust context representations and interactive relationships of dialogue utterances, so that the resulting model is capable of adequately capturing the semantics of each utterance, and the relationships among all the utterances inside the dialogue. inspired by the effectiveness for learning universal language representations of prlms, there are increasing studies that employ prlms for conversation modeling @xcite @xcite @xcite . these studies typically model the response selection with only the context-response matching task and overlook many potential training signals contained in dialogue data. although the prlms have learned contextualized semantic representation from token-level or sentence-level pre-training tasks like mlm, nsp, they all do not consider dialogue related features like speaker role, continuity and consistency. one obvious issue of these approaches is that the relationships between utterances are harder to capture using word-level semantics. besides, some latent features, such as user intent and conversation topic, are under-discovered in existing works @xcite . therefore, the response retrieved by existing dialogue systems supervised by the conventional way still faces critical challenges, including incoherence and inconsistency. in this work, we present spider (structural pre-trained dialogue reader), a structural language modeling method to capture dialogue exclusive features. motivated to efficiently and explicitly model the coherence among utterances and the key facts in each utterance, we propose two training objectives in analogy to the original bert-like language model (lm) training: 1) utterance order restoration (uor), which predicts the order of the permuted utterances in dialogue context; 2) sentence backbone regularization (sbr), which regularizes the model to improve the factual correctness of summarized subject-verb-object (svo) triplets. experimental results on widely used benchmarks show that spder boosts the model performance for various multi-turn dialogue comprehension tasks including response selection and dialogue reasoning. ## approach this section presents our proposed method spi-der (structural pre-trained dialogue reader). first, we will present the standard dialogue comprehension model as the backbone. then, we will introduce our designed language modeling objectives for dialogue scenarios, including utterance order restoration (uor) and sentence backbone regularization (sbr). in terms of model training, we employ two strategies, i.e., 1) domain adaptive post-training that first trains a language model based on newly proposed objectives and then finetunes the response selection task; 2) multi-task finetuning that trains the model for downstream tasks, along with lm objectives. ## use of spider objectives in this section, we introduce two training methods to take the newly proposed language modeling objectives into account, namely domain-adaptive post-training and multi-task fine-tuning, as illustrated in figure 3 . ## conclusion in this paper, we focus on the task-related adaptation of the pre-trained language models and propose spider (structural pre-trained dialogue reader), a structural language modeling method to capture dialogue exclusive features. to explicitly model the coherence among utterances and the key facts in each utterance, we introduce two novel dialogue-aware language modeling tasks including utterance order restoration and sentence backbone regularization objectives. experiments on widely-used multi-turn dialogue comprehension benchmark datasets show the superiority over baseline methods. our work reveals a way to make better use of the structure learning of the contextualized representations from pre-trained language models and gives insights on how to adapt the language modeling training objectives in downstream tasks.