Learning-by-Narrating: Narrative Pre-Training for Zero-Shot Dialogue Comprehension (2022)

Abstract: Comprehending a dialogue requires a model to capture diverse kinds of key information in the utterances, which are either scattered around or implicitly implied in different turns of conversations. Therefore, dialogue comprehension requires diverse capabilities such as paraphrasing, summarizing, and commonsense reasoning. Towards the objective of pre-training a zero-shot dialogue comprehension model, we develop a novel narrative-guided pre-training strategy that learns by narrating the key information from a dialogue input. However, the dialogue-narrative parallel corpus for such a pre-training strategy is currently unavailable. For this reason, we first construct a dialogue-narrative parallel corpus by automatically aligning movie subtitles and their synopses. We then pre-train a BART model on the data and evaluate its performance on four dialogue-based tasks that require comprehension. Experimental results show that our model not only achieves superior zero-shot performance but also exhibits stronger fine-grained dialogue comprehension capabilities. The data and code are available at https://github.com/zhaochaocs/Diana .

## introduction dialogue comprehension @xcite aims to capture diverse kinds of key information in utterances, which are either scattered around or implicitly implied in different turns of conversations. therefore, it requires different capabilities such as paraphrasing @xcite , summarizing @xcite , and commonsense reasoning @xcite . recent advances in pre-trained language models (plms) @xcite have been applied to the problem @xcite . however, these plms are generally pretrained on formal-written texts, which are different from dialogue data in nature. specifically, dialogues are composed of colloquial languages from multi-speakers, and utterances usually have complex discourse structures @xcite . therefore, applying these models directly to dialogue comprehension, especially in low-resource settings, is sub-optimal. to learn better dialogue representations, recent studies have designed several dialogue-specific pretraining objectives such as speaker prediction @xcite , utterance prediction @xcite , response selection @xcite , and turn order restoration @xcite . these methods, albeit improve over the vanilla plms, usually rely on surface-level dialogue information. in particular, they still fail to train the models to explicitly learn the aforementioned capabilities which are critical for dialogue comprehension (e.g., linguistic knowledge, world knowledge, and commonsense knowledge). furthermore, it was not able to incorporate knowledge beyond dialogue (e.g., non-verbal communications between speakers, as well as time and location information), which are also crucial for dialogue comprehension. to pre-train a zero-shot dialogue comprehension model with the aforementioned features, we develop a novel generative pre-training strategy that learns by narrating the key information from a dialogue input (see figure 1 for an example). in particular, the generated narrative text is supposed to not only (i) paraphrase the gists of the dialogue but also (ii) carry certain inferred information (e.g., the time and location of a scene and relations between speakers) that are not explicitly mentioned in the dialogues. learning to narrate such information helps the model to learn varied lexical, syntactic, and semantic knowledge of dialogue. it also enhances the model's ability to infer extra information beyond the literal meaning within dialogues, which will benefit the model's capability of dialogue comprehension. movies & tvs s: one hundred two point two. it's like you're not trying to get better. a: sheldon, you don't get over the flu in half an hour. s: i just want you to get better as soon as possible. s: let me ask you a question. do you believe in the placebo effect? a: of course i do. there have been many studies proving its validity. s: great. now, this may look like a tic tac... but it is really powerful a: sheldon, this isn't helping. why don't you just let me get some rest sheldon is checking amy's temperature of 102.2 °f and it is the same as half an hour before. he asks if she believes in placebo and then gives her a tic tac. amy doesn't like this and tries to get to sleep… however, the learning-by-narrating strategy would require a dialogue-narrative parallel corpus, which, to our best knowledge, is not publicly available. for this reason, we first create diana, a large-scale dataset with (dialogue, narrative) pairs automatically collected from subtitles of movies and their corresponding plot synopses. we consider dialogues from movie subtitles as they are close to daily human-to-human conversations @xcite . in addition, the movie synopses include rich narrative information, which is helpful for dialogue comprehension. after data collection and strict quality control, we obtain a dataset with 243k (dialogue, narrative) pairs written in english. as the automatic data construction procedure is language-independent, it can be applied to low-resource languages as well. we then pre-train a bart model @xcite on the constructed corpus with the proposed learning-by-narrating strategy, and evaluate it on four dialogue-based tasks that require comprehension. in zero-shot settings, our pre-trained model outperforms the bart baseline on all tasks by a large margin (e.g., +8.3% on dream @xcite ), demonstrating the success of our approach. the contributions of this paper are three-fold: ## diana: a dialogue-narrative corpus in this section, we describe the procedure to create the dialogue-narrative parallel dataset. ## pre-training: learning-by-narrating during pre-training, we aim to inject the knowledge contained in diana into pre-trained models. one option is to ask the model to distinguish between a correct narrative and an incorrect narrative via a classification objective. however, it requires carefully designing additional non-trivial negative (dialogue, narrative) pairs. therefore, we propose to directly generate a narrative text from the given dialogue by maximizing the generative probability: there are two main advantages of using the generative objective. first, it can fully leverage the narrative information from each token of the narrative text with no need to construct negative pairs. second, the pre-trained model can be directly applied to both generative and discriminative downstream tasks without further fine-tuning. for discriminative tasks, we calculate the probability of each candidate according to equation 2 and choose the most probable candidate as the predicted answer. ## experiments in this section, we evaluate the performance of the pre-trained model on four downstream tasks that require dialogue comprehension. ## conclusion we propose a learning-by-narrating strategy to pretrain a zero-shot dialogue comprehension model. we first construct a dialogue-narrative dataset named diana, which contains 243k (dialogue, narrative) pairs obtained by automatically aligning movie subtitles with their corresponding synopses. we then pre-train a dialogue comprehension model based on diana and evaluate its performance on four downstream tasks that require dialogue comprehension abilities. experiments show that our model outperforms strong pre-trained baselines, demonstrating that the learning-by-narrating strategy is a promising direction for dialogue comprehension. we also hope that diana will promote future research in related areas.