Representation Learning for Conversational Data using Discourse Mutual Information Maximization (2022)

Abstract: Although many pretrained models exist for text or images, there have been relatively fewer attempts to train representations specifically for dialog understanding. Prior works usually relied on finetuned representations based on generic text representation models like BERT or GPT-2. But such language modeling pretraining objectives do not take the structural information of conversational text into consideration. Although generative dialog models can learn structural features too, we argue that the structure-unaware word-by-word generation is not suitable for effective conversation modeling. We empirically demonstrate that such representations do not perform consistently across various dialog understanding tasks. Hence, we propose a structure-aware Mutual Information based loss-function DMI (Discourse Mutual Information) for training dialog-representation models, that additionally captures the inherent uncertainty in response prediction. Extensive evaluation on nine diverse dialog modeling tasks shows that our proposed DMI-based models outperform strong baselines by significant margins.

## introduction representation learning has transformed how we can apply machine learning to solve real-world problems. however, despite a vast body of research on pretrained language representations, there have been relatively fewer attempts to train representations specifically for dialog understanding. prior works mostly relied on finetuned representations based on generic models like bert @xcite or gpt-2 @xcite . in our experiments, we demonstrate that such representations do not perform uniformly across various dialog understanding tasks such as dialog-act classification, intent detection or dialog evaluation. on the other hand, prior works on pretraining large-scale dialog models focused mainly on opendomain generation. these works evaluated their models only on dialog generation @xcite @xcite or tasks related directly to the pretraining objective @xcite . their effectiveness on other dialog understanding tasks like act classification or intent detection remains unexplored. so we ask the following research question: can we learn enriched representations directly at the pretraining phase that are specifically helpful for dialog understanding? existing language modeling (causal or masked) pretraining objectives unfortunately are not the best to model dialogs for these reasons: (1) the model is not directly trained to learn the content discourse structure (e.g., context-response in dialogs). (2) such models are trained to generate the response word-by-word rather than predicting a larger unit. hence, in this paper, we propose pretraining objectives for improved dialog modeling that turn the discourse-level organizational structure of texts from natural sources (e.g., documents, dialogs, or monologues) into a learnable objective. we call this objective the discourse mutual information (dmi). the key insight towards the design of our pretraining objective is to capture representations that can account for a meaningful conversation out of a specific ordered sequences of utterances. we hope that a discourse-level pretraining objective with conversational data would guide the model to learn complex context-level features. for example, in fig. 1 , we illustrate the differences between standard language modeling (causal or masked) based pretraining objectives and a discourse-level reasoning task. the second research question that we ask is whether discourse-level features learned using selfsupervised pretraining outperform word-level prein a discourse-level reasoning task, the immediately preceding utterance may not be enough for understanding the full context. to predict the correct response, the model will need to capture both the larger context, in this case the topic of discussion, and the intent (e.g., asking for details) of the preceding utterance. in comparison, word-level reasoning is often easier and can be solved using local reasoning. each of the three masked-words, in the left image, could have been predicted with reasonable confidence without any more information than the utterance itself. overall, our main contributions are as follows. ## discourse mutual information we define discourse mutual information (dmi) as the mutual information foot_0 between two random variables representing two different segments within the same discourse. this is a general concept that can be applied to any form of discourse, no matter the domain or type of signal. in this paper, we focus on dialog type discourses and representation learning for conversational texts. we define two random variables for the contexts (c) and responses (r) that jointly construct a valid conversation. conversations between humans represent samples from the joint distribution p cr of c and r. we pose the following learning problem, "learn continuous representations for the textual random variables c and r such that the true mutual information between c and r can be closely estimated." in the remainder of this section we show that, if the lower bound on mi estimated by some representations of context and response is close to the true value, the representation of the context would be as predictive of the response as the natural language form itself. existing generative training objectives as used in dialogpt or blenderbot are extremely focused on predicting target response only. per-word cross-entropy loss, used for training these models, fails to take into account the inherent uncertainty in the context-to-response generation function. adapting context representations so as to predict the target responses optimally, helps our proposed dmi-based models learn better dialog representations applicable to a versatile set of dialog understanding tasks. objective function formulation let e c and e r be the representations foot_1 for c and r based on some encoder. using the data processing inequality from information theory (cover, 1999), we have this tells us that mi between any encoded version of c and r will always be less or equal than the true mutual information. the equality will hold if e c and e r are both fully-invertible encoding processes (as opposed to representations which are lossy or compressive, and inversion is thus not possible). however, neural networks generally embed the data points in a low dimensional manifold by learning robust features that can represent the data points efficiently. because of this, neural representations are usually not invertible. 4 now, the exact computation of mi is not possible for continuousvalued random variables. in recent years, various variational lower bounds have been proposed for estimating mi between continuous-valued random variables. including the mi estimator ( îθ ), the overall relation becomes this leads us to the proposed learning objective dmi: ## dmi vs. language modeling objectives in this work, we focus on utilizing dmi for pretraining dialog representations incorporating strong discourse-level features. but why should the dmi objective learn better discourse-level features than models trained on conversational data using mlm or lm objectives? we can find the answer by looking at various lm-based objectives through the lens of infomax, as shown by @xcite . they connected various pretraining objectives for natural language representations, including the ones used for training skipgram, bert and xlnet, to the infomax learning principle. if we consider an input text t and a masking function m that returns a masked text t and the masked word w, the mlm objective is equivalent to compared to these lm objectives, dmi focuses on optimizing i(e c , e r ), where c and r are two structural components from the discourse with des-ignated roles. this enables dmi to discover more important features at the discourse level. ## conclusions and future work in this paper, we proposed the concept of discourse mutual information (dmi) which is better suited for learning dialog-specific features in a self-supervised manner. using the infomax principle we formulated a pretraining method for dialog-specific representation learning. across 9 downstream dialog understanding tasks, our 12layer model outperforms state-of-the-art methods. further, we showed that on most of these tasks, even our 8-layer model outperforms standard 12layer pretrained models. these experiments show the potential of the proposed dmi objective towards building dialog understanding models. we will make the code and pretrained model checkpoints available on request, instructions can be found here https://bsantraigi.github. io/dmi . although we experimented only with dialog modeling in this paper, we believe that the proposed dmi objective is generic enough to be applied to any type of discourse in any domain. in the future, we would like to explore how to harness dmi representations for generative conversation modeling.