Deep Embedding of Conversation Segments (2021)

Abstract: We introduce a novel conversation embedding by extending Bidirectional Encoder Representations from Transformers (BERT) framework. Specifically, information related to “turn” and “role” that are unique to conversations are augmented to the word tokens and the next sentence prediction task predicts a segment of a conversation possibly spanning across multiple roles and turns. It is observed that the addition of role and turn substantially increases the next sentence prediction accuracy. Conversation embeddings obtained in this fashion are applied to (a) conversation clustering, (b) conversation classification and (c) as a context for automated conversation generation on new datasets (unseen by the pre-training model). We found that clustering accuracy is greatly improved if embeddings are used as features as opposed to conventional tf-idf based features that do not take role or turn information into account. On classification task, a fine-tuned model on conversation embedding achieves accuracy comparable to an optimized linear SVM model on tf-idf based features. Finally, we present a way of capturing variable length context in sequence-to-sequence models by utilizing this conversation embedding and show that BLEU score improves over a vanilla sequence to sequence model without context.

## introduction embedding of natural language units (word, sentence or paragraph) deals with the problem of finding a vector space representation of these units that can be used in downstream applications of classification, summarization or token identification. for example word embeddings @xcite have found application in information retrieval @xcite , document classification @xcite , question answering @xcite , named entity recognition @xcite and parsing @xcite . extending the same concept to sentences and documents one can also find the corresponding vector representations independently @xcite or by suitably averaging the word vectors @xcite . while aforementioned embeddings are created without optimizing for (or even considering) downstream applications there are recent approaches that seek optimal representations based on pre-training @xcite @xcite . these applications can be at sentence-level such as natural language inference @xcite and paraphrasing @xcite where the semantic relationship between sentences are captured or at word-level tasks @xcite . there are two different approaches for applying pre-trained embeddings, (a) feature-based ( @xcite , where the model architecture is task-specific and pre-training is a feature of the architecture) and (b) fine-tuning @xcite (where the pre-training architecture is quite generic to handle a variety of downstream tasks and model parameters are later fine-tuned for specific tasks). while most of the pre-training architectures use unidirectional language models @xcite , bidirectional encoder representations from transformers (bert, @xcite ) uses a different strategy to learn sentence/paragraph representations and achieves best scores on a variety of tasks. even though there are different strategies for creating word, sentence and document level embeddings, there is no study available in literature that deals with conversation embedding. while a piece of conversation may look very similar to a paragraph (and one can probably start with paragraph embedding to embed conversations) it has two important additional pieces of information, namely, turns and roles. a turn can consist of a single word, sentence or multiple sentences (all belong to a single role) and a conversation can have many participants (roles) where it is crucial to distinguish who is saying what. an efficient representation of a complete conversation (or part of it) should take into account the role and turn information (and their congruence) for downstream applications. the most important application of conversation embedding is in the area of automated dialogue generation. starting with the vanilla sequence-tosequence model @xcite there are different approaches to capture the "context" so that meaningful responses can be generated @xcite . the "context" continuously grows as the conversation progresses and can be defined in terms of everything that has happened in the conversation so far or key words from earlier turns extracted by some attention based algorithms @xcite . there could be different approaches to capture a context, e.g., (a) separate rnns for previous turns and roles, (b) attention over previous turns or (c) a global vector representing counts of tokens from previous turns etc. however, all of them have limitations either in capturing all the required information or in their ability to deal with a continuously increasing context length. an embedding that can map a variable length context (i.e., a conversation segment) into a numeric vector while including key pieces of information required for generating the next response would be immensely helpful in automatic dialogue generation. this is what has been attempted in this work where we create conversation embedding using bert and apply to various downstream tasks. our contributions are 1. extension of bert based sentence representation to conversation representation by adding the notion of roles and turns and thus creating an embedding of conversation segments hitherto unavailable in literature. ## related work very little work is available in the literature on conversation embedding, especially that treats conversations with all its associated complexities. most of the work has been on word embedding (nonneural, @xcite @xcite and neural @xcite ), sentence embedding (le and @xcite and embedding of paragraphs @xcite . recent approaches involving pretraining and fine-tuning also deal with sentences and sentence pairs @xcite @xcite and the downstream tasks are mostly classifications (and not sequence generation). on conversation embedding, the closest that we see is from @xcite where multiple pretraining objectives are explored. conversations are encoded using recurrent neural network (rnn) and no information from roles or turns are included. the importance of capturing "context" for relevant response generation is well understood. @xcite tried capturing the context initially using bag-of-words representation @xcite and later by a hierarchical recurrent encoder-decoder (hred) approach @xcite applied to the movie dataset @xcite with only one previous utterance appearing as the context. here, a dialogue d consisting of a series of utterances {u 1 , u 2 , . . . , u m } was decomposed as in the next section we describe our approach of creating conversation embedding from bert. subsequently, we demonstrate three applications of this embedding, namely, clustering, classifications and dialogue generation. finally, conclusions are drawn. the corresponding text, role and turn tokens for input to a bert model will be as shown in fig. 1 . while the role in the present case takes only two values, i .e., agent and customer (although there is no restriction) the turn can be as high as 200 in our conversation data. thus, the turn embedding can be similar to the position embedding used in transformer, i .e., sines and cosines. however, in the present work we have projected both the turn and role values to a fixed-dimensional vector and learnt the corresponding embeddings @xcite . these embeddings are added together along with the position and segmentation embeddings defined in the original bert paper. the pre-training steps of bert are partly based on tasks defined earlier, namely, masked language model (mlm) and next sentence prediction (nsp). however, in case of mlm the masked word can be from different roles and turns. similarly, the sentence pairs in nsp can span across multiple roles and turns. thus, both mlm and nsp will drive better understanding of the conversation structure. ## experiments the data for all the experiments presented here are from conversations between customer service agents and existing/new customers who contact the customer service for their order related issues. the conversations between customer and agents are divided into sessions which we merged together to generate a single conversation. for bert model pre-training we have randomly selected 100,000 chats of various different topics. these conversations are of varying number of turns and tokens (as can be seen in ## conclusion we have introduced an embedding (representation) of a conversation (or conversation segment) by augmenting role and turn information to word tokens and utilizing bert for pre-training. this pre-trained model can be used either to generate features from new conversations or can be finetuned further on specific tasks. in this work we have explored both the options. pre-trained model based conversation features are used for (a) conversation clustering and (b) for representing contexts in a conversation for predicting the next response. in case of clustering we show that embedding based features result in higher accuracy when compared to tf-idf based features. similarly, for conversation modeling embedding feature based context representation drove higher bleu score when compared to a vanilla seq2seq model without any contextual information. we also fine-tune a pre-trained model for conversation classification on new dataset and obtain accuracy similar to what is given my a linear svm model trained on tf-idf based features. with these examples we show the general applicability of the current approach on modeling various tasks involving conversation data.