Mix-and-Match: Scalable Dialog Response Retrieval using G aussian Mixture Embeddings (2022)

Abstract: Embedding-based approaches for dialog response retrieval embed the context-response pairs as points in the embedding space. These approaches are scalable, but fail to account for the complex, many-to-many relationships that exist between context-response pairs. On the other end of the spectrum, there are approaches that feed the context-response pairs jointly through multiple layers of neural networks. These approaches can model the complex relationships between context-response pairs, but fail to scale when the set of responses is moderately large (>1000). In this paper, we propose a scalable model that can learn complex relationships between context-response pairs. Specifically, the model maps the contexts as well as responses to probability distributions over the embedding space. We train the models by optimizing the Kullback-Leibler divergence between the distributions induced by context-response pairs in the training data. We show that the resultant model achieves better performance as compared to other embedding-based approaches on publicly available conversation data.

## introduction retrieval-based response predictors @xcite @xcite @xcite @xcite @xcite retrieve the response from a predefined set of responses given the dialog context. such methods find application in a variety of real-world dialog modeling and collaborative human-agent tasks. for instance, dialog modeling frameworks typically utilize the notion of "intents" and "dialog flows" which aim to model the "goal" of a user-utterance @xcite . to make task of building and identifying such intents easier, some tools mine conversation logs to identify responses that are often associated with dialog contexts (intents) figure 1 : an example of a context with multiple valid responses. note that each response contains different information and hence must have embeddings that are far way from each other. however, embedding-based approaches for retrieval attempt to bring all such responses close to the context and hence, close to each other. @xcite and then surface these responses for review by humans. these reviewed responses are then modeled into the dialog flow for different intents. another instance, of humanagent collaboration powered by system returned responses is in 'agent assist' environments where a system makes recommendations to a customersupport or contact-center agent in real-time @xcite . the success of a good response retrieval system lies in learning a good similarity function between the context and the response. in addition, it also needs to be scalable so that it can retrieve responses from the universe of responses efficiently. these two requirements present a trade-off between the richness of scoring and scalability, as discussed below. trade-off between scoring and scalability: typically, in neural dialog retrieval models, the contexts and the responses in the conversation logs are embedded as points in the embedding space @xcite . approaches such as contrastive learning @xcite are then used to ensure that the context is closer to the ground-truth response than the other responses. figure 1 shows a dialog context followed by multiple responses. despite the apparent diversity among responses, all the responses are valid for the dialog context. similarly, a generic response may be a valid response for several dialog contexts. a typical embeddingbased approach for retrieval would bring the embedding of the dialog context close to the embedding of all the valid responses @xcite @xcite . however this has the undesirable effect of making the valid, but diverse, responses gravitate towards each other in the embedding space. thus, typical embedding-based approaches for retrieval fail to capture the complex, many-to-many relationships that exist in conversations. more complex matching networks such as sequential matching networks @xcite and bert @xcite based cross-encoders jointly feed the context-response pairs through multiple layers of neural networks for generating the similarity score. while these approaches have proven to be effective for response retrieval, they are very expensive in terms of inference time. specifically, if n c is the total number of dialog contexts and n r is the total number of responses available for retrieval during inference, these methods have a time complexity of o(n c n r ). hence, they can't be used in a real-world setting for retrieving from thousands of responses. ## mix-and-match we consider a dialog to be a sequence of utterances (u 1 , . . . , u n ). at any time-step t, the set of utterances prior to that time-step is referred to as the context. the utterance that immediately follows the context foot_1 is referred to as the response. instead of modeling the context and response as point embeddings, we use probability distributions induced by the context and the response on the embedding space, denoted as p c (z) and p r (z) foot_2 respectively, where z is any point in the embedding space r d . ## conclusion by modelling contexts and responses as multimodal distributions, we allow the network to be more expressive without forcing the representations of unrelated responses to move closer, as would have been the case with traditional dual-encoder learning objectives. we derived and presented a closed form expressions for efficiently computing the kl-divergence based distance measures and showed its suitability for real-world settings. we demonstrated the effectiveness of our retrieval systems on three different datasets -ubuntu, twitter and an internal, real-world tech support dataset. additional experiments for response relevance, including a human study were performed on the publicly available datasets. we found that not only is our model able to retrieve more relevant responses as compared to recent retrieval systems, it also presented more diverse results.