We’ve had this conversation before: A Novel Approach to Measuring Dialog Similarity (2021)

Abstract: Dialog is a core building block of human natural language interactions. It contains multi-party utterances used to convey information from one party to another in a dynamic and evolving manner. The ability to compare dialogs is beneficial in many real world use cases, such as conversation analytics for contact center calls and virtual agent design. We propose a novel adaptation of the edit distance metric to the scenario of dialog similarity. Our approach takes into account various conversation aspects such as utterance semantics, conversation flow, and the participants. We evaluate this new approach and compare it to existing document similarity measures on two publicly available datasets. The results demonstrate that our method outperforms the other approaches in capturing dialog flow, and is better aligned with the human perception of conversation similarity.

## introduction measuring semantic textual similarity lies at the heart of many natural language and text processing tasks, such as sentence classification, information retrieval, and question answering. traditional text representation approaches, such as high dimensional and sparse feature vectors, have been boosted by the introduction of efficiently learned embeddings @xcite @xcite , unleashing the full power of the dense semantic representation of words. subsequently, new methods were developed for contextual representation of words, sentences, paragraphs, and documents, facilitating the assessment of semantic similarity between larger portions of text @xcite @xcite . conversations 1 differ from documents that compound multiple sentences or paragraphs in a 1 although not strictly equivalent, we use the terms conversation and dialog interchangeably hereafter. related work in this field is relatively sparse. @xcite derive two different similarity functions between conversations, one for content similarity based on tf-idf, and the other for the conversation structure, using engineered features related to the dialog flow. the scores, computed independently for the two dimensions, are further combined to infer the overall conversation similarity ranking. @xcite learn a distance function between utterances and conversations based on expert judgments and later use this function to cluster conversations. their approach is specifically tailored for conversations with an automatic dialog system (i.e., bot), limiting its applicability to the much wider conversational domain. in this work we draw on the concept of edit distance, the family of metrics and algorithms @xcite widely used for sequence analysis. this analysis is done mainly at the character level for strings or at the nucleotide or amino-acid level in computational biology @xcite . the edit distance has also been applied to sequences of sentences to detect the differences between documents (zhang and litman, 2014; @xcite . we propose combining the power of edit distance in assessing sequence similarity with the power of distributional semantics to form a novel similarity measure between conversations. this new measure -conved -takes into account the utterance semantics, as well as the dialog flow and its unique traits. we suggest and evaluate a framework for seamless, non-intrusive, and elegant adaptation of the widely-used edit distance metric to the scenario of conversation similarity. 2 the suggested approach can be practically leveraged for downstream applications, such as those in the domain of dialog pattern mining. ## model in this section, we present a brief reminder of what the edit distance metric involves (section 2.1), followed by the unique adaptations designed for the scenario of conversation similarity (section 2.2). ## evaluation and results we evaluated the effectiveness of our model through two distinct approaches: intrinsic evaluation, assessing the ability of the model to capture dialog flow (section 3.3), and external human evaluation via crowd-sourced annotations (section 3.4). we compared our model to two competitive baselines used for estimating text similarity. ## conclusions and future work we presented a novel approach for measuring dialog similarity, capturing both conversation semantics and its structural properties. our evaluation shows that this measure outperforms other baselines with respect to both intrinsic evaluation and agreement with human judgements. the framework is easily adaptable to different settings by manipulating the cost functions. for example, addressing multi-party chat by assigning lower distance to utterances coming from parties that share the same role, or reducing the cost for general chit chat utterances, thus focusing on semantic similarity of the conversations' essence. our future work includes enhancements of the proposed measure with additional dialog-related traits, as well as its application to downstream tasks and adapting the framework to multiple conversation alignment.