T ask D iff: A Similarity Metric for Task-Oriented Conversations (2023)

Abstract: The popularity of conversational digital assistants has resulted in the availability of large amounts of conversational data which can be utilized for improved user experience and personalized response generation. Building these assistants using popular large language models like ChatGPT also require additional emphasis on prompt engineering and evaluation methods. Textual similarity metrics are a key ingredient for such analysis and evaluations. While many similarity metrics have been proposed in the literature, they have not proven effective for task-oriented conversations as they do not take advantage of unique conversational features. To address this gap, we present TaskDiff, a novel conversational similarity metric that utilizes different dialogue components (utterances, intents, and slots) and their distributions to compute similarity. Extensive experimental evaluation of TaskDiff on a benchmark dataset demonstrates its superior performance and improved robustness over other related approaches.

## introduction task-oriented conversational assistants have become increasingly popular in multiple industries enabling users to perform tasks such as travel reservations, banking transactions, online shopping, etc., through multi-turn conversations. the increased use of these assistants has led to the availability of valuable user-assistant conversation logs @xcite , prompting efforts to extract insights from them. a key aspect of such conversational analytics is identifying similarities and dissimilarities between conversations. this will enable developers to improve the user-experience including personalized response generation, next-action recommendations, and information retrieval @xcite @xcite . the popularity of large language models like chatgpt and llama 2 @xcite has resulted in a race to create custom task-oriented conversational assistants in enterprise domains like finance and retail @xcite . however, evaluating these assistants has become an important challenge and requires effective metrics that can measure their performance across similar user-assistant conversations. measuring semantic textual similarity has been extensively studied for textual sources like documents, social media, transcripts, etc. however, there has been limited prior work studying similarity in task-oriented conversation settings @xcite . most approaches leverage popular word embeddings like word2vec @xcite , glove @xcite or pre-trained models like universal sentence encoder @xcite , sentence-bert @xcite to obtain vector representations of utterances, and then use distancebased approaches such as cosine and edit-distance to compute the similarity between text snippets. while such approaches can identify semantic relationships between texts, task-oriented conversations present several challenges that limit their effectiveness. firstly, they consist of distinct components -intents, slots, and utterances -that impact the similarity and overlap between conversations. for instance, users can have different objectives (e.g., booking travel vs. product returns), or even have the same intents but provide different levels of slot information @xcite . additionally, information is typically provided over multiple conversation turns, and each turn could involve multiple user intents and slots. finally, the same set of tasks can be expressed using numerous possible utterances by users, depending on their choice of phrasing, order of sentences, use of colloquialisms, introducing digressions, etc. @xcite . hence, relying solely on distance based similarity of utterance embeddings would adversely impact performance. in this work, we present taskdiff, a novel similarity metric designed for task-oriented conversations to address the above challenges. figure 1 shows multiple users having similar conversations about making bookings for a trip but with re-ordered tasks or paraphrased utterances with different slot values. it also shows that prior work is not robust to such differences that commonly occur in task-oriented conversations. an ideal metric to measure conversational similarity should be able to identify that the overall goal of these conversations in figure 1 is the same. taskdiff represents the structure of conversations as distributions over the different task-oriented components and combines the geometry of the distributions with optimal transport to measure the similarity between conversations. our approach is inspired by prior work in topic modelling @xcite that have shown the effectiveness of comparing the structure of distributions, albeit for different settings. we evaluate taskdiff on a benchmark task-oriented conversation dataset and demonstrate its effectiveness while presenting examples illustrating its improvement over existing approaches. ## conclusion in this paper we present taskdiff, a novel metric to measure the similarity between task-oriented conversations. it not only captures semantic similarity between the utterances but also utilizes dialog specific features like intents and slots to identify the overall objective of the conversations. we demonstrate that unlike existing metrics, taking advantage of these unique components is critical and results in significantly improved performance. as part of future work, we will investigate the inclusion of additional dialog features on open domain dialog datasets and the utilization of taskdiff to improve the performance of various downstream conversational tasks.