Improving Hierarchical Text Clustering with LLM -guided Multi-view Cluster Representation (2024)

Abstract: In this work, we present an approach that introduces different perspectives or views to improve the quality of hierarchical clustering of interaction drivers in a contact center. Specifically, we present a multi-stage approach that introduces LLM-guided multi-view cluster representation that significantly improves the quality of generated clusters. Our approach improves average Silhouette Score by upto 70% and Human Preference Scores by 36.7% for top-level clusters compared to standard agglomerative clustering for the given business use-case. We also present how the proposed approach can be adapted to cater to a standard non-hierarchical clustering use-cases where it achieves state-of-the-art performance on public datasets based on NMI and ACC scores, with minimal number of LLM queries compared to the current state-of-the-art approaches. Moreover, we apply our technique to generate two new labeled datasets for hierarchical clustering. We open-source these labeled datasets, validated and corrected by domain experts, for the benefit of the research community.

## introduction contact centers record interactions between their agents and customers and store them in the form of text transcripts for multiple downstream use cases like quality assurance, business analytics and insights. the primary reason for an interaction, often referred to as an interaction driver is an essential data point for some of these downstream use cases. identifying these drivers at an interaction level can be automated with multiple techniques ranging from simple classification based on key phrases @xcite ) to large language model (llm) based generation in recent times @xcite ). however, with contact centers handling hundreds of thousands of interactions a day, each with a unique driver, making sense † equal contribution as third authors. contact center interaction drivers are often thought of as having a two-level hierarchical structure consisting of a few main categories with several sub-categories under each main category. these are often referred to as level-1 (l1) and level-2 (l2) categories respectively. this makes it conducive to applying hierarchical clustering techniques to organize them into l1 & l2 clusters. however, while contact center business use cases call for the best quality especially at the top-level for l1 clusters, current state-of-the-art clustering techniques fall short of this. specifically, l1 categories surfaced by existing methodologies often fail to bring out the right abstraction and multifaceted similarities within the l2 categories. meanwhile, we notice that capturing this abstraction comes naturally to humans, and in recent times the best of large language models (llms) (ai@meta, 2024) have demonstrated this capability as well. different businesses have unique perspectives on how they prefer to cluster their l1 and l2 drivers. in this context, we present a methodology that generates l1 clusters that are better and more aligned with human-preferences from given l2 clusters for contact center interaction drivers. specifically, we present a multi-stage clustering approach that introduces an llm guided multi-view representation of l2 clusters to improve quality of l1 clusters. our method employs standard agglomerative clustering to first derive the l2 clusters, and perspective 1 perspective 2 documents cluster name cluster description cluster name cluster description customer called in to cancel niagara falls tour niagara falls tour customers calling in to inquire, book, modify and cancel bookings for niagra falls tour cancellation drivers related to tour cancellations due to various reason customer wanted to add one more family member to the niagara falls boating experience niagara falls tour customers calling in to inquire, book, modify and cancel bookings for niagra falls tour modification customer calling in to modify a booking they made with the tours company we also present how this approach can be adapted for standard non-hierarchical clustering approaches as an alternative to current state-ofthe-art approaches @xcite ; raedt et al. ( 2023 )). benchmarking experiments on public intent-classification datasets which are used for evaluation of clustering techniques, specifically banking77 @xcite ) and clinc150 @xcite ) shows that our approach achieves close to state-of-the-art performance measured by normalized mutual information (nmi) @xcite , @xcite ) and clustering accuracy (acc) @xcite ) scores. our approach is also cost effective with number of llm queries limited to twice the number of l2 clusters, while llm queries needed by the above mentioned existing approaches increases linearly with number of documents in the dataset. moreover, we apply our approach to generate l1 clusters on top of existing base clusters for bank-ing77 and clinc150 datasets. the new datasets consist of 7 and 15 l1 clusters respectively, and we call them banking7 and clinc15. the l1 clusters generated are validated and corrected by domain experts, and we open-source these datasets as part of this work for the benefit of the research community. to summarize, below are our specific contributions in this work. ## methodology the proposed approach consists of four key stages as illustrated in figure 1 and detailed below. ## extending to non-hierarchical clustering text clustering research in recent times proposing llm-guided approaches have reported state-of-theart performance on labeled public datasets. while our methodology in this paper is primarily targeted towards hierarchical clustering, we posit that this approach can be adapted to improve quality of clusters generated for standard clustering use-cases, and can provide a more efficient alternative to the current state-of-the-art techniques. to adapt our approach to standard clustering use-cases, we assume that the final output clus-ters required are l1 clusters, and there exists a layer of hidden l2 clusters. we apply the proposed multi-view cluster representation to the hidden l2 clusters before they are clustered again to generate the l1 clusters. this approach brings in different perspectives through multi-view representation as a light-weight one-time intervention during a bottomup clustering process to improve quality of the final clusters generated. ## open-source labeled dataset for two-level hierarchical clustering we applied our proposed approach to banking77 and clinc150 datasets to generate a 7 and 15 l1 clusters respectively. the number of l1 clusters is determined by optimizing for silhouette scores. as this optimization is performed after generating names and descriptions using the llm, this step do not significantly impact our computational costs. the generated clusters were validated/ corrected through the following annotation process. annotators were given names and descriptions of the existing intent classes, derived using our proposed approach along with text samples from the intent class and the corresponding l1 cluster generated. they were asked to verify if the tagging of an intent class to an l1 cluster was correct and if not, to reassign the intent class to the correct l1 cluster. we open-source the labeled two-level hierarchical dataset thus created as an additional contribution to the community † . ## limitations and future work our research showcases the effectiveness of the proposed methodology in generating hierarchical clusters, but there are several key areas for future exploration and limitations to consider. first, we limited our experiments to agglomerative clustering. however, our methodology is clustering algorithm-independent, suggesting that future work could investigate various algorithms, such as k-means, dbscan, or spectral clustering, to enhance l1 and l2 cluster formations across diverse datasets. second, our current framework employs a specific external llm for generating cluster names and descriptions. future research could benchmark different llm architectures and sizes to determine which configurations yield the most meaningful cluster representations. lastly, determining the optimal number of l2 clusters remains a challenge in unsupervised clustering. future work could focus on developing efficient methods for this task, potentially employing advanced heuristics or hybrid approaches to improve robustness and applicability. in summary, while our study provides a strong foundation, there are ample opportunities to extend this research by exploring diverse clustering techniques, evaluating llm performance, and optimizing the clustering process ## conclusion in this paper, we present a multi-stage approach for the hierarchical clustering of interaction drivers in contact centres that achieves significantly better quality for the top-level clusters. we propose to leverage llm-guided multi-view intermediate cluster representations as part of the clustering process to obtain more coherent and meaningful top level clusters. our approach despite using out-ofthe-box embedding models and requiring minimal llm queries (twice #l2 clusters), achieves better silhouette scores for our internal datasets, and state-of-the-art nmi and acc scores on public datasets. we also release two labeled datasets for hierarchical clustering for the benefit of the research community.