Generating Meaningful Topic Descriptions with Sentence Embeddings and LDA (2022)

Abstract: A major part of business operations is interacting with customers. Traditionally this was done by human agents, face to face or over telephone calls within customer support centers. There is now a move towards automation in this field using chatbots and virtual assistants, as well as an increased focus on analyzing recorded conversations to gather insights. Determining the different services that a human agent provides and estimating the incurred call handling costs per service are key to prioritizing service automation. We propose a new technique, ELDA (Embedding based LDA), based on a combination of LDA topic modeling and sentence embeddings, that can take a dataset of customer-agent dialogs and extract key utterances instead of key words. The aim is to provide more meaningful and contextual topic descriptions required for interpreting and labeling the topics, reducing the need for manually reviewing dialog transcripts.

## introduction topic models are statistical tools for discovering the hidden semantic structure in a collection of documents/dialogs. one such widely used topic model is latent dirichlet allocation (lda, @xcite . lda is a hierarchical probabilistic model that represents each topic as a distribution over terms/words and represents each document/dialog as a mixture of the topics. one of the main issues with the standard lda bag-of-words approach is that the discovered topics can be difficult to interpret, as the user is presented with only the key words per topic. due to this, the user often needs to go through the documents/dialogs for each topic to gather more context. the elda (embedding based lda) approach attempts to produce more interpretable topics by running the topic modeling at an utterance level. the resulting topics can be represented by the most relevant utterances per topic, giving more context to the analyst so they can better understand the topic, with little to no manual inspection of the dialogs. another issue with bag-of-word approaches is that they fail to capture co-reference resolution, homonymy, and polysemy. for example, the words "leave" and "depart" mean the same thing in similar contexts but will be treated as having different meanings. conversely, one word, for example "right", can mean different things given the context but will be treated as having the same meaning. representing text as embeddings can overcome these issues to some extent. for example, word and sentence encoders such as (google's) multilingual universal sentence encoder @xcite , sentence-bert (sbert, @xcite , etc. can capture the meaning of sentences and words in context with no need for any text pre-processing (e.g. stop word removal, part-of-speech tagging, lemmatization etc.). a further challenge in running lda is that it requires to specify in advance the number of topics to generate, which can be hard to determine in cases where the domain or data is not known in detail. the elda approach includes a novel technique to automatically estimate the number of topics to generate for a given dataset. we compared the topic descriptions of the elda approach with that of standard lda on the multi-woz dataset @xcite . our work specifically targets topic discovery in customer call conversations rather than general documents, such as news articles or publications, as in most of the related work. we have also created novel techniques in: (i) automatically deciding on the number of topics to produce and (ii) to measure the interpretability and accuracy of the produced topics. ## method given a collection of dialogs segmented into utterances, either by a speech-to-text system that includes diarization or based on metadata provided by a text-messaging system (see ## results in this section we first report the overlap scores of the baseline lda model and the different elda models, using the oracle topic labeling for all the models. then we report the results of the comparison between the oracle labels and manual labels (from the annotators) for both the baseline lda and optimal elda (best grid-search model). next, we show a comparison of the overlap scores resulting from manual labeling obtained from the baseline lda with the same obtained from the optimal elda. we also compare the true intent distribution of multiwoz with that produced by the manual labels of the baseline lda and optimal elda. lastly, we exhibit the topic descriptions of the three biggest topics from the baseline lda and optimal elda and compare their individual level of interpretability. ## conclusions and future work in this work we developed elda, an embeddingbased lda method, that represents each document or dialog in a dataset as a bag of utterance clusters instead of a bag of words. as a result, this approach represents each lda topic as a probability distribution over utterance clusters which are labeled by the utterances closest to the cluster centroids. unlike key words, the key utterances (cluster labels) provide more context to each topic, which helps to better interpret and label the topics. the elda and lda approaches were evaluated and compared using the multiwoz dataset. the results indicate elda is on par with the standard lda in accurately identifying the existing topics or dialog intents, while producing easier-to-interpret topic descriptions that facilitate and accelerate the task of manually labeling the resulting topics. the optimal elda hyperparameter values presented here may be extrapolable to other datasets, given that the semantic similarity distance magnitudes are given by the sentence embedding and not by the dataset. we continue testing elda with other (proprietary) datasets to verify this hypothesis. one proposal for improving this work is to use a more stringent overlap metric in order to force the hyperparameter fine-tuning process to converge to better values. note that the current approach considers a match between the predicted and true intents if any one of the intents match. hence better hyperparameter values than the ones selected in this paper may be yet found.