Sentiment-Aware Dialogue Flow Discovery for Interpreting Communication Trends (2024)

Abstract: Customer-support services increasingly rely on automation, whether fully or with human intervention. Despite optimising resources, this may result in mechanical protocols and lack of human interaction, thus reducing customer loyalty. Our goal is to enhance interpretability and provide guidance in communication through novel tools for easier analysis of message trends and sentiment variations. Monitoring these contributes to more informed decision-making, enabling proactive mitigation of potential issues, such as protocol deviations or customer dissatisfaction. We propose a generic approach for dialogue flow discovery that leverages clustering techniques to identify dialogue states, represented by related utterances. State transitions are further analyzed to detect prevailing sentiments. Hence, we discover sentiment-aware dialogue flows that offer an interpretability layer to artificial agents, even those based on black-boxes, ultimately increasing trustworthiness. Experimental results demonstrate the effectiveness of our approach across different dialogue datasets, covering both human-human and human-machine exchanges, applicable in task-oriented contexts but also to social media, highlighting its potential impact across various customer-support settings.

## introduction dialogue systems are increasingly pervasive, playing a crucial role in communication with customers in many companies. monitoring and visualizing conversations produced by such systems offers a deeper comprehension of dialogue interactions, unveiling communication patterns, and providing valuable insights into the user experience. it is thus essential to ensure high-quality service. here, the analysis of frequent dialogue flows plays an important role, as they will depict the organic evolution of interactions, enhancing human interpretability. obtaining dialogue flows from black-box systems, such as chatbots based on large language models (llms) or other encoder-decoder frameworks, can be challenging due to their generative and open-domain nature. nonetheless, the ability to represent the conversation progression and consider emotional aspects such as the sentiment of the speakers is valuable, especially in activities requiring real-time assistance from responsible agents. we propose an approach for automatic dialogue flow discovery from a history of written dialogues, and their representation in a transition graph. we begin by grouping similar utterances into clusters, which may be seen as dialogue states. then we represent possible paths with their respective probabilities from the beginning to the end of the dialogue. furthermore, we enrich the states with the average sentiment of the included utterances. this has applications in a wide range of services and products involving dialogue or customer support, including call centers, emergency services, and virtual assistantsit also serves as an assessment tool, offering stakeholders a way to compare dialogue systems based on how they handle client requests while maintaining or improving their sentiment. moreover, this approach can potentially identify topics that often result in negative sentiment. the main contributions of this work are summarized as: the remainder of the paper is structured as follows: section 2 reviews work related to dialogue flow discovery and sentiment analysis; section 3 describes the proposed approach for sentimentaware dialogue flow discovery; section 4 clarifies the meaning of each element in the sentimentaware dialogue flows, and describes the experimental setup, the used datasets, and the flow metrics proposed; section 5 presents and analyses the resulting flows; finally, section 6 concludes the paper and provides cues for future work. ## related work the categorization of utterances in dialogue systems may help in understanding user intentions and facilitating effective interactions @xcite . generally, utterances are classified according to user intentions @xcite or dialogue acts @xcite , both providing valuable insights for task-oriented systems. however, the automatic classification of utterances is typically supervised and thus relies on annotated data, which is not always available. on the other hand, encoder-decoder systems, including those based on llms (e.g., chatgpt foot_0 ), do not rely on such classifications, but their flexibility comes at the cost of higher data demand and less control. traditional task-oriented dialogue systems are sustained by the design of flows to guide conversations towards specific goals. this entails defining specific user intentions and training phrases, and can be facilitated by tools like google's di-alogflow foot_1 , microsoft luis foot_2 , or rasa foot_3 . automating this process involves grouping semantically-similar utterances and representing them in a vector space, towards efficient intent discovery @xcite @xcite . representing dialogue flows as transition graphs offers insights on topics and other trends @xcite ). an earlier approach @xcite for flow discovery uses hidden markov models on twitter conversations. it introduces features like clustering similar utterances, vertices for marking the beginning and end of dialogues, as well as a threshold for ignoring low-probability transitions. towards interpretability, clusters were labelled manually. @xcite developed a similar approach with automatic labelling. by analysing communication trends, flow discovery may assist in the design of dialogue systems. this is the main goal of graph2bots @xcite , which adopts co-clustering for discovering dialogue states and transitions in human-human conversations. an alternative approach (sastre martinez and nugent, 2022) clusters utterances with dbscan and relies on finite-state automata for discovering ranked flows, based on the frequency of question-response sequences. sentiment analysis (sa) @xcite aims to extract sentiments from texts. in dialogues, it may help in identifying situations of sentiment degradation, which may then be acted upon, e.g., through a fallback system that replaces an artificial agent by a human; or by collecting information for later retraining the human or artificial agent. sa has been combined with other tasks, such as dialogue act recognition, which reinforce one another. for instance, detecting agreement often corresponds with the expression of the same sentiment, while transitions from negative to neutral tend to coincide with changing to a statement. works that tackled these tasks jointly @xcite @xcite benefited from it, and achieved high or state-of-the-art (sota) performances in datasets like mastodon @xcite . moreover, @xcite outperformed several sota methods for user satisfaction estimation in task-oriented dialogue systems by exploiting sa in a multi-task adversarial strategy. the seemingly symbiotic relationship between sa and other tasks motivated its application to dialogue flow discovery. yet, to the best of our knowledge, no other work has combined these tasks. ## proposed approach we propose a generic approach for automatically discovering the most common flows in a history of dialogues, while simultaneously associating sen-timent with their transitions. it comprises three distinct steps, outlined in figure 1: 1. utterance clustering clusters semantically similar utterances, represented by their embedding. discovered clusters may be seen as approximations to dialogue states. identifying communication trends i.e., the discovery of flows from any type of dialogue promotes the identification of common and/or undesired topics or transitions, which can be used to improve the agent, e.g., by changing intents, reviewing protocols, or adjusting human resources; interpreting black-box dialogue systems i.e., the discovery of flows in human-machine dialogues adds an interpretability layer that increases understanding of the agent and promotes the identification of issues. potential strategies for addressing such issues may include retraining the agent or implementing additional rules; planning and developing dialogue systems i.e., the analysis of human-human dialogues towards the identification of potential dialogue states and representative words or sentences, valuable to the agent's development process. in any scenario, the dialogue collection should be as comprehensive as possible and, ideally, cover all relevant intents. the set of applications attests to the versatility of the approach. still, in this paper, we focus on the interaction between a customer and an agent, where the ability to understand and efficiently manage interactions is essential for improving the quality of service and, consequently, customer satisfaction. ## experimentation in order to confirm the applicability of the sentiment-aware dialogue flows, extensive experimentation was conducted. this involved the implementation of each step of the proposed approach, introduced in figure 1 , with adequate tools, as well as the application to a range of dialogue datasets. this section details the implementation of the underlying processes but, before delving into the previous steps, we provide some clarifications on the visual notation used throughout the paper, aided by the illustrative diagram in figure 2 . the diagram (g) showcases the ideal scenario, in which an agent successfully manages to switch the customer's sentiment from negative, at the start of the dialogue (sod), to positive, by the end of the dialogue (eod). sod and eod are represented by specific nodes, which can be seen as states, represented as yellow boxes. the others correspond to the discovered clusters (c) and are represented by ellipses labelled with representative words in their utterances. states representing clusters by different speakers are also differentiated. in g, user clusters are coloured in light blue and agent clusters in dark blue. the diagram is complemented with examples of clustered utterances, on the right-hand side. edges represent transitions (t ) between clusters and have an associated weight, corresponding to their probability. for instance, g shows a 58% probability of moving from @xmath0. the sum of all probabilities of t originating from the same cluster is 1. nevertheless, in order to simplify the flow, a threshold can be applied for ignoring low-probability transitions, as carried out in this example. the colour of each transition represents the average sentiment within the destination cluster. red corresponds to a more negative sentiment, green to a positive, and yellow to a neutral sentiment. for eod, however, transitions represent the sentiment of the origin cluster, thus making the sentiment in the final interactions clearer and contributing to more immediate conclusions. ## results and discussion flows were discovered from every considered dataset and the designed metrics were computed as well. together, they provide insights into interpretability, communication trends, and limitations of the agents, among others. this section discusses some of the discovered flows and reports on the metrics computed for all. due to lack of space, we do not present the flows for all datasets, but include them in the appendix a. the dialogue flow for retailpt data is presented in figure 3 foot_6 . various interactions between the user and the (artificial) agent can be observed. we immediately note that the first interaction of the agent (sod's outgoing transition), is always the same, with probability 1.0. the label of the initial cluster suggests an offer of assistance, which is confirmed by the data: in fact, all dialogues start with the how can i help? utterance. sod sys -> can help how 1.0 user -> webform thanks order 0.19 user -> expected arrival books 0.4 user -> order shipment delayed 0.36 0.13 sys -> rephrase question ask 0.3 sys -> message please rephrase 0.13 sys -> vouchers should contact 0.11 sys -> can help questions 0.15 sys -> deadline delivery books 0.22 sys -> scholar books can 0.23 0.27 0.16 0.23 0.2 sys -> cancellation order voucher 0.24 0.17 0.12 eod 0.63 0.36 0.17 0.16 0.31 0.27 0.23 0.2 0.31 0.41 0.18 0.19 0.22 0.19 0.67 0.36 0.13 0.11 0.4 0.31 0.24 0.37 with the help of the labels, we see that this interaction is followed by a user message: thanking for the order; querying about the arrival of the books; or informing on a shipment delay. as the probabilities of each transition from the can help how state do not sum up to 1.0, there is at least one low-probability transition (i.e., @xmath1.1) not represented. afterwards, the agent replies and, in some cases, asks the user to rephrase the question. interactions continue until the eod, which marks the end of the conversation. in this case, non-negative sentiments (i.e., shades of green) predominate. additionally, there are edges with neutral colours leading to the can help questions state and red edges associated with negative sentiments in the rephrase question ask and message please rephrase states. we may deduce that the agent is failing to process the requests, which could potentially increase the user's frustration as some end the conversation afterwards. this negative sentiment could potentially be mitigated by retraining the agent to better handle the queries that lead to those clusters. it is important to note that the colour of the edges represents the average sentiment of utterances in each cluster, which may not fully capture the sentiment of the entire cluster. hence, we created an advanced analysis layer, shown in figure 4 , which considers the sentiment's standard deviation for each cluster via a three-layered cluster. in the can help questions state, the average sentiment is represented by the colour yellow (i.e., neutral). its left layer (red) represents the sentiment at its lower deviation value and its right layer (green) represents the sentiment at its highest deviation value. in this case, the average does not accurately represent the sentiment within that cluster as it also includes strong negative and positive values (i.e., deep shades of red and green). in states such as scholar book can or vouchers should contact, there is minimal sentiment deviation, as each layer of the node appears uniformly green, suggesting that the sentiment within the utterances of underlying clusters is accurately represented by their average. emowoz has eod -= 0, meaning no dialogue ends with negative sentiment. moreover, it has the highest sod ↛ eod, meaning that, with the applied threshold (0.1), most utterances are lost along the way. as this is the largest dataset (seven times larger than retailpt) it makes sense that it would be challenging to represent each utterance in it. sentiment variation is the lowest for this dataset. telecomsapt has the highest eod -, meaning it is the dataset that mostly finished with negative sentiment, followed by retailpt. this means that the involved (artificial) agents could benefit from an in-depth analysis, possibly culminating in reviewing and/or retraining. these are also the only datasets with a negative sentiment variation, i.e., by the end of the dialogue, sentiment gets lower. they also show high sod ↛ eod, as does mastodon, meaning these three datasets lose over half of their utterances throughout the flow. mastodon and tdsapt show intermediate values overall and the latter has the lowest sod ↛ eod, meaning that more than half the utterances are represented in the flow. both datasets have a positive sentiment variation, suggesting an improvement by the end of the conversation. in both cases, it is not easy to speculate more. mastodon has social media dialogues, where sentiment can flow, without clear negative consequences as in customer-support. moreover, tdsapt includes dialogues with a broad range of entities, and would benefit from a future analysis of the flows for each, independently. retailpt has the highest ais, however, afs suggests that sentiment gets worse by the end of the dialogues. it has also the highest fcc, meaning that the data is well-fitted to the clusters. mastodon has the highest fsc, meaning that, contrary to emowoz, sentiment diverges considerably within each cluster. however, ais and afs suggest that it increases by the end of the dialogue. it also presents the lowest fcc, meaning that data may not be well-fitted to the clusters, which aligns with the high divergence of sentiment within them. telecomsapt and tdsapt display intermediate results in flow cohesion and variation of sentiment within clusters. however, whereas the former's ais and afs suggest sentiment across the dialogues is predominantly positive, for the latter, ais and afs have the lowest values, suggesting a more neutral sentiment. for tdsapt, the difference between ais and afs is low, as is the ∆sentiment, but in different directions. the former value should be more accurate as it computes the variation by utterance instead of cluster. finally, two factors could impact the discovery of sentiment-aware dialogue flows and, thus, their analyses: (i) the sentiment classifier, and (ii) the flow discovery process, including the clustering and labelling methods. the low performance of any of these can cause a chain reaction, decreasing the quality of the final analysis. as such, performance at each level should always be considered. ## conclusion technological advances have increased reliance on artificial intelligence, including for customersupport services. while efficient cost-wise, customers can tell they are interacting with an artificial agent or a human following a mechanical protocol, and this degrades their interaction and deteriorates the customers' loyalty. our goal is to mitigate that by providing additional interpretability, also contributing to increased trustworthiness. we proposed a novel approach for automatically discovering the most common flows in a history of dialogues, while considering the sentiment. these are useful for various applications, from identifying communication trends to interpreting black-box dialogue systems, and contribute to uncovering the triggers of problematic situations. our solution is independent of domain and language, and does not require dialogues labelled with intents or acts. its implementation enabled the discovery of flows from a diverse set of dialogue datasets, out of which interesting insights were gathered, also with the help of computed metrics. for instance, in dialogues with artificial agents (retailpt, telecomsapt), sentiment gets worse throughout the flow. the automation of such agents results in more mechanical answers and, thus, more cohesive clusters (fcc), when compared to other datasets. mastodon and tdsapt were collected from social media and cover multi-ple domains, which contributes to a higher variation of sentiment (fsc). metrics also reveal that, with the parameters set (i.e., probability threshold of 0.1 and maximum 10 clusters for speaker), a large portion of utterances is lost in the flow discovery process. these regard low-probability transitions, but may degenerate interpretation, especially for large datasets as emowoz. yet, the alternative would be either to: reduce the number of clusters, with an impact on cohesion; or increase both the number of clusters and the threshold, with a negative impact in interpretability. therefore, we plan to test alternative implementations and analyze their impact on the previous, including clustering and labelling methods, and sentiment classification, where new trends @xcite ) can be explored. the computation of more metrics should also be considered, e.g., for assessing the coverage of discovered flows in unseen dialogues from the same domain. finally, towards stronger conclusions, flows should be discovered from additional datasets. another focus will be on flow visualization. while moving away from a graph-based model is unlikely, we consider integrating additional elements (e.g., reflecting the number of utterances in the node's size) and interactivity, towards improved interpretability (e.g., selecting the best threshold; highlighting the path taken in a specific dialogue).