Exploring User Dissatisfaction: Taxonomy of Implicit Negative Feedback in Virtual Assistants (2024)

Abstract: The success of virtual assistants relies on continuous performance monitoring to ensure their competitive edge in the market. This entails assessing their ability to understand user intents and execute tasks effectively. While user feedback is pivotal for measuring satisfaction levels, relying solely on explicit feedback proves impractical. Thus, extracting implicit user feedback from conversations of user and virtual assistant is a more efficient approach. Additionally, along with learning whether a task is performed correctly or not, it is extremely important to understand the reasons behind any incorrect execution. In this paper, we introduce a framework designed to identify dissatisfactory conversations, systematically analyze these conversations, and generate comprehensive reports detailing the reasons for user dissatisfaction. By implementing a feedback classifier, we identify conversations that indicate user dissatisfaction, which serves as a sign of implicit negative feedback. To analyze negative feedback conversations more deeply, we develop a lightweight pipeline called an issue categorizer ensemble with multiple models to understand the reasons behind such dissatisfactory conversations. We subsequently augment the identified discontented instances to generate additional data and train our models to prevent such failures in the future. Our implementation of this simple framework, called AsTrix (Assisted Triage and Fix), led to significant enhancements in the performance of our smartphone-based In-House virtual assistant, with successful task completion rates increasing from 83.1% to 92.6% between June 2022 and March 2024. Moreover, by automating the deeper analysis process targeting just five major issue types contributing to the dissatisfaction, we significantly address approximately 62% of the negative feedback conversation data.

## introduction the efficacy of virtual assistants (vas) depends upon their ability to accurately interpret user inputs and execute appropriate tasks. while assessing their success rate is crucial, it is equally important to scrutinize their failure rate, as this presents opportunities for improvement. hence, continuous evaluation of the virtual assistant's performance plays an important role in the system's improvements. there has been abundant amount of work done to evaluate virtual assistants @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite but most of them is related to taking reference from user feedback data (click data, data with likes/dislikes) to derive implicit feedback information @xcite , @xcite , @xcite , @xcite . the availability of explicit feedback data may be challenging and may be restricted to only a few data patterns. moreover, most of this work focused on estimating user satisfaction and improving classification accuracy, overlooking the interpretability. in this work, we aim to estimate user satisfaction and dissatisfaction in conversations between users and our smartphone-based in-house virtual assistant using only the interaction patterns, without any dependency on explicit user feedback data. additionally, we also focus on the interpretability of negative feedback conversations where we identify reasons behind user dissatisfaction. a few examples of satisfactory (sat) and dissatisfactory (dsat) conversations between users and our in-house virtual assistant (in-house-va) are illustrated in in the first stage, our objective is to find the user dissatisfaction sessions within the system. we assess user satisfaction or dissatisfaction through interactions between users and our smartphone-based in-house-va, involving dialog-style conversations with user requests and in-house-va responses. we build a feedback classifier that can analyze patterns in user and in-house-va conversations to determine whether the user experience is satisfactory or not. after identifying negative feedback sessions, in the second stage, it is important to determine the reasons for user dissatisfaction. therefore, we build a pipeline to identify various types of issues that can happen during user-va interactions, we call the pipeline auto-issue categorizer. smartphone-based in-house-va data is analyzed by language experts from 2022-10-07 to 2022-10-19. the analysis focused on the top 25 smartphone domains that were most frequently used during this period. they analyzed around 8750 utterances in total, with an average of 350 frequently used utterances from each domain. by examining these utterances, five major issue types were identified, shown in figure 1 . these issue types include automatic speech recognition (asr) failure, multi-intent failure, domain router failure, intent router failure, and network/server issues. we develop a lightweight pipeline ensemble with issue-type classifiers to address these failures. after identifying various issue types, the next step is how to utilize this analysis and results to improve the system. the idea is simple, we generate similar erroneous samples and train the va system to handle such scenarios in the future. we call it training variant generator. for data generation, we leverage mistral-7b-v0.2 (ai, 2024). by identifying the specific issues that are causing in-house-va to fail, developers can address these issues directly, resulting in an improved user experience. this may involve modifying in-house-va's algorithms, language-processing capabilities, or other features to better align with user needs and preferences. additionally, this process can provide valuable insights into how users interact with in-house-va, allowing developers to identify areas for improvement and optimization. ultimately, the goal of the astrix framework, shown in figure 2 , is to enhance in-house-va's ability to meet user needs and provide more accurate, reliable and efficient performance. ## proposed methodology this section introduces our approach in developing the integrated framework. our comprehensive framework astrix (assisted triage and fix) comprises three distinct stages shown in figure 2. (i) feedback classifier (ii) auto-issue categorizer (iii) training variant generator. ## conclusion our proposed framework, astrix, presents a streamlined approach to identify and analyze unsatisfactory interactions between users and smartphone-based virtual assistants. there could be multiple reasons behind the user dissatisfaction, and by addressing just five main issue types, we tackle around 62% of the total negative feedback conversations. this leads to a notable improvement in the success rate of user-in-house-va interactions. while we have demonstrated the application of our framework on the smartphone-based virtual assistant, it is important to note that this method is not confined to smartphones alone. the framework is versatile and can be applied to any type of conversational agent, whether it's integrated into smart speakers, wearables, home automation systems, or customer service chatbots. since the core focus of astrix is on improving the quality of interactions and addressing user dissatisfaction, it is adaptable to different platforms and devices that rely on conversational interfaces. in the future, we aim to identify additional negative issue types, such as out-of-scope intents and slot errors, that could contribute to conversation failures. furthermore, while retaining our non-llm methods, we plan to harness the capabilities of large language models (llms) to automatically identify issues based on the conversation's context. this will enable the framework to dynamically detect and categorize new or unforeseen failure types. additionally, by utilizing advanced prompt refinement techniques like soft prompting and multi-task prompt tuning, we aim to address the limitations observed in our experiments, where single-prompting methods in zero-shot or few-shot settings with llms did not yield optimal results.