LLM -guided Plan and Retrieval: A Strategic Alignment for Interpretable User Satisfaction Estimation in Dialogue (2025)

Abstract: Understanding user satisfaction with conversational systems, known as User Satisfaction Estimation (USE), is essential for assessing dialogue quality and enhancing user experiences. However, existing methods for USE face challenges due to limited understanding of underlying reasons for user dissatisfaction and the high costs of annotating user intentions. To address these challenges, we propose PRAISE (Plan and Retrieval Alignment for Interpretable Satisfaction Estimation), an interpretable framework for effective user satisfaction prediction. PRAISE operates through three key modules. The Strategy Planner develops strategies, which are natural language criteria for classifying user satisfaction. The Feature Retriever then incorporates knowledge on user satisfaction from Large Language Models (LLMs) and retrieves relevance features from utterances. Finally, the Score Analyzer evaluates strategy predictions and classifies user satisfaction. Experimental results demonstrate that PRAISE achieves state-of-the-art performance on three benchmarks for the USE task. Beyond its superior performance, PRAISE offers additional benefits. It enhances interpretability by providing instance-level explanations through effective alignment of utterances with strategies. Moreover, PRAISE operates more efficiently than existing approaches by eliminating the need for LLMs during the inference phase.

## introduction dialogue systems play an increasingly crucial role in enabling users to interact with intelligent agents to fulfill their needs. user satisfaction estimation (use), the process of predicting how satisfied a user is in dialogue interactions @xcite , is crucial for evaluating the quality of dialogue systems and ensuring a positive user experience @xcite ; deng † corresponding author. @xcite @xcite @xcite . effective use methods should not only accurately classify user satisfaction but also provide interpretable results to guide the improvement of dialogue systems. by understanding and quantifying user satisfaction, dialogue systems can be continuously improved to better meet user expectations. the development of use methods has evolved through three main approaches: content-based, dialogue act-based, and language model-based. content-based methods, such as sentiment analysis @xcite and response quality assessment @xcite , evaluate dialogue content to estimate user satisfaction. however, these methods often struggle to accurately capture user intentions and whether user goals are fulfilled. dialogue act-based methods incorporate dialogue acts, which represent user intentions at each turn @xcite @xcite , leveraging the relationship between these acts and user goal achievement @xcite . however, these methods often require complex pre-training procedures and accurate dialogue act labels, which can be challenging and time-consuming to obtain for real-world conversations. language model-based methods have shown a promising direction for use. @xcite employs large language models (llms) and iterative prompting to summarize dialogues and extract rubrics of user satisfaction from natural language utterances. however, this approach does not provide utterance-level interpretability and relies heavily on advanced llms like gpt-4 for the entire process, which can be expensive and impractical for large-scale applications. in this paper, we introduce praise (plan and retrieval alignment for interpretable satisfaction estimation), a framework that leverages llms to generate and refine strategies for classifying user satisfaction levels in conversational systems. these strategies serve as interpretable natural language criteria that indicate situations for identifying user satisfaction (sat), dissatisfaction (dsat) or neutrality (neu) in dialogues. for instance, as shown in figure 1 , a strategy like "user thanks for solving problem quickly" can be an indicator of user satisfaction. the framework consists of three key modules: strategy planner, feature retriever, and score analyzer (figure 2 ). the strategy planner generates interpretable natural language strategies for classifying user satisfaction by leveraging effective and ineffective strategies stored in memory. the feature retriever quantifies the relevance between user utterances and generated strategies, converting the llm knowledge into measurable features for analysis. the score analyzer evaluates the effectiveness of strategies and categorizes them according to their contributions to use improvement. through iterative refinement of this process, praise aims to identify optimal strategies to maximize overall performance. praise achieves state-of-the-art performance on use benchmark datasets like mwoz, sgd, and redial, while providing interpretable results. beyond its high performance and interpretability, praise brings additional advantages, including efficient inference and scalable deployment. the key contributions of our paper are: ## problem statement consider a dataset d of conversations, where each conversation c i is represented as a sequence of t utterance pairs between the user and the assistant, denoted as c @xmath0,1 , a i,1 ), ..., (u i,t , a i,t )]. here, u i,n represents the n-th user utterance in the ith conversation, and a i,n represents the corresponding assistant response. each user utterance u i,n is associated with a satisfaction label y i,n , which can take one of three values: sat, dsat, or neu. the user satisfaction estimation problem can be formulated as learning a satisfaction estimator e that maps the current user utterance u i,n and the preceding conversation context to the corresponding satisfaction label y i,n . ## related work user satisfaction estimation prior works on user satisfaction estimation in dialogue systems have evolved from content-based methods, such as sentiment analysis @xcite and interaction quality assessment @xcite , to leveraging pre-trained language models @xcite and dialogue action tasks @xcite , modeling satisfaction dynamics @xcite , and using multi-task adversarial method @xcite . recent efforts have explored using llms as simulators @xcite and augmenting datasets with counterfactual dialogue samples @xcite . despite recent advancements, many of these methodologies still struggle with interpretability. to address these limitations, @xcite proposed a framework using llm, but it falls short in providing instance-level interpretability and incurs high costs when summarizing and classifying data using gpt-4. praise achieves high efficiency by using llm only to generate strategies during training, without requiring it for inference. additionally, this approach provides interpretability for overall use classification and utterance-level analysis. retrieval models text embeddings that capture semantic similarity and context have been developed, ranging from early models @xcite @xcite to larger models (behnamghader et al., 2024) trained on large-scale corpora. diverse retrieval tasks utilize these embeddings, such as dpr @xcite , contriever @xcite , ance @xcite , and rocketqa @xcite . however, applying these retrievers to domain-specific data is challenging due to the lack of relevance supervision data. to address this issue, studies have utilized llms to generate pseudo-labels, such as hypothetical documents @xcite or generated relevant queries @xcite @xcite for model training. to resolve the scarcity of relevance data, we employ llms to generate passages that align with the generated strategies. we then utilize the search results between these generated passages and actual user utterances as features in our framework. ## conclusion in this paper, we presented praise, a novel framework for user satisfaction estimation (use) in dialogue systems that leverages large language models (llms). our work addresses the crucial challenge of developing interpretable and scalable methods for assessing user satisfaction, a key factor in improving conversational ai. praise consists of three main modules: the strategy planner, which generates natural language strategies for classifying user satisfaction; the feature retriever, which provides multi-level interpretability by aligning user utterances with strategies; and the score analyzer, which evaluates strategy effectiveness and enables efficient inference by transforming llm knowledge into a structured representation. our experimental results across three benchmark datasets demonstrate praise's superior performance compared to existing use methods.