Expectation Confirmation Preference Optimization for Multi-Turn Conversational Recommendation Agent (2025)

Abstract: Recent advancements in Large Language Models (LLMs) have significantly propelled the development of Conversational Recommendation Agents (CRAs). However, these agents often generate short-sighted responses that fail to sustain user guidance and meet expectations. Although preference optimization has proven effective in aligning LLMs with user expectations, it remains costly and performs poorly in multi-turn dialogue. To address this challenge, we introduce a novel multi-turn preference optimization (MTPO) paradigm **ECPO**, which leverages Expectation Confirmation Theory to explicitly model the evolution of user satisfaction throughout multi-turn dialogues, uncovering the underlying causes of dissatisfaction. These causes can be utilized to support targeted optimization of unsatisfactory responses, thereby achieving turn-level preference optimization. ECPO eliminates the significant sampling overhead of existing MTPO methods while ensuring the optimization process drives meaningful improvements. To support ECPO, we also introduce an LLM-based user simulator, **AILO**, to simulate user feedback and expectation confirmation during conversational recommendations. Experimental results show that ECPO significantly enhances CRA’s interaction capabilities, offering notable improvements in both efficiency and effectiveness over existing MTPO methods.

## introduction conversational recommendation systems (crss) leverage multi-turn natural language interactions to gradually uncover user interests and subsequently recommend items aligned with their preferences @xcite . powered by the advanced text generation and toolcalling capabilities of large language models (3) simulated environmental noise in the expansion and evaluation may be incorporated into preference relations, leading the model to update in the wrong direction. to address this challenge, aligning cras with human expectations presents a viable solution. preference optimization has demonstrated success in aligning llm outputs with user preferences @xcite @xcite . its core principle involves sampling multiple candidate outputs from the llm and increasing the probability of those that align with user expectations. however, conversational recommendation is a multi-turn dialogue task, and applying preference optimization to this process presents great challenges. the main difficulty is that user preferences change in each dialogue turn and dynamically evolve as the conversation progresses. most existing multi-turn preference optimization (mtpo) methods simply treat each turn equally, failing to capture turn-level preference relationships @xcite . several recent works @xcite try to infer turn-level preference relationships through tree-based simulations. as illustrated in fig. 1 , these approaches introduce three inherent challenges: (1) to obtain turn-level preference, it is necessary to sample multiple candidate responses at each turn and simulate the entire conversation to evaluate preferences for intermediate turns, resulting in significant sampling overhead. (2) in multi-turn conversational recommendation tasks, llms struggle to generate effective positive outputs through self-sampling. (3) evaluating preferences for intermediate turns relies on the simulated environment, whose randomness may introduce additional noise into preference relationships, leading to suboptimal performance of the aligned cra. overcoming these limitations is essential to aligning cras with human expectations. this leads to a critical question: is there a way to construct highquality turn-level preference relationships without additional sampling and evaluation? a problem well stated is a problem half solved. the core idea of this paper is to explicitly model how user satisfaction evolves throughout multi-turn dialogues and uncover the underlying causes of dissatisfaction. by identifying and addressing the root causes of low satisfaction, we can naturally construct responses that better align with user expectations. expectation confirmation theory (ect) @xcite tells us satisfaction is a subjective feeling that arises from the comparison between an individual's initial expectations and the perceived actual performance or outcomes. when applied to the context of conversational recommendation, this can be understood as: during a dialogue, a user has specific expectations for the system's response in each turn. upon receiving the actual response, the user evaluates it by comparing it with their initial expectations, assigning a subjective satisfaction score based on the perceived gap. confirmation preference optimization (ecpo), which comprises three key steps: (1) forward expectation confirmation to identify unsatisfactory responses and uncover their root causes; (2) backward expectation derivation to rewrite the unsatisfactory responses based on these causes; (3) preference optimization using the original and rewritten responses. considering the high cost and potential bias associated with real users participating in the expectation confirmation (ec) process, we further introduce ailo, an llm-based agent that simulates real users' activities, interests, language, and orientations. during the dialogue, ailo acts as a user, providing diverse and realistic feedback as well as performing the ec process. our contributions are summarized as follows: to better align multi-turn cras with human expectations, we propose expectation confirmation preference optimization (ecpo). its core idea is to leverage ect to explicitly model the evolution of user satisfaction throughout multi-turn dialogues and construct turn-level preference relationships by identifying and addressing the root causes of dissatisfaction. a detailed description of ecpo is provided in section 2.2. additionally, we introduce a novel user simulator, ailo, which generates diverse and realistic user feedback while performing expectation confirmation (see section 2.3). ## experiments to thoroughly evaluate the effectiveness of ecpo in enhancing multi-turn cras, we conduct extensive experiments, which are outlined as follows: the "#calls" column represents the number of llm calls required to complete an entire dialogue. n denotes the number of dialogue turns, and m represents the number of times the llm generates retrieval queries (@xmath0. sr (success rate) and r (recall rate) are recommendation metrics, while wr reflects the interactive capabilities. ## conclusion and future works in this work, we propose ecpo, a novel mtpo paradigm designed to enhance the interaction capabilities of llm-based cras, thereby improving user satisfaction. our core design principle is to explicitly model the evolution of user satisfaction in multi-turn dialogues and achieve turn-level preference alignment by identifying and addressing the root causes of dissatisfaction. to support ecpo, we introduce a new llm-based user simulator, that enable more diverse and realistic simulations as well as expectation confirmation. extensive ex-periments on three recommendation datasets fully demonstrate the superiority of our proposals. although ecpo is designed for cras, we believe it can be extend to broader dialogue assistants by modeling the ec process across different domains. another promising direction is enabling llms to generate simulated user expectations. this capability can establish an internal feedback mechanism, integrating the ec process into the reasoning phase to help o1/r1-style @xcite dialogue assistants in refining responses and further enhancing user satisfaction.