Chain of Strategy Optimization Makes Large Language Models Better Emotional Supporter (2025)

Abstract: The growing emotional stress in modern society has increased the demand for Emotional Support Conversations (ESC). While Large Language Models (LLMs) show promise for ESC, they face two key challenges: (1) low strategy selection accuracy, and (2) preference bias, limiting their adaptability to users’ emotional needs. Existing supervised fine-tuning (SFT) struggles to address these issues, as it rigidly trains models on single gold-standard responses without modeling nuanced strategy trade-offs. To overcome these limitations, we propose a novel two-stage framework that optimizes strategy selection preferences at each dialogue turn. We first leverage Monte Carlo Tree Search to construct ESC-Pro, a high-quality preference dataset with turn-level strategy-response pairs. Then training on ESC-Pro with Chain-of-Strategy Optimization (CSO) improves both strategy accuracy and bias mitigation, enabling LLMs to generate more empathetic and contextually appropriate responses. Experiments on LLaMA-3.1-8B, Gemma-2-9B, and Qwen2.5-7B demonstrate that CSO outperforms standard SFT, highlighting the efficacy of fine-grained, turn-level preference modeling in ESC.

## introduction in modern society, people increasingly face emotional stress due to mounting work and life pressures. as a result, the demand for emotional support conversations (esc) has grown significantly, providing individuals with psychological relief and guidance @xcite @xcite . high-quality esc can help alleviate emotional distress, offering comfort and constructive advice * equal contribution † corresponding author 1 our code and data can be found in https://github. com/xingyusss/cso . ( @xcite . with the rapid advancement of large language models (llms) @xcite @xcite , their exceptional conversational abilities have opened up new possibilities for esc. however, achieving high-quality esc with llms remains a significant challenge due to two core limitations: 1) llms struggle with low strategy selection accuracy, often failing to choose contextually appropriate support strategies @xcite @xcite . 2) llms exhibit strategy preference bias, favoring certain strategies rigidly rather than adapting to users' dynamic emotional needs @xcite . due to the strategy-driven nature of esc, nuanced tradeoffs between strategies are critical @xcite @xcite . while supervised fine-tuning (sft) is the mainstream training approach @xcite , its inherent rigidity exacerbates these challenges. by training exclusively on single golden strategies and responses, sft teaches llms which strategies to apply but fails to clarify why certain strategies are inappropriate in specific contexts. this leaves models unable to grasp the contextual nuances required for dynamic adaptation. to tackle these challenges, we advocate for a more fine-grained and turn-level approach to optimizing esc. specifically, we propose a two-stage framework that first constructs rich preference data at the (strategy, response) level, and then refines model behavior through preference learning. in the first stage, we utilize monte carlo tree search (mcts) to expand gold-standard esc conversations into a conversation tree, where each layer represents a dialogue turn, and sibling nodes correspond to different strategic responses. as illustrated in the middle of figure 1 , we design a comprehensive value function that evaluates each response along four key dimensions-empathy, information, humanoid, and strategy-to ensure high-quality exploration. from this tree, we extract a refined esc dataset for preference optimization, esc-pro, as shown in the right of figure 1 . specifically, low-scoring nodes are categorized as nonprefer samples, identifying suboptimal strategies, while both the original gold-standard nodes and newly discovered high-scoring nodes are included as prefer samples, enhancing the dataset with highquality strategy examples. in the second stage, we introduce chain-of-strategy optimization (cso), a preference optimization approach that explicitly targets strategylevel improvements. as shown in the right of figure 1 , training on esc-pro with cso enables llms to not only select better strategies over weaker ones at each dialogue turn, but also to explore diverse conversation paths and avoid rigid preference patterns. this chain-like optimization across turns improves both local adaptability and long-range strategic coherence. our comprehensive experiments on llama-3.1-8b @xcite , gemma-2-9b @xcite , and qwen2.5-7b @xcite demonstrate that cso is highly effective in raising the accuracy of strategy selection and mitigating strategy preference bias. to further validate cso, we instantiated multiple preference optimization methods @xcite on esc-pro and consistently observed superior performance compared to standard sft. this consistent improvement underscores the effectiveness of esc-pro as a high-quality dataset and highlights the importance of fine-grained, turn-level preference modeling in achieving effective esc. the main contributions of this work are summarized as follows: (1) we present esc-pro, a high-quality turn-level preference dataset for esc, constructed using mcts. (2) we propose cso, enabling llms to learn nuanced strategy trade-offs at each dialogue turn. (3) extensive experiments on three llms demonstrate cso is effective in improving both esc quality and adaptability. preference optimization algorithms preference optimization algorithms align model outputs with user preferences by training on pairs of positive and negative samples, enhancing the finetuning of llms. traditional approaches use policybased reinforcement learning (rl) to train reward models @xcite , followed by online rl methods like proximal policy optimization (ppo) @xcite for llm fine-tuning. the direct preference optimization (dpo) algorithm @xcite streamlined this process by unifying reward modeling and rl into a single step, reducing computational costs. subsequent methods further optimized dpo: simpo @xcite ) eliminates dpo's reference model, @xcite enhances robustness via regularization, orpo @xcite integrates instruction tuning and preference alignment, and kto @xcite operates without paired data by leveraging prospect theory. these advancements simplify training, reduce resource a c b mcts iteration 🏆 b c a 🏆 a a b c a b b a c selection expansion simulation backpropagation reward compute a c a b low score expanded high score end node c a unexpanded start node a b a b c c a c b c ... ... ... demands, and broaden data applicability. existing esc models rely on sft, which lacks the ability to differentiate between optimal and suboptimal strategies. our work bridges this gap by applying preference optimization to esc, enhancing strategy accuracy and reducing bias. ## methodology we introduce a novel framework that refines strategy selection at each dialogue turn through structured preference modeling. as shown in figure 2 , this optimization paradigm consists of three key components: (1) dialogue expansion with mcts, (2) preference data construction and (3) chain-of-strategy optimization (cso). ## conclusion in this work, we address low strategy selection accuracy and preference bias challenges in esc. we propose a novel two-stage framework that optimizes strategy selection preferences at each dialogue turn. we first expand existing esc datasets with monte carlo tree search, constructing a conversation tree where different strategy-response pairs are evaluated to generate esc-pro, a refined preference dataset. by training llms on esc-pro, chain-of-strategy optimization (cso) improves strategy accuracy, reduces bias, and enhances adaptability to user emotions. extensive experiments on llama-3.1-8b-instruct, gemma-2-9b-it, and qwen2.5-7b-instruct demonstrate that cso significantly outperforms standard sft and decodingbased methods, validating the efficacy of turn-level preference modeling in improving esc quality.