D ecoupled ESC : Enhancing Emotional Support Generation via Strategy-Response Decoupled Preference Optimization (2025)

Abstract: Recent advances in Emotional Support Conversation (ESC) have improved emotional support generation by fine-tuning Large Language Models (LLMs) via Supervised Fine-Tuning (SFT). However, common psychological errors still persist. While Direct Preference Optimization (DPO) shows promise in reducing such errors through pairwise preference learning, its effectiveness in ESC tasks is limited by two key challenges: (1) Entangled data structure: Existing ESC data inherently entangles psychological strategies and response content, making it difficult to construct high-quality preference pairs; and (2) Optimization ambiguity: Applying vanilla DPO to such entangled pairwise data leads to ambiguous training objectives. To address these issues, we introduce Inferential Preference Mining (IPM) to construct high-quality preference data, forming the IPM-PrefDial dataset. Building upon this data, we propose a Decoupled ESC framework inspired by Gross’s Extended Process Model of Emotion Regulation, which decomposes the ESC task into two sequential subtasks: strategy planning and empathic response generation. Each was trained via SFT and subsequently enhanced by DPO to align with the psychological preference. Extensive experiments demonstrate that our Decoupled ESC framework outperforms baselines, reducing preference bias and improving response quality.

## introduction mental health is essential to well-being @xcite , yet rising stress and fast-paced life have increased related issues @xcite @xcite . according to who, 1/8 people suffer from mental disorders (organization, 2022). amid a shortage of professionals, this underscores the need for scalable solutions, where large language models (llms) offer promising potential. to enhance the performance of llms in emotional support conversation (esc), prior works @xcite have constructed several large-scale, high-quality dialogue datasets and applied supervised fine-tuning (sft) to improve model responses. among them, liu et al. @xcite built the esconv dataset based on hill's helping skills theory @xcite and filtered out failedesconv dataset. the esconv dataset follows a three-phase structure (@xmath0, each paired with corresponding responses, details are provided in appendix a and c.1. this structured design significantly enhances a model's ability to generate empathetic dialogue. observation currently, sft has become the mainstream approach in the esc field. however, we observe that models still frequently exhibit common psychological errors @xcite during inference, which align with those identified in the failedesconv dataset (obs 1). in addition, zhao et al. @xcite found that sft's focus on single gold strategy-response pairs limits adaptability to nuanced contexts, weakening empathetic support. to mitigate this, they use monte carlo tree search (mcts) to collect pairwise preference data linking strategies and responses, and apply direct preference optimization (vanilla-dpo) to guide the model in choosing appropriate strategies, thereby partially reducing preference bias and improving response quality. challenges however, as shown in figure 1 and 4 , our further analysis reveals that the limitations of current work lie not in the sft or dpo training methods themselves, but rather in two overlooked challenges (obs 2): (1) entangled data structure: existing esc datasets heavily entangle psychological strategies with response content, making it difficult to construct high-quality preference pairs. for instance, penalizing responses with correct strategies but flawed content may degrade data quality. (2) optimization ambiguity: applying vanilla-dpo directly to such entangled data can blur training objectives and even lead to negative optimization outcomes. approach to address these issues, we first introduce the inferential preference mining (ipm) method, which automatically constructs preference samples decoupled from strategy-response. specifically, we use dynamic data routing to route four types of psychological error samples identified from the sft model's inference data to the dpo training stage of either strategy planning or response generation, depending on the error type. these samples are then paired with human-annotated ground truth samples to form the inferential preference mining preference dialogues (ipm-prefdial) dataset, containing 21k strategy preference pairs and 11k response preference pairs. this dataset provides disentangled and high-quality supervision signals for two separate dpo models. building on this, we propose a decoupled esc optimization framework (de-coupledesc), grounded in the extended process model of emotion regulation (epmer) @xcite , which divides emotion regulation into three sequential stages: (1) @xmath1, details are provided in appendix a.1. accordingly, we explicitly split the esc task into two subtasks: strategy planning (sp) and response generation (rg). results across multiple evaluation metrics, our decoupled optimization framework significantly outperforms joint training baselines. it not only enhances the diversity of strategy selection but also improves response quality and empathy. contributions our key contributions are summarized as follows: reinforcement learning for llm. reinforcement learning (rl) was initially introduced into llm training to align with human preferences @xcite . this approach uses a reward model to guide the optimization of the policy model via the proximal policy optimization (ppo) algorithm @xcite . recently, the group relative policy optimization (grpo) algorithm was proposed to enhance model reasoning capabilities @xcite , which eliminates the need for a critic model by using within-group rewards as advantages. while these online reinforcement learning methods are effective, they suffer from high computational costs and reliance on accurate reward modeling. as a simpler offline optimization algorithm, dpo optimizes the policy model from pairwise preference data directly without the need for reward modeling. due to its simplicity and effectiveness, dpo has achieved significant success across multiple domains, including mathematical reasoning, code generation, and recommendation systems @xcite @xcite . ## preliminary observations to investigate the causes of low response quality in the esc task, we analyzed outputs from six models: base, sft foot_1 , and dpo versions of qwen2.5-7b-instruct (team, 2024) and llama3.1-8b-instruct @xcite . ## experiments in this section, we conduct extensive experiments to address the following research questions: baselines. we compare vanilla coupled models (base, sft, dpo) with prompt-optimization baselines such as direct-refine, self-refine @xcite , and emotional cot @xcite . datasets. the esconv @xcite dataset is split into train, valid, and test sets in an 8:1:1 ratio, with the training set used for sft. the ipm-prefdial dataset is used for dpo training. evaluation metrics. we evaluate model performance using the following metrics: (1) automatic metrics, including bleu-1 (b-1) @xcite , distinct-1 (d-1) @xcite , f1-score (f1), and rouge-l (r-l) @xcite ; (2) llmbased metrics, including fluency (flu.), professionalism (pro.), empathy (emp.), and helpfulness (hel.). all metrics are rated on a 5-point likert scale @xcite ; (3) strategy metrics, including preference bias (b) @xcite and strategy prediction accuracy (weighted-f1 qw and macro-f1 q). detailed definitions of the evaluation metrics, the prompt, and the bias calculation formula are provided in appendix e. implementation details. for sft, we train for 3 epochs using a batch size of 32 and a learning rate of 1e-5. dpo is trained for 1 epoch under the same settings. all experiments are conducted on 4×24gb rtx 4090 gpus. for llm-based evaluation, we randomly sample 100 test instances and evaluate them using gpt-4.1-mini-2025-04-14 4 alongside 3 psychology experts. additional implementation details are provided in appendix d.1. ## conclusion in this paper, we propose a decoupled esc framework that separates strategy planning from empathetic response generation, enabling targeted optimization and avoiding mutual interference. extensive experiments demonstrate that our decoupled esc framework significantly outperforms joint optimization baselines, effectively reducing preference bias and improving response quality in emotional support conversation tasks.