Steering Conversational Large Language Models for Long Emotional Support Conversations (2025)

Abstract: In this study, we address the challenge of consistently following emotional support strategies in long conversations by large language models (LLMs). We introduce the Strategy-Relevant Attention (SRA) metric, a model-agnostic measure designed to evaluate the effectiveness of LLMs in adhering to strategic prompts in emotional support contexts. By analyzing conversations within the Emotional Support Conversations dataset (ESConv) using LLaMA models, we demonstrate that SRA is significantly correlated with a model’s ability to sustain the outlined strategy throughout the interactions. Our findings reveal that the application of SRA-informed prompts leads to enhanced strategic adherence, resulting in conversations that more reliably exhibit the desired emotional support strategies over longer conversations. Furthermore, we contribute a comprehensive, multi-branch synthetic conversation dataset for ESConv, featuring a variety of strategy continuations informed by our optimized prompting method. The code and data are publicly available on our Github.

## introduction in the rapidly evolving domain of conversational ai, the creation of emotionally intelligent conversational agents is becoming increasingly important as it opens up new possibilities for more natural and helpful interactions between humans and machines. central to this transformative journey is the challenge of empowering large language models (llms) not only to partake in natural dialogues but also to adeptly navigate and influence the conversation flow using expert strategies derived from psychology and emotional support literature. this paper delves into the critical aspect of integrating emotional support strategies into conversational llms, a domain that remains largely uncharted yet holds significant promise for a range 1 https://github.com/navidmdn/esconv-sra figure 1 : a sample continuation of a conversation using "provide different perspectives" strategy, given by three different prompt templates sorted by the sra metric increasing from bottom to top using llama-70b-chat model. the model output using the prompt template with higher sra adheres better to the given strategy. the advent of the emotional support conversations dataset (esconv) @xcite has marked a significant milestone, providing a rich resource for researchers to delve into and enhance emotional support dialogue systems. despite this advancement, there remains a notable gap in the state-of-the-art evaluation methods for such sys-tems. researchers have tried to build and improve systems that either align closely with the gold standard responses in the dataset (responses from amazon mturk workers certified as emotional supporters) or focus on enhancing the model's ability to plan subsequent strategies. however, the predominant metric for comparison in these works remains the alignment with these gold standard responses. we argue that this approach may not be the most effective for several reasons. first, in the realm of emotional support, there is often no single 'correct' strategy for continuing a conversation. second, even when a model bases its response on a specific strategy, there are numerous potential high-quality responses that could be equally effective. in our research, we adopt a different perspective, reevaluating the core problem in the context of recent advancements. with the advent of large language models (llms), generating natural and fluent text has become less of a challenge. our focus, therefore, shifts to a more nuanced aspect: the degree to which we can effectively guide these llms to adhere to specific emotional support strategies during extended conversations, and importantly, how we can evaluate and quantify their proficiency in following these strategies. this approach acknowledges the proficiency of llms in text generation while emphasizing the critical need for strategic control and direction in prolonged interactive scenarios. the challenge extends beyond merely directing the conversation, delving into the realm of assessing and quantifying the model's adherence to the predefined emotional support strategies. below are the main contributions of our work: ## evaluation of strategy following and sra metric in the following section we propose methods for evaluating the efficiency and usability of our proposed sra metric in designing efficient prompts for prolonged strategy adherence in emotional support conversations. we first outline two automatic evaluation approaches in section 5.1 and 5.2. we also conduct a human evaluation experiment which will be described in section 5.3. ## conclusion in this paper, we introduced the strategy-relevant attention (sra) metric, a novel, model-agnostic approach designed to optimize zero-shot prompt generation for adhering to emotional support strategies within conversational ai systems. our findings demonstrate that sra significantly correlates with the capability of large language models (llms) to maintain strategic alignment with emotional support strategies. our study uncovers a key challenge in conversational ai: the reduction in large language models' (llms) adherence to emotional support strategies with increasing conversation length. we found that naive prompts to llms often result in decreased strategic focus in extended dialogues. the strategy-relevant attention (sra) metric we introduced not only facilitates the crafting of prompts but also the ongoing monitoring of adherence to strategy throughout the conversation. this ensures that the models maintain a consistent strategic direction.