SAGE : Steering Dialog Generation with Future-Aware State-Action Augmentation (2025)

Abstract: Recent advances in large language models have enabled impressive task-oriented applications, yet building emotionally intelligent chatbots for natural, strategic conversations remains challenging. Current approaches often assume a single “ground truth” for emotional responses, overlooking the subjectivity of human emotion. We present a novel perspectivist approach, SAGE, that models multiple perspectives in dialogue generation using latent variables. At its core is the State-Action Chain (SAC), which augments standard fine-tuning with latent variables capturing diverse emotional states and conversational strategies between turns, in a future-looking manner. During inference, these variables are generated before each response, enabling multi-perspective control while preserving natural interactions. We also introduce a self-improvement pipeline combining dialogue tree search, LLM-based reward modeling, and targeted fine-tuning to optimize conversational trajectories. Experiments show improved LLM-based judgments while maintaining strong general LLM performance. The discrete latent variables further enable search-based strategies and open avenues for state-level reinforcement learning in dialogue systems, where learning can occur at the state level rather than the token level.

## introduction recent advances in large language models (llms) have shown impressive capabilities in task-oriented applications like question-answering, mathematics, and coding @xcite @xcite . however, effective human-ai interaction requires not just problem-solving abilities but also emotional intelligence and interpersonal skills. this has led to emotion-based chatbots like character ai @xcite and pi (pi, 2023), which focus on emotional connection and engagement. these models are designed to understand emotions, provide empathy, and build trust through natural conversations, finding applications in education, healthcare, counseling, and gaming @xcite . building effective emotional chatbots presents unique challenges that highlight the limitations of traditional single-perspective approaches in nlp. unlike task-oriented interactions, emotional conversations require strategic navigation without clear objectives and involve inherent subjectivity-what constitutes an appropriate emotional response varies significantly across individuals, cultures, and contexts. this subjectivity represents a fundamental challenge that aligns with the growing recognition of human label variation (hlv) in nlp tasks @xcite . current approaches often aggregate diverse human perspectives into a single "ground truth," potentially overlooking the richness and validity of multiple viewpoints. the challenges in emotional dialogue systems manifest in two key areas: (1) chatbots need to steer conversations strategically by planning ahead-an ai therapist must balance various conversational moves to build trust and facilitate disclosure, while an ai negotiator needs to integrate multiple techniques to achieve desired outcomes. (2) the chatbot should be proactive and interactive by asking clarifying questions and providing customized information based on real-time feedback. a career counselor might probe into a client's experiences to offer personalized guidance, while a fitness coach might assess energy levels to adjust workout recommendations. these capabilities require modeling multiple valid perspectives rather than assuming a single correct approach. these capabilities require effective learning mechanisms for handling sparse rewards across multiple conversation turns. while rlhf @xcite and deepseek r1 @xcite have succeeded in complex reasoning tasks @xcite @xcite ; wang ": wow! how did your dog react to being used as a mode of transportation? and how is the baby doing? they sure have quite an imagination! !: my dog seemed surprised at first but then started wagging its tail and running around the yard with the baby on its back. ## results llm-judge based evaluation following §3.6, we compare the various versions of the models, namely sage 1 , sage 2 , sage 3 , with two judge llms, namely gpt-3.5 and mixtral 8x7b. the llm selector in the tree search used mixtral, potentially introducing bias towards mixtral's inductive bias. to mitigate this, we incorporated both mixtral and gpt-3.5 for the judgement, and primarily rely on the assessment by gpt-3.5. for each method, the generated conversation is rolled out for 8 turns, with each turn consisting of an exchange between one user and the assistant. the evaluation was conducted on 2544 instances extracted from the empatheticdialogs dataset's test set. the results are shown in the average length of responses for sage 0 is 86.2, while for sage dp o it is 21.8. we show some examples of the comparison between the initial sage 0 with sage dp o in figure 1 and appendix (figure 5 and 6 ). generally, sage dp o appears to be more concise, interactive, engaging, sympathetic, and resembling a more human-like tone. we provide several additional examples of multi-turn conversation rollouts of sage dp o in the appendix. these examples showcase its capacity to produce coherent (figure 7 ), humorous (figure 8 ), and empathetic dialogues (figure 9 ), highlighting the contrast in style compared to a utility-oriented chatbot such as sage 0 . the overall judgements of gpt-3.5 and mixtral are consistent. gpt-3.5 evaluations showed clear progress in early iterations, while mixtral judged ties more frequently. llm benchmarks we evaluated our model on standard llm benchmarks, including hellaswag @xcite , arc (challenge and easy) @xcite , mmlu @xcite , winogrande @xcite , open-bookqa @xcite , piqa @xcite , siqa @xcite , commonsenseqa @xcite , and gsm8k @xcite (see we hypothesize that the performance degradation (particularly on gsm8k) occurs because our model became more colloquial and conversational, making exact match more challenging (see appendix figure 10 for an example). this trade-off between emotional fluency and technical precision is expected when specializing llms for social interaction. to mitigate this performance gap, several strategies could be employed: (1) incorporating instruction-tuned datasets @xcite during training to maintain reasoning capabilities, (2) using a hybrid approach that combines emotional coarse generation control by state manipulation generally, the model uses various motivations and emotions to guide the conversation. in appendix figure 3 , we present the distribution of generated actions in simulated rollouts from our model, based on the situational prompts from @xcite . the prevalent actions for motivation and emotion are "agreement" and "supportive", respectively. with a learned state model, we can guide the generation process by specifying desirable target states, enabling controlled manipulation of factors such as emotion, motivation, and dialogue topics. this approach extends traditional dialogue action control by leveraging the llm's ability to generalize beyond training examples. examples demonstrating this capability are presented in tables 3 , 4 , and 5 (in appendix). ## conclusion we propose the state-action chain (sac) framework for emotional dialogue generation, enabling explicit state modeling and controllable conversation flow. a key insight is our future-looking annotation strategy, which labels states and actions based on the full dialogue context rather than isolated utterances. this allows the model to develop strategic thinking by associating state-action pairs with future consequences and utility. using iterative dialogue tree search and preference learning, sac generates more engaging and emotionally intelligent responses. it enables flexible control of conversation dynamics during inference without retraining. while there are trade-offs on traditional benchmarks, sac enhances human-like dialogue and lays groundwork for state-level reinforcement learning to train emotionally intelligent chatbots that reason about long-term outcomes.