Dream to Chat: Model-based Reinforcement Learning on Dialogues with User Belief Modeling (2025)

Abstract: World models have been widely utilized in robotics, gaming, and autonomous driving. However, their applications to natural language tasks are relatively limited. In this paper, we construct the dialogue world model, which could predict future utterances and user beliefs, including emotion, sentiment, and intention. In this paper, we propose a framework called DreamCUB, which shows that this user belief modeling and the entire dialogue world model can be established by LLM post-training. By defining a POMDP, we apply model-based reinforcement learning to the dialogue system and solve it by maximizing the information bottleneck. Experiments show that the pretrained dialogue world model can achieve state-of-the-art performances on emotion classification and sentiment identification, while dialogue quality is also enhanced by joint training of policy, critic and dialogue world model. Further analysis reveals that DreamCUB holds a reasonable exploration-exploitation balance and also transfers well to out-of-domain scenarios such as empathetic dialogues.

## introduction due to strong capabilities, modern large language models (llm) have obtained remarkable progress on dialogue systems @xcite . among the training pipeline of conversational llm, reinforcement learning from human feedback (rlhf) @xcite is an important post-training stage that bootstraps the human preference and achieves a deeper alignment by interactive sampling. although ppo @xcite is employed as the usual approach, its variants, such as dpo and grpo, are also proposed to improve the dialogue policy. however, reinforcement learning (rl) is often subject to low sampling efficiency, high performance variance, * internship at geely and high computational overhead. when applied to dialogue systems, these issues become more challenging when the model size is large and the annotation is consuming. to alleviate these issues, model-based reinforcement learning (mbrl) @xcite is proposed, which enables the agent to learn the environment model and use it to simulate, plan, and act. combining with recent progress on world models (wm) @xcite , mbrl has been a powerful solution for visual control @xcite , game @xcite , auto-driving @xcite and also dialogue system @xcite . for example, ddq @xcite proposes the world model of dialogue which can predict the dialogue contents. nevertheless, dialogues are highly sensitive to human psychological states, such as emotion and sentiment @xcite . people's reasoning, expression and intention can be affected and influenced by these inner states. however, such states are unobservable, while current mbrl studies on dialogues are based on observable states only, i.e., utterances. on the other hand, previous research on empathetic dialogue systems has mostly focused on generating responses given certain emotions. however, being empathetic not only requires responding based on self-emotions, but more importantly, calls for the understanding of user emotions and intentions, to respond appropriately @xcite . to bridge these gaps, in this paper, we introduce the user belief modeling into the mbrl framework, to provide a more thorough understanding of the dialogue policy. such user beliefs may include emotion, sentiment and intention, which are unobservable states for the agent, forming a partially observable markov process (pomdp). correspondingly, our dialogue world model (dwm) can not only generate future dialogue utterances, but also recognize user beliefs and behave as the reward model. to solve this problem, we refer to the theoretical derivations of pomdp-based mbrl studies @xcite , and deduce the dwm-rl algorithm based on the information bottleneck. combining user belief modeling, dwm and mbrl, we propose the framework called dream to chat with user belief (dreamcub). dreamcub simulates user belief and emotional dynamics over the course of interaction. rather than relying on static emotion classification or purely supervised generation, dreamcub enables an agent to imagine possible future dialogue trajectories, reason about long-term emotional impact, and plan supportive responses accordingly. figure 1 illustrates the paradigm of dreamcub. we summarize our contributions as follows: reward modeling. application of rl on textual environments requires reward model (rm) @xcite , which is trained from pairwise preference data (x, y + , y -) with x as the input, y + and y -are positive and negative responses. rm is usually implemented by an llm with the classification head added, which produces a 0-1 score. its loss can be derived from human preference distribution by the bradley-terry @xcite rlhf. the generative policy on language tasks solves the following problem: recent studies usually bootstrap and annotate the agent's reply strategy, to have enhanced response grounded by strategy. in this work, we further argue that the user's state, called belief , can also be modeled and behaves as the contextual information of subsequent strategy and response. such belief may include the user's emotion, sentiment, and intention. in this formulation, the determination pipeline becomes hist ⊕ @xmath0. the above formulation suggests query, resp, hist and strategy are observable to the agent while the user's emotion, sentiment and intention are unobservable. the system can then be described as a 5-tuple pomdp (o, s, a, r, t ): ## conclusion in this paper, we propose a framework called dreamcub to introduce the mbrl on the dialogue system, with user belief modeling of emotion, sentiment and intention. we first pretrain a dialogue world model which allows the user emotional identification and the next-query prediction, then jointly train this world model with dialogue policy, to achieve better performance on the daily dialogues. we further verify the effectiveness of user belief both in the world model and the reward model, as well as the typical conversation cases.