Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition (2025)

Abstract: We propose a large language model based reward decomposition framework for aligning dialogue agents using only a single session-level feedback signal. We leverage the reasoning capabilities of a frozen, pretrained large language model (LLM) to infer fine-grained local implicit rewards by decomposing global, session-level feedback. Our first text-only variant prompts the LLM to perform reward decomposition using only the dialogue transcript. The second multimodal variant incorporates additional behavioral cues, such as pitch, gaze, and facial affect, expressed as natural language descriptions. These inferred turn-level rewards are distilled into a lightweight reward model, which we utilize for RL-based fine-tuning for dialogue generation. We evaluate both text-only and multimodal variants against state-of-the-art reward decomposition methods and demonstrate notable improvements in human evaluations of conversation quality, suggesting that LLMs are strong reward decomposers that obviate the need for manual reward shaping and granular human feedback.

## introduction as dialogue agents become increasingly deployed in open-ended, long-form social interactions, the challenge of aligning model behavior with trajectory-level human preferences becomes more pressing. while large language models (llms) have achieved strong linguistic competence, alignment techniques such as reinforcement learning with human feedback (rlhf) @xcite remain fundamentally constrained by their reliance on fine-grained, turn-level annotations. in many realistic settings, such as therapy, education, or companionship, annotators provide feedback only at the session level, reflecting their overall impression of the entire interaction. this presents a major obstacle: how can one use sparse, global feedback to improve local behavior? in this paper, we first introduce llm-geli, a large-language model based solution to decompose global explicit feedback into local implicit turn-level supervision signals that can be used for rlhf. we leverage a frozen pretrained llm as a reward decomposition oracle. given the full dialogue and a scalar global reward, the llm is prompted to infer how utterances contributed to the final outcome score. the result is a turnlevel reward decomposition derived purely from the llm's reasoning over the linguistic context. these decomposed signals are then used to train a lightweight, text-only reward function that can guide downstream policy learning. furthermore, we introduce multimodal-llm-geli, an extension of our core framework that further incorporates multimodal behavioral signals-such as facial expressions, gaze, and prosody-into the reward decomposition process. building on recent advances in multimodal prompting, we represent the listener's nonverbal reactions as natural language descriptors and append them to each utterance in the dialogue history. this textual augmentation enables the language model to interpret affective and social cues in context, allowing it to reason about the listener's internal state and how it evolves throughout the conversation. by grounding decomposition in the listener's multimodal feedback, multimodal-llm-geli produces turn-level reward estimates that more accurately reflect the listener's extralinguistic reactions. we evaluate our approach on the candor dataset @xcite , which features longform video conversations and post-session survey ratings, and an out-of-distribution dataset, soda @xcite . our results show that llm-geli and multimodal-llm-geli outperform prior baselines across a range of human and automatic metrics. notably, multimodal-llm-geli achieves the strongest gains on affective and socially grounded human evaluation measures figure 1 : multimodal llm-geli framework. a global reward r ge (τ ) is decomposed into turn-level pseudorewards using an llm prompted with dialogue and multimodal listener cues. these local rewards supervise a text-only reward model r θ , distilling crossmodal reasoning into a learnable function for alignment (sec. 4). @xcite , which highlights the value of llms not only as generators, but as competent interpreters of both global human preferences and rich, real-world behavioral signals to serve as reward decomposition oracles. llms for reward decomposition recent work has explored using large language models (llms) to support reward decomposition in reinforcement learning (rl). for instance, lare @xcite prompts an llm to generate a latent code that is then decoded into scalar rewards which sum to the total episodic return. calm @xcite employs an llm to identify when subgoals are achieved, assigning auxiliary rewards in a zeroshot manner. in contrast, recent alignment methods like spin @xcite and @xcite treat llms as reward functions, using them to rank or evaluate outputs via self-play or preference modeling-requiring dense or per-turn supervision. on the other-hand, geli @xcite is closely related to reward decomposition for dialogue modelling, however, it does not utilize an llm, instead it utilizes a manually designed reward shaping function. our method, llm-geli, is the first to leverage a pretrained llm that decomposes a global end-of-dialogue reward into fine-grained, turn-level signals. furthermore, by prompting an llm with multimodal behavioral feedback (e.g., facial affect, prosody), we infer fine-grained turn-level rewards from sparse, trajectory-level supervision. ## background: aligning language models we consider the problem of generating conversational responses using an autoregressive language model in a multi-sensory setting. we model the language generator as a dialogue agent with a policy π ϕ , parameterized by ϕ, following prior work @xcite @xcite . at each turn t, the agent observes the dialogue history s t -which includes all preceding utterances s 1 , a 1 , . . . , s t-1 , a t-1 -and produces a response a t as its action. for notational simplicity, we denote the full dialogue context up to turn t as s @xmath0:t-1] . the policy π ϕ (s t ) thus defines a distribution over possible responses a t conditioned on s t . to align the agent's behavior with human preferences, reinforcement learning from human feedback (rlhf) is commonly employed @xcite @xcite . given a reward function r θ (s t , a t ), parameterized by θ, that scores the quality of a generated response in context, we optimize the policy via reinforcement learning. a regularization term based on the kullback-leibler (kl) divergence from the original pretrained model is typically included to prevent the policy from straying too far from its initialization. the resulting objective takes the form: ## methods: llm-geli in many long-form dialogue settings, we often lack fine-grained annotations for individual utterances. instead, we may only have access to a single global explicit (ge) reward, a scalar score assigned at the end of an interaction that reflects the overall impression or outcome, such as how positively the user felt about the conversation. a key challenge is how to decompose this global reward into perturn feedback that can train a reward model capable of assigning appropriate values to individual utterances. we address this challenge using a large language model geli (llm-geli) framework, which proceeds in two stages (1) global explicit (ge) reward decomposition using the zero-shot prompting capabilities of a pretrained llm and (2) local implicit (li) reward modeling. ## experiments in this section, we evaluate the effectiveness of the proposed llm-geli framework for preferencebased learning, namely reward modeling and language model adaptation in long-form dialogue. therefore, our evaluation consists of two stages. first, we train reward functions using the various methods of global explicit reward decomposition, which we denote as "ge", and/or local implicit reward supervision, which we denote as "li". second, we fine-tune a dialogue language model using proximal policy optimization (ppo) @xcite , guided by the learned reward functions to improve response generation. due to computational restraints, all reward function training and rl fine-tuning procedures are executed with a single run. we assess each method's impact using both automatic and human-centric metrics. ## conclusion we introduce a new paradigm for preference-based alignment of conversational agents by offloading reward decomposition to large language models. unlike prior methods that rely on manually engineered functions to interpret nonverbal cues, llm-geli leverages the zero-shot reasoning capabilities of pretrained llms to infer turn-level rewards from a single global trajectory score. our multimodal-llm-geli extension further integrates nonverbal feedback through natural language descriptors, enabling rich, socially grounded reward modeling without requiring architectural changes or multimodal training. empirical results on the candor dataset show that our approach outperforms the state-of-the-art in human and automatic evaluations. these findings underscore the potential of llms as scalable and domain-agnostic reward decomposers, bridging the gap between sparse feedback and fine-grained behavioral alignment in real-world dialogue systems. this work opens the door to more generalizable, interpretable, and low-friction alignment methods for long-form, multimodal interactions.