Global Reward to Local Rewards: Multimodal-Guided Decomposition for Improving Dialogue Agents (2024)

Abstract: We describe an approach for aligning an LLM based dialogue agent for long-term social dialogue, where there is only a single global score given by the user at the end of the session. In this paper, we propose the usage of denser naturally-occurring multimodal communicative signals as local implicit feedback to improve the turn-level utterance generation. Therefore, our approach (dubbed GELI) learns a local, turn-level reward model by decomposing the human-provided Global Explicit (GE) session level reward, using Local Implicit (LI) multimodal reward signals to crossmodally shape the reward decomposition step. This decomposed reward model is then used as part of the RLHF pipeline to improve an LLM-based dialog agent. We run quantitative and qualitative human studies on two large-scale datasets to evaluate the performance of our GELI approach, and find that it shows consistent improvements across various conversational metrics compared to baseline methods.

## introduction developing social dialogue agents that can interact and collaborate with humans over a long horizon remains a longstanding goal of artificial intelligence. large language models (llm) pretrained at scale on the next-word prediction objective and then aligned to human preference via rlhf (reinforcement with human feedback) represent a significant step in this direction @xcite , leading to successful commercial applications. however, existing methods for alignment usually assume that preference labels are annotated at the turn-level (i.e., after each utterance). this makes it difficult to extend this framework to cases where human preference labels are only available at the session-level, i.e., after an entire dialogue session (which could span 30 minutes or more). insofar as we are interested in developing dialogue agents that can continually learn from session-level dialogue data "in the wild" (e.g., through in-person conversations), there is a need to develop techniques that can (1) align agents based on global rewards at the session level and (2) take into account extralinguistic multimodal signals that are pervasive in naturally-occurring conversations. concretely, a session-level score obtained postconversation is a form of global explicit feedback, which provides a holistic assessment of a conversation session. such feedback can be obtained naturally at scale by, for example, asking participants to rate how they felt about the dialog session. however, it is not possible to use such data directly as part of an rlhf pipeline, since current methods generally require local, turn-level signals for aligning an llm-based agent to human preferences. moreover, in real world settings, agents are deployed in multisensory environments @xcite where they have access to rich multimodal signals (e.g., facial expressions during a video conversation). an ideal agent should leverage these signals as proxy rewards to improve its behavior. in dialogue, previous work attribute many multimodal cues such as body mimicry, vocal accommodation, and emotion, as implicit measures of conversation quality @xcite . hence, we can utilize multimodal signals as local implicit feedback, which presents an opportunity to use multimodal local implicit feedback as signals to crossmodally guide the decomposition of the single global explicit (ge) post-interaction score. in this paper, we describe a joint framework called geli, which integrates global explicit (ge) and local implicit (li) feedback. geli makes it possible to align an llm-based dialogue agent based on global rewards, while simultaneously taking into account naturally-occurring multimodal signals. our formulation brings together the idea of training a reward model which decomposes a single global explicit annotation score that is shaped by local implicit multimodal signals, which is sub-sequently used to align an llm-based dialogue agent via rlhf. specifically, we use geli to learn a reward function based on the overall affect of the user (i.e., how positive the user felt at the end of the conversation) from a large-scale long-horizon multimodal dialogue dataset @xcite and evaluate on two datasets for the generated dialogue. our local implicit multimodal signal comes from an affect classifer based on facial expression. we find that the reward function learned via geli can be used train a dialogue agent that has improved ability across various metrics of conversational quality including sensibleness, reusability, and specificity @xcite . temporal credit assignment temporal credit assignment (tca) is a concept within the field of reinforcement learning and artificial intelligence that addresses the challenge of attributing credit to actions over time. it involves determining the extent to which past actions contributed to the current outcome, allowing an intelligent agent to understand the consequences of its decisions. one way to apply tca to reinforcement learning is by manipulating the λ-discount factor and investigating how this affects policy learning @xcite . recently, a line of works have been proposed to treat tca as a return decomposition. rudder @xcite assigns step-wise credit by the predictive difference between two consecutive states. ircr @xcite is an instantiation of uniform reward redistribution. randomized return decomposition (rrd) @xcite formulate a surrogate problem through monte-carlo sampling estimating step-wise rewards via least-squares estimation. ## background language models as conversational agents. we are interested in generating conversational responses with an autoregressive language model in a multi-sensory setting. we treat a conversational language model as an agent with a policy π ϕ @xcite @xcite , which is parameterized by ϕ. the utterance generated at turn t, given access to the textual dialogue history s t is defined to be the action a t . to be more specific, the dialogue until turn t-1 is defined as s 1 , a 1 ..., s t 2 , a t-2 , s t-1 = s [:t-1] , for brevity we will call this s [:t-1] = s t . therefore, the auto-regressive llm policy, π ϕ (s t ), takes in as input s t and outputs a distribution over a t . reinforcement learning with human feedback (rlhf). rlhf is commonly used to adapt an agent π ϕ to be aligned to human feedback. given a reward function which can gauge the quality of individual generated utterances, we can perform adaptation via reinforcement learning with human feedback (rlhf) @xcite @xcite . for turn t, our reward function r θ (s t , a t ) parameterized by θ takes in as input the context utterance s t and the generated response a t to predict the reward at the utterance level. it is common to use a kl term to penalize the policy from diverging from the pretrained model, resulting in the following objective: ## methods: geli the reward function r θ in standard adaptation techniques relies on intermediate fine-grained annotations, requiring manual human annotations at each generated utterance. however, in many longterm dialogue settings there is only a single global explicit (ge) annotated reward for each session. given a trajectory of the multi-turn dialogue τ , the global explicit reward r ge (τ ) is a scalar reward at the end of the interaction, such as how positively the user felt about the conversation. this ge reward can be decomposed via sum decomposition (more details in sec. 4.1) with the ge loss function l ge . a core novelty of our proposed geli approach is that the decomposition of the ge reward will be guided by some local implicit (li) feedback. concretely, in many dialog applications/datasets of interest there are rich multimodal signals, which is can provide intermediate signals that are useful for the decomposition of the single global explicit reward. we thus perform crossmodal distillation of the signals from such multimodal signals into the individually decomposed text-only reward function via the li loss function l li (more details in sec. 4.2). in practice, our reward function r θ is optimized with a joint objective which enables the (1) redistribution of the global explicit (ge) reward and (2) inclusion of local implicit (li) reward signals as a reward shaping function. in the following sections, we share more details about the global explicit decomposition and local implicit crossmodal reward shaping. ## experiments in this section, we describe our experiments to evaluate our proposed geli framework which performs reward function training with global explicit reward decomposition and local implicit visual then, we utilize the decomposed reward function to update the language model, where the language model generates utterances and the reward function assigns a score to be optimized via ppo @xcite . best viewed zoomed and in color. ## results in this section, we discuss the quantitative results and human evaluation of our experiments. ## discussion we describe components of geli with ablations and further analysis and visualizations. ## conclusion we introduce geli, which automatically decomposes a single global explicit post-interaction score, incorporating local implicit feedback from multimodal behaviors. geli performs global alignment of multi-turned interactions by locally reward-ing parts of the interaction, shaped by multimodal local implicit feedback. our proposed approach complements previous alignment approaches, such as rlhf, alleviating the need for fine-grained manual reward annotations.