CPO : Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization (2025)

Abstract: Reinforcement Learning Fine-Tuning (RLFT) has achieved notable success in tasks with objectively verifiable answers (e.g., code generation, mathematical reasoning), yet struggles with open-ended subjective tasks like role-playing dialogue. Traditional reward modeling approaches, which rely on independent sample-wise scoring, face dual challenges: subjective evaluation criteria and unstable reward signals. Motivated by the insight that human evaluation inherently combines explicit criteria with implicit comparative judgments, we propose Comparative Policy Optimization (CPO) . CPO redefines the reward evaluation paradigm by shifting from sample-wise scoring to comparative group-wise scoring. Building on the same principle, we introduce the CharacterArena evaluation framework, which comprises two stages: (1) Contextualized Multi-turn Role-playing Simulation , and (2) Trajectory-level Comparative Evaluation . By operationalizing subjective scoring via objective trajectory comparisons, CharacterArena minimizes contextual bias and enables more robust and fair performance evaluation. Empirical results on CharacterEval, CharacterBench, and CharacterArena confirm that CPO effectively mitigates reward ambiguity and leads to substantial improvements in dialogue quality.

## introduction role-playing dialogue systems aim to support immersive multi-turn interactions by simulating specific character personas @xcite . a core challenge lies in generating responses that are not only coherent with the character profile but also rich in narrative appeal and stylistic diversity @xcite . existing approaches primarily adopt supervised fine-tuning (sft) @xcite over high-quality dialogue corpora @xcite , following an imitation learning paradigm. while sft has proven effective to a degree, it is inherently constrained by the distribution of the training data-leading to overfitting on observed dialogue patterns, limited creative flexibility, and often underwhelming performance in delivering truly engaging and immersive role-playing experiences @xcite . ## preliminary this section introduces the group relative policy optimization (grpo) algorithm @xcite , a reinforcement learning method commonly employed in the rl fine-tuning (rlft) stage of llms. grpo optimizes the policy by leveraging relative reward comparisons among a group of generated responses. formally, let q denote the input query, and o represent the response generated by a policy. we denote the current and old policy models as π θ and π θ old , respectively. for a given query q, grpo samples a set of g responses, {o 1 , o 2 , . . . , o g }, from the old policy π θ old . a reward model rm then assigns a scalar reward to each response, resulting in g corresponding rewards: the grpo objective is then defined as: @xcite where ## experimental setup in this section, we provide a brief overview of the datasets (section 5.1), baselines (section 5.2), implementation details, and evaluation protocols (section 5.4). the comprehensive experimental setups are available in appendix a. ## conclusion this paper proposes comparative policy optimization (cpo), a simple yet effective reinforcement learning framework that addresses the challenge of reward ambiguity in open-ended role-playing dialogue. cpo replaces conventional sample-wise reward estimation with comparative group-wise scoring, aligning more closely with human evaluative behavior. to facilitate this, we introduce character-arena evaluation framework, which enables fairer trajectory-level comparisons under shared context. experiments on charactereval, characterbench, and characterarena show that cpo outperforms existing rlhf methods. this work offers a novel and efficient pathway for reward modeling in subjective and creative tasks.