CINEMETRIC : A Framework for Multi-Perspective Evaluation of Conversational Agents using Human- AI Collaboration (2025)

Abstract: Despite advances in conversational systems, the evaluation of such systems remains a challenging problem. Current evaluation paradigms often rely on costly homogeneous human annotators or oversimplified automated metrics, leading to a critical gap in socially aligned conversational agents, where pluralistic values (i.e., acknowledging diverse human experiences) are essential to reflect the inherently subjective and contextual nature of dialogue quality. In this paper, we propose CINEMETRIC, a novel framework that operationalizes pluralistic alignment by leveraging the perspectivist capacities of large language models. Our approach introduces a mechanism where LLMs simulate a diverse set of evaluators, each with distinct personas constructed by matching real human annotators to movie characters based on both demographic profiles and annotation behaviors. These role-played characters independently assess subjective tasks, offering a scalable and human-aligned alternative to traditional evaluation. Empirical results show that our approach consistently outperforms baseline methods, including LLM as a Judge and as a Personalized Judge, across multiple LLMs, showing high and consistent agreement with human ground truth. CINEMETRIC improves accuracy by up to 20% and reduces mean absolute error in toxicity prediction, demonstrating its effectiveness in capturing human-like perspectives.

## introduction what makes a conversation good? if we ask ten people, we might get ten different answers. as shown in figure 1 (human evaluators), a response that one person finds relatively empathetic might strike another as less empathetic or even offensive. these differences highlight that the quality of the dialogue is inherently subjective and multifaceted @xcite . yet the way we evaluate conversational systems today often assumes that there is an objective fact (by using automatic evaluation in the cinemetric evaluation, each character reflects a unique personality profile, resulting in diverse ratings and subjective commentary across various dimensions. metrics) or a single definitive measure of quality (by aggregating ratings from crowdworkers or domain experts) to rate qualities such as coherence or overall satisfaction @xcite . however, this "one-size-fits-all" human annotation approach has its own blind spots. it implicitly assumes a homogeneous pool of evaluators, averaging out individual differences. in reality, the background, values, cognitive styles, or personality of an annotator may significantly influence how they perceive the quality of a conversation @xcite . thus, both purely automatic metrics and aggregated human ratings risk missing the plurality of perspectives inherent in dialogue quality, struggling to capture the nuanced dimensions in a scalable and robust way. to address this gap, we introduce cinemet-ric, a novel framework grounded in perspectivist principles for pluralistic alignment @xcite . this idea breaks away with from the singularity of current methods by embracing pluralism @xcite . as illustrated in figure 1 , the core idea behind this framework is to simulate a panel of diverse evaluators (i.e., movie and public characters), each embodied as a distinct perspective-driven persona, capable of assessing a conversation through different lenses. these personas are defined by interpretable attributes such as gender, personality traits (e.g., mbti types), thinking style (e.g., analytical vs. intuitive), and more. we then task the llm to role-play these personas and evaluate conversational turns along multiple dimensions, including but not limited to toxicity, persuasiveness, clarity, and empathy. this work is guided by the following research questions: rq1. how can the perspectivist role-playing of diverse personas by large language models enhance the pluralistic alignment of conversational agents? rq2. how can human-ai collaboration be used to design an evaluation framework that captures the diversity of human values and preferences in llm outputs? ## background in this section, we discuss how perspectivism can be operationalized by the role-playing technique. we investigate the current approaches for evaluating conversational systems, and finally, we explore the pluralistic alignment in conversational agents. ## methodology: cinemetric the cinemetric framework proceeds in three steps as shown in figure 2 : (i) perspective source (i.e., selecting representative human evaluators from multi-perspective datasets (section 3.1)), (ii) perspective making (i.e., creating a set of "persona" movie characters and their corresponding perspectives for each sampled human evaluator (section 3.2)), and (iii) perspective taking (i.e., leveraging the movie characters and their perspectives by llms to make predictions on held-out human evaluator annotations (section 3.3)). ## experimental setup to comprehensively evaluate the effectiveness of our proposed approach, we conduct experiments across a diverse set of tasks and models, employing various techniques for comparison. ## results & analysis we evaluate the performance of cinemetric and competing methods on two datasets across a diverse set of llms. as shown in tables 1 and 2 , cinemetric consistently outperforms all baseline approaches, demonstrating its robustness and adaptability across different model families and evaluation formats. ## conclusion in this work, we introduced cinemetric, a novel evaluation framework that operationalizes perspectivist alignment by simulating diverse evaluative standpoints through llm role-play. by drawing on a rich set of character-based personas, our approach provides a scalable, pluralistic alternative to monolithic evaluation practices. through comprehensive experiments on two diverse benchmarks and across four leading llm families, we demonstrated that cinemetric consistently outperforms existing evaluation strategies in both accuracy and mae. our results highlight the value of perspective-driven simulation in enhancing the human-likeness and value-diversity sensitivity of automated evaluations. in particular, cinemet-ric achieves stronger agreement with human judgments than standard llm-based or personalized-llm evaluation baselines.