Can LLM be a Personalized Judge? (2024)

Abstract: As large language models (LLMs) gain widespread adoption, ensuring they cater to diverse user needs has become increasingly important. While many researchers have studied LLM personalization and role-playing, they primarily use LLM-as-a-Judge for evaluation without thoroughly examining its validity. This paper investigates the reliability of LLM-as-a-Personalized-Judge—asking LLMs to judge user preferences based on persona. Our results suggest that LLM-as-a-Personalized-Judge is less reliable for personalization than previously believed, showing low agreement with human ground truth. We observed that the personas provided to the LLM often have limited predictive power for the tasks, leading us to introduce verbal uncertainty estimation. We find that powerful LLMs are aware of the certainty of their prediction and can achieve high agreement with ground truth on high-certainty samples, indicating a promising approach for building reliable and scalable proxies for evaluating LLM personalization. Our human annotation reveals that third-person crowd worker evaluations of personalized preferences are even worse than LLM predictions, highlighting the challenges of evaluating LLM personalization.

## introduction as large language models (llms) gain widespread adoption among global users with diverse backgrounds, it is imperative to ensure these models designed to reflect their values and preferences @xcite . however, the current alignment process often assumes a homogeneous set of human preferences and ignores individual perspectives, even in context-dependent, subjective tasks @xcite . therefore, efforts have been made to fine-tune llms to encode individual preferences or enhance role-playing capabilities @xcite @xcite @xcite with "llmas-a-judge" as the main evaluation metric @xcite , often without adequate validation. despite "llm-as-a-judge" showing high agreement with human annotators in many tasks, its effectiveness for personalization tasks remains largely unscrutinized. mt-bench @xcite includes a role-playing component but only considered simplistic personas, such as "imagine you are a doctor," without addressing more complex personas that encompass demographics, user descriptions, and prior interactions-settings increasingly employed in recent research. furthermore, a persona description may not always be contextually relevant. knowing that someone is a doctor, for instance, provides little insight into their favorite types of beverages. we refer to this issue as the persona sparsity issue. 1 in this paper, we examine the validity of llmas-a-judge for personalization, where the objective is to generate personalized outputs based on a given user persona (see figure 1 ). we assess performance on tasks where ground truth data is available, including prism @xcite , opinionqa @xcite , public reddit @xcite , and empathetic conversation @xcite . to address the issue of persona sparsity, we then propose a verbalized confidence component into the judge pipeline. by verbalizing its own certainty levels, an llm can assign lower certainty to samples for which it perceives insuffi-figure 1 : overall workflow of personalized judge. given a subjective question and two distinct responses, we ask an llm to infer the preference of a real user based on a user persona. we also ask the llm to estimate its certainty level in this prediction. the inferred preference is then compared against the user's self-reported ground truth to evaluate the performance of the judge. in the context of llms, personalization has become even more critical due to the vast, diverse, and ever-growing user base. the necessity to align llms to a pluralistic set of user needs is discussed in @xcite . however, the current alignment processes typically assume a single set of human preferences and researchers are just beginning to explore methods to address the varied preferences and values of different users, either at the collective level @xcite or at individual level @xcite @xcite @xcite . our study focuses on the evaluation aspect of personalized alignment approaches. a challenging issue in this domain is the definition of personas. not all variables are universally applicable or useful @xcite . for instance, while knowing an individual's profession as a doctor may offer some insights about this individual, it does not necessarily inform us about their preferred types of beverages. ideally, we would include demographic, behavioral, and contextual factors that are relevant to the specific task at hand. however, defining the relevant set of variables a priori is inherently difficult. additionally, even if surveys are designed to gather this information, acquiring such detailed information on a large scale is often impractical and can frequently result in incomplete responses. we refer to this challenge as the persona sparsity issue. in practice, this means that in some cases, we cannot reasonably infer preferences based on the available persona information and should therefore deprioritize such samples. this motivates us to explore verbalized confidence estimation as a method to filter out cases where persona information is insufficient for the llm-judge to make well-informed judgments. evaluation of llms evaluating natural language generation (nlg) systems is challenging, but the evaluation of llms arguably presents even greater difficulties. this is due to the advanced capabilities and versatility of current-generation llms, as well as the diverse ways in which they are employed in practice. recently, "llm-as-judge" @xcite ) is introduced as a versatile and reference-free evaluation metric that shows high agreement with human annotators on various nlp tasks. despite concerns over issue such as positional bias, self-enhancement bias, length bias, sensitivity to prompting, and cost @xcite @xcite @xcite , it is becoming the new paradigm for llm evaluation @xcite @xcite , and have been used in llm personalization works such as @xcite ; @xcite . however, there is little work in validating llmas-a-personalized-judge. while mt-bench @xcite included a role-playing component, it focused only on simplistic cases such as roleplaying specific professions and did not account for the complex personas typically used in llm personalization works, encompassing diverse demographics, user descriptions, and prior interactions. our work considers more realistic cases of llm-as-a-personalized-judge and carefully examines its validity. calibration of llms pretrained llms are wellcalibrated but preference tuning can degrade this calibration @xcite . recent studies have shown that verbalized confidence levels in llms are typically more reli-able than token-level confidence scores @xcite . additionally, llms possess some intrinsic capabilities to assess the answerability of questions @xcite . by selectively answering questions when confidence levels are high, we can maintain high levels of predictive accuracy @xcite @xcite . building on these findings, our research introduces verbalized confidence within the context of llm-as-personalized-judge. ## methodology in this work, we study llm-as-a-personalized-judge (figure 1 ), building on @xcite . we condition an llm with a persona profile, which can include information ranging from demographic data and socio-behavioral indicators to free-form user descriptions, as well as any other pertinent details that could enrich the persona profile. using this conditioned persona, we task the llm with selecting the preferred response to a subjective question in a binary choice setting, aiming to reflect the preferences that the persona would likely have. as is done in @xcite , we also consider a setting where a "tie" option is allowed. as highlighted in section 2, persona sparsity can lead to instances where an llm struggles to assess certain questions accurately. however, we hypothesize that the llm possesses some notion of its confidence in these instances. therefore, we instruct the llm to estimate the certainty in its answer. the overall workflow is shown in figure 1 . the prompts used are detailed in appendix a.7, while the experimental setups are described in section 4.2. ## conclusion in this paper, we formalized and examined the validity of llm-as-a-personalized-judge. contrary to previous assumptions, we demonstrated that the standard llm-as-a-judge setting is not sufficiently reliable for personalization tasks, showing low agreement with human ground truth. we identified persona sparsity as a major cause of this unreliability. we then introduced verbalized confidence estimation and found that powerful llms (e.g. gpt-4) are capable of effectively assessing the certainty of their own responses. this led to the observation that high-certainty samples indeed exhibit high accuracy (80%). we additionally conducted a human annotation experiment and found that llm-as-a-personalized-judge achieves comparable accuracy as third-person human judge and surpasses humans on high-certainty samples. while we advocate for the collection of more first-person personalization data, we also believe that a certainty-aware llmas-a-personalized-judge is a promising proxy for evaluation, particularly in cases first-person preference data are not available, provided that personas are as fine-grained as possible. we hope our work helps the community recognize the challenges in evaluating llm personalization and ultimately leads to the development of llms that better serve each individual's preferences and needs.