Learning Personalized Alignment for Evaluating Open-ended Text Generation (2024)

Abstract: Recent research has increasingly focused on evaluating large language models’ (LLMs) alignment with diverse human values and preferences, particularly for open-ended tasks like story generation. Traditional evaluation metrics rely heavily on lexical similarity with human-written references, often showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences. To address these challenges, we introduce PerSE, an interpretable evaluation framework designed to assess alignment with specific human preferences. It is tuned to infer specific preferences from an in-context personal profile and evaluate the alignment between the generated content and personal preferences. PerSE enhances interpretability by providing detailed comments and fine-grained scoring, facilitating more personalized content generation. Our 13B LLaMA-2-based PerSE shows a 15.8% increase in Kendall correlation and a 13.7% rise in accuracy with zero-shot reviewers compared to GPT-4. It also outperforms GPT-4 by 46.01% in Kendall correlation on new domains, indicating its transferability

## introduction large language models (llms) have recently demonstrated impressive generative capabilities across various tasks, with rapid improvements in language qualities such as fluency and consistency @xcite @xcite . however, evaluating their performance in open-ended generation tasks remains challenging due to the diversity of responses. traditional automatic metrics struggle with the oneto-many problem in open-ended generation @xcite and often show poor correlation with human judgment @xcite ...ben becomes increasingly worried for his mother and reaches out to jesse and isabelle for support but fails. i like plot a because it has an uplifting ending. plot b is sadder. there isn't a happy resolution to the story. i found plot b more interesting because it involves a family dynamic, which adds a layer of complexity to the story. the ending is more relatable and empathetic. ## learning personalized alignment in evaluating open-ended generation we propose an llm-based evaluation model for assessing the alignment between generated content and personal preferences. perse provides a reference-free, interpretable evaluation from a specific reviewer's perspective. given an open-ended generation as the query x and the personal preference p u of reviewer u, the goal is to provide a scalar or pairwise rating y u along with an explanation e u to indicate how well the query aligns with the reviewer's personal preference. the reviewer's preference is defined by their profile, which includes their historical comments p @xmath0, e as demonstrated in figure 2 , perse provides a scalar rating for individual input text and a pairwise rating between two inputs. for the scalar rating, the reviewer's comment serves as the explanation e u , and their rating (ranging from 1 to 10) represents the alignment score y u . for a more interpretable comparison between two inputs, we conduct a fine-grained assessment across a set of aspects a. for each aspect, perse determines the winner between the two inputs and uses this as the rating. the aspects and their ratings collectively form the explanation e u . the win rate across all aspects is considered the alignment score y u . ## experiment we introduce our two datasets in section 4.1. in section 4.2, we describe the implementation of perse and several baselines. further details on the fine-tuning process are listed in appendix d. ## results and analysis we report the performance of the scalar rating on the test set of per-mpst and the pairwise rating on per-doc. the reviewers in the test set u test have no overlap with those in the instruction-tuning set u ift . ## conclusion and discussion in this paper, we focus on learning personalized alignment in evaluating open-ended generation. we introduce perse, an llm-based evaluation model that provides an interpretable evaluation from the perspective of an unseen reviewer. it infers the reviewer's preferences based on their profile and applies these preferences to the evaluation of open-ended generation. in addition to providing a score, perse also offers a detailed explanation. by being instruction-tuned on personalized alignment data, the llama-2-based perse outperforms gpt-4 in both scalar and pairwise ratings. our comprehensive analysis of personalized alignment underscores the importance of personalized fine-tuning to avoid over-alignment with common human values, often a result of rlhf. the interpretability of perse makes it particularly wellsuited for personalized generation and recommendation systems. reviewer profile query [review] i have to admit however, that it was pretty interesting, and it made for some pretty good viewing. its not a happy movie by any means, but it is thought provoking. wealth, and the lack thereof is the main focus, and how it can distort ones sense of right and wrong...[score] 8 perse response [the start of plot 0] two police officers respond to a car crash, where two people are killed, but one miraculously survives. the story then cuts to the title credits. [the end of plot 0] [review] folks, let me tell you, this is one great weird little thriller, not much blood, but a whole lot of creepiness, a great story, and very british. [score] 10 [the start of plot 1] angela is a call girl who is terrorized by a man named frank, who was just released from prison. [the end of plot 1] [review] the trouble with anthology films like this, is that you're usually gonna get stories that are better then the others. [score] 7 richard hayes is a financial whiz who's about to earn a million dollars through a bonds scheme. he has a wife, julianna, and daughter, grace, and is engaged in an affair with sophia taylor, a southern belle gold digger... after much turmoil, richard and sophia are both acquitted of all charges and are allowed to leave the courtroom, where richard thanks the jury for their decision.