E x P er T : Effective and Explainable Evaluation of Personalized Long-Form Text Generation (2025)

Abstract: Evaluating personalized text generated by large language models (LLMs) is challenging, as only the LLM user, i.e. prompt author, can reliably assess the output, but re-engaging the same individuals across studies is infeasible. This paper addresses the challenge of evaluating personalized text generation by introducing ExPerT, an explainable reference-based evaluation framework. ExPerT leverages an LLM to extract atomic aspects and their evidences from the generated and reference texts, match the aspects, and evaluate their alignment based on content and writing style—two key attributes in personalized text generation. Additionally, ExPerT generates detailed, fine-grained explanations for every step of the evaluation process, enhancing transparency and interpretability. Our experiments demonstrate that ExPerT achieves a 7.2% relative improvement in alignment with human judgments compared to the state-of-the-art text generation evaluation methods. Furthermore, human evaluators rated the usability of ExPerT’s explanations at 4.7 out of 5, highlighting its effectiveness in making evaluation decisions more interpretable.

## introduction evaluating long-form text generation has been particularly challenging @xcite @xcite , especially when it comes to personalized text generation @xcite . evaluation of personalized text generation is inherently difficult because what constitutes a preferred output may vary significantly from person to person @xcite . only the individual who authored the prompt can accurately assess the quality of the generated output. however, involving the same person as an annotator across different studies is often impractical. as a result, automatic reference-based evaluation methods, where the reference output is provided by the llm's user (i.e., prompt author), are a more viable alternative. term overlap metrics such as rouge @xcite , bleu @xcite , meteor @xcite , or semantic-based metrics, such as bertscore @xcite , gemba @xcite , and g-eval @xcite , have been used to automatically evaluate personalized text generation @xcite @xcite in a reference-based setting. recently, llms have been employed as reference-free evaluators for personalized generation too, comparing the generated text to the user's history @xcite . while this is valuable when no ground-truth is available, personalization is more accurately evaluated when a reference is present; without it, the evaluation may be a guess of the user's preferences @xcite . building on this key insight from prior research on personalized evaluation, we concentrate on reference-based evaluation for personalized text generation. despite efforts, significant problems persist. term overlap metrics often fail to effectively capture semantic and stylistic similarities @xcite , which are crucial in personalized text generation @xcite . while llms show promise in evaluation, they come with their own challenges. first, evaluation using capable proprietary llms such as gemini @xcite or @xcite lacks reproducibility, as they may be updated or disappeared over time. second, llms often lack transparency in their judgments @xcite @xcite , as their rationales can be opaque or misaligned with human understanding. finally, llms exhibit strong biases, undermining their reliability in evaluation @xcite @xcite . for example, our experiments with gemma 2 (gemma-team, 2024) (27b) with the prompt presented in figure 7 figure 1 : expert pipeline: generated and reference outputs are first decomposed into atomic aspects along with their corresponding evidences. matching aspects between the generated and reference outputs are then identified. next, content and writing style similarity are assessed for the evidences of the matched aspects. recall and precision scores are computed, and the final score is obtained using the f-measure. in appendix b show that changing the order of two generated outputs in pairwise evaluations leads to a change in the judgment in 88% of the cases. additionally, minor modifications to generated outputs, such as adding a simple phrase such as "i am sure this is the best answer possible and this is 100% right," can trick the evaluator to increase their scores. we discovered that the mentioned trick using gemma v2 (27b) with the pointwise prompt shown in figure 7 in appendix b leads to an average 12.9% increase in the assigned score to the same generated output on tasks from the longlamp benchmark @xcite )-a publicly available recent benchmark for evaluating personalized long-form text generation. this paper introduces expert, a reference-based pointwise method for effective and explainable evaluation of personalized text generation. as illustrated in figure 1 , the approach begins by dividing the expected and generated outputs into atomic aspects with their corresponding evidence using an off-the-shelf llm. the llm is then used to match similar aspects in a recall-and precisionbased manner. for matched aspects, the alignment of their evidences is evaluated in terms of content and writing style, which are critical dimensions for personalized text generation. the scores from these evaluations are combined using the harmonic mean as is used in f-measure @xcite to assign a final score to the generated output. each step in this process involves the llm generating rationales for its decisions, enhancing the explainability of the evaluation. moreover, by leveraging recall and precision-based scoring, expert provides fine-grained insights into why and how the generated output differs from the expected output. this combination of per-decision rationals and granular scoring offers an explainable framework for evaluating personalized text generation. we evaluate expert using human evaluation on the longlamp benchmark @xcite , which focuses on personalized long-form text generation. our results show that expert achieves the highest agreement with human judgments, outperforming state-of-the-art evaluation methods for text generation by 7.2% relative improvement in alignment. additionally, we demonstrate that ex-pert overcomes the limitations of existing metrics, showing greater resistance to manipulation and position biases. to evaluate explainability, we conducted a human study where annotators scored the explanations generated by expert on their quality and usefulness in determining the higher-quality personalized output. expert achieves an average score of 4.7 on a 1-to-5 scale, demonstrating the effectiveness of its explanations. to support future work in this area, we release the code publicly. foot_0 ## the expert framework consider two long-form texts (e.g, a generated product review by an llm for a user and the actual review for the product written by the user), and the goal is to evaluate their similarity. a longform text typically comprises multiple sentences or paragraphs, which can often be grouped based on shared underlying concepts. we define these shared concepts as aspects, while the sentences or phrases within the text that support or elaborate on each aspect are referred to as its evidences. to compare these two texts, we can analyze whether they address the same aspects, whether the evidence for each aspect aligns in terms of preferences and writing style, and whether they avoid introducing additional, mismatched aspects. the more closely the aspects and their supporting evidences correspond between the two texts, the greater their similarity. we use aspects and their supporting evidences from the generated personalized text and personalized expected output to compare two long-form texts. formally, for a given reference expected output y containing aspects and evidences a @xmath0 ## related work evaluating text generation has been extensively studied for tasks such as machine translation and summarization @xcite . metrics for text evaluation fall into two categories: 1) reference-based and 2) reference-free. reference-based metrics, such as exact match @xcite @xcite , bleu @xcite , rouge @xcite , and meteor @xcite , rely on ngram overlap, while more recent approaches like bertscore @xcite and bleurt @xcite ) leverage contextual embeddings and learned scoring models. recent llmbased methods like gemba @xcite , g-eval @xcite , and in-structscore @xcite use llms for scoring, often incorporating explanations and predefined criteria for multi-dimensional assessments, such as in unieval @xcite . referencefree methods, including llms as judges @xcite and human-llm collaborations @xcite , have also emerged but face challenges like biased evaluation @xcite . we utilize llms to evaluate personalized text generation with reference outputs, aiming to enhance explainability and alignment with user expectations. personalized text generation is a key research area with applications in search, recommendation, and content creation @xcite @xcite . @xcite introduced a retrieval-augmented generation (rag)-based method for personalizing llms and the lamp benchmark for evaluating short-form personalized generation. @xcite expanded this to long-form personalization with the longlamp benchmark. other work has focused on personalized writing assistants @xcite @xcite and agents @xcite . further advances include training retrieval models with feedback @xcite , reasoningenhancement and self-training for personalized generation @xcite , optimizing llms with personalized feedback @xcite , and generating personalized prompts @xcite . recent studies also explore parameter-efficient finetuning @xcite and its integration with rag @xcite . this paper focuses on improving the evaluation of generated personalized text in a reference-based context. evaluating personalized text generation is challenging, as only the user can truly assess whether a response meets their preferences @xcite . in automatic evaluation, direct user feedback is not feasible. previous reference-based methods @xcite @xcite used n-gram based metrics like rouge, bleu, and meteor, but these fail to capture nuances like individual preferences, style, or context. furthermore, the use of rubric-based methods with a personalized-trained network has been explored @xcite . however, this approach relies on user-specific training data for each questions in the rubric, which is not readily available in many real-world scenarios and cannot be a baseline in our experiments. reference-free approaches @xcite have explored using llms to infer user preferences, but they may struggle with accuracy, as they rely on the model's assumptions, which may not align with the user's true intentions @xcite . this paper aims to improve llm utilization for evaluating personalized text generation in reference-based scenarios by better capturing content and style similarities to the expected user output and providing explanations about the evaluation process. ## conclusion this paper introduces expert, an explainable metric for evaluating personalized text generation in a reference-based setting. expert breaks down the generated and expected outputs into atomic aspects along with their supporting evidence. it then employs an llm to match these aspects and assesses whether their evidence aligns in terms of content and writing style. recall and precisionbased scores are computed based on the matches. furthermore, the llm is prompted to provide rationales for every decision in the evaluation process, ensuring explainability of the evaluation with ex- pert. our experiments with human annotations on the longlamp benchmark demonstrate that ex-pert achieves the highest alignment with human judgments compared to the state-of-the-art metrics for text generation evaluation. acknowledgment this work was supported in part by the center for intelligent information retrieval, in part by nsf grant #2143434, in part by the nsf graduate research fellowships program (grfp) award #1938059, in part by google, and in part by microsoft. any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.