From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLM s-based Applications (2024)

Abstract: Evaluating large language models (LLMs) is fundamental, particularly in the context of practical applications. Conventional evaluation methods, typically designed primarily for LLM development, yield numerical scores that ignore the user experience. Therefore, our study shifts the focus from model-centered to human-centered evaluation in the context of AI-powered writing assistance applications. Our proposed metric, termed “Revision Distance,” utilizes LLMs to suggest revision edits that mimic the human writing process. It is determined by counting the revision edits generated by LLMs. Benefiting from the generated revision edit details, our metric can provide a self-explained text evaluation result in a human-understandable manner beyond the context-independent score. Our results show that for the easy-writing task, “Revision Distance” is consistent with established metrics (ROUGE, Bert-score, and GPT-score), but offers more insightful, detailed feedback and better distinguishes between texts. Moreover, in the context of challenging academic writing tasks, our metric still delivers reliable evaluations where other metrics tend to struggle. Furthermore, our metric also holds significant potential for scenarios lacking reference texts.

## introduction you can't manage what you can't measure well. @xcite with the continuous development of large language models (llms) such as chatgpt 1 , gpt-4(openai), and llama @xcite , a plethora of application research and development work based on llms has emerged. edit distance between "kitten" and "sitting" is 3 1.@xmath0.@xmath1.@xmath2 ## related work the text evaluation methods can be categorized into human evaluation and automated approaches. human evaluation is widely recognized as the most natural way to evaluate the quality of a given text, which often involves human annotators and qualitative analyses @xcite . this method is often expensive and timeconsuming work and requires extensive domain expertise for domain-specific scenarios. on the other hand, current automated evaluation methods tend to generate a comprehensive score that is facilitated in comparing new models with established stateof-the-art approaches. these include metrics such as rouge @xcite , bleu @xcite , bertscore @xcite , mover-score @xcite , bartscore @xcite , and discoscore @xcite typically compute a similarity (or dissimilarity) score between a model-generated text and a reference text. large language models have been adeptly utilized for roles such as aiding in data annotation @xcite and delivering feedback that mirrors human preferences @xcite @xcite . for the evaluation stage, @xcite found that the llm evaluation is consistent with the human evaluation results. the gptscore @xcite has been proposed to score the model-generated text. similarly, @xcite also studied the efficacy of llms as multi-dimensional evaluators. in conclusion, current metrics tend to yield a comprehensive score that detaches the task context { "action_name": "simplify", "revision_description": "simplified the text by removing details ", "revision_level": "reference" our proposed metric shifts the text evaluation to a human-centered perspective, which incorporates the iterative process of user-driven text revision. ## revision distance as depicted in figure 2 , we frame the context of ai-powered writing assistance, with llms serving dual functions: as the proxy of the user (llm u ser ) and as the generator (llm gen ). the llm gen is the pivotal component of the writing assistant application. given the user input content, the llm gen generates a y draf t , such as emails, letters, articles, and "related work" sections. the d revision quantifies the number of edits from llm u ser (y draf t , y ). to avoid the generated revision focusing excessively on surface-level textual features, we have explicitly required llm u ser to focus on coherence and logical flow, as shown in appendix d. additionally, we have calibrated the revision generation process by setting the temperature parameter to 0, aiming to reduce the likelihood of generating unstable outputs. we also separated complex revisions that contained multiple revisions, such as "simplify and reorganize," into individual actions to ensure consistency in the granularity of revisions. by setting a consistent scope and clear objectives for revisions, our metric promotes a uniform revision process that is less prone to variance, thus enhancing reproducibility and reliability. for the reference-based evaluation setting, we utilize the human-written text or chatgpt output as the ground truth. the llm u ser is designed to produce structured revision edits, improving the consistency of the y draf t to the ground-truth text y . in scenarios where no ground truth text is available, we require the llm u ser to refine the given text towards an ideal form, as envisioned by the llm u ser itself. 3 these revision edits are produced to improve y draf t to closer align with the ideal version, which mimics the revision process of human writers. to validate the utility of our proposed metric, we have constructed two distinct datasets to address both the easy-writing task and the challengewriting task. the challenge-writing task refers to the scenario that requires heavy knowledge reasoning and complex concept understanding. for the easy-writing task, we use emails, letters, and article generation as a testbed. for the challenge-writing task, we employ academic writing as the testbed. the test dataset details in this evaluation setting are described in appendix a. for the more challenging writing task, it offers stable evaluations and better distinguishes model quality, whereas other metrics struggle. the limited input length of bert-score, capped at 512 tokens, precluded its use in the easy-writing task where many texts exceeded this limit. ## conclusion with the rapid advancement of llm-based applications, the pivotal question arises: "how can we evaluate llm-based applications from a humancentered perspective?" existing evaluation metrics, typically used for model development, merely yield a context-independent numerical score, lacking user relevance. our research shifts text evaluation from a predominantly model-centered perspective to a human-centered one. using the llm-powered writing assistant as a test scenario, we take a comprehensive experiment on diverse writing tasks to validate the effectiveness and reliability of our "revision distance" metric. this metric converts text evaluation into contextualized text revisions, highlighting textual discrepancies and offering users a detailed, transparent rationale for the scores. our findings demonstrate the metric's applicability and dependability in both reference-based and reference-free contexts.