ABSE val: An Agent-based Framework for Script Evaluation (2024)

Abstract: Recent research indicates that large language models (LLMs) possess a certain degree of script planning capability. However, there is still a lack of focused work on evaluating scripts generated by LLMs. The evaluation of scripts poses challenges due to their logical structure, sequential organization, adherence to commonsense constraints, and open-endedness. In this work, We introduced a novel script evaluation dataset, MCScript, consisting of more than 1,500 script evaluation tasks and steps, and developed an agent-based script evaluation framework, ABSEval, to collaboratively evaluate scripts generated by LLMs. Our experiments demonstrate that ABSEval provides superior accuracy and relevance, aligning closely with human evaluation. We evaluated the script planning capabilities of 15 mainstream LLMs and provided a detailed analysis. Furthermore, we observed phenomena like the key factor influencing the script planning ability of LLM is not parameter size and suggested improvements for evaluating open-ended questions.

## introduction script is a structure that describes an appropriate sequence of events in a particular context @xcite . in daily routines, individuals often rely on meticulously outlined steps to realize their objectives. for instance, figure 1 illustrates the process of opening a can with a spoon. recent studies have applied llms to script-related tasks, demonstrating that these models have scripting knowledge inside it @xcite , and they can effectively decompose high-level tasks @xcite . however, scripts generated by llms may contain errors, making it crucial to evaluate the quality of these llm-generated scripts. a script is a predetermined, stereotyped sequence of actions that define a well-known situation @xcite , which is not only logically and sequentially organized but also adheres to commonsense. script evaluation is to evaluate whether a script meets the aforementioned characteristics. additionally, entirely different steps can achieve the same goal, highlighting the open-ended nature of script tasks. traditional approaches to script evaluation, such as manual evaluation, require considerable time and expense @xcite . automated evaluation methods like bertscore @xcite and rouge @xcite assess script correctness by calculating semantic similarity which is a struggle to evaluate the sequential order of scripts. these methods require a gold answer for comparison, but it is difficult to obtain a gold answer for scripts. furthermore, these methods have been shown to exhibit a relatively weak correlation with human judgment @xcite . recent breakthroughs achieved by llms spurred a wave of research utilizing llm as evaluator @xcite @xcite . even though a single llm has demonstrated the ability to serve as an evaluator, recent research indicates that employing multiple llms can enhance evaluation performance @xcite . assigning distinct roles to llms leads to more effectively identifying problems in text @xcite . existing script datasets are not sufficiently close to the tasks encountered in real-life scenarios, this paper introduces the multi-constrained script planning dataset, i.e., mcscript, which includes more than 1,500 real-life script planning tasks and steps. in addition, we propose the agent-based script evaluation framework (abseval), an evaluation system that integrates answer synthesize agent, critic agent, execute agent and commonsense agent to comprehensively evaluate the scripts based on their different characteristics. we designed an answer synthesize agent to act as a learner, learn scripts generated by llms being evaluated, and produce a more refined answer. then, a critic agent compares the scripts under evaluation with the gold answer provided by the answer synthesize agent, identifying mistakes such as missing, redundant, and duplicate steps. moreover, an execute agent verifies whether the scripts meet the implicit constraints of tasks, achieve the desired goals, and maintain a logical sequence by executing each step of the scripts. finally, a commonsense agent assesses whether each step of the script conforms to commonsense. this paper evaluated 15 widely used llms and analyzed their script planning capabilities. from the evaluation results, we observed some interesting phenomena, like the fact that the key factor influencing the script planning ability of llm is not the llm's parameters, providing gold answers within appropriate metrics can improve the assessment performance of open-ended questions, etc. our contributions are as follows: 1) we developed a high-quality script evaluation dataset mc-script, which simulates a real-world situation by adding multiple constraints and contains over 1,500 script tasks and answers. 2) we propose abseval, an agent-based evaluation framework that exhibits superior alignment with human evaluations compared to current script assessment methods. 3) using abseval, we assessed the script planning capabilities of 15 llms, offering insights into the advancements in llms' script planning abilities. ## data construction currently, multiple large-scale script datasets are developed via crowdsourcing or automatic methods @xcite @xcite . however, these datasets concentrate on abstract tasks (e.g., create a decision tree.). we aim to create a set of evaluation data that is more closely aligned with real-life specific tasks (e.g., create a decision tree on computer to help you choose a holiday destination.). we utilized wiki-how @xcite , a comprehensive database of how-to guides on a wide range of subjects, as the primary source for our data. from this resource, we selected abstract questions across ten different topics as shown in figure 2 . as is shown in obtain abstract task: how to buy disney world tickets add constraints: online, for a family of four, during peak season. generate constraint question: research and purchase disney world tickets online for a family of four during peak season. ## evaluation methodology this section provides an in-depth explanation of the abseval evaluation framework. the discussion includes a breakdown of the evaluation metrics, the components in the evaluation framework, and a detailed explanation of the entire evaluation process. the overall workflow is illustrated in figure 3 . ## deepthinking abseval we present the performance of all llms be evaluated across various metrics in figure 11 in the appendix. to enhance the clarity of our observations, we employ a consistent color scheme to delineate llms within the same series (e.g., llama3 is shown in red), with varying shades denoting differences in llm parameters. our analysis has several interesting observations. ## conclusion in this study, we introduced a new script evaluation dataset, mcscript, comprising over 1,500 script tasks and steps. we proposed a more fair, fine-grained, and human-aligned script evaluation method known as abseval. by utilizing abseval, we conducted a comprehensive analysis of the script planning abilities of 15 current llms and identified the shortcomings of existing script evaluation methods. our discussion and analysis provide insights for the evaluation of open-ended tasks similar to scripts. our objective is to establish a new framework within the llm community for assessing and analyzing the script planning capabilities of llms.