Hierarchical Evaluation Framework: Best Practices for Human Evaluation (2023)

Abstract: Human evaluation plays a crucial role in Natural Language Processing (NLP) as it assesses the quality and relevance of developed systems, thereby facilitating their enhancement. However, the absence of widely accepted human evaluation metrics in NLP hampers fair comparisons among different systems and the establishment of universal assessment standards. Through an extensive analysis of existing literature on human evaluation metrics, we identified several gaps in NLP evaluation methodologies. These gaps served as motivation for developing our own hierarchical evaluation framework. The proposed framework offers notable advantages, particularly in providing a more comprehensive representation of the NLP system’s performance. We applied this framework to evaluate the developed Machine Reading Comprehension system, which was utilized within a human-AI symbiosis model. The results highlighted the associations between the quality of inputs and outputs, underscoring the necessity to evaluate both components rather than solely focusing on outputs. In future work, we will investigate the potential time-saving benefits of our proposed framework for evaluators assessing NLP systems.

## introduction human evaluation is crucial for assessing the quality, validity, and performance of natural language processing (nlp) systems especially as automatic metrics are usually not sufficient @xcite . human evaluation can deal with complex generated natural language and its nuances such as pragmatics, context and semantics which often requires some expert knowledge @xcite . automatic evaluation may be used to assess individual dimensions (e.g., fluency, accuracy) of natural language, however, may often lose to humans in terms of accuracy and understanding. various methodologies are often employed in human evaluation such as ranking, pairwise compari-son, or a state-of-the-art machine translation metric that was used in @xcite . they can provide valuable insights into the strengths and limitations of an nlp system; however, it is notably timeconsuming and expensive and significant trade-offs may exist in consideration of different goals or requirements @xcite . the human evaluation also comes with its own set of limitations, such as fatigue effect (van der @xcite and inconsistencies between evaluators. the role of human evaluators should also be considered as some tasks may require domain expert knowledge or provide specific training evaluators. there is currently a lack of consensus on which metrics to use for the human evaluation of nlp systems @xcite . as there tend to be different research goals, requirements and task-dependent metrics, there exists the challenge of standardizing human evaluation metrics and essentially reaching an overall consensus. a unique combination of metrics can be used for a more comprehensive assessment depending on the desired objectives. these combinations can be grouped based on different evaluation aspects @xcite . metrics may also vary depending on the task (e.g., machine translation, sentiment analysis) and thus task design can affect the criteria used for evaluation @xcite . to identify gaps in the literature pertaining to human evaluation, we conducted a scoping review to systematically examine various aspects of human evaluation experiments in nlp tasks, including the characteristics of evaluators, evaluation samples, scoring methods, design of evaluation and statistical analysis. the findings of our literature review revealed three significant gaps: (i) the absence of evaluation metrics for nlp system inputs, (ii) the lack of consideration for interdependencies among different characteristics of assessed nlp systems, and (iii) a limited utilization of metrics for extrinsic evaluation of nlp systems. we hope to bridge the aforementioned gaps by providing a standardized human evaluation framework that can be used across different nlp tasks. our proposed framework employs a hierarchical structure that divides the human evaluation process into two phases: testing and evaluation. this division enables evaluators to assess the quality of inputs used by testers when evaluating nlp systems. furthermore, the hierarchical design of the evaluation metric allows for the computation of a composite score that reflects the overall quality of the nlp system. this paper is organized as follows. section 2 presents the analysis from a scoping review that included more than 200 papers published within the last three years in the top 5 nlp venues. the results of the aforementioned analysis informed the development of the proposed hierarchical evaluation framework, which is presented in section 3. section 4 presents the results of adopting the proposed framework for the human evaluation of the machine reading comprehension (mrc) system developed as a part of the human-ai symbiosis model. finally, section 5 concludes the paper. ## hierarchical evaluation framework the review of existing literature identified 3 gaps: the analysis of existing literature revealed that the majority of papers (88%, 153/173) focused solely on an intrinsic evaluation of nlp systems. to avoid conducting an evaluation merely for the sake of it, we suggest that first a clear purpose for an nlp system is defined, and subsequently, an extrinsic evaluation is designed to gauge the systems' performance in fulfilling that specific purpose. additionally, the evaluation of various aspects of nlp systems' outputs (e.g., truthfulness) is usually conducted independently, without providing a composite score for the overall system performance. we suggest adopting a hierarchical approach, where the characteristics of the systems are interdependent, and the evaluation process continues only if the preceding characteristic(s) is deemed satisfactory. conversely, if a characteristic is unsatisfactory, the evaluation can be discontinued, allowing evaluators to save time by not evaluating all characteristics for the low-quality outputs. lastly, to date, the existing literature has focused solely on the human evaluation of nlp systems' outputs, assuming that the inputs provided to these systems were of good quality. however, this assumption may not always hold true. we thus propose a two-phase approach for human evaluation, wherein testers initially assess nlp systems, followed by evaluators who evaluate both the inputs and outputs of the systems. by dividing the evaluation process into two phases, we enable evaluators to also assess the quality of the inputs used by testers during the testing phase of nlp systems. in essence, our hypothesis is that the quality of the outputs may not only be influenced by the system itself but also by the quality of the inputs. in order to address those gaps, we propose a framework as shown in figure 4 . by defining a system's purpose as the first step, our framework supports extrinsic evaluation. the second step is to define interdependencies between the evaluated characteristics and consequently to design a hierarchical evaluation metric that supports calculating a composite score that encompasses the overall quality of an nlp system. namely, the evaluation stops if any of the evaluated characteristics is deemed unsatisfactory and, in this case, the composite score is "bad" as the system did not pass the evaluation. otherwise, if the evaluation goes to the end, then the composite score is "good". we hypothesize that our framework facilitates a shorter evaluation time for evaluators by allowing early termination of evaluation in cases where any evaluated characteristic does not meet satisfactory quality. the third step is to do testing of the system according to the defined purpose. testers are independent of evaluators who evaluate the system's inputs and outputs using the designed hierarchical evaluation metric in the fourth step. this allows for independent evaluation of the system's inputs as well. consequently, our framework enables an examination of whether the quality of a system's outputs is influenced by the quality of its inputs. ## case study: hierarchical evaluation for an mrc system we evaluated a machine reading comprehension (mrc) system using the framework outlined in the previous section. in an mrc system, answers come in the form of short text spans which are directly extracted from the text corpus (i.e., relevant text database). questions asked, on the other hand, need to be relevant to the topic that the text corpus covers, factoid, answerable and mistake-free (i.e., no spelling or grammar mistakes). ## discussion and conclusions in this study, we conducted a scoping review to identify gaps in the literature regarding human evaluation in nlp. the findings revealed three significant gaps that need to be addressed: the lack of evaluation metrics for nlp system inputs, limited consideration for interdependencies among different characteristics of nlp systems, and a scarcity of metrics for extrinsic evaluation. to bridge these gaps and enhance human evaluation in nlp, we proposed a hierarchical evaluation framework. our framework offers a standardized approach that considers both the inputs and outputs of nlp systems, allowing for a more comprehensive assessment. moreover, our hierarchical approach considers the interdependencies among different characteristics of nlp systems. rather than evaluating characteristics independently, our framework emphasizes their interconnectedness and the impact they may have on each other. this approach enables a more holistic evaluation that captures the overall performance of nlp systems. to validate the effectiveness of our proposed framework, we conducted a pilot rct evaluating an mrc system. the evaluation phase of our study involved 10 health coaches who evaluated a total of 387 question-answer pairs generated during the rct. the evaluation metrics developed for inputs focused on aspects such as relevance, factoid nature, answerability, spelling, grammar errors, and difficulty levels of the questions. for outputs, the evaluation criteria included clarity, relevance, clinical accuracy, and usefulness of the retrieved short answers and explanations. the results of the evaluation provided valuable insights into the strengths and weaknesses of the mrc system and demonstrated the practical application of our hierarchical evaluation framework. the findings supported the notion that evaluating both inputs and outputs is crucial for obtaining a comprehensive understanding of the performance and effectiveness of nlp systems. future research should focus on validating the scalability and timesaving benefits of our proposed framework.