Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation (2024)

Abstract: Human evaluation serves as the gold standard for assessing the quality of Natural Language Generation (NLG) systems. Nevertheless, the evaluation guideline, as a pivotal element ensuring reliable and reproducible human assessment, has received limited attention. Our investigation revealed that only 29.84% of recent papers involving human evaluation at top conferences release their evaluation guidelines, with vulnerabilities identified in 77.09% of these guidelines. Unreliable evaluation guidelines can yield inaccurate assessment outcomes, potentially impeding the advancement of NLG in the right direction. To address these challenges, we take an initial step towards reliable evaluation guidelines and propose the first human evaluation guideline dataset by collecting annotations of guidelines extracted from existing papers as well as generated via Large Language Models (LLMs). We then introduce a taxonomy of eight vulnerabilities and formulate a principle for composing evaluation guidelines. Furthermore, a method for detecting guideline vulnerabilities has been explored using LLMs, and we offer a set of recommendations to enhance reliability in human evaluation. The annotated human evaluation guideline dataset and code for the vulnerability detection method are publicly available online.

## introduction natural language generation (nlg) has found extensive applications across diverse domains. nevertheless, evaluating the quality of generated outputs has posed a longstanding and formidable challenge due to the inherent diversity of expressions capable of conveying the same meaning @xcite . this abundance of possible variations complicates the development of automated evaluation methods @xcite , thus necessitating the reliance on human evaluation as the gold standard and regarding it as a more reliable evaluation method in nlg @xcite @xcite @xcite . however, the evaluation guidelines, which play a crucial role in ensuring the reliability of human evaluation, have not received adequate emphasis. the transparency issues inherent in human evaluation guidelines raise concerns regarding the validity and reproducibility of the evaluation results @xcite . to investigate this issue, we conducted a study based on 3,233 papers that we crawled from acl, emnlp, and naacl conferences in the last three years. surprisingly, we indicate that only 29.84% of the papers involving human evaluation release their human evaluation guidelines. human evaluation guidelines are crucially important for ensuring that human assessments are conducted reliably. however, when papers fail to release the evaluation guidelines, there is no guarantee of the reliability and reproducibility of their evaluation results. moreover, our analysis of the guidelines released by these papers uncovered a significant concern: a striking 77.09% of the released guidelines exhibited noticeable vulnerabilities foot_0 , which could potentially have a detrimental impact on the correctness of human evaluation outcomes @xcite . the ultimate goal of establishing reliable human evaluation guidelines comprises several essential steps, which include detecting potential vulnerabilities in the guidelines, identifying specific vulnerability types, marking the precise segments with vulnerabilities, providing modification suggestions, and finally correcting identified vulnerabilities in the guidelines. in this paper, we conduct a preliminary study on defining and detecting vulnerabilities in human evaluation guidelines, marking an initial step towards reliable guidelines. specifically, we first constructed a human evaluation guideline dataset by collecting annotations of guidelines extracted from existing papers as well as generated via llms. based on the analysis of the guidelines, we identified eight main categories of vulnerabilities including ethical issues, unconscious bias, ambiguous definition, unclear rating, edge cases, prior knowledge, inflexible instructions, and others. the guidelines with vulnerabilities can result in issues such as annotators being unclear about task requirements, misunderstanding specific scoring standards, or erroneously directing annotators to assign higher scores to particular systems, which leads to incorrect and irreproducible evaluation results. to detect these vulnerabilities, we explored several prompt strategies to evoke the capability of current llms in vulnerability detection for human evaluation guidelines, and recommend an llm-based method employing chain of thought (cot) strategies. the main contribution of this paper is as follows: 1) we are the first to study vulnerabilities in human evaluation guidelines and release the first human evaluation guideline dataset with annotated vulnerabilities for advancing reliable human evaluation; 2) we analyze the existing human evaluation guidelines and introduce a taxonomy of eight vulnerabilities for evaluation guidelines; furthermore, we establish a principle for writing a reliable human evaluation guideline; 3) we explore an llm-based method for detecting guideline vulnerabilities. we recommend employing this method to assess the reliability of the guidelines before conducting human evaluations; 4) we present a set of recommendations designed to elevate the reliability of human evaluation by offering guidance on writing robust guidelines and identifying potential vulnerabilities. ## raw guideline dataset due to the lack of existing work related to human evaluation guideline assessment, we construct the first human evaluation guideline dataset through two methods: extracting from existing papers and generating from gpt3.5, referred to as the authentic guidelines and synthetic guidelines, respectively. note that we collect and analyze synthetic guidelines because llms has been proved as pow-erful tools for synthetic data generation @xcite @xcite . ## experiments in this section, we investigate utilizing llms to detect the specific vulnerability types in each eval-uation guideline, which is taken as a multi-label vulnerability type classification task. ## conclusion in this paper, we propose and analyze significant issues in the evaluation guidelines of gold-standard human assessments. we conduct a preliminary study on defining and detecting vulnerabilities in guidelines to advancing reliable human evaluation. by proposing a taxonomy of guideline vulnerabilities, we constructed the first annotated human evaluation guideline dataset. we then explored llms with few-shot prompting and cot strategies for automatic vulnerability detection. recommendations include employing llms to assist in writing human evaluation guidelines and modifying them based on the proposed principles. utilizing the proposed llm-based vulnerability detection method is suggested for assessing the reliability of the guidelines. in future work, we will delve into the precise annotation of spans containing vulnerabilities in guidelines, providing correction suggestions, automatically correcting identified vulnerabilities in the guidelines, and generating reliable guidelines by ai models. this advancement aims to contribute towards the ultimate goal of establishing dependable gold-standard human evaluation guidelines, thereby enhancing the reliability of nlg assessments.