A Needle in a Haystack: An Analysis of High-Agreement Workers on MT urk for Summarization (2023)

Abstract: To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.

## introduction natural language generation (nlg) tasks like text summarization are challenging to evaluate both in terms of automatic metrics and human evaluations @xcite . although automatic metrics are inexpensive proxies for human annotations for tasks like dialog evaluation @xcite , they may have problems dealing with paraphrases, capturing distant dependencies, or identifying nuances in human languages @xcite @xcite . thus, it is still crucial to obtain high-quality human annotations as gold labels for evaluation. amazon mechanical turk (mturk) 1 is a commonly used crowdsourcing platform for collecting human annotations on designed tasks, known as human intelligence tasks (hits). however, finding qualified workers for high-quality annotations with a better inter-annotator agreement (iaa) is challenging, especially for difficult tasks such as text summarization. best practices for recruiting high-quality workers are also poorly understood, and the relationship between high quality and high agreement needs further investigation. to tackle the above issues, we design a recruitment pipeline to identify workers who are able to produce high-agreement annotations for the evaluation of text summarization on mturk. it comprises a qualification task and an endurance task, followed by a reference-based task (see figure 1 ). in the qualification task, workers who meet predefined qualification settings receive instructions and qualification questions, including an attention check @xcite . the qualification questions are designed to assess the annotator's ability to evaluate multiple dimensions of a summary correctly. performance on this task determines whether they are categorized into gold, silver, bronze, or block. only the best workers (gold and silver) move on to the endurance task, which consists of 10 hits with 4 summaries in each to evaluate. this task only tests the summary's saliency, which is the most subjective dimension @xcite , but it challenges the annotator's capacity for handling a heavy annotation workload. gold and silver workers who complete all hits are added to a maintained worker list as high-agreement annotators for future tasks. to ensure their general performance for the true annotation task, a reference-based task to evaluate information coverage between summaries is conducted with these workers later. while serving as a best practice beyond its scope, our study has the following contributions: challenges of human evaluation compared to automatic evaluation metrics for nlg tasks like bleu @xcite and rouge @xcite , human annotations from non-expert annotators on mturk can reach an agreement with gold standards or expert judgments @xcite . although recent works leverage language models like bert @xcite to get better automatic evaluations @xcite , human judgments are still indispensable in identifying nuances in specific language tasks @xcite . finding qualified workers to carry out the evaluations is crucial. this is especially true for tasks like text summarization, which lacks consensus on evaluation protocols @xcite and is often inconsistent with previous human evaluations @xcite . however, human evaluation from non-expert crowdsourcing platforms have low quality @xcite and a sim-ple qualification filter is not sufficient to identify qualified workers @xcite . some studies applied quality control mechanisms to filter out poor quality annotations, resulting in a relatively low pass rate for a variety of tasks @xcite @xcite . the fact that up to 70% of the hits are eventually discarded indicates a huge resource waste. even with qualified workers, human annotations might still be adversely affected by factors like incomplete instructions or unfair wages paid to annotators @xcite , and workers need clear references, schemes, or standards to follow @xcite . thus, our study serves as a detailed reference for finding qualified mturk workers for a summarization evaluation task and further identifying those who can assist in a large number of annotations. inter-annotator agreement for annotations without true labels or those evaluated with a qualitative scale such as likert scale @xcite , the inter-annotator agreement (iaa) among mturk workers measures the reliability of the annotations. for example, cohen's kappa @xcite measures iaa between a pair of results of the same length from two annotators, while krippendorff's alpha @xcite measures the agreement of a set of results from any number of annotators, even with unequal sample sizes. both range from -1 to 1, with 1 indicating complete agreement. further studies also continue to mitigate annotator bias through complementary methods to iaa @xcite , aimed at high-quality annotations. in our study, we utilize both cohen's kappa and krippendorff's alpha as the measurement of annotation reliability. ## methods in this section, we detail how the workers were recruited and which tasks were carried out. 2 ## statistical test for stability of pipeline we next examined whether there was a difference in the probability of passing the qualification and endurance task among mturk workers. thus, we started by assuming the probability of passing each task for each round came from the same distribution, and we performed a statistical test as follows. let x denote the random variable representing the mturk worker. for the qualification task, let q @xmath0, and 0 otherwise. similarly, let e @xmath1. given 50 mturk workers in each round, we use q to denote the binary random variables in a round as (1). it can also be regarded as examples sampled from q @xmath2. among the samples, the probability of a thus, we used the bootstrap method (efron, 1992) with 10,000 iterations to estimate the mean and standard deviation of the probability of passing the qualification and endurance task. ## conclusion in this paper, we present a two-step recruitment pipeline that yields 12 qualified workers (4 gold and 8 silver workers) out of 200 mturk workers with basic qualification settings in our experiments. we show that workers identified by our pipeline can (i) achieve a higher inter-annotator agreement than expert annotators in the endurance task, (ii) outperform the statistical filter (mace) that discards annotation after the reference-based task, and (iii) replicate a proxy of cloudresearch annotations in the correctness analysis. though the 6% yield rate is not as expected, our pipeline serves as the best practice to deliver high-agreement annotations and addresses the widespread waste of resources on low-quality annotations through filtering out subpar workers before they embark on large-scale tasks. in the future, we plan to build up a pool of reliable annotators who can deliver high-quality (both high agreement and correctness) evaluations on a large scale and in multiple tasks, languages, and platforms. this research creates a relatively complete pipeline to identify qualified mturk workers for highquality human evaluations based on existing techniques, and thoroughly tests the effectiveness of this pipeline both qualitatively and quantitatively compared to other methods. however, there are several limitations of this work: