The Perils of Using M echanical T urk to Evaluate Open-Ended Text Generation (2021)

Abstract: Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text quality (e.g., Likert scores of coherence or grammaticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a survey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a series of story evaluation experiments with both AMT workers and English teachers and discover that even with strict qualification filters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rating model-generated text.

## introduction recent advances in neural language modeling have spurred research into open-ended text generation tasks such as story generation @xcite , style transfer @xcite , and pun generation @xcite . since the space of possible outputs for these tasks is huge compared to more constrained problems such as machine translation, automatic metrics such as bleu @xcite and rouge @xcite that measure similarity to reference texts are mostly uninformative @xcite . 1 human evaluation of model-generated text, which is critical for openended tasks given the unreliability of automatic metrics @xcite @xcite , is frequently conducted on amazon's popular mechanical turk platform (amt) to minimize cost and time. most existing amt studies ask crowdworkers to provide likert scale ratings of various properties of generated text, such as fluency and likability. in this paper, we study the reliability and reproducibility of amt evaluations of open-ended text generation. we first conduct a survey of papers on open-ended text generation between 2018-2020 and find many critical details often go unreported (e.g., worker qualifications, payment, task descriptions, annotator agreement), a finding in line with prior reproducibility studies outside open-ended text generation @xcite @xcite . next, we perform a series of story generation evaluations with both amt workers and expert raters (english teachers), applying a variant of the most common task configuration that appeared in our survey (5 point likert scale ratings of 200 examples with three annotators per example) to the paragraph-length writingprompts dataset of @xcite . unlike prior work in this area, we ask raters to evaluate both stories generated by a fine-tuned gpt-2 language model @xcite and human-written reference stories on the same scale, as we expect the latter to consistently score higher on all evaluations. our experiments expose and quantify several troubling trends: for future human evaluations of open-ended text generation tasks, we urge researchers to obtain expert raters whenever possible. if amt is the only feasible option, we recommend that available reference outputs also be evaluated alongside modelgenerated ones to improve rating calibration, and also that heavy filtering of the worker population (possibly through qualification tasks, or post-hoc removal) is performed prior to reporting results. ## a survey of papers that evaluate open-ended text generation with amt we begin with a survey of 45 papers that use amt to evaluate the output of open-ended englishlanguage text generation models, which includes generated stories, metaphors, paraphrases, puns, sarcasm, and sentences with transferred style or attributes foot_1 . each paper was published between 2018 and 2020 at acl, naacl, or emnlp, and we exclude papers that use amt to evaluate more wellestablished generation tasks like machine translation, or summarization. 3 unlike previous surveys of evaluating generated text @xcite , we focus specifically on amt evaluations of open-ended text generation. in this section, we provide an overview of the different types of evaluation task setups present in our survey; later, we experiment with several variants of the most common setup. evaluation criteria: as in the survey of @xcite , we observe a variety of different evaluation criteria and definitions of these criteria across the 45 papers. the most common criteria include fluency and/or grammaticality (19), overall "quality" (12), and relevance (10) to a corresponding prompt. furthermore, stories in particular tend to also be evaluated on some notion of coherence (9) and likability (4). rating scales: more than half of the papers (24) employ a 5-point likert scale to evaluate the above criteria; of these, 19 provided labels for just the end points of the scale (e.g., "lowest" vs. "highest"), while 5 labeled all points on the scale. the next most common evaluation type is ranking two or more system outputs (23). less common are other likert scales (3, 4, or 6-point), pass/fail tasks, and output-prompt matching tasks. number of raters and rated items: an alarming number of papers ( 14 ) do not even report the number of raters and/or items (6) used for evaluation. of the remaining, most papers (16) obtain ratings from 3 separate amt workers per item. the most common number of items per evaluation is 100 ( 14 ). the number of raters per item in other studies ranges from 2 to 11, while the number of items ranges from 12 to 1,000. ## evaluation by expert teachers the experiments in the previous section demonstrate the unreliability of amt ratings for openended text generation, even when qualifications are used to restrict the task to ostensibly reliable workers. in this section, we compare the ratings produced by amt workers to those of expert raters, specifically a set of three english teachers, and discover significant deviations between the two groups. though they rated both types of stories separately, their ratings clearly distinguish between human-written references and machine-generated stories. we also conducted post-task interviews with the teachers and organized a mediation session to discuss stories with high disagreement, observing that they reach consensus after discussion in about 80% of cases. teachers need to see many examples to properly calibrate their ratings: in post-task interviews, all teachers reported that it took them 10-20 stories on average to calibrate their ratings. since most amt workers complete only one to two hits, they do not have similar time to get acquainted with the task; this may suggest that having a pre-task training phase can improve worker calibration. coherence is difficult to rate for machinegenerated text: the teachers unanimously report that while coherence is easy to rate for reference stories (since most of them are largely coherent), it is the most difficult property to rate for gpt-2 generated stories. since they did not know that they were rating machine-generated text, they spent time trying to make sense of the author's possible intent in producing many of the strange artifacts and hallucinations common to output of neural language models @xcite . in contrast, relevance turned out to be the easiest property of machine-generated text for teachers to rate, which is expected as many of gpt-2's stories deviate very quickly from the prompt (see figure 1 ). gpt-2 generated stories are much harder for teachers to rate overall: all teachers reported struggling more when rating gpt-2 stories, a fact reflected in their average rating time per story increasing significantly from 69.8 seconds to 87.3 seconds (@xmath2.05). in contrast, the average rating time of amt workers decreased from 135.3 seconds for human-written text (day 1) to 91.5 seconds for gpt-2 text (@xmath3.05) foot_13 . teachers also reported having to recalibrate their scale when rating the gpt-2 generated stories, as the stories were significantly worse than the human-written text. consequently, they suggested that it would be easier to calibrate their scale had the gpt-2 output been presented beside the human-written text, which supports the results from our joint rating task with amt workers. finally, the teachers suggested that creating a standardized rubric would greatly facilitate the rating process. this step is even more important as machine-generated text faces different issues than human-written text. resolving teacher disagreement: one advantage of using human expert raters is that we can easily have them discuss examples on which they disagree. we arranged a mediation meeting between two of the three teachers to discuss 60 stories on which they showed the highest disagreement (3 attributes × 10 stories × 2 types, we excluded likability due to its subjective nature). in this meeting, they were first asked to rate the stories again, without being provided their previous rating. in about 20% of cases, one of the teachers disagreed with their own previous rating due to honest lapses of judgment. another common reason for disagreement was missing world knowledge (see figure 1 , right). one more reason for disagreement, a confusion about how to rate slang in terms of grammaticality. while the text was not correct in the view of the official grammar, it was appropriate for the prompt, so one teacher rated it high while the other rated it low. overall, after discussing examples that they still disagreed on after re-rating, teachers were able to come to a consensus on 80% of the stories; the remaining disagreements persisted due to individual differences in strictness. see appendix c for details on the mediation meeting. replicating the study on upwork: we recognize that replicating our study is difficult without access to a network of english teachers. as such, we performed the same experiment using three certified teachers recruited on a freelance platform, upwork. 16 the teachers were paid @xmath0144 to rate 200 stories with amt vs. @xmath1262.5 with upwork.