Incorporating Worker Perspectives into MT urk Annotation Practices for NLP (2023)

Abstract: Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics shared among NLP researchers. However, without considering the perspectives of MTurk workers, these approaches are susceptible to issues regarding workers’ rights and poor response quality. We conducted a critical literature review and a survey of MTurk workers aimed at addressing open questions regarding best practices for fair payment, worker privacy, data quality, and considering worker incentives. We found that worker preferences are often at odds with received wisdom among NLP researchers. Surveyed workers preferred reliable, reasonable payments over uncertain, very high payments; reported frequently lying on demographic questions; and expressed frustration at having work rejected with no explanation. We also found that workers view some quality control methods, such as requiring minimum response times or Master’s qualifications, as biased and largely ineffective. Based on the survey results, we provide recommendations on how future NLP studies may better account for MTurk workers’ experiences in order to respect workers’ rights and improve data quality.

## introduction amazon mechanical turk (mturk) is an online survey platform that has become increasingly popular for nlp annotation tasks @xcite . however, data collection on mturk also runs the risk of gathering noisy, low-quality responses @xcite or violating workers' rights to adequate pay, privacy, and overall treatment @xcite @xcite @xcite . we conducted a critical literature review that identifies key concerns in five areas related to nlp data collection on mturk and observed mturk worker opinions on public forums such as reddit, then developed a survey that asked mturk workers about open questions in the following areas: task clarity. how can requesters ensure that workers fully understand a task and complete it accurately? surveyed workers expressed a desire for more examples and more detailed instructions, and nearly half desired more information about the downstream context in which an annotation will be used. in addition, they indicated that task clarity is a major factor in their judgment of how responsible and reliable a requester is (section 3). ## survey design we conducted a literature review and examined posts on r/mturk, a reddit forum that mturk workers use to discuss the platform, to identify the areas of uncertainty discussed in sections 3-7. then, to collect data on mturk workers' experiences, we conducted a qualtrics survey posted as a task on mturk. our survey was broadly subdivided into sections on payment, sensitive questions, response quality, and miscellaneous issues (mainly task clarity and context). as a preliminary filter for response quality, we required that all respondents have a 97% hit approval rate, at least 100 hits completed, minimum of 18 years of age, and english fluency. this is a lower hit approval rate than is typically used, which can increase the rate of spam; however, we aimed to collect opinions of workers who might be excluded by high approval rate filters, and therefore manually reviewed the data afterwards to remove spam responses by examining responses to the required free text fields. we collected 207 responses from our survey over one week, of which 59 responses were dropped for spam or incompleteness (we discuss further implications of high spam rates in section 8). each annotator was paid @xmath015/hour. appendices a and b contain details on informed consent, the full text of the survey, and data cleaning details. 1 the following sections address each area of concern raised in section 1, discussing previous research on mturk annotation practices, the open questions raised by the literature, the survey questions we included on those topics, and the results and implications of the survey responses. ## task clarity on mturk, requesters publish "human intelligence tasks" (or hits) for mturk workers to complete in exchange for a monetary incentive. best practices for question phrasing and ordering are consistent with overall survey design guidelines, which encourage including clear instructions for how to use the platform, clearly outlining requirements for acceptance, including label definitions and examples, and avoiding ambiguous language (gideon, 2012). to look for strategies for minimizing confusion in survey questions, our survey asked what additional provided information, such as context and purpose of the task, would be most helpful for reducing ambiguity or confusion. to ensure that results are accurate and useful, it is important to carefully conceptualize the task (define what quality is being measured by the study) and operationalize it (decide how the study will measure the construct defined by the conceptualization) along with clear compensation details (discussed further in section 4 and @xcite . in a reddit thread on r/mturk, worker u/dgrochester55 (2022) stated that "we are not likely to take your survey seriously if you as a requester aren't taking your own survey seriously" due to issues such as "poor pay, irrelevant subject" or being "badly put together." thus, task clarity not only ensures that the worker can provide accurate answers, but also serves to increase their effort levels and willingness to complete the task itself. our survey sought to further clarify the importance of task clarity by asking whether it influences effort level and willingness to start a task. in addition, we asked a free response question about common areas of confusion to observe common pitfalls in designing mturk tasks. ## payment mturk's platform, which enables researchers to collect large amount of data at low cost, does not currently regulate compensation beyond a @xmath13.13 and @xmath215 per hour (u/sharpsilverz, 2022). @xcite found that workers are likely to overestimate their work time by 38% of the observed task time. thus, the amount of time used to determine minimum wage can roughly account for this overestimation if using workers' self-reported work times as a metric. previous research is divided on whether extra pay beyond minimum wage improves response quality. @xcite gave anecdotal evidence that unusually high payments, such as @xmath32, ya know?" thus, it is vital for researchers to identify potentially sensitive content and take steps to protect workers from possible harm. generally, researchers are aware of the presence of offensive content, as reflected in content warnings included in the beginning of published papers (e.g., @xcite . however, there is a clear need for standardized best practices due to the highly vulnerable nature of such mturk work. a relevant example is facebook's partnership with sama, a company that pays workers to identify illegal or banned content on social media @xcite . an interview with the workers revealed high levels of psychological distress due to insufficient breaks and subpar in-person resources. in an online setting such as mturk, such personal-ized care is even less accessible, making it more difficult to provide support for vulnerable workers. these workers' experiences exemplify the difficulties faced by the large, invisible workforce behind mturk and powerful nlp applications, also described by @xcite in their discussion of the typically unregulated "ghost work" that powers machine learning. common safeguards include listing clear descriptions of the benefits and risks of the task, providing mental health resources for workers, using data safety monitoring boards, adhering to institutional review board policies, and following confidential data code and informed consent procedures @xcite . we look to further add to this list of best practices by asking survey respondents whether there are questions that will immediately deter them from completing a task or to answer untruthfully and preferred actions or resources provided by requesters to minimize possible harm. ## recommendations we consolidate our findings on worker perspectives to arrive at several recommendations.