Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing (2021)

Abstract: Extensive work has argued in favour of paying crowd workers a wage that is at least equivalent to the U.S. federal minimum wage. Meanwhile, research on collecting high quality annotations suggests using a qualification that requires workers to have previously completed a certain number of tasks. If most requesters who pay fairly require workers to have completed a large number of tasks already then workers need to complete a substantial amount of poorly paid work before they can earn a fair wage. Through analysis of worker discussions and guidance for researchers, we estimate that workers spend approximately 2.25 months of full time effort on poorly paid tasks in order to get the qualifications needed for better paid tasks. We discuss alternatives to this qualification and conduct a study of the correlation between qualifications and work quality on two NLP tasks. We find that it is possible to reduce the burden on workers while still collecting high quality data.

## introduction workers using amazon mechanical turk earn a median wage of @xmath07.25. many researchers pay workers a higher wage, estimating the time spent on a task and giving bonuses when the time required is higher than expected. at the same time, researchers try to maintain the quality of work completed using a variety of methods @xcite . one common approach, used by 19% of tasks (hits) on the platform @xcite , is to restrict tasks to workers who have had a certain number of hits approved. tasks with this restriction have a median wage of @xmath1$@xmath212 / hour). reviews on turkerview ( https://turkerview.com/ ) indicated that workers effective hourly rates were @xmath311.25, @xmath414.59. we measure performance by comparing with the ontonotes annotations. an f-score of 80% or above was considered acceptable, to allow for minor errors and points of confusion. sentiment analysis this task is very intuitive and has been crowdsourced extensively in the past. we closely followed the set up used to annotate the stanford sentiment treebank @xcite , with the same task instructions. workers were shown ten examples whose true scores were evenly spread across 0 to 1. we estimated that the task would take 4 minutes and paid workers 80 cents (@xmath522.15, @xmath650.53, suggesting that workers were faster than anticipated. each value is based on fifteen workers, except for sentiment there were fourteen for (98-99%, 500-1,000), and for coreference there were fourteen for (97-98%, 500-1,000), (97-98%, 1,000-2,500), (98-99%, 2,500-5,000), (98-99%, 5,000+), thirteen for (97-98%, 5,000+), and twelve for (97-98%, 2,500-5,000). to evaluate, the labels are mapped to [0, 1] and compared with the sts values. an average value of below 0.15 was considered acceptable. this cutoff was chosen based on the scores achieved by two nlp students in our lab (0.11 and 0.09). ## ethics and impact statement this work involved consideration of several potential impacts. in terms of privacy, all data from workers is aggregated for the purpose of presenting results, and information from worker discussions were only sourced from publicly shared content. in terms of payment, we estimated the effort involved and aimed to pay workers $12 usd an hour. see the main text for worker reported values of hourly earnings on the two tasks. this was approved by the michigan irb under study id hum00155689. one potential harm of this work is that it may encourage higher values of the percentage of hits accepted qualification, making workers more vulnerable to requesters who unfairly reject work. ## conclusion and recommendations this paper identifies the issue of qualification labour: the implied labour created by the qual-ifications we define. based on a range of sources, we found that 5,000 approved tasks is one common threshold. that takes approximately two months to achieve and the tasks are poorly paid. we conducted a study of two tasks to understand how work quality correlates with these qualifications. we found that trends are task dependent, but lower thresholds can often be used. we recommend either not using the "hits accepted" qualification, or running preliminary tests to identify the lowest suitable threshold for your task. this calibration is necessary because worker performance depends on many factors, including the task type, data (including which language), user interface, and instructions. one particularly promising method is to use controlled crowdsourcing @xcite with a low threshold: run a short task with low or no qualifications to identify workers, then for the full task only allow those workers to participate. this reduces the burden on workers while maintaining high quality work.