Mining Crowdsourcing Problems from Discussion Forums of Workers (2020)

Abstract: Crowdsourcing is used in academia and industry to solve tasks that are easy for humans but hard for computers, in natural language processing mostly to annotate data. The quality of annotations is affected by problems in the task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. To learn about the major problems, we provide a short but comprehensive survey based on two complementary studies: (1) a literature review where we collect and organize problems known from interviews with workers, and (2) an empirical data analysis where we use topic modeling to mine workers’ complaints from a new English corpus of workers’ forum discussions. While literature covers all process phases, problems in the task evaluation are prevalent, including unfair rejections, late payments, and unjustified blockings of workers. According to the data, however, poor task design in terms of malfunctioning environments, bad workload estimation, and privacy violations seems to bother the workers most. Our findings form the basis for future research on how to improve crowdsourcing processes.

## introduction crowdsourcing refers to the idea of providing collaborative services using the intelligence, skills, and creativity of a huge crowd @xcite . it has become a prevalent method in academia and industry to solve tasks that are complex for computers and yet easy for humans with low cost and high creativity. a typical use case in natural language processing (nlp) is the annotation of data. crowdsourcing platforms create an extensive network of people and enable organizations and individuals (the so called task requesters) to hire other individuals (the crowdworkers or just workers) in an anonymous and distant manner. among the most widely used platforms are upwork, appen, and above all amazon mechanical turk (mturk) which host annotation tasks and collect huge sets of annotated data from workers. conceptually, the general crowdsourcing process includes three main phases which are illustrated in figure 1 and further detailed in section 2: (1) in the task design, requesters create and post tasks on the given platform. (2) in the task operation, workers take on and solve tasks to then submit their results. workers may ask for clarifications of task details, and requesters may give feedback to ensure correct results @xcite . and (3) in the task evaluation, requesters decide whether to accept results and hence to pay workers, while workers may ask about feedback, payment, or reasons for rejections. to achieve mutual benefit in this process, workers need to understand the tasks designed by the requesters and their expected results. in practice, however, workers face diverse problems in the process and the interactions @xcite . this does not only affect the workers' feelings of pride and satisfaction @xcite , but it also negatively impacts the quality of the results, which in turn leads to a lack of trust between requesters and workers @xcite . despite the importance of crowdsourcing for nlp, such problems are not yet well-explored within the community. in the long run, we aim to create technology that supports making crowdsourcing processes in nlp and other contexts more successful. towards this goal, a first substantial step is to understand what problems exist and which of them are most dominant problems. to this end, we combine two complementary this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . (2) task operation: workers accept and solve the task, and submit the results. rq1. what problems do workers face in the different phases of crowdsourcing processes? rq2. which of these problems are most dominant in the literature and in the data respectively? in section 5, we compare the results from the literature review and the data analysis. literature covers problems in all three phases of the crowdsourcing process, along with platform-related problems. missing responses and unfair rejections without explanation from the requesters are the most discussed problems in literature. data reflects many problems from literature, however, the most dominant problems in the workers' complaints refer to task design: workload misestimation, malfunctioning environments, low payment, and privacy violations. our findings imply that a thorough task design by the requesters leads to significant improvements in crowdsourcing and a better technological support may be needed. with our research, we contribute to a better understanding of the current challenges relating to crowdsourcing processes, which forms the basis of any process improvements and best practices. beyond the insights of this paper, the provided corpus allows for studying related questions, e.g., regarding the difference between the topics discussed in positive and in negative reviews. also, future work may additionally consider the requesters' perspective on crowdsourcing problems. so far, however, barely any literature or data reflecting this perspective is available. crowdsourcing is prototypical for distant work employment processes, which become increasingly common within the digitization of society. we believe that an understanding of its mechanisms is crucial to shape the work life of the future. ## crowdsourcing processes crowdsourcing outsources tasks of a requester (i.e., some organization or individual) to a crowd of workers around the world. all communication and interaction between the requesters and workers happens via a crowdsourcing platform. on amazon mechanical turk (mturk), for instance, requesters may post a wide range of micro-tasks, including text annotation, translation, writing, and similar. a typical crowdsourcing process consists of the three main phases illustrated in figure 1 , often at least partly conducted in an iterative manner: (1) the task design, (2) the task operation, and (3) the task evaluation. we detail each phase in the following. 2 task design first, the requester creates a task to be solved by the workers (e.g., the annotation of a set of english texts). usually, the task is instantiated multiple times with different data (e.g., once for each considered text). the task design includes instructions for the workers, specifications of deadlines and paid rewards, possibly a time estimation, and similar. besides, the requester may define qualification requirements to obtain more suitable workers (e.g., a worker may need to have a certain reputation score or to be a native english speaker). part of the design is also the implementation of an environment (in terms of a user interface) where workers are supposed to operate the task and/or to submit their results. task operation the (instances of) designed tasks are posted on the platform. workers that meet the requirements can accept tasks and solve them, depending on their incentives, such as rewards or task types. often, the actual work on a task is conducted directly via the task's interface; sometimes, only result submission takes place there. during task operation, workers and requesters may communicate regarding questions on the task or other inquiries by the workers. once finished, workers may be required (e.g., on mturk) to enter a completion code received from requesters to successfully submit their results. requesters may provide feedback to obtained results in order to improve future result submissions. ## literature review this section presents the literature review that we conducted to collect and organize crowdsourcing problems of workers covered by existing research through interviews and surveys. ## data analysis as a complement to the literature review, we empirically analyzed reviews from a workers' online discussion forum using topic modeling, in order to investigate the most discussed problems according to the experiences that workers complain about in practice. this section summarizes our analysis as well as the corpus that we created for this purpose and for future research in the community. our analysis aims to identify the problems that workers face in crowdsourcing processes rather than the extent to which the problems change over time. while such temporal aspects are worthy investigating given the rapid evolution of crowdsourcing, we leave them aside here for a focused discussion. ## comparison of literature review and data analysis finally, we compare the results obtained from the data analysis to those from the literature review, in light of the two research questions from section 1. to this end, we counted how often each problem was discussed in the literature, and we compared the relative proportions of these problems with the problem probabilities obtained from topic modeling. figure 3 illustrates the results discussed in the following. ## conclusion natural language processing heavily relies on crowdsourcing, for the annotation of corpora as well as for user studies as part of approach evaluations and similar. since the quality of crowdsourcing results is directly affected by the quality of the preceding process from task design to task evaluation, we investigate how to improve this process through computational methods. in this paper, we have combined a literature review with a data analysis, in order to identify the problems that workers face with requesters in the design, operation, and evaluation of crowdsourcing tasks. the literature covers findings from theory along with results of interviews and questionnaires that cover the workers' problems. in the data analysis, we have used topic modeling to mine respective problems from the workers' complaints in an online discussion forum, and we provide the underlying corpus for further research in the community. bringing findings from literature and data together, we aim to bridge and share knowledge between the two research fields of human computation and natural language processing. we found that the main problems workers complain about refer to the task design: not only do requesters seem to underestimate the workload with respect to the time, effort, and fair payment needed to accomplish a task. also, errors in their task environment implementation cause workers to waste time with unsuccessful submissions. some requesters are also said to violate the workers' privacy, not respecting the terms-of-service of the used crowdsourcing platform. unfair rejections without explanation are the dominant problem in the task evaluation, and the literature also extensively discusses communication problems between requesters and workers, combined with a poor platform support. a look at the requesters' perspective on the found problems would nicely complement our study, but literature and data for this purpose are missing so far. even though crowdsourcing is continuously evolving and certain aspects may change compared to the time we gathered our data, our findings show fundamental problems in different stages of the processes and are worth investigating more deeply. approaches to tackle some problems exist already: for task design, specific interfaces help requesters create high-quality tasks; for fairer rejections, also a requester's acceptance rate was introduced, letting workers find trustworthy requesters; and for fairer requester behavior in general, discussion forums (such as turkopticon) help support the workers' rights. however, workers still face many problems, so more improvements are required. in future work, we seek to improve the requester-worker relationship through a technological support of the task design. in particular, we envision an automated assistant system that helps improve the clarity of task descriptions. we believe that this a key step in assisting the requesters, specially the beginners, to learn lessons from the previous requesters' mistakes in order to create more clear task descriptions, which consequently leads to receive high quality results from workers. in line with our findings, we believe that this will solve many problems of today's crowdsourcing processes.