Developing a tool for fair and reproducible use of paid crowdsourcing in the digital humanities (2022)

Abstract: This system demonstration paper describes ongoing work on a tool for fair and reproducible use of paid crowdsourcing in the digital humanities. Paid crowdsourcing is widely used in natural language processing and computer vision, but has been rarely applied in the digital humanities due to ethical concerns. We discuss concerns associated with paid crowdsourcing and describe how we seek to mitigate them in designing the tool and crowdsourcing pipelines. We demonstrate how the tool may be used to create annotations for diagrams, a complex mode of expression whose description requires human input.

## introduction crowdsourcing is regularly used to create data for training and evaluating natural language processing and computer vision algorithms @xcite . these fields often rely on paid crowdsourcing, which means that the work is distributed through online platforms and the performers are paid for their work. in the digital humanities, however, crowdsourcing is often associated with use of volunteers who are motivated by personal interests and altruism @xcite . conversely, paid crowdsourcing is viewed as ethically problematic @xcite due to sweatshop wages @xcite and other exploitative practices, such as invisible labour @xcite . in this article, we present ongoing work on a tool for fair and reproducible use of paid crowdsourcing in the digital humanities. we argue that paid crowdsourcing offers a viable alternative to fields that fall under the umbrella of digital humanities, but are unlikely to attract a volunteer workforce. however, using paid crowdsourcing warrants attention to ethics. we demonstrate how ethical concerns may be addressed by incorporating mechanisms that discourage exploitative practices into the design of the tool and crowdsourcing pipelines. ## ethical issues related to crowdsourcing as a portmanteau of crowd and outsourcing, the term crowdsourcing inherently evokes ideas of exploiting cheap labour in the global economy @xcite . paid crowdsourcing typically involves requesters, who post tasks on an online platform, which then distributes the tasks to workers. the platform thus acts as a mediator between the requesters and workers, and charges a commission from the requester. in natural language processing, crowdsourcing has become an established way of creating corpora due to the availability of a large pool of workers, short turnaround time and perceived cost efficiency @xcite . digital humanities have been cautious of paid crowdsourcing due to ethical issues related to low wages and workers' rights @xcite . similar concerns have also been voiced in the field of natural language processing @xcite , which are increasingly supported by empirical evidence. @xcite show that only 4% of the workers on amazon mechanical turk earn more than the federal minimum wage in the united states (@xmath011.58 per hour, but requesters who pay less than the minimal wage outnumber those who pay fair wages @xcite . in addition to fair pay, recent research has highlighted issues arising from qualification labour, which refers to low-or non-paid work that workers must perform to qualify for tasks that pay more @xcite . qualification labour emerges as a result of an information asymmetry between the requesters and workers. the requesters want to recruit high-performing workers by paying more, but higher wages also attract spammers who do not take the work seriously. because the requesters cannot assess the quality of work in advance, they are inclined to pay less, which drives away highperforming workers @xcite . making tasks only available to highly-qualified workers mitigates this problem, but to qualify for these tasks, the workers must perform approximately two months worth of non-or low-paying work @xcite . other forms of invisible labour on crowdsourcing platforms include the time spent searching for tasks, interacting with requesters and managing payments. @xcite estimate that the median time spent on invisible labour accounts for 33% of active working time on crowdsourcing platforms. because the workers are not compensated for this effort, invisible labour drives down their hourly wage. additional forms of invisible labour include working on tasks that are rejected or expire, that is, the worker cannot complete the tasks within the timeframe set by the requester. ## crowdsourcing in digital humanities given the issues described above, it is not surprising that crowdsourcing in the digital humanities has mainly relied on volunteers who are motivated by personal interests and altruism @xcite . successful examples of volunteer-based crowdsourcing include platforms such as zooniverse foot_0 and the transcribe bentham project @xcite , which have been able to attract a large body of motivated volunteers. this form of crowdsourcing in the digital humanities can also be conceptualised as a form of citizen science and peer production @xcite . however, some fields of study in the humanities may not be able to attract a sufficiently large body of volunteers. one such example is the emerging discipline of multimodality research, which studies how human communication relies on intentional combinations of expressive resources (see e.g. @xcite . as an emerging discipline, multimodality research is not widely known among the public at large, and its objects of study -everyday communicative situations and artefacts -are arguably less likely to attract the kind of attention needed for recruiting volunteers. multimodality research is currently undergoing a turn towards empirical research, which has been accompanied by calls for creating larger corpora to support this effort @xcite . current multimodal corpora remain small, because creating multiple layers of cross-referenced annotations needed to capture multimodal phenomena requires time and resources @xcite . @xcite have recently argued that the size of multimodal corpora can be increased by combining crowdsourced and expert annotations. as researchers working in the field of multimodality research, our motivation to develop a tool for fair and reliable use of paid crowdsourcing arises from the prospect of building large multimodal corpora with multiple layers of rich annotation. at the same time, we acknowledge the ethical dimensions of using paid crowdsourcing and seek to address them in the design and use of the tool. ## system demonstration in this section, we demonstrate how our tool can be used to crowdsource descriptions for a complex mode of communication, namely diagrams. diagrams combine diverse expressive resources, such as natural language, photographs, illustrations, drawings, lines and arrows into a common discourse organisation @xcite . computational processing of diagrams is challenging, because their constituent parts are not fixed, but determined dynamically by the communicative goals set for the diagram @xcite . to exemplify, figure 1 shows a diagram that uses written language and lines to pick out parts of an illustration, but we cannot know how the illustration should be decomposed without first considering the diagram as a whole, as the written labels determine how the depicted object should be decomposed into its constituent parts. to demonstrate how our tool may be used to decompose diagrams into analytical units, we define a pipeline with four steps. the pipeline aims to identify written labels and the parts they describe (see figure 2 ). each step consists of multiple tasks and actions. we first establish whether the diagram contains text (1), before asking the workers to outline all instances of written text (2). next, we ask the workers to determine whether text elements refer to other parts of the diagram (3). finally, we request the workers to outline the part(s) of the diagram referred to by the text (4). essentially, steps 1 and 3 consist of binary classification tasks (yes/no) in which agreement between the three workers is evaluated computationally. steps 2 and 4, in turn, combine human verification with computational evaluation of agreement on the final decision between three workers (accept/reject). as figure 2 shows, each step combines a training with a paid examination, which is used to recruit the workforce needed for completing the step. workers who pass the examination are also granted a skill that allows them to access similar tasks in the future. in each step, the aggregate actions use the dawid-skene algorithm implemented in the crowdkit library @xcite to determine the most likely answer based on three responses from the workers. the forward actions, in turn, determine where each assignment should be sent based on the result. individual assignments are forwarded immediately upon completion. we used 100 diagrams from the ai2d-rst corpus @xcite as input to the pipeline. the pipeline and its configuration files can be found at: https://github.com/ thiippal/latech-clfl-2022 . we aimed to train at least 10 workers to perform each of steps 1-2 and 50 workers for each of steps 3-4 using paid examinations. the workers could take a paid examination if they passed the training with a 70% score. a score of 80% in the paid examination would grant access to the actual tasks. the total cost for paid examinations amounted to @xmath112 per hour. ## results and discussion based on the results of step 1, 96 out of 100 diagrams contained text elements. these 96 diagrams contained a total of 996 text elements, which were outlined and verified in step 2. 733 of these elements were classified as referring to another part of the diagram in step 3. their targets were also outlined and verified, which yielded 784 annotations for non-textual elements in step 4. figure 3 shows example outputs from step 4. whereas annotations for the diagram on the left are complete, annotations for the diagram on the right show considerable variation. in the right-hand diagram, stages 3, 5 and 6 feature rectangular bounding boxes which indicate that the numbers below refer to the text and illustration above. zooming in on other stages shows that their outlines are drawn twice, as the workers have associated both written labels (above) and numbers (below) with the illustration. this shows how multiple workers who work on the same diagram make different inferences about the task and the diagram itself. furthermore, the annotations for stage 8 are missing altogether. this results from a false decision in step 3 of the crowdsourcing pipeline. because three workers agreed that these written elements do not describe other elements in the diagram, they were not forwarded to step 4. these missing annotations could be created by adding a final verification step to the pipeline, which asks the workers to evaluate the completeness of the annotations. overall, the results suggest that paid crowdsourcing holds much potential for the digital humanities. as our results may also be used to estimate the cost of creating similar annotations for all 1000 diagrams in the ai2d-rst corpus @xcite . note, however, that the descriptions created above are partial, as they only target elements that consist of written text and the parts that they describe. decomposing entire diagrams into analytical units by targeting other expressive resources such as arrows and lines would increase the costs considerably. in short, paid crowdsourcing is not cheap if used in an ethically responsible manner, but can be used to produce descriptions needed for building multimodal corpora. finally, researchers are responsible for applying paid crowdsourcing in a fair and ethical manner, which emphasises the need for transparency in relation to how crowdsourcing is used in academic research. however, not all issues outlined in section 2 may be addressed by the requesters, as the platforms are ultimately responsible for designing the algorithms that distribute work and constrain the actions that workers and requesters can take. these are concerns that the research community should address together, as paid crowdsourcing has become a part of the research infrastructure in datadriven fields and beyond (cf. @xcite . ## conclusion in this article, we introduced a new tool for fair and reproducible use of paid crowdsourcing in the digital humanities. we showed how ethical issues associated with paid crowdsourcing can be mitigated by emphasising them in (1) tool development and (2) crowdsourcing pipeline design. we also demonstrated how the tool can be used to crowdsource descriptions of complex multimodal data. we conclude that paid crowdsourcing can be applied productively in the digital humanities, but its use warrants attention to ethical concerns at all stages of the process.