E asy T urk: A User-Friendly Interface for High-Quality Linguistic Annotation with A mazon M echanical T urk (2021)

Abstract: Amazon Mechanical Turk (AMT) has recently become one of the most popular crowd-sourcing platforms, allowing researchers from all over the world to create linguistic datasets quickly and at a relatively low cost. Amazon provides both a web interface and an API for AMT, but they are not very user-friendly and miss some features that can be useful for NLP researchers. In this paper, we present EasyTurk, a free tool that improves the potential of Amazon Mechanical Turk by adding to it some new features. The tool is free and released under an open source license.

## introduction in the last years, deep learning algorithms have achieved state-of-the-art results in most nlp tasks such as textual inference, machine translation, hate speech detection @xcite . despite their accuracy, deep learning algorithms have a major downside, i.e. they require large amounts of data to be trained, making the data bottleneck issue even more problematic than with other machine learning algorithms like svm @xcite . the need to leverage large amounts of manually annotated data has become a major challenge for the nlp community, since linguistic annotation performed by domain experts is both expensive and time-consuming. this explains why crowdsourcing platforms, offering access to a large pool of potential annotators, have been successfully used for the creation of annotated datasets. amazon mechanical turk (amt) is probably the most widely used platform of this kind, enabling the distribution of low-skill but difficult-toautomate tasks to a network of humans who could 1 https://youtu.be/omkjornpgss work in parallel, when and where they prefer, for a certain amount of money. the availability of a lot of workers at the same time allows researchers all over the world to annotate large datasets in a fraction of the time and the money needed doing it through the recruitment of domain experts. furthermore, crowd-workers are spread all over the world, offering the possibility to have annotation performed in different languages by native speakers. in the last years, amt turned out to be successful in a wide range of nlp annotations, such as named entities from e-mails @xcite or medical texts @xcite , subjectivity word sense disambiguation @xcite , image captioning @xcite , and much more. unfortunately, annotations obtained by amt workers are often of low quality, since: (i) they are non-expert and therefore they can make mistakes in annotations; (ii) some of them are spammers who try to maximise the earnings by submitting random answers as quickly as possible. mitigating the effect of errors in datasets annotated by crowd-workers is one of the biggest challenge in using amt. one mitigation strategy adopted by researchers is usually to collect multiple annotations of the same instance, and apply different methods to deal with this information redundancy. most of the times, majority voting seems to be an appropriate strategy, i.e. the final label assigned to an instance is the one provided by the majority of the workers, even if they are not all in agreement. however, if spammers always choose the same answer to finish the task quicker, this strategy would finally assign a wrong label to the textual instance. while past works have described how to successfully deal with non-expertness @xcite , it is more challenging to identify spammers. some tools @xcite deal with the problem offline, when the task is completed, trying to identify spammers using redundant annotations and comparing the answers given by all crowd-workers. in this context, spammers are correctly identified, but they are nevertheless paid because their annotations are filtered out after the task is closed. another idea to find spammers is to use a gold standard, a set of very easy-to-understand instances, previously annotated by an expert, that a careful worker should not miss. in this paradigm, when a worker gives the wrong answer to a gold question, one may infer that the annotator is trying to cheat and should be blocked. the amt api provide a way to do it automatically, but the feature is not included in the web interface, therefore the only way to get this result is by writing a program (in python, php, or any supported language) that checks whether the gold instances have been answered correctly or not. in this paper, we describe easyturk, both a web interface and a powerful api that tackles all these issues and enhances the experience of using amt. the tool can aggregate more than one instance of a task in a single page shown to the worker, concealing also gold standard instances. furthermore, easyturk can be configured to take an action, e.g. block a worker when he or she misses too many gold answers, marking the already-given questions as not reliable. finally, the software is open source and its user-friendly interface has been implemented using most recent guidelines for usability and responsiveness. amt has two kinds of users: requesters and workers. the formers create the hits (using the api or the web interface) and upload them to the amazon servers, along with the fee that will pay for each of them to be completed. the latters search the hit database, choose the preferred tasks and complete them in exchange for monetary compensation. requesters can restrict the range of workers allowed to complete the task, based on demography, school level, spoken languages, and so on. some requirements are free for the requester (for example the living country of the worker), but normally they raise the price of the hits. requesters can also assign custom qualifications to workers in order to filter out them during the submission of the hits to the system. the platform also provides an automatic mechanism that allows multiple unique workers to complete the same hit. this is useful, for example in nlp tasks, for which requesters usually need more than one answer for each hit, so that the majority label can be selected, resulting in a higher-quality final annotation thanks to the 'wisdom of the crowd'. each annotation instance (a pair worker-hit) is called assignment. requesters have the option of rejecting the answer of a particular worker, in which case they are not paid. the above-described custom qualifications can be used to filter out, for a particular task, workers who did not reach sufficient accuracy in previous hits. in specific cases, for example as a consequence of particularly sloppy annotations, a worker can be blocked and is not able to perform hits for the requester anymore. one of the main issues with using amt is that some features are available only using the api, while others can be used only in the web interface. for example, through the web interface a requester can upload a tsv file with the data to be annotated, or select which qualifications the workers should have to complete the hits. these two features are not available in the api, but one can automatise acceptance/rejection of the worker job only through it. given the above constraints, we developed easy-turk so to allow non-skilled users to submit hits without using a specific programming language, such as python or java, while using the features available through apis. ## description of easyturk easyturk is composed of three modules: (i) the web interface; (ii) the api; (iii) the server. most of the features included in easyturk are accessible directly from the web interface, but are managed by the server. ## release easyturk is completely free, available on github, 9 and released as open-source under the apache 2.0 license. 10 the web interface is developed in vuejs and needs nodejs foot_8 to be compiled and launched. both the api and the server are written in php foot_9 and need a machine with at least version 7 of the interpreter and mysql server foot_10 installed. the server can be run as a service and does not need other particular dependencies to work. the api, instead, must be configured to work in a web server (such as apache foot_11 or nginx foot_12 ). ## related work since 2005, when amt was released, an increasing number of researchers has used this platform for research purposes. in particular, the nlp community has taken advantage of amt to bring linguistic resources to a new scale, also with the support of amazon. for example, in 2010 amazon sponsored a workshop during the naacl conference, where researchers were given 100 dollars of credit on the platform to run an annotation task and answer some meta-research questions, such as how nonexpert workers can perform complex annotations, or how can one ensure high quality annotations from crowd-sourced contributors. some past works have dealt with the abovementioned issues related to crowd-worker quality. in @xcite , the authors present a software that, after a round of annotations using amt, tries to understand which workers perform better and, consequently, which are the best annotations to consider and which to discard when there is redundancy, in an unsupervised fashion. in @xcite , the efficiency of amt is analysed over 100,000 local business listings for an online directory. a mechanism for filtering low-quality workers in order to build a reliable workforce that has high accuracy is described, to understand better the problem of quality control in crowdsourcing systems. some attempts have also been done to improve the potential of amt by writing new frameworks on top of the amt api. cloudresearch, formerly turkprime, @xcite was born for this purpose and at the time of launch was free to use for researchers. now it is part of a bigger company and is not free any more. lingoturk @xcite is an open-source, freely available crowdsourcing client/server system aimed primarily at psycholinguistic experimentation, where custom and specialized user interfaces are required but not supported by popular crowdsourcing task management platforms. openmturk @xcite is a free and open-source administration tool for managing research studies using amt. turkit (little et al., 2010) is a toolkit for prototyping and exploring truly algorithmic human computation, while maintaining a straightforward imperative programming style. turktools (erlewine and kotek, 2016) is a set of free, open-source tools that allow linguists to post studies online and simplify the interaction with amt. turkgate foot_13 provides better control and verification of workers' access to an external site and allows the grouping of hits, so that workers may only access one survey within a group. amti, foot_14 developed at the allen institute for ai, is a command-line interface for amt that emphasizes the ability to quickly iterate on and run reproducible crowdsourcing experiments. finally, amt is integrated to add human annotations in more complex tools. qurk @xcite , for example, is a query system for managing annotation workflows. ## conclusion and future work in this paper, we presented easyturk, a free program that improves the potential of amazon mechanical turk by adding some features which are not present out-of-the-box. in particular, the requester has now the ability to insert multiple instances of the task in a single hit, and optionally mix them with a gold standard, that can be used to track the accuracy of the workers. finally, when some events are triggered (for example a worker answering too quickly to a hit or missing the gold standard), easyturk can be programmed to take an action such as reject the assignment, or block/restrict the worker. the tool is free and open source, and can be downloaded from github and installed locally. in the future, we are planning to implement new features. for example, the system can intercept spammers using also a particular pattern of answers (for example a set of hit where the same answer is always selected). we also would like to include in easyturk a collection of templates for basic annotations (for example, yes/no, a set of possible answers, a free text, and so on), so that requesters do not need any more to create their template on the amt website.