Easy, Reproducible and Quality-Controlled Data Collection with CROWDAQ (2020)

Abstract: High-quality and large-scale data are key to success for AI systems. However, large-scale data annotation efforts are often confronted with a set of common challenges: (1) designing a user-friendly annotation interface; (2) training enough annotators efficiently; and (3) reproducibility. To address these problems, we introduce CROWDAQ, an open-source platform that standardizes the data collection pipeline with customizable user-interface components, automated annotator qualification, and saved pipelines in a re-usable format. We show that CROWDAQ simplifies data annotation significantly on a diverse set of data collection use cases and we hope it will be a convenient tool for the community.

## introduction data is the foundation of training and evaluating ai systems. efficient data collection is thus important for advancing research and building time-sensitive applications. @xcite data collection projects typically require many annotators working independently to achieve sufficient scale, either in dataset size or collection time. to work with multiple annotators, data requesters (i.e., ai researchers and engineers) usually need to design a user-friendly annotation interface and a quality control mechanism. however, this involves a lot of overhead: we often spend most of the time resolving frontend bugs and manually checking or communicating with individual annotators to filter out those who are unqualified, instead of focusing on core research questions. another issue that has recently gained more attention is reproducibility. dodge et al. (2019) and pineau (2020) provide suggestions for system reproducibility, and bender and friedman (2018) and @xcite crowdsourcing with automated qualifcation; https: // www.crowdaq.com/ 2 this holds not only for collecting static data annotations, but also for collecting human judgments of system outputs. gebru et al. (2018) propose "data statements" and "datasheets for datasets" for data collection reproducibility. however, due to irreproducible human interventions in training and selecting annotators and the potential difficulty in replicating the annotation interfaces, it is often difficult to reuse or extend an existing data collection project. we introduce crowdaq, an open-source data annotation platform for nlp research designed to minimize overhead and improve reproducibility. it has the following contributions. first, crowdaq standardizes the design of data collection pipelines, and separates that from software implementation. this standardization allows requesters to design data collection pipelines declaratively without being worried about many engineering details, which is key to solving the aforementioned problems (sec. 2). second, crowdaq automates qualification control via multiple-choice exams. we also provide detailed reports on these exams so that requesters know how well annotators are doing and can adjust bad exam questions if needed (sec. 2). third, crowdaq carefully defines a suite of pre-built ui components that one can use to compose complex annotation user-interfaces (uis) for a wide variety of nlp tasks without expertise in html/css/javascript (sec. 3). for non-experts on frontend design, crowdaq can greatly improve efficiency in developing these projects. fourth, a dataset collected via crowdaq can be more easily reproduced or extended by future data requesters, because they can simply copy the pipeline and pay for additional annotations, or treat existing pipeline as a starting point for new projects. in addition, crowdaq has also integrated many useful features: requesters can conveniently monitor the progress of annotation jobs, whether they are paying annotators fairly, and the agreement level of different annotators on crowdaq. finally, sec. 4 shows how to use crowdaq and amazon mechanical turk (mturk) foot_0 to collect data for an example project. more use cases can be found in our documentation. ## standardized data collection pipeline a data collection project with multiple annotators generally includes some or all of the following: (1) task definition, which describes what should be annotated. (2) examples, which enhances annotators' understanding of the task. (3) qualification, which tests annotators' understanding of the task and only those qualified can continue; this step is very important for reducing unqualified annotators. (4) main annotation process, where qualified annotators work on the task. crowdaq provides easy-to-use functionality for each of these components of the data collection pipeline, which we expand next. instruction a markdown document that defines a task and instructs annotators how to complete the task. it supports various formatting options, including images and videos. tutorial additional training material provided in the form of multiple-choice questions with provided answers that workers can use to gauge their understanding of the instruction. crowdaq received many messages from real annotators saying that tutorials are quite helpful for learning tasks. exam a collection of multiple-choice questions similar to tutorial, but for which answers are not provided to participants. exam is used to test whether an annotator understands the instructions sufficiently to provide useful annotations. participants will only have a finite number of opportunities specified by the requesters to work on an exam, and each time they will see a random subset of all the exam questions. after finishing an exam, participants are informed of how many mistakes they have made and whether they have passed, but they do not receive feedback on individual questions. therefore, data requesters should try to design better instructions and tutorials instead of using exam to teach annotators. we restrict tutorials and exams to always be in a multiple-choice format, irrespective of the original task format, because it is natural for humans to learn and to be tested in a discriminative setting. @xcite an important benefit of using multiplechoice questions is that their evaluation can be automated easily, minimizing the effort a requester spends on manual inspections. another convenient feature of crowdaq is that it displays useful statistics to requesters, such as the distribution of scores in each exam and which questions annotators often make mistakes on, which can highlight areas of improvement in the instruction and tutorial. below is the json syntax to specify tutorial-s/exams (see fig. 3 and fig. 4 in the appendix). task for example, if we are doing sentencelevel sentiment analysis, then a task is to display a specific sentence and require the annotator to provide a label for its sentiment. a collection of tasks are bundled into a task set that we can launch as a group. unlike tutorials and exams where we only need to handle multiplechoice questions in crowdaq's implementation, a major challenge for task is how to meet different requirements for annotation ui from different datasets in a single framework, which we discuss next. ## customizable annotation interface it is time-consuming for non-experts on the frontend to design annotation uis for various datasets. at present, requesters can only reuse the uis of very similar tasks and still, they often need to make modifications with additional tests and debugging. crowdaq comes with a variety of built-in resources for easily creating uis, which we will explain using an example dataset collection project centered around confirmed covid-19 cases and deaths mentioned in news snippets. ## usage we have already deployed crowdaq at https: // www.crowdaq.com with load balancing, backend cluster, relational database, failure recovery, and user authentication. data requesters can simply register and enjoy the convenience it provides. for users who need to deploy crowdaq, we provide a docker compose configuration so that they can bring up a cluster with all the features with one ## related work crowdsourcing platforms the most commonly used platform at present is mturk, and the features crowdaq provides are overall complementary to it. crowdaq provides integration with mturk, but it also allows for in-house annotators and any platform that provides crowdsourcing service. other crowdsourcing platforms, e.g., crowdflower/fig-ureeight, @xcite hive, @xcite and labelbox, 10 also have automated qualification control as crowdaq, but they do not separate the format of an exam from the format of a main task; therefore it is impossible to use its built-in qualification control for non-multiplechoice tasks like question-answering. in addition, crowdaq provides huge flexibility in annotation uis as compared to these platforms. last but not least, crowdaq is open-source and can be used, contributed to, extended, and deployed freely. customizable ui to the best of our knowledge, existing works on customizable annotation ui, e.g., mmax2 11 (müller and strube, 2006), palinka (orȃsan, 2003), and brat 12 (stenetorp et al., 2012), were mainly designed for in-house annotators on classic nlp tasks, and their adaptability and extensibility are limited. amti is a command line interface for interacting with mturk, 13 while crowdaq is a website providing one-stop solution including instructions, qualification tests, customizable interfaces, and job management on mturk. amti also addresses the reproducibility issue by allowing hit definitions to be tracked in version control, while crowdaq addresses by standardizing the workflow and automated qualification control. sprout by bragg and weld (2018) is a metaframework similar to the proposed workflow. they focus on teaching crowd workers, while crowdaq spends most of engineering effort to allow requesters specify the workflow declaratively without being a frontend or backend expert. ## conclusion efficient data collection at scale is important for advancing research and building applications in nlp. existing workflows typically require multiple annotators, which introduces overhead in building annotation uis and training and filtering annotators. crowdaq is an open-source online platform aimed to reduce this overhead and improve reproducibility via customizable ui components, automated qualification control, and easy-to-reproduce pipelines. the rapid modeling improvements seen in the last few years need a commensurate improvement in our data collection processes, and we believe that crowdaq is well-situated to aid in easy, reproducible data collection research.