Crowdsourcing has emerged as a new method for obtaining annotations for training models for machine learning. While many variants of this process exist, they largely differ in their method of motivating subjects to contribute and the scale of their applications. To date, however, there has yet to be a study that helps a practitioner to decide what form an annotation application should take to best reach its objectives within the constraints of a project. We first provide a faceted analysis of existing crowdsourcing annotation applications. We then use our analysis to discuss our recommendations on how practitioners can take advantage of crowdsourcing and discuss our view on potential opportunities in this area. It is an accepted tradition in natural language processing (NLP) to use annotated corpora to obtain machine learned models for performing many tasks: machine translation, parsing, and summarization. Given that machine learners can only perform tasks as good as their input annotation, much work in annotation centered on defining high quality standards that were reliable and reproducible, and finding appropriately trained personnel to carry out such tasks. The Penn Treebank and WordNet are probably the most visible examples in this community. Even now, this high quality route continues to be used in sister projects in extending these resources to other languages: HowNet (Dong and Dong, 2006) and EuroWordNet (Jacquin et al., 2007). An alternative to high quality annotation is to make use of quantity and the rule that redundancy in large data would act to filter out noise. The emergence of the Web made this a real possibility, where raw monolingual and parallel corpora, term counts and user generated content enabled the mining of large amounts of statistical data to train NLP models. In Web 2.0, it is also clear that the Web has made people available as resources to take advantage of. This trend reaches one logical conclusion when the web serves to network human service providers with those seeking their services. Although this process is described by many different terms, we use the term crowdsourcing throughout this paper. Crowdsourcing is a strategy that combines the effort of the public to solve one problem or produce one particular thing. "Crowdsourcing" has been used in the popular press to emphasize that the workers need not be experts but laymen or amateurs. While human subjects can be used to provide data or services in many forms, we limit our attention in this work on annotations for data useful to NLP tasks, and do not focus on the distributed nature of crowdsourcing. Crowdsourcing takes many forms that require different forms of motivation to achieve the end goal of annotation. In Games with a Purpose (hereafter, GWAP), the main motivator is fun (von Ahn and Dabbish, 2008a,b). Annotation tasks are designed to provide entertainment to the human subject over the course of short sessions. In Amazon Mechanical Turk (MTurk), the main motivator is profit. Providers create and list batches of small jobs termed Human Intelligence Tasks (HITs) on Amazon's Mechanical Turk website, which may be done by the general public. Workers who fulfill these tasks get credited in micropayments. While certainly not the only paid labor sourcing environment, Mechanical Turk's current ubiquity make "MTurk" a useful label to refer to this and other forms of computer mediated labor. Wisdom of the Crowds (WotC) is another form of crowdsourcing. WotC deployments allow members of the general public to collaborate to build a public resource, or to predict event outcomes or to estimate difficult to guess quantities. Wikipedia, the most well-known fielded WotC application, has different motivators that have changed over time. Initially, altruism and indirect benefit were factors: people contributed articles to Wikipedia to help others but also to build a resource that would ultimately help themselves. As Wikipedia matured, prestige of being a regular contributor or editor also slowed the ranks of contributors from the crowd to a more stable formalized group (Suh et al., 2009) . It is important to recognize that these different motivators crucially shape each form of crowdsourcing, changing key characteristics. Equally important is to note that the space of possible motivations and dimensions of crowdsourcing have not been fully 1 explored. Given raw linguistic data, what vehicle for annotation would be most fruitful to pursue? Thus far, there has been no systematic scheme for potential projects to follow. These shortcomings of the current work are serious, and we attempt to address them. In particular, we: • deconstruct crowdsourced annotation applications into pertinent dimensions and then give our subjective analysis of published crowdsourced NLP applications using the above dimensions (Section 0.3); • analyze the characteristic of the three genres of crowdsourcing approaches and make recommendations for the most effective means of obtaining annotations for sample NLP tasks (Section 0.4); • analyze these crowdsourcing instances and propose better crowdsourcing platforms (Section 0.5); • discuss the future of crowdsourcing annotations in the conclusion (Section 0.6). Although crowdsourcing is a fairly recent development, it is recognized as a growing and burgeoning research area, as evidenced by this journal's special issue as well as several works that have produced an overview of these methods from different perspectives. Most related in scope is Quinn and Bederson (2009) who describe a taxonomy of crowdsourcing (termed "Distributed Human Computation" in their paper). They divide crowdsourcing into seven genres: GWAP, Mechanized Labor (which we term "MTurk"), Wisdom of Crowds, Crowdsourcing, Dual-Purpose Work, Grand Search, Human-based Genetic Algorithms and Knowledge Collection from Volunteer Contributors. They deconstruct these genres along six dimensions and also discuss future directions towards utilizing crowdsourcing to solve other computational problems. While certainly a useful starting point, the genres are generally defined and not specific towards the task of annotation. As such, some of their genres are irrelevant and we feel are better combined from an annotation perspective. In parallel, Yuen et al. (2009) also surveyed crowdsourcing applications, categorizing them into six classes: initiatory human computation, distributed human computation, social game-based human computation with volunteers, paid engineers and online players. Their survey analyzes crowdsourcing from a social gaming perspective, differentiating the classes based on game structure and mechanisms. They also touch on the performance aspects in such systems, presenting references to primary studies that describe methods for best measuring the reliability of results coming from crowdsourcing frameworks. Yuen et al.'s work, however, does not go beyond its survey to suggest any critiques or analyses of the existing crowdsourcing literature. Aside from these two surveys that examined crowdsourcing in its wider definition, studies have also analyzed specific theoretical aspects. von Ahn and Dabbish (2008a) present general design principles for GWAPs, They articulate three game classes ( 2 "templates"). Each template defines the basic rules and winning conditions for a game in a way to make it in the players' interest to perform the intended computation. They also describe a set of design principles to complement the templates, and propose metrics to define GWAP computation success in terms of maximizing the utility obtained per player hour spent. In a similar vein, Jain and Parkes (2009) surveyed existing gametheoretic models for their use in crowdsourcing models, outlining challenges towards advancing theory that can enable better design. Focusing on the user's perspective, Roi (2009) suggests that participation can be improved through either useful tools or entertaining distractions that result in the production of desired data. They developed tools to assist users in managing their web bookmarks in a social setting, providing a tagging game for users to connect resources from each other. This environment helps the system acquire additional knowledge about tags and resources implicitly. Kazai et al. (2009) proposes a method for the collective gathering of relevance assessments using a social game model to instigate users' engagement. They discuss the approach in detail, and present the results of a pilot study conducted on a book corpus. The analysis revealed the relationships between the affordances of the system, the incentives of the social game, and the behavior of the assessors. Although there has been clear interests in harnessing crowdsourcing, the summative work thus far has concentrated on its mechanisms and design. Surveys have described past work in each crowdsourcing mechanisms but have not compared these application instances across a uniform set of criteria. Also, there has been little work to unify these frameworks with respect to annotating natural language data; although there clearly is much explicit interests in this area, as evidenced by recent workshops on the MTurk framework (Callison-Burch and Dredze, 2010) and on collaboratively constructed semantic resources (Gurevych and Zesch, 2009, 2010) To our knowledge there has been little that describes which crowdsourcing model is appropriate in which situation or task. It is also not clear when the traditional form of manual annotation by local experts is appropriate -are the days of highly-paid expert annotation over? A practitioner in need of annotations cannot rely on the existing literature to recommend which form of annotation her specific task should take. WordNet sense definitions as the possible labels. They obtained more than 100,000 labeled instances from a crowd of volunteers. Given these shortcomings, we first revisit crowdsourcing applications to distill a set of dimensions suitable for characterizing them. This work differs from Quinn and Bederson (2009), as we construct our five dimensions from the practitioner's perspective. However, there is certainly overlap, as some of their dimensions are preserved in our analysis, which are marked with asterisks. There is also certainly correlation between dimensions. Many of our five dimensions -motivation, annotation quality, setup effort, human participation and data character -have internal facets, which we briefly discuss in turn. A key part of our work is to also assign a range of possible values for a facet. This 3
In the REF 's taxonomy of the collaborative approaches for acquiring annotated corpora, our approach could be related to the Wisdom of the Crowds (WotC) genre where motivators are altruism or prestige to collaborate for the building of a public resource.
Perspectives on Crowdsourcing Annotations for Natural Language Processing
Computer Science