In this article, the authors first provide a practical yet rigorous definition of crowdsourcing that incorporates "crowds," outsourcing, and social web technologies. They then analyze 103 well-known crowdsourcing web sites using content analysis methods and the hermeneutic reading principle. Based on their analysis, they develop a "taxonomic theory" of crowdsourcing by organizing the empirical variants in nine distinct forms of crowdsourcing models. They also discuss key issues and directions, concentrating on the notion of managerial control systems. Keywords crowdsourcing; outsourcing; e-business; social web; social media; advanced internet technologies; Web 2.0 Taking advantage of the growing acceptance of social web technologies, 1 entrepreneurs big and small are continually creating and experimenting with innovative sourcing models. One of the most buzzworthy models is "crowdsourcing," whereby businesses use the web to harness the efforts of a virtual "crowd" to achieve specific organizational tasks. Crowdsourcing thus takes advantage of many of the same technological features that characterize "social media," the technology that enables online communities through which users can interact with those of similar interests. However, crowdsourcing is distinct from pure social media applications in that it not only actively involves a diverse crowd of users but actively controls the online community through sophisticated management schemes involving compensation, copyright protection, and the like. Simply put, while social media sites place emphasis on the social aspect of community, crowdsourcing involves the management of a community via web-based collaborative technologies to elicit the community's knowledge and/or skill sets, and thus fulfill a pre-identified business goal. Address correspondence to Rajiv Kishore, Department of Management Science & Systems, School of Management, Jacobs Management Center, SUNY at Buffalo, Buffalo, NY 14260, USA. E-mail: rkishore@buffalo.edu In effect, despite its "buzzword" status, there is an identifiable core to crowdsourcing and this important goal-oriented strategic micro-outsourcing model has received scant attention from academic audiences. This article aims to correct this deficiency by developing an empirically grounded taxonomy of crowdsourcing models that can drive future research. The specific goals of this article are as follows: First, because of the confusing state of academic and popular discussion of crowdsourcing, we begin by putting forward a simple yet comprehensive definition that serves to differentiate crowdsourcing from e-business, e-commerce, outsourcing, Web 2.0, social media, and other germane concepts. We conceptualize crowdsourcing as a type of sourcing model that lies at the intersection of outsourcing and sophisticated internet technologies. In building the conceptual framework, we elaborate on how outsourcing, the web, and virtual "crowds" are harnessed and transformed by businesses big and small alike to produce the crowdsourcing model. Second, we undertake a comprehensive empirical investigation of the crowdsourcing domain by examining over 100 crowdsourcing organizations. This enumeration shows the range and variety of crowdsourcing models, and creates an exhaustive categorization of the types of these organizations, particularly with regard to the type of service or product that is outsourced and the role of online community users. Based on this empirical examination and our analysis of key inter-organizational differences, we then build a comprehensive taxonomy of nine distinct types of crowdsourcing models. Third, we analyze several key elements, issues, and unanswered questions surrounding the crowdsourcing model that arose from our empirical examination. Here we focus special attention on the organizations' variation in terms of three key elements: the extent of collaboration, the type of human intelligence tasks, and the use of systems of managerial control. We end by discussing the implications of our conceptual framework, our taxonomy, and our analysis for future empirical investigations of crowdsourcing. RULES OF CROWDSOURCING 3 Jeff Howe, who coined the term "crowdsourcing" in a seminal article appearing in Wired magazine (Howe, 2006) , defined it as "the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call." Howe's definition is sufficiently clear and comprehensive to serve as a useful beginning point. We especially stress how, in the original definition, the intersection of outsourcing and amorphous crowds-or the use of "open" crowds to complete any feasible organizational taskis explicit. Our conceptual elaboration thus builds on this base, with the major initial difference being our explicit incorporation of advanced internet technologies into the definition. One of the initial driving forces of this article, in fact, is to help academics approach crowdsourcing from a clear, defensible conceptual base that provides a framework for future research-a goal necessitated by the extensive conceptual "stretching" and confusion regarding crowdsourcing. Much of the confusion likely flows from the understandable yet inordinate attention by the media on several high-profile examples such as Innocentive. The problem is that these "sexier" examples are almost always in the "innovation" or "problem-solving" or "research and development (R&D)" arena, which represents only one small sub-category of crowdsourcing. The media has generally ignored the more "mundane" crowdsourced tasks such as transcription services, bookkeeping, or web site development. As a result, many have concentrated on the innovation facet and come to equate crowdsourcing with decentralized innovation. The existing academic literature almost exclusively concentrates on this facet of crowdsourcing, some even building their definitions around the "crowdsourcing-as-innovation" model. For example, Axel Bruns (2007) focused on the "ideas" aspect of this phenomenon, though he prefers his alternate term, produsage: "Such practices have also been described as 'crowdsourcing' (playing on the corporate term 'outsourcing'): the employment-usually in a figurative rather than monetary sense-of users as produsers of ideas" (p. 103). Similarly, Daren Brabham, one of the academic pioneers in this burgeoning field, has written a number of valuable conceptual pieces and case studies. Brabham delineated crowdsourcing according to "for-profit" or "nonprofit" (not-for-profit and governmental) applications; for the latter, he focused exclusively on the innovation and problem-solving role: "crowdsourcing is a legitimate, complex problem-solving model, more than merely a new format for holding contests and awarding prizes" (Brabham, 2008, p. 76). The problem with this definition is that it unnecessarily mixes an intended goal for crowdsourcing's future with its essential core; that is, Brabham intends to expand the crowdsourcing model beyond the for-profit sector, and wants to "establish a model that can have profound influence in the way we solve our world's most pressing social and environmental problems" (p. 75). Though this is indeed a laudable goal, we must not limit ourselves to the "crowdsourcing as innovation" approach. Nor do we believe that an explicit delineation into "for-profit" and "not-for-profit" uses is needed. As a result, we present the following practical but rigorous definition of crowdsourcing: Crowdsourcing is a sourcing model in which organizations use predominantly advanced internet technologies to harness the efforts of a virtual crowd to perform specific organizational tasks. As defined above and highlighted in Figure 1 , crowdsourcing can best be understood as lying at the intersection of three key elements: the "crowd," outsourcing, and advanced internet technologies. Though simple, this conceptualization allows for the clearest conceptual delineation of crowdsourcing from other relevant phenomena and existing e-business models. Accordingly, before turning to an analysis of the implications of crowdsourcing and our empirical examination, we briefly discuss these three "defining elements" and how crowdsourcing brings the three together to create a unique web-driven crowdsourcing model. Consistent with Jeff Howe's original definition, crowdsourcing can first be viewed through the lens of outsourcing. According to Lacity and Hirschheim (1993, p. 74), "outsourcing, in its most basic form, can be conceived of as the purchase of a good or service that was previously provided internally" from outside providers. Many information systems researchers also define it as contracting of various internal business needs or functions to outside service providers (Kishore et al., 2003) . Except for the fact that crowdsourcing is an emerging type of "small-scale outsourcing" (Gefen & Carmel, 2008) , both outsourcing and crowdsourcing share similar objectives in that they source in their business needs from outside entities to achieve their business goals. Despite this outward similarity, the characteristics of service providers and suppliers are radically different in the two models. While with traditional outsourcing, an entity subcontracts a business process or need-such as product design and manufacturing-with a handful of professional third-party companies (Lacity et al., 2010) , the crowdsourcing model turns to scale via an undefined, non-professional, and heterogeneous online "crowd" to source in these needs. In the crowdsourcing model, it is the online community that is expected to play the role of "service providers" as producers, innovators, and problem solvers. This implication is not trivial, in that crowdsourcing depends on the broad anonymous "masses" found on the web, with the expectation that a large-scale virtual crowd can outperform a handful of professionals.  Lastly, in crowdsourcing, as a means to reach a large scale, state-of-the-art web-based technologies, of which Web 2.0 is the current manifestation, are used to find and control the potential "crowd" of workers, negotiate contracts, and monitor work progress in real time. In fact, excluding the Web 2.0 capabilities, the basic idea of crowdsourcing-on a very limited scale-has hundreds of years of history. For example, in 1714, the British government needed a solution to "The Longitude Problem" that made sailing difficult and dangerous, killing thousands of sailors every year. The British government offered 20,000 pounds (worth $4.7 million in 2010) for lay people to find a solution, and the problem was ultimately solved by a working-class person with little formal education (O'Donnell, 2002) . As seen from this example, although the idea of crowdsourcing in itself may not be entirely new-at least in prototypical form-advanced internet technologies have made crowdsourcing practicable for an immeasurably wider audience, at a larger scale, for a greater number of products and services, and at greatly enhanced speed. Web 2.0 effectively allows firms to tap into these large-scale, latent virtual work forces in a way that was previously impossible. Advanced web technologies have, in effect, enabled companies to reach and search a tremendous number of potential workers at low cost and, as a result, any organization big or small can take advantage of these technologies to outsource a wide variety of organizational tasks to an online crowd. Overall, advanced internet technologies (notably Web 2.0 applications such as Wikipedia.org, Flickr, YouTube, Facebook, and Del.icio.us) represent a "rich" media source (Daft & Lengel, 1986 ) that is cheap, easy to use, interactive, and decentralized (e.g., McAfee, 2006) . This attracts large user bases with a minimal learning curve and low cost, and accordingly facilitates massive amounts of user-created content. Moreover, in a profound move from the "pre-Web 2.0" platform, Web 2.0 does not "impose on users any preconceived notions about how work should proceed or how output should be categorized or structured" (McAfee, 2006, p. 25). The technology effectively has no pre-defined knowledge structure; it is instead is a highly flexible, interactive, and user-centric system that readily accommodates changing human activity and knowledge. The social bookmarking web site del.icio.us is a prime example. With no predefined categories, knowledge categorization emerges dynamically and changes in structure as people interact with the system. The "folksonomy," a taxonomy that a multitude of anonymous individuals collaboratively make on the del.icio.us platform over time, works opposite to the established "digital economy," which has an up-front knowledge taxonomy predefined by a handful of professional business/system analysts. A unique strength of the social web is thus its capacity for collectively extracting the tacit knowledge latent in the crowd's brain, and for aggregating it into a structured and usable knowledge format. Unlike conventional knowledge management systems (e.g., internet portals or intranets), which process information from the perspective of a handful of business professionals, the social web platform virtually opens the system to the online community to aggregate the crowd's collective intelligence. What the crowdsourcing model is able to do is harness these key features of advanced web technologies to fulfill previously difficult and costly organizational endeavors. Crowdsourcing applications effectively funnel the crowd's energies with a clear business orientation and set of goals to be achieved. In short, the crowdsourcing model utilizes advanced internet technologies to tap the collective knowledge of the community and/or to harness the crowd to directly produce goods and services. Rather than managing and monitoring the complex contractual processes and working behavior of thirdparty companies (as in the traditional outsourcing model), the crowdsourcing model actively involves the community in the process of online production activity. However, to do this successfully, the crowdsourcing model embeds into the web-based platform control mechanisms such as intellectual property and copyright protection measures, compensation schemes (e.g., virtual money, credits, royalties, or real money), quality control procedures (e.g., peer or specialist review, commenting systems), competition schemes (e.g., voting or bidding), and escrow systems. Through the employment of such strategically value-adding, sophisticated "managerial control systems," the crowdsourcing model assimilates into a business strategy the crowd's collective human intelligence, wisdom, and skills. 4 It thus converts the web into a technology-driven sourcing model. Amazon's crowdsourcing web service, Mechanical Turk (mTurk), illustrates the nature of crowdsourcing. Amazon describes its mTurk crowdsourcing service as an "artificial intelligence" service that performs "HITs" (human intelligence tasks), tasks that cannot be easily duplicated or replaced by machines but are easy for human intelligence. Simply put, this service applies real persons' intelligence and skills in and through the artificial environment of heterogeneous computing RULES OF CROWDSOURCING 5 networks for solving some problem that is difficult for machines but easy for human. It means the crowdsourcing platform is a networked virtual production space where human beings interact with each other and perform economic activities for their own advantage. It is a production platform through which people and firms send requests and other people ("the crowd") return responses. Rather than being an automated information processing machine responding to remote procedure calls, it is a human knowledge-production, processing, and control platform. In simpler terms, the crowdsourcing platform connects people, employs their skill sets, and/or aggregates their knowledge, with the responding "crowd" being strategically managed by the firm for the purpose of production, innovation, or problem solving. With its unique characteristics and structure, crowdsourcing promises to be a new, useful business model for new types of tasks (HITs, innovation, problem-solving, collective intelligence), to have a unique set of pre-requisites (e.g., managerial control systems), to lead to new questions (e.g., optimal level of user collaboration and aggregation of collective knowledge), and to generate new controversies (e.g., exploitation, intellectual property rights). Developing a taxonomy to classify the many types of crowdsourcing applications currently being used in the real world and identifying some of the issues surrounding crowdsourcing are the central concerns of this article. In laying out our framework, it is especially important to note the differences between crowdsourcing and the still highly important "Web 1.0"-centered e-business model. The reason is that crowdsourcing has outward similarity in that it appropriates many of the tactics of e-business to buy and sell work forces instead of finished products. One of the ways crowdsourcing and e-business are distinct is in how the former engenders "meta-innovation," by which Brynjolfsson means "innovation not just in the technology but innovation in some of the institutions that manage the collaboration and that manage a global community working on problems" (Brynjolfsson and McAfee, 2007, p. 51). The context of "meta-innovation" is fundamentally predicated on the Web 2.0 technology platform, which has demonstrated its potential in harnessing the collective participation of online users at a global level. The "crowdsourcing as meta-innovation" concept indicates a radical shift from established e-business models, which have defined their own internal organization and assume external web users as passive consumers of a given product or service. Differently put, traditional e-business firms have their own internal staff and passive, external customers. E-business thus assumes strict boundaries, in contrast to the blurred boundaries of the crowdsourcing environment. Crowdsourcing also implies a modification and extension of the social web into a virtual production platform. The crowdsourcing model approaches web community users as a latent virtual labor force, and actively manages the online activity of engaged user communities to elicit the crowd's latent productivity and creativity in order to achieve a pre-defined business goal. Crowdsourcing effectively shows a big shift from the traditional e-business model in understanding the web and approaching the role of web users. While the social web has transformed web sites into collaborative social platforms and changed the concept of the consumer into that of content creator, crowdsourcing is now forging the social web platform into a collaborative production platform and the role of content creators into producers of goods and services. In brief, while the crowdsourcing model takes advantage of the social web to tap into the knowledge and skills of the "crowd" for the purpose of production, innovation, and problem solving, traditional ebusiness models use web technology as an additional channel of supply chain sourcing/logistics and marketing/distribution to reach potential suppliers and end customers, respectively. To help better understand these differences, it is useful to compare crowdsourcing with specific manifestations of ebusiness. Most useful is Rappa's taxonomy of traditional web business models (Rappa, 2011) . In Rappa's eight modelsBrokerage, Advertising, Infomediary, Merchant, Manufacturer, Affiliate, Community, and Utility-users' contribution to the production of goods and services is minimal. Although Amazon (Merchant model) and eBay (Brokerage model) have taken advantage of the aggregate information available in user reviews and seller-buyer ratings, users are not directly involved in the production, innovation, or problem-solving activities. Users' activity in online reviews and ratings is confined to the evaluation of finished products, sellers, or buyers. They are excluded from the process of innovation or the production of goods and services. In brief, in e-business, users are not considered as a potential work force to perform outsourced tasks or produce goods or services. Perhaps foreshadowing the rise of crowdsourcing, Rappa pointed out that his community model "is one of the more fertile areas of development, as seen in the rise of social networking" (Rappa, 2011). For his community-based business model, he gave examples of Web 2.0 web sites such as Flickr, Friendster, Facebook, and Orkut. As he correctly mentions, however, these web sites' revenues are largely "tied to contextual advertising and subscriptions for premium services" (Rappa, 2011) . Although online users play a critical role in this "community" model, community members are still not involved in the process of creating, innovating, or producing ideas, goods, and services. At best, they gather into a virtual space for social activities with those who share similar interests. Overall, as shown in Rappa's web business models, in the e-business model users are defined as consumers only-with the one exception being the community model, where users play the role of social content creators through postings, tagging, rating, and so on. In contrast, crowdsourcing appropriates certain advantages of the social web, as discussed above, and has established itself as a collaborative platform for facilitating collective content creation. It extends or modifies social web features into an outsourcing platform through which potential online workers are involved in the process of production (Bruns, 2007) , innovation, and problem-solving (Brabham, 2008) . At the same time, it transforms the role of content creators into producers of goods and services, and incorporates into the social web platform strategically designed managerial control systems. This role of active producer is in sharp contrast to the traditional e-business model that assumes web users to be passive consumers (Bruns, 2007) , and where the web site is considered largely as part of the firm's supply chain, logistics, and marketing functions for the efficient procurement of materials and other inputs from suppliers as well as for the supply of finished goods and services to end customers. As stated earlier, the crowdsourcing model instead uses web technology not only as a medium of e-commerce, but as a production, innovation, and labor platform. We summarize these different assumptions of the e-business and crowdsourcing models in Table 1 . It is important to also differentiate crowdsourcing from a related phenomenon, the "wisdom of crowds" (Surowiecki, 2004) . 5 The key assumption in the "wisdom of crowds" is that the opinions, insights, ideas, and knowledge of the "many" can be better than that of a given expert. To have a "wise" crowd, in Surowiecki's (2004) framework, there are four prerequisites: (1) cognitive diversity, by which each individual involved has some private information; (2) independence, wherein each person's opinion or decision is not influenced by those around them; (3) decentralization, through which individuals can specialize and tap into local fonts of knowledge; and (4) aggregation, which stresses the importance of structural mechanisms for translating many private opinions or decisions into a collective decision (p. 10). If one compares these four prerequisites with the three defining elements of crowdsourcing, one may see that the two phenomena overlap in certain respects but diverge in important ways. On the one hand, although both phenomena explicitly rely on the "crowd," the wisdom of crowds does not necessarily rely on either advanced internet technologies or on outsourcing. For instance, governments and nonprofit organizations and informal work groups, among others, have implemented the "wisdom of crowds" in their organizations in order to tap the crowd's (i.e., employees') collective wisdom. This is an informal and internal organizational process that is removed from a formal "outsourcing" endeavor. Moreover, many such temporary, internal efforts rely on in-person, low-tech formats (e.g., citizen juries, open space forums, design "charettes," future search, appreciative inquiry, etc.). The wisdom of crowds, in short, is not necessarily a web-based business model and, as a result, does not always represent a form of outsourcing, Web 2.0, nor crowdsourcing. On the other hand, not all manifestations of crowdsourcing are examples of the "wisdom of crowds." To start, many of the "mundane" tasks outsourced (e.g., transcription services, bookkeeping) are not designed to tap into the crowd's "wisdom" or opinions but rather its skills. Moreover, many of the "knowledge-building," "innovation," or "solution" manifestations of crowdsourcing do not employ the four prerequisites of wise crowds. There are almost always element #1 (diversity) as well as element #3 (decentralization). However, there are not always elements #2 or #4 (independent decisionmaking and aggregation of opinions). Most notably, many of the crowdsourcing sites that at first glance appear designed to tap into the crowd's collective wisdom fail completely on element #2. For instance, on most of the consumer "ratings" sites and "investing" sites we examined, as well as many of the "idea-generation" sites, users can only add their opinion/rating after seeing the existing ratings. This is a stark violation of the wisdom of crowds concept. It is thus "crowdsourcing" but not "wisdom of crowds." In short, though many think of the "wisdom of crowds" when they hear the word "crowdsourcing," in part due to the press given to high-profile "innovation" and "problem-solving" crowdsourcing cases such as Innocentive, the two phenomena are only partially ovelapping. Instances of wisdom of crowds with a clear business purpose that rely on advanced internet technologies conform to our definition of crowdsourcing; likewise, instances of crowdsourcing that fall in the "innovation" and "knowledge-building" area and employ Surowiecki's four Overall, the wisdom of crowds is a fascinating, potentially far-reaching phenomenon that is worthy of further, focused research. However, the current study only concerns those manifestations of the wisdom of crowds that overlap with crowdsourcing. For the above reasons, a complete examination of the wisdom of crowds phenomenon is beyond the scope of the present article. Our sample comprises 103 of the most well-known Englishlanguage crowdsourcing web sites in 2009. In this section, we detail the specific procedures we undertook for selecting cases and gathering the data. Our primary goal was to ensure that we had sufficient numbers and variability in examples of crowdsourcing organizations, a term that highlights several conditions we placed on candidates for inclusion. First, we were only interested in "pure" examples of crowdsourcing-that is, organizations for which crowdsourcing is their core business, rather than organizations that have merely used crowdsourcing as part of their daily business (e.g., the pharmaceutical firm Eli Lilly has used crowdsourcing to help vet ideas and assist with innovation, but it is certainly not a core part of their business). We likewise excluded from our sample Dell's IdeaStorm community (www.ideastorm.com), which is used to collect online users' product ideas. We also excluded small-scale crowdsourcing "projects" or one-shot "competitions" for a similar reason, such as a "How Many SUVs are on Your Block?," a 2007 crowdsourcing project of the radio station WNYC's Brian Lehrer Show, 6 and one-time idea competition events by RedHat (http://redhatchallenge.com) and RuckusNation (http:// ruckusnation.com/). We also excluded sites that were not in English, such as www.a-better-tomorrow.com (German), www. myfab.com (French), www.battleofconcepts.nl (Dutch), and www.idealiike.fi (Finnish). The data for the crowdsourcing organizations were gathered and analyzed in June and July 2009. In the absence of a comprehensive list of the universe of crowdsourcing organizations, it was not possible to conduct a random sample of the population of crowdsourcing organizations. Instead, we aimed for a sample that was sufficiently large and diverse, and in order to obtain such a sample we undertook a multiple-search strategy for finding and analyzing cases for the study. Web sites were a major source of sample gathering; given that we are looking at a Web-based phenomenon, our search for crowdsourcing organizations was effectively a search for crowdsourcing organizations' web sites. First, we employed a form of "convenience sampling" that can be characterized as a web version of "snowball sampling," with new candidates being "nominated" by existing subjects as they are mentioned and/or hyperlinked in the existing subjects' web sites. We began our search for candidate organizations in the "birthplace" of crowdsourcing, Jeff Howe's seminal Wired article and his crowdsourcing web site: 1. http://www.wired.com/wired/archive/14.06/crowds.html 2. http://crowdsourcing.typepad.com We also looked at Howe's recent book (Howe, 2008) . We considered all of the crowdsourcing organizations mentioned in these places candidates for inclusion and proceeded to look through the candidate organizations' web sites both to determine whether they should be considered "crowdsourcing" organizations consistent with our definition and to look for mentions of additional candidate crowdsourcing organizations. This was where the "snowball" analogy came into play. For instance, Howe's Wired article and web site mentions several crowdsourcing web sites, including istockphoto, ShutterStock, DreamsTime, GettyImages, iFilm (which later merged with spike.com), InnoCentive, YourEncore, NineSigma, and mTurk. All of these sites conformed to our definition of crowdsourcing and were included in our sample. In the course of assessing the nature of the crowdsourcing business for each of these sites, we also looked for mentions or hyperlinks to additional crowdsourcing organizations. 7 We also looked at three sites we knew to contain directories (albeit incomplete) of a variety of crowdsourcing endeavors: 1. http://www.crowdsourcingdirectory.com 2. http://www.openinnovators.net/list-open-innovationcrowdsourcing-examples 3. http://en.wikipedia.org/wiki/Crowdsourcing Similar to our procedure noted above, we analyzed all of the organizations mentioned on the directory in order to assess whether they were actually examples of crowdsourcing and to see if they mentioned any other examples of crowdsourcing. In conjunction with this "snowball" approach to building our sample, we undertook the following additional steps in order to search for candidate crowdsourcing examples: • We looked at all organizations noted in all academic articles found via ScienceDirect and Articles+ searches on the term "crowdsourcing." • Google search (including web site search and GoogleBlog search) for "crowdsourcing" and "wisdom of crowds." For each of these search terms, we looked at the first 100 sites returned by the search. • Blog searches using Technorati and BlogPulse for "crowdsourcing" and "wisdom of crowds." For each of these search terms, we looked at the first 100 sites "returned" by the search. • We looked at the Wikipedia entries for "crowdsourcing" and "wisdom of crowds." • Alexa search for the "10 related sites" for each of the organizations in our sample; for example, on the Alexa home page (www.alexa.com), we entered "Innocentive.com" in the "Lookup sites" box, hit enter, then clicked on the "Related Links" tab, which returned 10 "similar" sites; Innocentive's related links included an additional candidate for our sample, (SellaBand.com). The reason for extending the search beyond "crowdsourcing" is our supposition that there is considerable "conceptual stretching" of the term-that is, that many use the term to describe web phenomena that do not conform to our definition. At the same time, as noted earlier, many web phenomena described using the term "wisdom of crowds" do represent crowdsourcing endeavors. Accordingly, we employed multiple search terms. These methods resulted in close to 200 candidate organizations' web sites for our study. In line with our methods noted earlier, to verify if they contain the characteristics of the crowdsourcing model, we reviewed each web site-including its "Mission Statement," "FAQ," "About Us" or equivalent pages, the role of web components, and the overall 'business flow'-to determine if it conformed to our definition. We then removed those sites from our list that either did not conform to our definition of crowdsourcing or were in a language other than English. Our final sample of 103 organizations represent the most well-known English-language crowdsourcing sites at the time of the study. One interesting finding from our search is that some crowdsourcing web sites were affiliated with each other. For example, a crowdsourcing Wiki (Knol.google.com) and GWAP.com were owned by Google, mTurk (www.mturk.com) and QuestVille (askville.amazon.com) by Amazon, and a realname based Wiki, Citizendium (www citizendium.org) by Wikipedia. In addition, a successful T-shirt crowdsourcing web site (www.threadless.com) was also running a necktie design crowdsourcing site (www.nakedandangry.com), and a business trend watching site was affiliated with a similar crowdsourcing web site (www.springwise.com). Understanding the potential constraints of our sampling method, we believe the data offer a meaningful portrait of variation in crowdsourcing organizations, and that our data and findings provide an important general view of crowdsourcing that allows us to identify areas where opportunities and challenges exist. In a situation such as this where little is known about the phenomenon in question, and where the boundaries between crowdsourcing and its context are not yet clear, the literature would benefit greatly from what Gregor (2006) referred to as "taxonomic theory." One of our key goals in this article is thus to add to the literature and provide a framework for future studies by constructing a comprehensive taxonomy built on a rigorous empirical analysis. To categorize the organizations, we first conducted a comprehensive content analysis (Krippendorf, 2004) 9 of the nature, forms, and uses of crowdsourcing on each organization's web site. Specifically, we analyzed all relevant mission, background, business flow, and other key web documents on each web site from four key perspectives: (1) the type of service or product being outsourced, (2) the role of community users, (3) the level of collaboration, and (4) the type of managerial control systems employed. Using these data, the web site content interpretation and taxonomy-building process was also guided by the hermeneutic reading principle, wherein "the harmony of all details with the whole is the criterion of correct understanding" (Gadamer, 1976, cited in Klein and Myers, 1999, p. 71). 10 Application of this principle involved situating each type of crowdsourcing task, each community role and level of collaboration, and each element of the managerial control system within the contextual whole of existing literatures on e-business taxonomies (e.g., Rappa, 2011), outsourcing (e.g., Lacity & Hirschheim, 1993) , crowdsourcing (e.g., Howe 2006), and new media and the social web (e.g., Brynjolfsson & McAfee, 2007) . This situated, or contextualized, approach was critical to avoid adding another de-contexualized metaphor to the list of crowdsourcing buzzwords, such as "open collaboration." In short, through a contextualized analysis of organizational variation along four key dimensions, we developed a taxonomy consisting of nine distinct crowdsourcing models. Table 2 summarizes our taxonomy along with the outsourcing area, community user role, level of collaboration, and type and level of managerial control systems that characterize each model. 12 In the following section, we discuss each of the nine models in turn. In the subsequent section, we then turn to an analysis of the broader conceptual, theoretical, and practical issues raised by organizational variation on the key dimensions that formed the basis for differentiating the models. TAXONOMY OF CROWDSOURCING MODELS The crowdsourcing model embeds major business and control processes in its web platform. In the intermediary model, such as Amazon's mTurk, web users serve as a virtual work force who follow the process of (1) Find, (2) Finish, and (3) Earn through the web. 13 They search through the lists of "HITs" posted by labor seekers, choose and finish the tasks they feel qualified to accomplish, and earn monetary rewards. The InnoCentive crowdsourcing web site has a similar process. Web users around the world find research problems posted by solution seekers, submit their proposed solution, and potentially earn rewards for their work from the solution-seeking company. In the meantime, as an intermediary, InnoCentive is actively involved in the various stages of documentation and working processes to facilitate communication and address intellectual property and copyright issues. In this model, R&D problem-solving activity is outsourced to the online community, and community users accomplish tasks by playing the role of innovative problem solvers such as scientists, researchers, and engineers. $0 to $99, $100 to $4,999, and over $5,000, respectively.Escrow: Low, Medium, and High indicate low-level intervention (e.g., checkbox on the web), medium-level intervention (e.g., e-mail exchange of contract), and high-level intervention (e.g., PDF, online form, and fax exchanges of contract), respectively. Buyer-Worker rating: Low, Medium, and High indicate no rating system, more than two steps of browsing needed to find the rating, and ratings immediately visible when clicking user IDs, respetively. Voting and commenting (for products and services): Low, Medium, and High indicate no rating systems, more than two steps of browsing needed to find the voting and/or comment result, or immediately visible when clicking the completed task list, respectively. 9 10 G. D. SAXTON ET AL. The citizen media production crowdsourcing model generates revenue through user-generated news, TV programs, or commercials, and shares profits with citizen reporters or media producers. In this model, the crowdsourcing media web site is not a supply chain to resell the paper-based news to online users after processing them into digital format; instead, the crowdsourcing web space is a collaborative media content production platform, where heterogeneous users create or directly post news content from their perspectives. So, in the citizen media model, the "readers, or news audience, are no longer passive consumers of news produced by a few privileged, and sometimes arrogant, reporters. They are active producers of the news they will consume at the end of the day" (Min, 2005, p. 20), and earn profits for the media content they contribute. At the same time, citizen media companies are involved in the collaborative media production process by addressing the issues of copyright, intellectual property, and quality control, utilizing sophisticated managerial control systems. CambrianHouse, a collaborative software development company, embeds into a web platform sophisticated software development processes, from idea selection to software development to product sales. They source in resources necessary for software development from their online community. The sourcing areas range from business idea initiation through software design to product development and sales. Once a community member posts a product idea on the web site, other members vote weekly for the best idea, and the selected product idea is entered in a quarterly tournament designed to select the most feasible product items. Once an idea is officially selected, the idea contributor is given virtual seed money (called "Cambros"), which can be used to hire the necessary workforce from among online community members, and a separate virtual "project room" is assigned where a team of software developers share or store ideas. In the meantime, the crowdsourcing company conducts market research with the selected product ideas, facilitates the product development process, and shares profits from the successful product with involved project members. In this model, the processes of product design, decision making, market research, marketing, and software development are outsourced to online users, who perform the roles of web designer, programmer, project manager, and even marketer. As in the citizen media production model, the royalty-free stock photo crowdsourcing sites (e.g., iStockPhoto.com and ShuttersStock.com) are not supply chains to resell pictures supplied by a limited number of professional photographers. Rather, the web site is a platform to source in digital pictures from community users around the world. Approved community users upload and sell their digital pictures through the company-provided web platform, and they are rewarded based on the download count of their pictures. To become a member, community users not only have to pass the quality-review process but comply with rigorous copyright policies. Also, some record-proven community members make contracts with the crowdsourcing web site to screen the quality of new members' pieces, and they share a higher percentage of profit from the sale of their pictures. In the digital goods sales model, the photo creation, photo tagging, and quality-control processes are outsourced to community users. The company is involved in the production process by providing market demand information and policies to address potential copyright and intellectual property issues. Threadless, Zazzle, and Fluevog manufacture consumer products (e.g., T-shirts, mugs, shoes, calendars, and so on) out of designs submitted by online community members. Threadless, a community based T-shirt company, outsources Tshirt design to online users through an ongoing open call for design submissions. Once a T-shirt design is submitted, community members begin voting, scoring, and commenting about the design. Each week, Threadless tabulates and presents the results to the community, picks around 10 designs from which to manufacture T-shirts (based on voting scores, comments, and the "I'd buy it" count), and rewards the selected designers with credit points that they can use to buy T-shirts or exchange for cash. Also, Threadless rewards 10 points to community members who submit their pictures wearing a Threadless T-shirt, and promotes these pictures as T-shirt models on the web site. In this crowdsourcing model, the process of T-shirt design is outsourced to an online community, and the T-shirt models are sourced in from the community using a monetary reward. The peer-to-peer social financing model, such as Kiva.org (see Anderson & Saxton, 2012 , for a detailed analysis), bypasses traditional banks and financial institutions by directly connecting lenders and borrowers. Through the web site interface, lenders and borrowers negotiate interest rates directly with each other, and lenders make the individual loan decisions and use their own funds to make the loans. In the peer-to-peer social financing model, web users thus play the role of both banker and borrower, and the lending decision-making and funding processes are effectively outsourced to a geographically dispersed crowd of virtual lenders. The participatory social web technology has created a plethora of product-recommendation social web sites (ReeVoo.com, Buzzillions.com, ThisNext.com, Zebo.com, and Crowdstorm.com, etc.) where users share their product reviews RULES OF CROWDSOURCING 11 or recommendations. An Intelliseek survey highlighted the influence of such social web sites on consumer purchase behavior in showing that online peer recommendations exercise 50% more influence in shaping consumers' decisions of purchasing products than traditional TV and radio commercials. 14 By taking advantage of the social web, a consumer report model sources in consumer reviews or essays on products, local contractors, or healthcare professionals from online community users by using review quality control measures. Noteworthy is Epinions.com, which motivates consumers to provide product reviews through its "Income Share" program. According to the Epinions.com web site, "the best reviews are likely to earn five to ten times as much as mediocre reviews on the same product." 15 This program uses an "Eroyalties credits" system to decide users' compensation scale depending on the quality of the product review. Also, by differentiating mediocre reviewers from high-quality "category lead" or "advisor" reviews with different recognition and reward scales, they induce community users to provide high quality reviews. In this consumer report crowdsourcing model, the review process is outsourced to online consumers who, enticed by strategic compensation measures, play the role of consumer reporters. The knowledge base building model aggregates human intelligence information or knowledge of specific topics. This model has been demonstrated by "wikis" as an effective knowledgebuilding method, and is being followed by Citizendium.com and Knol.Google.com with different authoring procedures and measures. 16 In addition, domain-specific information or knowledge-building web sites implement diverse methods to elicit domain experts' contributions. For instance, a buildings data-providing web site, Emporis.com, uses a rigorous peer-review system to evaluate the competence of membership applicants who are supposed to submit buildings-related data, while a business trendspotting web site, Trendwatching.com, employs reward schemes to draw quality business-trend information from local people around the globe. In the knowledgebuilding crowdsourcing model, the information-or knowledgegeneration process is outsourced to community users, and diverse types of incentive measures and quality control mechanisms are utilized to elicit quality knowledge and information that may be latent in the virtual crowd's "brain." Although computer algorithms have evolved to automatically identify or match data patterns out of massive data sets, many areas still require human input, interaction, and evaluation to train or refine the data. Especially when the amount of data is huge, its relations are complex, or its attributes are emotional, massive human interactions with computer systems may be beneficial. For example, realizing that computer algorithms cannot decipher distorted text images perfectly, reCAPTCHA outsources the text-deciphering job to an online crowd by opening its API (Application Programming Interface) to other web sites around the world. 17 Also, a social search engine web site, Scour.com, lets users vote or comment on the search results returned by Yahoo, Google, and MSN in a single web interface by providing Visa gift cards based on the user's contribution. This web site purports to enhance search research by filling the gap between search results and relevant information, based on user feedback. In a similar vein, GWAP.com provides a game interface with web users to tag images and audio, and the aggregated tag information is used for machine training purposes to improve search results. The collaborative science project model takes advantage of human participation to complement shortcomings of computer algorithms by introducing large-scale, diverse online communities into the machine learning process. In this model, web users play the role of research assistant, distorted text transcriber, search result evaluator, and so on. Having discussed our taxonomy of crowdsourcing models, we now discuss the broader implications of three salient issues and points of differentiation that arose from our empirical analysis of crowdsourcing firms: (1) the product or service that is being outsourced, (2) the level of collaboration, and (3) managerial control systems. In an important sense, all crowdsourcing endeavors deal with what Amazon mTurk calls HITs, or tasks that even in an Information Age cannot be easily replaced or replicated by pure automated computing algorithms. As we have seen in the above analyses, there is a broad array of activities, in both the "labor and production" and "problem-solving and innovation" categories, being outsourced via the crowdsourcing platform. A key dimension differentiating project types is the level of task complexity. At the low end of complexity are the HITs prevalent on Amazon's mTurk service, such as photo tagging, simple data coding, transcription services, or data de-duplication, as well as the outsourcing of ratings/evaluations to community users commonly seen on many other crowdsourcing sites. At the other end of the spectrum, the current crowdsourcing market also covers much more complicated human intelligence work, including such business areas as R&D, accounting, product design, peer-to-peer finance, digital media production, software development, citizen journalism, and consumer reporting. This shows that combined with the low cost and easy-to-use social media technology platform, crowdsourcing is diversifying sourcing mechanisms, creating new markets and/or overriding the market share of traditional businesses by managing the scale of online communities in a flexible way for performing different tasks of varying levels of complexity. It is in this sense that crowdsourcing is, to paraphrase Innocentive's mission statement, "innovating the way we innovate" by sourcing the knowledge, creativity, and innovative ideas necessary for their business from an online community. Collaboration is one of the buzzwords frequently used to explain crowdsourcing. It pertains to the opening of internal information to outside online community members, with the intention of drawing answers or solutions from that community. However, we find great variation in the extent and nature of the collaboration that occurs on the crowdsourcing platform. For instance, most of the "intermediary" organizations do not have any collaboration. In effect, the key in crowdsourcing is not collaboration itself. Rather it is the control of the community-and, if it does exist, the collaboration within that community-to elicit the best skill sets, ideas, or solutions while simultaneously protecting intellectual property. For instance, although InnoCentive broadcasts research questions as open calls to online community members, they are allowed to solve the research questions individually or only among agreed-upon team members through the confidential Team Project Room. Thus, InnoCentive allows partial but not open collaboration. This serves to protect intellectual property, which could otherwise be compromised through broad collaboration with unknown community members. The case is similar with oDesk. To perform specific tasks assigned by customers, the service provider can work alone or invite other community members to form a project group. That is, while oDesk allows either "blind" or partial collaboration strategies to complete tasks, full collaboration is prohibited. In contrast, Threadless has adopted an open collaboration strategy. The company accepts T-shirt designs from an online community, and any community member can vote or comment on the submitted designs. Across all the crowdsourcing cases, the decision of whether to work alone or as a group is predicated upon the scope, copyright and intellectual property needs, and complexity of the given task. Although online community users play the role of producers, innovators, and problems solvers, they are, unlike in traditional organizations, not employees of crowdsourcing networks. They are only temporarily connected to the organization for a specific task or project. Therefore, the management focus in crowdsourcing is not on the enhancement of potential workers' skill or on employee retention. Rather, its management strategy is to uncover and leverage the skills and knowledge of potential workers and to draw more intelligent, diverse, and capable people into the crowdsourcing platform. It is here that managerial control systems become critical. In crowdsourcing, managerial control systems can be defined as processes, policies, and mechanisms designed to achieve specific business goals by effectively facilitating user participation and eliciting their best knowledge and skills. We evaluate three key areas of managerial control systems we found especially important in our examination: (1) compensation schemes, (2) trust-building systems, and (3) voting and commenting mechanisms. A sound reward/compensation scheme is generally a key element of any managerial control system (Kirsch, 2004) . Preliminary evidence (Lakhani et al., 2007) suggests that financial and other rewards are a key motivator of individual involvement in the crowdsourcing transaction. Lending support to this assertion, we found the great majority of the crowdsourcing organizations in our study (89 out of 103) employed some sort of compensation scheme to encourage user participation (see the Appendix for a full list of organizations and their managerial control systems). Moreover, roughly half (50 sites) provided a monetary or equivalent reward. In such schemes, the amount varies greatly by crowdsourcing model and by the type and complexity of the task. For instance, for complex and knowledge intensive R&D tasks, InnoCentive announces preset monetary awards that can be in the thousands of dollars. For trivial digital-image tagging work, in contrast, mTurk pays only a few cents. We also found a great variety of reward types, including virtual money, paper checks, gift cards, royalty points, etc. For example, OhmyNews motivates citizen reporters with a virtual "tip jar" that is filled according to the number of comments an article receives from readers. iStockPhoto employs a check-delivery method based on download counts of contributors' pictures. Threadless gives credit points for T-shirt design submissions or web site referrals, and pays cash for pictures of users wearing their Threadless brand T-shirt, which are then used as promotional images on the web site. Cambrian House provides virtual money (called "Cambros") for weekly and quarterly idea competition winners that can be used to hire other community members to assist in the launch of their projects. And among those that did not provide compensation, nonprofit organizations (e.g., Stardust@Home) were prominent; these organizations often choose instead to recognize highquality contributors' efforts with a prominent acknowledgment on the organization's web site. Online marketplaces are filled with uncertainty and risk. In the virtual working mode, a worker's skill set cannot be verified until the given task is delivered to buyers, and workers in turn are uncertain if they will be compensated by buyers for the work they complete. These uncertainties must be reduced to make crowdsourcing transactions successful. One strategy for reducing uncertainty and enhancing trust, predominant in the intermediary crowdsourcing model, involves market-driven buyer/worker rating systems. Findings from the e-commerce context (Ba & Pavlou, 2002) suggest that such rating systems not only create trust but also lead to premium prices even in the absence of previous buyer/seller interactions. Also RULES OF CROWDSOURCING 13 commonly employed are legally binding escrow systems, which have been found to significantly increase trust in online marketplaces (Pavlou & Gefen, 2004) . Such systems help ensure secure financial transactions through the use of third parties, who release funds only when both the buyer and the seller are satisfied with the quality of delivered goods or services as described in the contract. In our sample, third-party escrow services are a popular trust-building method especially in the intermediary crowdsourcing model, including R&D, application development, and office work, etc., as well as in the peer-to-peer financing model. Voting, rating, and commenting are popular tools in both social media and e-business to express community members' opinions or to evaluate the quality of others' ideas, products, and services. Crowdsourcing also employs these tools to exchange and evaluate ideas about products and services as well as to check buyers' and sellers' past history. Uniquely, crowdsourcing firms were also found to directly integrate the results of community-driven voting, rating, and commenting systems into their decision-making processes and compensation schemes. Such firms are dynamically employing these systems as strategic virtual management control tools to draw and refine product ideas, to predict consumer product preferences, to control product and community-member qualities, and to make compensation decisions. Threadless is a prime example of the use of such systems to create a "soft competition environment" within the user community. In their identification of the most popular usersubmitted T-shirt designs, Threadless uses participatory voting and commenting systems as a proxy for general ideas about consumer preferences. Votes and comments are thus used as the basis for rewarding community designers and product-selection decisions. Cambrian House also relies on voting and comment systems to bring to light the best product ideas, to refine selected product ideas and develop marketing strategies, and to make reward decisions. In the case of Connextions, a collaborative education material production platform, a commenting system is utilized to enhance peer-authors' textbook quality. As an open collaborative educational material product platform, participatory commenting and voting tools are not separable from their core business strategy of including faculty members, superintendents, and school administrators in the review process. The crowdsourcing model has been described from the perspective of a virtual production platform and new mediaenabled small-scale outsourcing. Along with examples, we have presented a comprehensive taxonomy of crowdsourcing models and identified several key issues in the employment of the crowdsourcing platform. A major contribution of this article is the identification of the role of managerial control systems, and it is deemed to be one of the most important research topics related to crowdsourcing. This is in line with research from Howe (2006 Howe ( , 2008 and McAfee (2006) , who argue that in a situation where the cost of hardware necessary for online activity is getting cheaper and web interfaces are getting easier to use, the real issues and challenges lie in the management of the online community as potential workers. Community users' direct involvement in the process of production or problem solving is a unique and essential characteristic of the crowdsourcing model, and this direct involvement necessitates a great role for community management. Still, crowdsourcing is an emerging area that is mainly driven by participatory social media technologies, and it has yet to draw much attention from researchers. Although Brabham (2008) and Bruns (2007) have published articles on the topic, they did not pay attention to the role of managing the community. Considering that (1) crowdsourcing is a sourcing mechanism directed toward the achievement of clear business goals, and (2) many crowdsourcing models employ diverse types of control mechanisms to manage collective intelligence and labor skills latent in the virtual space, more research about the nature of managerial control systems in crowdsourcing must be done. In particular, the identification of relationships between managerial control systems and production quality or communication quality could be an important research topic in the area of both empirical study and design science. Research on measurement methods also should follow to quantify the effectiveness of managerial control systems in relation to the virtual economy. Another issue we uncovered regards the importance of scale and the management of collective intelligence and labor. In crowdsourcing, information sharing and knowledge creation are not simply a matter of information processing out of enterprise resource planning (ERP) and customer relationship management (CRM) systems. It is more about management of human beings' collective intelligence and labor skills to achieve explicit business goals. As Nonaka (2007) illustrated through examples of successful Japanese firms, "it depends on tapping the tacit and often highly subjective insights, intuitions, and hunches" (p. 164) of others, and "making those insights available for testing and use by the company as a whole" (p. 164). However, the main difference lies in the context of its application. Whereas Nonaka's knowledge management context is predicated upon the proximity of geographical location and physical places of specific organizations, the crowdsourcing model acts upon the virtual space of global scale, which is dynamically redefined by the participatory new media technologies. Compared to the context of Nonaka's knowledge management, crowdsourcing takes advantage of the virtually limitless scale of online community at a global level to tap heterogeneous tacit knowledge and skill sets. In other words, the crowdsourcing model taps the productive potential of a crowd in the virtual space to accomplish 14 G. D. SAXTON ET AL. business goals, with the assumption that the crowd's collective intelligence is greater than that of a limited pool of professional intellects. It is here that our findings are instructive for those interested in the related "wisdom of crowds" phenomenon. Specifically, our identification and analysis of the notion of managerial control systems strongly suggests the fruitfulness of attempts to better understand the types of managerial control systems that can engender "wise crowds" in the online crowdsourcing platform. More broadly, future research should undertake in-depth studies of the different requirements and outcomes of various types of crowdsourcing projects. Our evidence shows that both the level of collaboration and the array of managerial control systems employed vary substantially with both the crowdsourcing model and the type and complexity of the task or project undertaken. For instance, in important ways, simple tasks such as transcription services will be organized much differently than tasks designed, for instance, to solve problems or to advance learning, understanding, and innovation. We now need to better understand the different control, collaboration, and organizational requirements of such tasks. We end by proposing potentially fruitful general avenues of research. The logical avenue of future research is twofold: (1) to study how variation in the components of managerial control systems impacts the "success" of crowdsourcing, and (2) to study how important different features of managerial control systems are to the distinct crowdsourcing models. In the first domain, we would be interested in predicting such phenomena as success or failure rates of crowdsourcing firms, number of users and visitors, and optimal levels of collaboration among different types of crowdsourcing, etc. And in the second domain, we might be interested in whether certain managerial control features-such as reward systems, different level of collaboration, voting and commenting or trustbuilding systems-are more important in determining success for "innovating," "producing," or "problem-solving" types of crowdsourcing organizations. The authors would like to thank the editor and the two annonymous reviewers of this journal for their valuable comments and suggestions. The reviewers and editors of the Fourth Global Sourcing Workshop (March 22-25, 2010, Zermatt, Switzerland) also provided helpful comments on an earlier version of this article.
A more fine-grained characterization has been used by Saxton et al. REF , who refers, among others, to the level of collaboration; compensation schemes, which reflect the amount of money one can earn; the control mechanisms; and trust-building systems.
Rules of Crowdsourcing: Models, Issues, and Systems of Control
Information Systems Management
Computer Science