Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial (2021)

Abstract: In this tutorial, we present a portion of unique industry experience in efficient natural language data annotation via crowdsourcing shared by both leading researchers and engineers from Yandex. We will make an introduction to data labeling via public crowdsourcing marketplaces and will present the key components of efficient label collection. This will be followed by a practical session, where participants address a real-world language resource production task, experiment with selecting settings for the labeling process, and launch their label collection project on one of the largest crowdsourcing marketplaces. The projects will be run on real crowds within the tutorial session and we will present useful quality control techniques and provide the attendees with an opportunity to discuss their own annotation ideas.

## outline our tutorial includes the following sessions: ## prerequisites for the attendees we expect that our tutorial will address an audience with a wide range of backgrounds and interests. thus, even a beginner, each participant will be able to practice their skills in producing language resources via a crowdsourcing marketplace (this practical part will constitute most of our tutorial timeline). our tutorial contains an introduction that positions the topic among related areas and gives the necessary knowledge to understand the main components of data labeling processes. thus, the entry threshold is shallow to start learning and understanding the topic. only minimal knowledge on collecting labels is required: no knowledge on crowdsourcing, aggregation, incremental relabeling, and pricing is needed. we plan to share rich experiences of constructing and applying large-scale data collection pipelines while highlighting the best practices and pitfalls. as a result, any person who develops a web service or a software product based on labeled data and nlp will learn how to construct a language data annotation pipeline, obtain high-quality labels under a limited budget, and avoid common pitfalls. ## reading list we offer an optional reading list for the tutorial attendees. these references allow one to understand crowdsourcing annotation basics for maximizing the learning outcomes from our hands-on tutorial. we will nevertheless cover these materials during the workshop. skene (1979