Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection (2021)

Abstract: Crowdsourcing from non-experts is one of the most common approaches to collecting data and annotations in NLP. Even though it is such a fundamental tool in NLP, crowdsourcing use is largely guided by common practices and the personal experience of researchers. Developing a theory of crowdsourcing use for practical language problems remains an open challenge. However, there are various principles and practices that have proven effective in generating high quality and diverse data. This tutorial exposes NLP researchers to such data collection crowdsourcing methods and principles through a detailed discussion of a diverse set of case studies. The selection of case studies focuses on challenging settings where crowdworkers are asked to write original text or otherwise perform relatively unconstrained work. Through these case studies, we discuss in detail processes that were carefully designed to achieve data with specific properties, for example to require logical inference, grounded reasoning or conversational understanding. Each case study focuses on data collection crowdsourcing protocol details that often receive limited attention in research presentations, for example in conferences, but are critical for research success.

## tutorial description crowdsourcing from non-experts is one of the most common approaches to collecting data and annotations in nlp. it has been applied to a plethora of tasks, including question answering @xcite , textual entailment @xcite , instruction following @xcite @xcite , visual reasoning @xcite @xcite , and commonsense reasoning @xcite . even though it is such a fundamental tool, crowdsourcing use is largely guided by common practices and the personal experience of researchers. developing a theory of crowdsourcing use for practical language problems remains an open challenge. however, there are various principles and practices that have proven effective in generating high quality and diverse data. this tutorial exposes nlp researchers to such data collection crowdsourcing methods and principles through a detailed discussion of a diverse set of case studies. the selection of case studies focuses on challenging settings where crowdworkers are asked to write original text or otherwise perform relatively unconstrained work. through these case studies, we discuss in detail processes that were carefully designed to achieve data with specific properties, for example to require logical inference, grounded reasoning or conversational understanding. each case study focuses on data collection crowdsourcing protocol details that often receive limited attention in research presentations, for example in conferences, but are critical for research success. we introduce the task of each case study, and do not assume prior knowledge. where possible, we highlight common trends, or otherwise key differences between the discussed case studies. relevance to the nlp community crowdsourcing techniques are commonly used, but rarely discussed in detail. this tutorial provides a detailed description of crowdsourcing decisions in complex scenarios and the reasoning behind them. nlp researchers aiming to develop new datasets, tasks and data collection protocols will find the content directly applicable to their own work. a strong understanding of data collection practices and the range of decisions they include will also aid researchers using existing dataset to critically assess the data they use, including its limitations. post-tutorial materials the tutorial videos, slides and other material will be made available publicly online following the tutorial. ## structure and content overview the tutorial spans three hours (180 minutes), and is divided into eight sections: introduction (10 min) a brief introduction to the tutorial structure, its goals, and the case studies. background (20 min) a high-speed recap of established crowdsourcing concepts and terms. we refer back to the content of this section in the case studies. this section includes the basic structure of a mechanical turk task (hit), typical incentive mechanisms, typical communication mechanisms, typical worker qualification and screening mechanisms, as well as relevant results about the demographics and expressed preferences of crowdworkers and the crowdworker community. case study i: multinli (45 min) we discuss the multinli @xcite corpus, with primary focus on experiments from subsequent papers that extend or evaluate the data collection protocol used to create this dataset. multinli is built around the task of natural language inference (a.k.a. textual entailment; @xcite : given two sentences, the task is to identify (roughly) whether the first sentence entails the second. we start with this case study not because of any unique success of the data collection protocol, but because multinli and the natural language inference task have emerged as a popular testbed for data collection methods and for relevant data analysis methods in nlp. topics include: case study iii: cerealbar (25 min) cereal-bar @xcite ) is a game designed for studying collaborative natural language interactions, released alongside a dataset of interactions between human players. 2 cerealbar emphasizes collaboration through natural language instruction between agents with differing abilities. each of the agents can be a human user or a learned model. cerealbar has been used to design and train systems that follow instructions by grounding them in the surrounding environment and acting in the environment. the game rules were explicitly designed with the intent of eliciting rich collaborative interactions across many instructions, for example by allowing a pair of players that is scoring well to continue playing for longer, thereby collecting more data from successful collaborations. the cerealbar data collection process included a development of a community of players, which has demonstrated behavioral and linguistic change over the crowdsourcing process. case study iv: quac (25 min) question answering in context is a dataset for studying information seeking dialogs between a student and a teacher @xcite . given a subject heading, a student questions a teacher, who responds by copying spans from a wikipedia article. the goal of the pair is to maintain a dialog of sufficient length without encountering too many unanswerable questions. the task is to play the role of the teacher: answering questions of an interested student. the collection protocol is unique in that two unreliable workers had to be coordinated for sufficient time to accomplish a meaningful dialog. quac collection relied on several strategies to keep partners from leaving interactions, such as allowing workers to simultaneously participate in multiple related dialogs, a feedback system teachers used to help students formulate questions, and scaling incentives that included punitive elements. case study v: socialiqa (25 min) so-cialiqa @xcite is the first large-scale benchmark to test model emotional and social reasoning through 38k questions about everyday situations. the distributional nature of social commonsense knowledge requires the answer candidates to cover the plausible and likely, as well as the plausible but unlikely, as opposed to right/wrong answer candidates as common in other qa benchmarks. socialiqa introduces a question-switching technique for crowdsourcing these unlikely answers, to overcome the possible stylistic artefacts in negative answers (e.g., negations, out-of-context responses; @xcite . additionally, to achieve large-scale and broad coverage, socialiqa used a multi-stage crowdsourcing pipeline to expand seed events from the atomic @xcite commonsense knowledge graph into full-fledged social situations. ## breadth the set of case studies covers a broad and diverse set of task types, including large-scale inference tasks (e.g., nli), small-scale interactive tasks (e.g., cerealbar), and multi-modal grounded tasks (e.g., nlvr). the aim of this broad distribution is to cover the most common task and data scenarios in nlp. we focus on details that are rarely discussed fully in papers. the set of case studies covers a broad and diverse set of task types, including largescale inference tasks (e.g., nli), small-scale interactive tasks (e.g., cerealbar), and multi-modal grounded tasks (e.g., nlvr). the aim of this broad distribution is to cover the most common task and data scenarios in nlp. the case studies cover the research of four distinct research labs. for each case study, we will also discuss related work from other authors as is relevant. for example, the multinli case study will include extensive discussion of followup work and the socialiqa case study will discuss related commonsense resources. in addition, we will discuss relevant existing work to provide all necessary background (e.g., @xcite @xcite . ## prerequistites broad familiarity with nlp tasks, empirical evaluation methods, and data collection practices. we introduce all the necessary terms and the specifics of each case study. ## reading list we recommend reviewing the 2015 naacl tutorial on crowdsourcing. 3 while we focus on unconstrained and complex case studies, the 2015 tutorial provides an overview of basic terms and methods that is a complementary background to our material. however, we review the required material in the background section, and do not assume a familiarity with the content of this prior tutorial. we also recommend reading the main papers describing each of the case studies @xcite @xcite @xcite . ## presenters alane suhr phd student, cornell university suhr@cs.cornell.edu https://alanesuhr.com alane's research focuses on grounded natural language understanding. alane has designed crowdsourcing tasks for collecting language data to study situated natural language understanding. alane co-presented a tutorial in acl 2018.