A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels (2020)

Abstract: Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy. In particular, we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels. Our three-stage workflow produces annotations with 95% accuracy for predicate labels and 93% for argument labels, which is comparable to expert agreement. Compared to prior work on crowdsourcing for SRL, we decrease expert effort by 4x, from 56% to 14% of cases. Our approach enables more scalable annotation of SRL, and could enable annotation of NLP tasks that have previously been considered too complex to effectively crowdsource.

## introduction high quality data is crucial in nlp, but difficult to collect for complex tasks such as semantic role labeling (srl). annotating propbank involved a team of annotators, each of whom took around three days to learn the annotation process @xcite . for tasks such as sentiment analysis @xcite ) and question answering @xcite , crowdsourcing has produced massive datasets that enabled the development of new, more sophisticated models. recent work introduced a hybrid workflow to allow crowd workers to usefully contribute to annotation of srl @xcite , but still required expert annotation in a third of cases. this paper introduces a new hybrid srl annotation workflow with the goal of minimizing expert annotation without sacrificing annotation accuracy. in order to develop our method, we first explored why srl annotations are hard for crowd workers. we found that workers had difficulty identifying the correct answer because the number of options for labels in srl can be overwhelming and workers lack the linguistic expertise to handle subtle cases. however, we also observed that (1) nonexpert workers are capable of reliably identifying many of the answers that are incorrect, and (2) when given the opportunity, crowd workers can accurately identify the limits of their knowledge. based on these observations, we developed a three phase workflow: (1) workers filter the set of options, reducing the complexity of the task, (2) workers select an answer or say they are unsure, and (3) difficult cases that workers disagreed on or were unsure of are decided by experts. the experts choose from the complete, unfiltered set of options. to measure the effectiveness of the approach we ran experiments at two scales. first, using 200 examples, we measured the effectiveness of each phase in the process and ran a comparison of end-toend performance against other workflows. second, using a larger set of 2,014 examples, we verified the end-to-end performance of our approach, showing that it achieves high accuracy while requiring experts for only 13% of cases. 1 our work shows that with careful workflow design, crowd workers can effectively contribute to annotation of complex tasks such as semantic role labeling. the key ideas of crowd filtering and a mechanism for expressing uncertainty could be used in other nlp annotation tasks to enable the creation of larger, more sophisticated resources. ## related work a range of previous studies have explored methods of crowdsourcing srl. most work has focused on crowd-only workflows, with comparatively low accuracy or extensive worker training @xcite @xcite @xcite . this work guided our user interface designs and our understanding of challenges in srl annotation. for example, we apply @xcite 's finding that cases where workers disagree are often more subtle or ambiguous. the most relevant work, @xcite , used a classifier to assign hard examples to experts and easy examples to crowd workers. they achieved high accuracy (95%), but required experts for 34% of cases. their classifier is complementary to the ideas we propose. another approach has used question-answering to annotate srl @xcite . this method is effective, but does not cover all roles and tends to have low recall. recent work has improved recall, but overall accuracy remains low, with an f-score of 82 on conll-2009 data @xcite . another approach used an automatic process to expand existing datasets and then used the crowd to check paraphrases @xcite . while effective, this approach is limited to expanding lexical coverage using sentences from an existing resource. word sense disambiguation (wsd) is related to the predicate sense labeling task we consider. prior work has explored crowdsourcing for wsd, but has mostly been unable to achieve high performance @xcite @xcite @xcite . there has been success on combining crowdsourcing with distant supervision for relation extraction @xcite @xcite . many other semantic parsing formalisms exist, such as amr and ucca, but we are unaware of work on crowdsourcing for them. more generally, a range of approaches have been proposed to increase crowdsourcing quality, including worker filtering @xcite , attention checks @xcite , and incentives @xcite . these are all complementary to our proposed method. ## proposed workflow srl can be divided into three parts: (1) identifying predicate and argument spans, (2) labeling predi-cate senses, and (3) labeling argument roles. we consider the latter two. 2 we describe each labeling decision as a task. in predicate sense classification tasks, a predicate in a sentence is given, and the goal is to identify the sense in which it is being used. in argument role classification tasks, an argument for a predicate with a known sense is given, and the goal is to identify the argument's role relative to the predicate. for example, for "john spoke .", there are five options for the sense of "speak", and between one and four options for the argument "john" depending on the sense of "speak". in this case, the correct sense is "speak.01 (speak, lecturing, talking)" for the predicate and "a0 (talker)" for the argument. we aim to use the crowd to annotate srl with high accuracy. this is difficult for two reasons. first, non-expert workers lack the linguistic expertise to understand some of the more complex role labels. second, there can be an overwhelming number of label options, with subtle differences in meaning. these issues increase the cognitive load of selection, reducing the likelihood that workers will select the true label. in a preliminary study, we measured the accuracy of asking five workers to choose a label. the crowd only outperformed a machine prediction when they were unanimous, which occurred in 1% of cases. however, we also found that workers could reliably identify the top few most likely labels, and could almost perfectly identify the most unlikely labels. these observations led us to design a three phase workflow for predicate and role labeling: this workflow addresses the two key challenges described above. first, consider the challenge that workers lack expert knowledge. the select phase separates out difficult cases by requiring majority agreement and no uncertainty. these difficult cases are then decided by experts with the necessary knowledge. second, consider the challenge that there can be an overwhelming number of options. the filter phase reduces the complexity of the task, focusing attention on likely options. this assumes that our filtering process removes unlikely options without removing the correct ones, which we verify experimentally in section 5.1. ## experimental setup we consider experiments on two sets of data, both from the english portion of the conll-2009 shared task @xcite . we use one set of 200 randomly chosen tasks (drawn from the training data) to evaluate components of our approach. we use a second set of 2,014 randomly chosen tasks to evaluate our workflow end-to-end. there are 459 predicates and 1555 arguments, covering 300 sentences from the conll test set. we did not include cases where there is only one frame for the predicate in propbank as there is no decision to be made. we evaluate against the expert-annotated shared task data, with edits based on errors we found in 39 cases. we recruited crowd workers from amazon mechanical turk via legiontools @xcite , and paid them us minimum wage ($7.25/hr). in all conditions, workers received two tutorial tasks with feedback before working on ten tasks. workers were randomly and independently assigned to tasks. n is five for both the filter phase and the select phase. the predicate word and argument spans are automatically identified using the @xcite system. we present the workers with spans by projecting the head-word, as we expected spans to be more intuitive for workers. the sense inventory and argument types are as defined in propbank. for argument labeling the sense of the predicate is the one produced by our workflow. if the span is incorrect, we expect workers would make a best effort to interpret the span (for example, if the span is one word too long or short they will probably still understand it correctly, especially since they see it in the context of the entire sentence). however, for evaluation, we label these cases with a special category, 'none', indicating that the span is incorrect or attached to the incorrect predicate. to confirm the consistency of our expert annotator, we had a second expert independently perform the annotations. the cohen's kappa score between the two experts was 0.92 for predicates and 0.85 for arguments, near-perfect agreement (altman, 1990). ## conclusion we propose a filtering process that can simplify complex selection tasks that arise in srl annotation. evaluating on 2,014 examples, we find that our workflow matches gold-standard data for 95% of predicates and 93% of arguments, with expert input for only 13% of cases. more broadly, our approach expands the applicability of crowdsourcing, enabling the creation of larger, more complex, high quality resources.