Controlled Crowdsourcing for High-Quality QA - SRL Annotation (2020)

Abstract: Question-answer driven Semantic Role Labeling (QA-SRL) was proposed as an attractive open and natural flavour of SRL, potentially attainable from laymen. Recently, a large-scale crowdsourced QA-SRL corpus and a trained parser were released. Trying to replicate the QA-SRL annotation for new texts, we found that the resulting annotations were lacking in quality, particularly in coverage, making them insufficient for further research and evaluation. In this paper, we present an improved crowdsourcing protocol for complex semantic annotation, involving worker selection and training, and a data consolidation phase. Applying this protocol to QA-SRL yielded high-quality annotation with drastically higher coverage, producing a new gold evaluation dataset. We believe that our annotation protocol and gold standard will facilitate future replicable research of natural semantic annotations.

## introduction semantic role labeling (srl) provides explicit annotation of predicate-argument relations. common srl schemes, particularly propbank @xcite and framenet @xcite , rely on predefined role inventories and extensive predicate lexicons. consequently, srl annotation of new texts requires substantial efforts involving expert annotation, and possibly lexicon extension, limiting scalability. aiming to address these limitations, question-answer driven semantic role labeling (qa-srl) @xcite labels each predicate-argument relationship with a question-answer pair, where natural language questions represent semantic roles, and answers correspond to arguments (see qa-srl carries two attractive promises. first, using a question-answer format makes the annotation task intuitive and easily attainable by laymen, as it does not depend on linguistic resources (e.g. role lexicons), thus facilitating greater annotation scalability. second, by relying on intuitive human comprehension, these annotations elicit a richer argument set, including valuable implicit semantic arguments not manifested in syntactic structure (highlighted in overall, qa-srl largely subsumes predicateargument information captured by traditional srl schemes, which were shown beneficial for complex downstream tasks, such as dialog modeling @xcite , machine comprehension @xcite and cross-document coreference @xcite . at the same time, it contains richer information, and is easier to understand and collect. similarly to srl, one can utilize qa-srl both as a source of semantic supervision, in order to achieve better implicit neural nlu models, as done recently by @xcite , as well as an explicit semantic structure for downstream use, e.g. for producing open information extraction propositions @xcite ). 1 around 47 people could be arrested, including the councillor. (1) who might be arrested? 47 people | the councillor perry called for the das resignation, and when she did not resign, cut funding to a program she ran. (2) why was something cut by someone? she did not resign (3) who cut something? perry previous attempts to annotate qa-srl initially involved trained annotators @xcite but later resorted to crowdsourcing @xcite for scalability. naturally, employing crowd workers is challenging when annotating fairly demanding structures like srl. as @xcite acknowledge, the main shortage of their large-scale dataset is limited recall, which we estimate to be in the lower 70s (see §4). unfortunately, such low recall in gold standard datasets hinders proper research and evaluation, undermining the current viability of the qa-srl paradigm. aiming to enable future qa-srl research, we present a generic controlled crowdsourcing annotation protocol and apply it to qa-srl. our process addresses worker quality by performing short yet efficient annotator screening and training. to boost coverage, we employ two independent workers per task, while an additional worker resolves inconsistencies, similar to conventional expert annotation. these steps combined yield 25% more roles than @xcite , without sacrificing precision and at a comparable cost per verb. this gain is especially notable for implicit arguments, which we show in a comparison to propbank @xcite . overall, we show that our annotation protocol and dataset are of high quality and coverage, enabling subsequent qa-srl research. to foster such research, including easy production of additional qa-srl datasets, we release our annotation protocol, software and guidelines along with a high-quality dataset for qa-srl evaluation (dev and test). 2 we also re-evaluate the existing parser @xcite against our test set, setting the baseline for future developments. finally, we propose that our systematic and replicable controlled crowdsourcing protocol could also be effective for other complex annotation tasks. 3 similar embeddings for semantically similar questions. these embeddings may be leveraged downstream in the same way as embeddings of traditional categorical semantic roles. corpora the original 2015 qa-srl dataset @xcite was annotated by hired non-expert workers after completing a short training procedure. they annotated 7.8k verbs, reporting an average of 2.4 qa pairs per verb. even though multiple annotators were shown to produce greater coverage, their released dataset was produced by a single annotator per verb. in subsequent work, @xcite employed untrained crowd workers to construct a large-scale corpus (2018) and used it to train a parser. in their protocol, a single worker ("generator") annotated a set of questions along with their answers. two additional workers ("validators") validated each question and, in the valid case, independently annotated their own answers. in total, 133k verbs were annotated with 2.0 qa pairs per verb on average. in a subset of the corpus (10%) reserved for parser evaluation, verbs were densely validated by 5 workers (termed the dense set). 4 yet, adding validators accounts only for precision errors in question annotation, while role coverage solely relies upon the output of the single generator. for this reason, both the 2015 and 2018 datasets struggle with coverage. also, while traditional srl annotations contain a single authoritative and non-redundant annotation (i.e., a single role and span for each argument), the 2018 dataset provides raw annotations from all annotators. these include many redundant overlapping argument spans, without settling on consolidation procedures to provide a single gold reference, which complicates models' evaluation. these limitations of the current qa-srl datasets impede their utility for future research and evaluation. next, we describe our method for creating a viable high quality qa-srl dataset. ## dataset quality analysis inter-annotator agreement (iaa) to estimate dataset consistency across different annotations, we measure f1 using our ua metric. 10 individual worker-vs-worker experiments yield 79.8 f1 agreement over 150 predicates, indicating high consistency across our annotators, in line with agreement rates in other structured semantic annotations, e.g. @xcite . overall consistency of the dataset is assessed by measuring agreement between different consolidated annotations, obtained by disjoint triplets of workers, which achieves f1 of 84.1, averaged over 4 experiments, 35 predicates each. notably, consolidation boosts agreement, indicating its necessity. for la agreement, averaged f1 was 67.8; however, it is likely that the drop from ua is mainly due to falsely rejecting semantically equivalent questions under the strict-match criterion, given that we found equal la and ua scores in a manual evaluation of our dataset (see ## baseline parser evaluation we evaluate the parser from @xcite on our dataset, providing a baseline for future work. as we previously mention, unlike typical srl systems, the parser outputs overlapping arguments, often with redundant roles ( ## conclusion applying our proposed controlled crowdsourcing protocol to qa-srl successfully attains truly scalable high-quality annotation by laymen, facilitating future research of this paradigm. exploiting the open nature of the qa-srl schema, our nonexpert annotators produce rich argument sets with many valuable implicit arguments. indeed, thanks to effective and practical training over the crowdsourcing platform, our workers' annotation quality, and particularly its coverage, are on par with expert annotation. we release our data, software and protocol, enabling easy future dataset production and evaluation for qa-srl, as well as possible extensions of the qa-based semantic annotation paradigm. finally, we suggest that our simple yet rigorous controlled crowdsourcing protocol would be effective for other challenging annotation tasks, which often prove to be a hurdle for research projects.