The Extraordinary Failure of Complement Coercion Crowdsourcing (2020)

Abstract: Crowdsourcing has eased and scaled up the collection of linguistic annotation in recent years. In this work, we follow known methodologies of collecting labeled data for the complement coercion phenomenon. These are constructions with an implied action — e.g., “I started a new book I bought last week”, where the implied action is reading. We aim to collect annotated data for this phenomenon by reducing it to either of two known tasks: Explicit Completion and Natural Language Inference. However, in both cases, crowdsourcing resulted in low agreement scores, even though we followed the same methodologies as in previous work. Why does the same process fail to yield high agreement scores? We specify our modeling schemes, highlight the differences with previous work and provide some insights about the task and possible explanations for the failure. We conclude that specific phenomena require tailored solutions, not only in specialized algorithms, but also in data collection methods.

## introduction crowdsourcing has become extremely popular in recent years for annotating datasets. many works use frameworks like amazon mechanical turk (amt) by converting complex linguistic tasks into easy-to-grasp presentations which make it possible to crowdsource linguistically-annotated data at scale @xcite @xcite . in this work, we attempt to use existing methodologies for crowdsourcing linguistic annotations in order to collect annotations for complement coercion @xcite , a phenomenon involving an implied action triggered by an eventselecting verb. specifically, certain verb classes require an event-denoting complement, as in: "i started reading a book", "i finished eating the task annotations explicit after a heartfelt vow, she agrees {officiating}, φ and the two begin kissing as the preacher tries to continue the ceremony. entailment hunter waited for max to finish his burger before asking him again. ; ## background complement coercion we are interested in the linguistic phenomenon of complement coercion. 1 in complement coercion, there is a clash between an expectation for a verb argument denoting an event, and the appearance of a noun argument denoting an entity. uncovering the covert event requires the comprehender to infer the implied event by invoking the comprehender's lexical semantics and/or world knowledge @xcite . consider examples 1 and 2 below, with an implicit event of reading or writing missing in the surface form. inferring the implicit event ( marked ) is necessary in order to construe the full semantics of this sentence. the reconstruction of the covert event requires an interplay between semantics 2 and world knowledge. in example 1 above, the prefix "i started " with the event-selecting verb started triggers expectations for some event-denoting object (reading, writing, eating, watching, etc). the object that follows, "a new book", narrows down the expectations -based on world knowledge. as @xcite puts it, "different nouns grant privileged access to different activities, particularly those which are most frequently performed with the entities they denote". although the entity narrows down the set of possible events, the implied event might remain ambiguous (in example 1, both reading and writing are plausible, but eating is not). as can be seen in example 2, additional context, as in "i bought last week", provides further world-knowledge cues, towards accessing a more specific event (in this case 1 complement coercion has been studied in linguistics from many theoretical viewpoints. see appendix a for background. complement coercion is particularly frequent with certain verb classes, including aspectual verbs -verbs that "describe the initiation, termination, or continuation of an activity" @xcite -such as: 'start', 'begin', 'continue' and 'finish' @xcite . this set of verbs is the focus of our work. note however, that such verbs may appear in similar constructions that do not imply any covert action or event. for instance, in the following sentence: 3. i started a new company. here, the verb 'start' is used as an entity-selecting (and not event-selecting) verb, a synonym of 'found' or 'establish'. see more examples of similar non-coercive constructions in appendix b. annotated data for complement coercion (pustejovsky et al., 2010) was collected in the past, based on a tailor-made annotation methodology @xcite , consisting of a multi-step process that includes word-sense disambiguation by experts. the annotation focused on coercion detection (as well as labeling the arguments type) and did not involve identifying the implied action. here, we aim to collect complement coercion data via non-expert annotation, at scale, to test whether models can recover the implicit events and resolve the emerging ambiguities. crowdsourcing nli nli, originally framed as recognizing textual entailment (rte), has become a standard framework for testing reasoning capabilities of models. it originated from the work by @xcite , where a small dataset was curated by experts using precise guidelines with a specific focus on lexical and syntactic variability rather than delicate logical issues, while dismissing cases of disagreements or ambiguity. @xcite ; @xcite then scaled up the task and crowdsourced large-scale nli datasets. in contrast to @xcite , the task definitions were short and loose, relying on the annotators' common sense understanding. many works since have been using the nli framework and the crowdsourcing procedure associated with it to test models for different language phenomena @xcite @xcite @xcite . ## discussion inherent disagreements in human textual inferences recently, pavlick and kwiatkowski (2019) discussed a similar trend of disagreements in five popular nli datasets (rte @xcite , snli @xcite , mnli @xcite , joci @xcite and dnc @xcite ). in their study, annotators had to select the degree to which a premise entails a hypothesis, on a scale @xcite (instead of discrete labels). @xcite show that even though these datasets are reported to have high agreement scores, specific examples suffer from inherent disagreements. for instance, in about 20% of the inspected examples, "there is a nontrivial second component" (e.g. entailment and neutral). our findings are related to theirs, although not identical: while the disagreements they report are due to the individuals' interpretations of a situation, in our case, disagreements are due to the difficulty in imagining a different scenario. while some works propose to collect annotator disagreements and use them as inputs @xcite (see @xcite for an elaborated overview), this will not hold in our case, because only one of the labels is typically correct. however, the bottom-line is the same: these dis-agreements cannot be dismissed as 'noise', they are more profound. we hypothesize that when tackling specific phenomena like the one we address in this work, which involve sources of disagreements that are often 'ignored' (not intentionally) during the collection of large datasets, foot_4 these sources of disagreements are highlighted and manifest themselves more clearly. this results in low agreement scores as we see in our study. ## conclusions in this work, we attempt to crowdsource annotations for complement coercion constructions. we use two modeling methods, which were successful in similar settings, but resulted in low agreement scores in our setup. we highlight some of the issues we believe are causing the disagreements. the main one being different construals @xcite of the utterances by different people -as well as the difficulty to consider a different one, once fixating on a specific construal -that led to different answers. we connect our findings to previous work that observed some inherent disagreement in human judgments in popular datasets, such as @xcite . although this issue is less prominent in these datasets (which is manifested as higher agreement scores), we notice that when tackling a specific phenomenon, e.g. involving implicit elements, these issues may arise. we also argue that the lack of detailed definitions in the commonly used nli tasks may lead to poor performance on small buckets of language-specific phenomena. this drop might be lost in large-scale datasets, but may have critical effects when modeling and studying specific phenomena. as a community, we claim, we should seek to identify those buckets and further investigate them, using more profound approaches for data collection, with clear and grounded definitions. we hope that our attempted trial in data collection will allow others to learn from our failure.