Crowdsourcing Preposition Sense Disambiguation with High Precision via a Priming Task (2022)

Abstract: The careful design of a crowdsourcing protocol is critical to eliciting highly accurate annotations from untrained workers. In this work, we explore the development of crowdsourcing protocols for a challenging word sense disambiguation task. We find that (a) selecting a similar example usage can serve as a proxy for selecting an explicit definition of the sense, and (b) priming workers with an additional, related task within the HIT improves performance on the main proxy task. Ultimately, we demonstrate the usefulness of our crowdsourcing elicitation technique as an effective alternative to previously investigated training strategies, which can be used if agreement on a challenging task is low.

## introduction crowdsourcing work relies on effective protocol design in order to elicit meaningful and accurate annotations from lay workers. traditional crowdsourcing protocols for challenging tasks train crowd workers, incentivize high quality work with bonuses, and filter out workers with pre-determined gold annotations. in this work, we demonstrate a novel crowdsourcing technique which frames the task such that the worker is able to provide a high quality annotation for challenging tasks without extensive prior training. we consider the challenging task of english preposition supersense disambiguation. prepositions are difficult to disambiguate due to their extreme polysemy and frequency @xcite @xcite . while some distinctions-such as the locative vs. temporal ambiguity of in seattle vs. in october-are quite intuitive, the semantic range of english prepositions makes fully disambiguating them a daunting task. consider the sentence "nice and quiet place with partportion cosy living room just outside the city." though with here denotes a part of a whole (a room in an abode), it is also describing a characteristic of the place/living room, suggesting that with characteristic would also be a reasonable annotation. with this being a difficult task because of the inherently categorical nature of word sense disambiguation, we aim to develop simple linguistic tasks which elicit annotations by proxy. prepositions are critical to language understanding @xcite , which makes preposition sense disambiguation an important task, impacting a range of downstream applications, including: relation extraction @xcite , paraphrasing of phrasal verbs @xcite and noun compounds @xcite , machine translation @xcite @xcite , semantic role labelling @xcite , and more. automatic supersense classifiers exist for preposition sense disambiguation @xcite , but are unable to achieve the precision levels of trained annotators. in order to tackle the challenging aim of preposition sense annotation, we design linguistic tasks to enable the elicitation of high-quality annotations. we demonstrate that a weak form of guidance, which we call a priming task, improves performance on the main task. in our case, the main task asks the worker to select the most similar usage to the prompt. however, the notion of meaning similarity between preposition usages may not be apparent to crowd workers. we find that first asking the worker to select an appropriate definition is an effective priming task: its mere presence enhances workers' ability to choose the correct exemplar in the second question. this related (priming) task serves as a high-precision, low-overhead alternative to training. finally, we compare the predictions of the crowd with those of a classifier, and find that they are largely complementary; for most prepositions tested, combining the two predictions gives perfect or near-perfect precision. related work has proposed the ability to crowdsource supersenses and tested a pilot on trained in-house annotators @xcite ; in this work, we develop new formulations and collect actual crowdsourced annotations from mechanical turk to ascertain the suitability of our formulations. our contributions include: ## related work on crowdsourcing the amazon mechanical turk online crowdsourcing platform enables crowd workers from around the world to perform short tasks published by requesters. mechanical turk has been used for a range of annotation tasks. mechanical turk is used in many cases where a large amount of annotations are collected. if these many annotations are of lesser quality, it is not necessarily the case that having more data is better than having a small amount of expert annotations would have been @xcite . ## results & analysis before coming to the task design presented in this work, we iterated over many approaches and techniques to crowdsourcing preposition sense disambiguation. in total, we elicited 4,080 annotations and developed 17 slight variations of the approach presented here, which primes the crowd workers by combining the definition-based and exemplarbased tasks. we saw consistent trends across these pre-study variations, resulting in the current approaches (presented in this work), which prime the crowd workers by combining the definition-based and exemplar-based tasks. for these results, we collect between 10 and 30 annotations for each of the 6 prepositions, via two methods: the exemplar matching design alone, and the combined design. ## conclusion this paper outlined a promising approach to crowdsourcing preposition supersense annotation (a particularly challenging form of word sense disambiguation). the crowd workers outperformed automatic supersense tagging on 5 of the 6 prepositions studied. we compared multiple designs, finding that prompting annotators to reason both explicitly and implicitly about meaning is most effective, even when the explicit question does not elicit a correct annotation. the crowdsourcing approach achieves very high precision and acceptable recall for 3 prepositions.