Supersense and Sensibility: Proxy Tasks for Semantic Annotation of Prepositions (2020)

Abstract: Prepositional supersense annotation is time-consuming and requires expert training. Here, we present two sensible methods for obtaining prepositional supersense annotations indirectly by eliciting surface substitution and similarity judgments. Four pilot studies suggest that both methods have potential for producing prepositional supersense annotations that are comparable in quality to expert annotations.

## introduction prepositions are highly ambiguous function words which can express a wide variety of relationships (litkowski and hargraves, 2005; tratz, 2011). supersenses have been proposed as an analytic framework for studying their lexical semantics, but extant gold-annotated corpora (e.g. schneider et al., 2018; peng et al., 2020) are small because preposition supersense annotation is a relatively complex annotation task that requires substantial training and time. we ask whether preposition supersense annotation could be made cheaper and quicker with crowdsourced labor. this will require "sensible" annotation tasks accessible to non-experts. in this work we present two possible designs for proxy tasks for crowdsourcing from which supersense labels can be recovered indirectly. these designs involve in-context substitution and similarity judgments. based on four in-house pilot experiments, we conclude that both designs are promising as methods for building a large preposition supersense-annotated corpus, and that they differ in how difficult they are for the workers and for the researchers. however, the setup should be considered a proof of concept, as we work with an idealized pool of workers (in-house computational linguistics graduate students). future research will be necessary to ascertain whether the paradigm is suited to naïve crowdworkers as well. ## preposition supersenses prepositions 1 can express many different kinds of semantic relations. schneider et al. (2018) present snacs, a coarse-grained annotation framework for prepositions encoding these relations. the meanings of prepositions are expressed in terms of supersenses, of which there are 50 in snacs v2.5. @xcite for instance, the preposition in can be used to express time, place, and other relations: "i rented an apartment in locus boston", "i hope to see you in time the future". @xcite compared to other tagging tasks, supersense annotation is relatively hard: the snacs guidelines, which are only for prepositional supersenses, are around 100 pages in length, and a single preposition can often have multiple plausible annotations which must be carefully considered before a final decision. instead, we explore what we term proxy tasks: foot_0 rather than teach and elicit supersenses (or semantic features associated with supersenses) directly, we elicit judgments of surface substitution/similarity, as has been done by previous work on word sense crowdsourcing ( §5). this approach leverages current annotated data in combination with the proxy annotations to infer supersense labels via crowdsourcing. here we outline two different approaches for framing a crowdsourcing task from which annotations can be derived. details will be explained in more depth when we turn to discuss our pilot studies. ## pilot studies in order to assess the quality and characteristics of these two designs before large-scale deployment, we conducted a series of pilot studies with a small number of participants. participants in each study were drawn from a group of several graduate students, all of whom had at least some familiarity with snacs. this is an unrealistic quality for crowdworkers to have, but our aim is to assess how well these designs can work in ideal conditions. ## related work crowdsourcing on amazon mechanical turk has been a popular method for scaling up linguistic annotation. early studies on its efficacy for semantic annotations like word sense disambiguation, textual entailment, and word similarity (snow et al., 2008) and psycholinguistic studies and judgment elicitation (munro et al., 2010) have shown that crowdsourced annotations can be as good as if not even better than annotations produced using traditional methods. several studies have focused specifically on word sense annotations for content-words like nouns, adjectives, and verbs (rumshisky, 2011; jurgens, 2013; biemann and nygaard, 2010; biemann, 2012; tsvetkov et al., 2014), but function words have received less attention. to our knowledge, the only work carried out specifically on prepositional sense annotation using a non-traditional annotation methodology is due to tratz (2011, §4.2), who describes a process by which existing prepositional sense annotations were refined by three annotators, two of which have unspecified levels of linguistic or other competencies. we conjecture that one reason why content-words have been favored over function-words for crowdsourced annotation is that their comparatively less abstract meanings make reasoning about their semantics more approachable to linguistically naïve crowdworkers, simplifying task design, while for function words, it can be difficult to tap into crowdworkers' intuitions without changing the task considerably. some work has investigated gamification with the hope that bringing gamelike elements would allow crowdworkers to produce good annotations without a traditional training process, in some cases achieving performance on par with or better than expert annotation (fort et al., 2020; hartshorne et al., 2014; schneider et al., 2014). independent from the matter of whether to crowdsource or gamify, some have modified their annotation schemes with an eye explicitly to annotation expense (in terms of time or money). in the qa-srl annotation scheme proposed by he et al. (2015), plain-language questions are used to describe the predicate-argument structures of verbs instead of formalisms such as frames or predicates, rendering it theory-and formalism-neutral and easier to explain to non-expert workers. our work is different from the work we have reviewed here in that we have attempted to have participants solve a task that is not the same task we would have given to an expert annotator. pursuing such a proxy task, as we have termed it, introduces the challenge of turning proxy data into gold data, but reduces the need for worker training. proxy tasks have been successfully pursued in other domains, like in the esp game designed by von ahn and dabbish (2004) for image labeling, where player data for a game played with an image-the proxy task-is used to infer image labels. ## conclusion we have presented two designs for deriving prepositional supersense tags from crowdsourced tasks, and we have investigated their efficacy through four pilot studies, finding that both hold promise for producing high-quality prepositional supersense annotations. we have seen that the two designs differ in their complexity and performance characteristics: the neighbor selection design, while consisting of only one task instead of two, requires a gold-annotated corpus of sufficient size to give every tag sufficient coverage, while the substitution design could reveal, bottom-up, clusters of usages that may not be well-represented in the training data. we have made several idealizations throughout this work: all data was drawn from streusle, guaranteeing that it would be homogeneous with respect to genre, and crowdworkers had some knowledge of the snacs guidelines which likely made them better at the tasks than real-world crowdworkers. moreover, we studied only 5 common prepositions covering 20 or so supersenses out of snacs's 50. in future work, we intend to implement these designs on platforms such as amazon mechanical turk to further investigate these designs' efficacy and the extent to which these idealizations affect our results.