Contrast Sets for Stativity of E nglish Verbs in Context (2022)

Abstract: For the task of classifying verbs in context as dynamic or stative, current models approach human performance, but only for particular data sets. To better understand the performance of such models, and how well they are able to generalize beyond particular test sets, we apply the contrast set (Gardner et al., 2020) methodology to stativity classification. We create nearly 300 contrastive pairs by perturbing test set instances just enough to change their labels from one class to the other, while preserving coherence, meaning, and well-formedness. Contrastive evaluation shows that a model with near-human performance on an in-distribution test set degrades substantially when applied to transformed examples, showing that the stative vs. dynamic classification task is more complex than the model performance might otherwise suggest. Code and data are freely available.

## introducing stativity aspectual properties of verbs, and the clauses they inhabit, have the potential to support a range of natural language processing tasks, such as event ordering @xcite and temporal relation classification @xcite , as well as contributing to speaker choices around situational construal @xcite . at the same time, verb and situational aspect are a complex set of interacting properties, in which the meaning of the verb, the nature of its arguments, adverbial modifiers, and grammatical features such as verb tense and nominal definiteness can all play a role in determining the aspectual make-up of a clause. this sensitivity of aspectual categorization to small shifts in linguistic form is one reason that automatic prediction of aspectual classes is an especially challenging computational problem. in this paper we explore the stability of automatic classification for one particular facet of aspect: stativity of 1 https://github.com/dchensta/se_ contrast english verbs. stativity reflects the degree to which a verb represents a static situation versus a situation that reflects some degree of dynamicity. dynamic verbs typically involve some change of state. (1-3) below show examples of the three classes relevant for our study: dynamic verbs, stative verbs, and verbs for which annotators cannot_decide. most verbs in english have a strong predominant category (stative or dynamic), yet allow for variable interpretation, depending on context. a smaller number of verbs (e.g. show), are highly flexible, with no strong statistical tendency in either direction @xcite . because of this variability, automatic classification of aspectual properties requires contextual input and instance-level classification. to better understand the ability of systems to automatically determine stativity, we produce contrast sets (section 2) for english verb stativity with 298 transformed instances. the contrast set instances and their ground-truth labels are extracted from the sitent corpus (section 2.2), and the transformed instances are produced using a range of linguistically-motivated transformation strategies, detailed in section 3. using a standard modeling configuration (section 4), we show that classification performance on the transformed instances is substantially lower than on the original instances. according to @xcite , observed annotator agreement for this task ranges from 79% to 82%. on the original instances, the model achieves micro-averaged accuracy of nearly 80%, approaching human agreement for this task. foot_0 on the transformed instances, micro-averaged accuracy is well below 60%. ## transformation strategies after building the contrast sets, we perform an analysis of the linguistic properties of the various strategies used. most transformations hinge on the main verb, either replacing the lexical item or changing the role of the verb so that it moves to a different structural and semantic configuration. ## model, results, and discussion having built contrast sets, we now evaluate performance compared to the original instances. ## related work on lexical aspect and verb stativity aspectual structure is complex and well-studied in the linguistics literature @xcite @xcite , among many others). classically, aspectual analysis involves the semantic properties of stativity, telicity, durativity, and iterativity. representation framework with aspectual features, and such aspectual information is a key feature of the uniform meaning representation framework @xcite . in addition, aspectual properties are relevant at both the clause level and the level of individual verbs. our current focus is the verb-level property of stativity, sometimes referred to as inherent lexical aspect. 7 stativity reflects the degree to which a verb represents a static situation versus a situation that reflects some degree of dynamicity. dynamic verbs typically involve some change of state. computational approaches to stativity. early approaches to computational analysis of verb stativity employ rule-based approaches based on known linguistic tests for stativity, such as the progressive test. 8 @xcite produce a type-level stativity rating for english verbs, based on the frequency with which verbs occur in various tenses in the brown and reader's digest corpora. @xcite treat stativity as one of several aspectual properties derivable from logical representations of verb meaning in the lexical conceptual structure (lcs) framework @xcite . @xcite use a wide range of linguistic indicators to derive type-level stativity values for english verbs. friedrich and palmer (2014a) extend siegel and mckeown's work to incorporate distributional features and perform classification in context. @xcite use distributional semantics to classify both stativity and telicity across genres. @xcite take a more finegrained approach to lexical aspect classification for 7 aktionsart @xcite also models lexical aspect. both aktionsart and stativity are subject to coercion at the clause-level, as described in the introduction. 8 generally, static verbs in english cannot occur in progressive form: e.g. *i am knowing thai. this is one of the most robust of the linguistic tests, but it too is subject to exceptions: e.g. i am liking this song! french verbs in context, categorizing verbs across a set of 13 different verbal readings. @xcite take a distributional approach to classifying german verbs for aktionsart, and @xcite provide a new annotated corpus and classification experiments for multiple components of aspect for german verbs. another important line of research @xcite takes a broader view of event meaning, treating stativity as one of a number of aspectual features which together compose the meaning of an event. similarly, work on clause-level semantic aspect classification (aka situation entity classification) @xcite @xcite considers stativity as a key semantic property for determining clause-level aspect. finally, @xcite use a sequence of rules to assign tense and aspect values to both verbal events and event nominals, making use of co-occurrence cues of partof-speech tags, special lexical items, and semantic configurations that help the classifier select the right shade of aspect for a given situation. ## conclusions we apply the contrast set methodology to the task of classifying english verbs in context as stative or dynamic. we see a serious performance degradation on the transformed examples, suggesting the model has not learned a clean decision boundary for stativity. this first analysis suggests a need to more clearly define features that may bias clauses toward stative or dynamic readings. the study would benefit from more data, across a wider range of text types. we would also like to investigate the effectiveness of contrastive evaluation for other semantic properties, using recentlydeveloped methods for partially-automatic contrast set creation @xcite @xcite .