Investigating Aspect Features in Contextualized Embeddings with Semantic Scales and Distributional Similarity (2024)

Abstract: Aspect, a linguistic category describing how actions and events unfold over time, is traditionally characterized by three semantic properties: stativity, durativity and telicity. In this study, we investigate whether and to what extent these properties are encoded in the verb token embeddings of the contextualized spaces of two English language models – BERT and GPT-2. First, we propose an experiment using semantic projections to examine whether the values of the vector dimensions of annotated verbs for stativity, durativity and telicity reflect human linguistic distinctions. Second, we use distributional similarity to replicate the notorious Imperfective Paradox described by Dowty (1977), and assess whether the embedding models are sensitive to capture contextual nuances of the verb telicity. Our results show that both models encode the semantic distinctions for the aspect properties of stativity and telicity in most of their layers, while durativity is the most challenging feature. As for the Imperfective Paradox, only the embedding similarities computed with the vectors from the early layers of the BERT model align with the expected pattern.

## introduction since the introduction of transformer architectures in nlp @xcite @xcite , their increasing success urged researchers to get more insights about the linguistic knowledge encoded in their internal representations. the literature on probing tasks is a clear example of this trend: a simple classification model is asked to solve a task requiring linguistic knowledge using embeddings representations extracted from a language model (lm) with little or minimal linguistic supervision, and if the classification model is successful, one can infer that the lm's representations do encode the targeted knowledge (e.g. @xcite al. (2019); hewitt and liang (2019); goldberg (2019); jawahar et al. (2019); wu et al. (2020); ravichander et al. (2020); madabushi et al. (2020); chen et al. (2021) ; @xcite ; belinkov (2022), inter alia). an alternative approach, especially popular for probing the semantic knowledge contained in the embeddings, involves mapping them onto humaninterpretable features @xcite @xcite . yet the probing methodology involves a trainable classifier, and therefore the relation between the probe results and the knowledge in the original representations is not always clear @xcite . moderate correlations with human ratings/norms can sometimes be obtained even by using random vectors as features @xcite , and thus alternative methods for directly analysing/modifying the structure of the semantic space have been proposed (e.g. indicator tasks, @xcite ). a recent study by @xcite introduced the usage of semantic projections to interpret the content of word embeddings, by constructing subspaces corresponding to human-interpretable semantic scales. such semantic scales were shown to be very useful in modeling human judgements for a variety of concepts in the semantics of nominals @xcite . in our paper, we focus on aspect, a concept in verb semantics that characterizes the temporal relationship of actions and events. aspect has been shown to be important in several nlp tasks, such as next event prediction @xcite and textual entailment @xcite . combining the usage of semantic scales and embedding similarity measurements, in two experiments, we address the question of whether and to what the extent the contextualized word embeddings produced by lms encode the aspectual properties of stativity, telicity and durativity. in the first experiment, we use semantic scales to quantify the values of the three aspectual properties in the verb token embeddings produced by different hidden layers of bert and gpt-2. we examine whether the projected scores reflect the binary distinction in the aspectual properties of the verbs described by @xcite , assuming that verbs having different values for a property (e.g. telic vs. a) should have significantly different scores. to our knowledge, we are the first to adopt the semantic scales method for modeling verb semantics. in the second experiment, we examine the similarity between simple past and past progressive forms of telic and atelic verbs that express activities and accomplishments. according to the imperfective paradox in @xcite , the past progressive of activity verb entails its simple past, while this entailment does not hold for accomplishment verbs. again, we extracted the verb token embeddings from different internal layers: if a bert/gpt-2 embedding from a given layer correctly encodes telicity, we expect that the similarity between past progressive and simple past of an activity verb will be higher -since the former entails the latter-than between the two corresponding past forms of an accomplishment verb. we found that both lms are capable of consistently encoding aspectual features, especially for stativity and telicity. however, bert was more sensitive to the nuanced difference in telicity, as we found in the imperfective paradox experiment. our findings reveal the extents to which prototypical lms encode core verb properties, which has important implications for selecting lms for downstream fine-tuning. for example, based on our results, we can hypothesize that fine-tuning bert-family models may be proven more beneficial for improving the performance of textual entailment. ## related work in the semantics literature, verb aspect is generally characterized in terms of three properties: stativity, telicity and durativity @xcite . stativity refers to the distinction between states and events.verbs of high stativity generally cannot be used in progressive forms: for example, it is not possible to use 'i am knowing/loving'. in comparison, verbs of low stativity can typically be used in progressive forms (e.g. 'i am running/swimming'). telicity refers to whether an event unfolds in time in an homogeneous way, and whether any part of the process is of the same nature as the whole. telic verbs can often be collocated with 'in' adverbial phrases but not with 'for' adverbial phrases; e.g. 'eat' can be used in 'he ate the apple in a minute' but not in 'he ate the apple for a minute'. notice that verbs of this type describe actions/events with a natural end point (e.g. the moment in which the apple is finished). the use of 'in' signifies that the action (of eating the apple) is completed within a specific timeframe. in contrast, atelic verbs usually collocate with 'for' but not with 'in', e.g. 'he was running for an hour' but not 'he war running in an hour'. finally, durativity refers to how long an event lasts. durative actions like 'love' can be questioned by 'how long have you loved her?', but punctual actions like 'recognize' cannot be questioned in a similar way ('how long have you recognized her?' sounds odd without additional context). these examples show that a verb can vary along the three dimensions. for example, 'love' is simultaneously stative, durative, and atelic. the work conducted by @xcite focuses on the automatic classification of verb stativity in context, using a combination of distributional and manually crafted linguistic features. it is one of the first to introduce a dataset of annotated sentences specifically for this feature. @xcite expanded on this work, by releasing datasets also for telicity and durativity with gold and silver annotations; the latter was automatically extracted from a parallel corpus between english and czech texts, exploiting the fact that czech aspectual features are signaled with specific morphological markers. @xcite proposed an approach based on compositional distributional models to distinguish between stative and dynamic verbs, and between telic and atelic ones. interestingly, their classification results confirmed that the tense is always a strong indicator of telicity; in particular, past tense is often correlated with telic events. cho et al. (2021) presented a study on using bert surprisal to model human typicality ratings of the location arguments in natural language sentences, which were shown in the studies by @xcite to be strongly related to verb aspect: humans show priming effects for typical locations in sentences, but only when the tense of the main verb is progressive (or, in other words, the description of an action as ongoing makes the location argument more salient for human conceptual representations). bert surprisal scores showed some sensitivity to the aspect of the verb, although they produced human-like patterns only when the entire sentence context other than the verb and the location were masked. more recently, @xcite reported a classification experiment on telicity and durativity on english and french, suggesting that transformer models encode a non-trivial amount of knowledge of aspect even before fine-tuning, although they have biases regards verb tense and word order. finally, @xcite presented a modeling study of the shortening effect that the usage of light verb constructions has on the perceived duration of event descriptions, and they also used the semantic scales method by @xcite to project bert vectors onto interpretable dimensions. they showed that certain type of events (e.g. punctive) have smaller values in their duration-related dimensions when expressed in the light verb form (e.g. to give a kiss takes less time than to kiss). ## experiment 1: measuring aspect properties with semantic scales in the first experiment, we select a set of verbs from the study by @xcite . for each of the three aspect properties, the verbs are divided into two groups: stative versus dynamic for stativity, telic versus atelic for telicity, and punctive versus durative for durativity. our primary goal is to construct a semantic scale for each property, and then to project the word embeddings of the verbs on the semantic scales, in order to assign them scores of stativity, telicity, and durativity. if the distributional space effectively captures the different value that a verb can express with respect to a given property (e.g. telicity), we expect the scores for the verbs of the two groups to be different (e.g. telic verbs should have considerably higher scores on the telicity scale compared to atelic verbs). ## experiment 2: modeling the imperfective paradox with distributional similarity our first experiment showed that the models generally have a good grasp of the semantic distinctions related to the three main aspectual properties. in our second experiment, we test if the distributional similarities between verb token embeddings reflect the entailment properties of telic and atelic verbs when we manipulate their tense. with this goal in mind, we aim at replicating the imperfective paradox described by @xcite . in his work, dowty focuses on the opposition of activities and accomplishments in the past progressive and in the simple past tense, as in the following example: given that our models encode telicity in the embedding representations, we extract the token verb embeddings for the verbs in the provided sentence pairs in a. and b., and for each verb we measure the distributional similarity to itself when used in the other tense. our hypothesis is that the similarity will reflect the entailment relation between the two statements. specifically, we expect the similarity to be significantly higher for activities than accomplishments, since the simple past is entailed by the progressive in the former, but not in the latter case. similar to the previous experiment, we used the 'accomplishment-activity' contrast to define telicity, e.g. accomplishment verbs are telic while activity verbs are atelic. for these two groups, we used the same verbs from experiment 1. for each group, we constructed 100 pairs of simple/progressive past sentence pairs, resulting in a total of 200 pairs. initially, we extracted sentences from the bnc that contained the target verbs in the simple past tense, and for each sentence we created an equivalent sentence in the past progressive by changing the verb's aspect. for telic verbs, we used word types from the 'accomplishment' verb class, while for atelic verbs, we used word types from the 'activity' verb class. in total, we collected 100 samples for each verb group. for each verb type in the lists, we randomly sampled 10 sentences in which the verbs are in the form of past particle, and filtered those sentences that are marked as passives rather than simple past sentences. the remaining sentences were evaluated by the authors and deemed less suitable for aspect conversion. we also made sure that each verb type occurred at most 5 times in the sample after filtering, to prevent the results from being too influenced by specific verb types. as a result, we obtained 100 instances for the telic (32 verb types) and atelic group respectively (21 verb types). 3 the sentences in each pair are exactly the same, except for the main verb tense foot_4 , and we also manually checked that they did not become incoherent due to the aspect conversion.once obtained the sentence pairs, we extracted the verb embeddings from each of them by using pytorch. for the sentences with the verb in the past progressive, we used the embedding of the progressive form, not including the auxiliary (e.g. from the children were building a sandcastle, we extract the embedding of building).once again, embeddings for multi-token verbs were obtained via mean pooling of the embeddings of the subtokens. the similarity between embeddings was computed with the standard cosine metric and with spearman correlation: we chose the latter as an additional measure because of the notorious issue of the anisotropy of contextualized vector spaces, as rank-based metrics were shown to be more robust to anisotropy and more consistently correlated with human similarity judgements @xcite . ## conclusion in our study, we presented an analysis of the contextualized verb embeddings of bert and gpt-2 to assess to what extent they encode semantic distinctions related to the three aspectual properties of stativity, telicity, and durativity. our first experiment, making use of the technique of the projection on a semantic scale by @xcite , showed that both models could consistently distinguish verbs with different values for stativity and telicity, but faced more challenges with durativity, and gave less consistent results. to our knowledge, this study is the first that applies the method of semantic scales to analyse features of verb semantics. as an additional contribution, we used the distributional similarities between the simple past and the past progressive of telic and atelic verbs to 'recreate' the imperfective paradox @xcite in a contextualized vector space. we showed that only the bert model in the early layers reflects the distinction proposed by the theory -progressive forms of atelic verbs, which entail their simple past, are more similar to the simple past than the corresponding forms of telic verbs.