On Quick Kisses and How to Make Them Count: A Study on Event Construal in Light Verb Constructions with BERT (2023)

Abstract: Psycholinguistic studies suggested that our mental perception of events depends not only on the lexical items used to describe them, but also on the syntactic structure of the event description. More specifically, it has been argued that light verb constructions affect the perception of duration in event construal, such that the same event in this type of constructions is perceived by humans as taking less time (to give a kiss takes a shorter time than to kiss). In our paper, we present two experiments with BERT using English stimuli from psycholinguistic studies to investigate the effects of the syntactic construction on event duration and event similarity. We show that i) the dimensions of BERT vectors encode a smaller value for duration for both punctive and durative events in count syntax, in line with human results; on the other hand, we also found that ii) BERT semantic similarity fails to capture the conceptual shift that durative events should undergo in count syntax.

## introduction temporality of event representations is at the core of human cognition, as the event duration in linguistic descriptions is likely correlated to the way we represent time in our mind @xcite . when we talk about an event, whether it is a hug between friends, an academic speech or the advice given by a doctor after a visit, we generally do not make explicit the duration information, since this information is supposed to be known by everyone from personal experience. moreover, duration is usually not encoded by grammar. however, previous studies in psycholinguistics suggested that grammatical cues in the message significantly affect the event representation built by language comprehenders for aspects such as causation and event structure @xcite . one of the main hypotheses of our research work is that the perceived duration of an event is also one of the aspects that is influenced by grammar. events can be individuated or not @xcite : she does runs and she does running describe something similar, but the count syntax in the first sentence makes us think about several occurrences of an activity, while the mass syntax in the second one makes us think about a more generic action whose temporal boundaries are not specified 1 . moreover, some events are more easily thought as atomic: when we hear that mary kissed john we can easily imagine that she did it several times, and each time could count as a separate kissing-event. on the other hand, speaking is non-atomic: if we say that the king spoke to the soldiers of the army, we might be plausibly describing a situation where the king had a break in his speech and then he started again, but this would still count as the same speaking-event. atomicity is the main criterion to classify events into punctive (like kissing) or durative. punctive events refer to verbs that tend to be instantaneous and are usually bounded by a natural end point (e.g. in mary kissed john, the set start and end point come with the contact and the separation of two lips). besides, they often receive iterative reading when taking place in progressive form or over a more protracted duration (e.g. by reading mary kicked the table for an hour, one could imagine that there should be more than one kicking action). durative events usually are not naturally bounded and not understood iteratively. for instance, the sentence mary talked for an hour describes a single protracted event rather than multiple events. a study by @xcite analyzed 1 according to the number asymmetry hypothesis by @xcite , the count syntax uses number as the uniform dimension of measurement (e.g., two cups, a dance, a jump). by contrast, the mass syntax is unspecified and open to comparison using various measuring dimensions, such as mass, volume and time (e.g., some water, some dancing, some jumping). the interaction of mass-count syntax with punctive and durative events when they are used within a light verb construction (e.g. to give a kiss, to give a speech). in the first experiment, the author asked human subjects to estimate the duration of events, and the events were denoted either by a verb in transitive construction, or by a corresponding light verb construction foot_0 , and the results revealed that light verb syntax has a shortening effect on the perceived duration of the event (to give a kiss takes a shorter time than to kiss). moreover, in a second experiment, human raters had to decide whether two events occurring in two different constructions were the same or not. noticeably, while an event in the transitive or in the light verb construction was rated to be similar to itself in the cases of punctive events (e.g. to kiss vs. to give a kiss) in count syntax and durative events in mass syntax (e.g. to advise vs. to give advice), durative events undergo a significantly greater semantic shift when they are described with count syntax. in other words, a higher semantic distance is perceived between event descriptions such as to talk and to give a talk. the event in the light verb construction, although it is still conceptually related to the one in the transitive frame, is conceived as of a different type. event and temporal knowledge are useful for many natural language processing (nlp) applications, including information retrieval, story generation, question answering and text summarization @xcite . performance in nlp applications vastly improved with the advent of language models (lms) based on transformer architectures @xcite , and consequently, a lot of research focused on analyzing their linguistic abilities, and questioning whether the representations they learn are compatible with linguistic theory @xcite . we would like to verify, therefore, whether the semantic representations of lms are able to capture changes in the representations of time that are encoded in subtle variations of the linguistic input. in our study, we used the popular bert language model @xcite to reproduce the experiments of @xcite . first, we analyzed the contextualized vector representa-tions produced by different versions of bert, and used the technique of semantic projection @xcite to test if the value of the semantic dimension of duration is actually shorter for events described with light verb constructions than in the corresponding ditransitive constructions; secondly, we measured the semantic similarity between the same events in the two constructions, to see if the distances between vectors capture the same meaning shifts that have been detected by humans. in our first experiment, we found that the shortening effect can be found across event types and projection conditions, with the contextualized vector of the light verb construction showing significantly shorter duration values. in the second experiment, however, the similarity between bert vectors largely fails to reproduce the pattern observed in the original study, as no significant meaning shifts were observed across event categories. we hypothesized this might be due to an inherent shortcoming of distributional similarity in distinguishing between fine-grained meaning relations between linguistic expressions @xcite @xcite . to the best of our knowledge, our study is the first to study meaning shifts in light verb constructions with contextualized vector spaces, and in general one of the first to analyze the representation of duration in distributional models. the materials to reproduce our experiments can be found at https: //github.com/xinxinlaoshi/quickkisses . ## the study by wittenberg and levy (2017): effects of light verb constructions on event duration and similarity the goal of @xcite was to investigate whether describing an event in mass or count syntax with a light verb construction affected the construals of event duration and similarity in comprehenders. the authors built sentences for three groups of verbs (see also the sentences were built to insert the event descriptions in natural contexts, and in each verb group, they differed only by construction type (e.g. after their first date, douglas kissed mary vs. after their first date, douglas gave a kiss to mary). to read the sentences and estimate how long the described event probably took. 3 while the results suggested that light verb constructions are generally associated with shorter durations in both experiments, consistent significant effects were found only in punctive events with count syntax (the tendency was present also for durative mass events, but findings were less consistent across settings). in experiment 4, the participants were asked to indicate the semantic similarity between the same event in transitive and ditransitive construction on a 7-point likert scale. the durative count pairs were rated significantly less similar to each other than the punctive count and durative mass pairs. combing the results from these experiments, the authors suggested that the light-verb encoding with count/mass syntax can also lead to a change in the general construal of an event, besides the shortening effect on the event duration. they argued that durative events are similar to mass nouns in terms of atomicity, as mass nouns like milk can also be partitioned arbitrarily, and one can get two portions of exactly the same substance by dividing @xcite @xcite . when mass nouns like wine or iron occur in a count context, the denotation of the resulting count noun phrase (several wines, an iron) is expected to change from the substance to another object that is different from but arbitrarily related to the substance, such as various types of wine or a piece of flatiron. given the analogy between the durative events and mass nouns, a similar conceptual shift would also occur when durative verbs are represented in the form of deverbal nouns with count syntax. for example, it is intuitive to think that giving a speech, while keeping the core meaning of utterance, is conceptually further from speaking than giving a hug is from hugging. however, the direction of the shift depends on the context and is hard to predict, thus any change in duration can be coincidental. therefore, the count light-verb encoding in this case resulted in a significant meaning difference rather than in a consistent shortening effect. ## experiment 1: modeling event duration with semantic projections the objective of our first experiment was to analyze the feature of duration in the embedding representations of events in language models and compare the results with the findings of experiments 1-2 in @xcite . but how to quantify duration in bert embeddings? for this goal, we adopted the semantic projection technique introduced by @xcite . the authors of the study suggested that one can infer semantic properties of objects and entities as semantic subspaces in a distributional model. subspaces were found with the following procedure: 1) identify multiple words that can represent extreme values of those properties on a scale, e.g. for size they could be big, huge, gigantic on one extreme, and tiny, small, minuscule on the other extreme; 2) average the word vectors at the two extremes, and then connect the two extremes with a line. this line will represent the human mental scale for size; 3) given a list of words/concepts to be ordered by their size, project their embeddings onto the size line and take the relative ordering of their values. applying this simple method to glove embeddings @xcite , the authors were able to predict human judgements across different semantic categories and for different types of properties (e.g. temperature, speed, arousal, intelligence etc.). our idea is to apply the same method to contextualized embeddings representing concrete usages of the verbs and the constructions from @xcite , by projecting them onto a duration subspace. to generate the embeddings, we used the popular bert model @xcite , in its base, uncased version for english. the target events are adapted from those identified by @xcite , and our dataset consisted of descriptions of those events either in bare verb forms or in ditransitive constructions led by give. there are 3 categories of phrase pairs in total: 1a. punctive events in count syntax: kiss -give a kiss, hug -give a hug, kick -give a kick, shake -give a shake, cuddle -give a cuddle, wink -give a wink; we first extracted sentences in which the target phrases occurred from the british national corpus (bnc) and obtained a total of 161,752 sentences. generally, all three categories of events have higher frequency when occurring as a transitive verb, and punctive count events have the lowest frequency in both transitive 4 and ditransitive contexts (detailed statistics for context extraction can be found in for each target event, we sampled 40 sentences where it occurs in a transitive context and 40 sentences where it occurs in a ditransitive context from 4 indeed, wink, talk, and speak are not transitive verbs as they precede prepositional phrases (wink *(at) the girl). @xcite used "transitive" to refer to the two-place argument structures with either a direct object or a prepositional phrase, and we follow their usage. the british national corpus (leech, 1992), then generated their vector representation via the bert architecture, using the minicons library foot_2 @xcite . for transitive verb sentences, we extracted the embeddings of the verbs from the last layer; for light verb constructions, we used the embedding of the nominal (e.g. we used the contextualized --→ kiss vector to represent to give a kiss). ## experiment 2: investigating the conceptual shift of events in count/mass syntax our second experiment aims at reproducing with bert the finding that the durative events in count syntax (give a talk) are conceptually further apart from their transitive verb counterparts (to talk) than punctive events in count syntax, or durative events in mass syntax. the prediction is drawn from the analogy to mass nouns, such as glass or iron, when they undergo mass-to-count coercion. when the mass nouns are coerced to be used in the count contexts, the denotation of the noun is enriched in a way that is conceptually related but further from its original sense. if durative events in count syntax behave similarly, they are expected to be conceptually further apart from their transitive counterparts than the other two categories of events. we used again bert and the minicons library to generate semantic representations of the target events in context: the idea is to measure the semantic similarity scores of each event (punctive count, durative count, or durative mass) to itself for randomly sampled sentences. we carried out the sampling either i) by selecting context pairs where the target events occur in both cases in its transitive or ditransitive contexts (within the same context type); or ii) by selecting context pairs where the target event occurs once in the transitive context and once in a ditransitive context (between context types). this means that each noun type will have its occurrences sampled in three different ways: the similarity comparison between (1) and ( 3 ) is the most relevant one for our study: we expect that similarities in (3) to be much lower than in (1), to an extent proportional to the meaning shift that the event undergoes when the count/mass syntax is introduced. conceptually, the difference between (2) and ( 3 ) should be similar to the difference between (1) and (3), since this difference approximates the degree of meaning shift that the events undergo when going from a ditransitive frame to the transitive contexts. however, given the relatively low frequency of the three categories of events in ditransitive constructions, we deemed more appropriate to compare (3) with (1) rather than (2), as the limited sentences in (2) may not fully represent the meaning of the events in count/mass syntax. for each category of event, we repeat the sampling ten times for each group, and for each time we randomly extract 10 different context pairs to generate the vectors. for context pairs classified as occurring in transitive contexts, we simply used the vector of the bare verb as the semantic representation of the target events. for context pairs in the ditransitive frame, we used the embedding of the nominal representing the target event, rather than the whole give construction. this choice is motivated by the fact that, according to the linguistic theory @xcite @xcite , give in the light verb construction only communicates the directionality of the action whereas the bulk of the action meaning is conveyed by the event nominal. therefore, we deemed it better to use the vectors of the deverbal noun instead of the whole phrase to represent the meaning of the target events. as a similarity score, instead of cosine, we used the spearman correlation between vectors. the reason is that contextualized vector spaces are affected by anisotropy @xcite , with a small number of dimensions having disproportionately high variance. metrics like cosine have been shown to be severely affected by outlier dimensions, while rank-based metrics like spearman are better correlated with human similarity judgements @xcite . the results are reported in the event talk in s1 and s2 refer in both cases to the means of communication or conveying information by spoken words, while talk in s3 is tended to be interpreted as a more formal activity. accordingly, the similarity of s1 and s2 is 0.60, and the similarity of s1 and s3 is 0.51. the difference between the similarity scores should reflect the degree of conceptual shift when entering in the count syntax. notice that, when the sentence pairs are sampled across contexts, it is intuitive to hypothesize that their average similarity will be lower than that of the same contexts as the partof-speech of the word representing the event has changed (e.g. speak -speech). however, if our results are to replicate the findings of @xcite , we expect the pairs of durative count events to undergo a greater meaning shift than the other categories. from a possible reason for the absence of significant conceptual shifts might be an inherent shortcoming of vector similarity in distributional spaces: metrics like cosine and spearman tell us that two word meanings are related, but they fail to tell us in which way those meanings are related. a large body of literature in distributional semantics focused on this issue regarding nominals, and pointed out the struggle in teasing apart relations such as synonymy and hypernymy/hyponymy @xcite @xcite , but the same problem could apply to the relation between durative count verbs and the corresponding light verb constructions. for example, the meaning of to give a lecture is still related to the meaning of to lecture, while being a more specific type of lecturing-event @xcite , similarly to the hyponymy relation in nominals. it is thus possible that such relations are not clearly distinguishable from near-synonymy between verb phrases on the basis of distributional similarity. ## conclusions in our paper, we presented an analysis of subtle meaning changes in event construal, comparing transitive verbs and light verb constructions, using the bert model to represent the meaning of events in a distributional semantic space. in experiment 1, we focused on event duration, by identifying a duration dimension in our bert vector space via the semantic projection technique @xcite . similarly to the original study by @xcite , we found that the light verb construction has a general shortening effect, with the vectors for the construction having generally lower values along this dimension than the ones of the corresponding transitive verbs. we take this result as initial evidence that the bert vector space encodes subtle meaning nuances related to the representation of time in natural language sentences. however, in experiment 2, we compared the distributional similarities of transitive verbs vs. constructions pairs, to see if the model was sensitive enough to spot the meaning shift that durative count events undergo in light verb constructions (cf. the experiment 4 in @xcite ). in this case, the answer was negative, and no significant differences in the meaning shifts across event categories was observed. we suggested that the lack of specificity of vector similarity as a semantic relation may explain this negative result. future work for specializing contextualized vector spaces, similarly to what has been done for static models @xcite , may be needed to handle fine-grained semantic distinctions.