About Time: Do Transformers Learn Temporal Verbal Aspect? (2022)

Abstract: Aspect is a linguistic concept that describes how an action, event, or state of a verb phrase is situated in time. In this paper, we explore whether different transformer models are capable of identifying aspectual features. We focus on two specific aspectual features: telicity and duration. Telicity marks whether the verb’s action or state has an endpoint or not (telic/atelic), and duration denotes whether a verb expresses an action (dynamic) or a state (stative). These features are integral to the interpretation of natural language, but also hard to annotate and identify with NLP methods. We perform experiments in English and French, and our results show that transformer models adequately capture information on telicity and duration in their vectors, even in their non-finetuned forms, but are somewhat biased with regard to verb tense and word order.

## introduction aspect is a linguistic concept that characterizes how an action, event, or state (expressed by a verb phrase) relates to time, beyond the scope of the verb's tense; via aspect, information such as frequency, duration, and completion is conveyed. some verbs express events or actions that have or do not have a clearly-defined endpoint because of their meaning (lexical aspect or aktionsart), while others can express different temporal properties in different contexts and forms (grammatical aspect). languages may express aspect in various ways, e.g. by using grammatical verb tense (incomplete actions with continuous/progressive, perfect progressive and imperfect, complete actions with perfect), morphemes (e.g. finnish, czech) or with aspect markers (e.g. mandarin chinese). however, certain aspectual features cannot simply be deduced from morphosyntax and require some degree of semantic knowledge. in this paper, we focus on two of these aspectual features: telicity and duration. telicity is related to the goal-oriented nature of the verb phrase. the verb's action is said to be telic if it has an endpoint; for example, verbs which demonstrate an action such as kick, eat ("i kicked the ball.", "i eat an apple.") are telic, because the action described has a perceived ending. when the verb denotes a state, e.g. exist, or when the completion of the verb's action is either indefinite, impossible or irrelevant, e.g. agree, stay ("i agree with you.", "we stayed at the hotel."), then the verb phrase is characterized as atelic. duration is another aspectual feature, different from telicity: it distinguishes between verbs that describe a state (stative, e.g. occupy, lie) or an action (durative, e.g. run, knock) regardless of whether they have a perceived endpoint or not. the perception of telicity and duration is the outcome of the entire verbal phrase, and not solely the verb's features @xcite . besides, the context can also place constraints on the aspectual class of a verb @xcite . therefore, making sound judgments on aspectual features such as telicity and duration, especially in a morphologically-poor language like english, is not always an easy taskour datasets in section 4.1 provide some examples of sentences where these features are hard to assess, even for a human. aspect has been exploited for tasks where semantic knowledge is necessary, since it provides information on temporal relations @xcite , textual entailment @xcite and event ordering @xcite . in recent years, transformer-based models have shown great success in nlp tasks which traditionally require in-depth language analysis and complex strategies on capturing dependencies, semantic information, and world knowledge. however, it remains unclear whether the success of these models is due to a genuine capability to accurately model linguistic meaning, or whether the models are just very good at picking up statistical correlations, but fail to capture fine-grained semantic distinctions @xcite . with this research question in mind, our goal is to investigate whether transformer-based architectures (both with and without fine-tuning) are able to capture the semantic information related to telicity and duration. to do so, we make use of two datasets annotated for telicity and duration @xcite , and we conduct a range of experiments using several pretrained transformer architectures in two languages (english and french). we extend our experiments from @xcite , where we only made use of the friedrich and gateva dataset and only in english. we aim to explore the capabilities of transformer architectures in classifying aspect beyond mere quantitative evaluation: we made custom qualitative datasets in order to observe how complex context, verb tense and prepositional phrases affect classification. 1 we find that classification with fine-tuned models is very successful-both for telicity and duration-and this success can be largely attributed to the knowledge built up during pre-training, as contextual word embeddings by themselves are already quite capable of capturing this information. we noticed that complex cases where the context was conflicting with the verbal aspect were harder for the models to classify, and we provide evidence that misclassification in complex sentences is related to verb tense and word order. finally, comparing the two languages we investigate, even though the french models show lower accuracy, they were more successful in classifying more difficult cases of telicity and duration, because of the properties of verbal tense in french. ## acquisition of telicity and duration before examining how transformer models handle telicity and duration, it is important to briefly present how humans learn to identify and express these concepts. complex semantic features are learned by humans with the use of multiple exemplars in the speaker's l1 (mother language), in order to create constructions which encapsulate abstract concepts, such as the perceived duration of an action and the presence or absence of an outcome @xcite . frequency @xcite and distributional bias @xcite are crucial for the acquisition of a language's spe-cific patterns of expressing these concepts, however, their semantics and lexical properties are separate from the grammar of the language and interact with it, to understand and express concepts. focusing on lexical aspect, @xcite and @xcite present the aspect hypothesis, claiming that children associate past and perfective marking to telic verbs (applying it to activity, accomplishment and achievement verbs in this order) and avoid such marking with stative verbs. @xcite confirm this hypothesis experimentally, showing that there is a strong negative correlation between telicity and progressivity (e.g. speakers will mostly avoid using progressive tenses with telic verbs). @xcite observed, in a self-paced reading experiment, that the combination of aspectually conflicting predicate and temporal modifiers in sentences produced a delay in processing -this suggests that humans have some preferred temporal association with verbs and modifiers, and when there is contradicting context, there is a need for reassessment of the given structure. @xcite also conducted experiments of self-paced reading, with sentences with verbs whose telicity degree depends-to some extent-on the verb's object (e.g. consumption verbs with a finite/infinite object), and observed that there was no time cost in the processing of these sentences (also pointed out by @xcite , which leads to the conclusion that the processing of a predicate, even with conflicting telicity marking, is simpler than the additional information of a temporal preposition. however, van hout (1998) claims that prepositions are mentally learned as markers of telicity earlier in life than the presence of bound/unbound objects (in experiments with dutch as l1), meaning that some function words are also considered important for the final telicity degree of an utterance. regarding duration, in earlier stages of language acquisition, it has been observed that children may erroneously assign stativity to an action without immediate change at the time of utterance @xcite , and such mistakes also occur in l2 learners of english (i.e. people who are learning as a foreign language). @xcite also noted that l2 learners of chinese acquired the perfectivity markers before the duration markers. such findings further support the aspect hypothesis, showing that the perception of time requires a significant amount of processing and contextualizing for humans, and that the lexi-cal aspect of a verb (and therefore, the telicity and duration of its presented action/state) is eventually learned and preferred, but can be overwritten (intentionally, in complex cases, at a computational cost, or erroneously, in earlier stages of language acquisition). friedrich and palmer (2014) use a semisupervised approach for learning lexical aspect, combining linguistic and distributional features, in order to predict a verb's stativity/duration, and also released two datasets of annotated sentences for stativity. @xcite extended this approach by classifying verbal lexical aspect into multiple categories of duration, habitual/episodic/static, and @xcite expanded their datasets and categories, achieving 76% accuracy on supervised classification compared to the 80% of their human baseline. in their most recent work, @xcite have released two datasets in english with gold and silver annotations of telicity and duration (gold is human annotated; silver is obtained from parallel english-czech corpora where aspectual features were extracted from czech morphological markers). with these datasets and an l1-regularized multi-class logistic regression model, they report significant improvement on automatic telicity classification. loáiciga and grisot (2016) exploit telicity in order to improve on french-english machine translation; they are using verb classification of telicity (defined as boundedness) and notice improvement on the translation of tense. falk and martin (2016) also use a machine learning approach, alongside morpho-syntactic and semantic annotations, to predict the aspect of french verbs in different contexts (verb readings). moving away from hard-coded annotations and lexical aspect, peng (2018) uses two different compositional models to classify aspect, exploring the entire clause and not only the verb, with the use of distributional vectors and with-out annotated linguistic features, and highlights the importance of the verbal phrase and the verb's dependents in the interpretation of telicity. @xcite propose modeling aspect of english verbs in context, with the use of compositional distributional models, and confirm that a verb's context and closed-class words of tense are strong features for aspect classification. ## discussion transformer models were quite successful in the classification tasks, outperforming our baselines to a large extent, and they proved to be quite successful even without fine-tuning. contextual embeddings proved to be an efficient way to encode the aspectual information of a verb and its interaction with its context, and this knowledge is probably already learned in the pretraining process. the superior performance of the duration classification with fine-tuned models did raise a question: from our datasets, most stative questions came from the friedrich dataset and most durative sentences from the captions dataset; did the models learn to classify duration or to identify the different corpora? with our qualitative analysis on two languages, we can conclude that the models are indeed able to classify duration and were successful because of the little overlap between stative and durative verbs and contexts. however, the models struggled with sentences for which world knowledge is crucial, which is a known issue @xcite . from our experiment with verb tenses and prepositional phrases, we noticed that perfect and continuous tenses are beneficial to classification by the models, and leading a sentence with a prepositional phrase of time sometimes improved predictions. however, conflicting context will almost always confuse the models. in addition, our findings on the french datasets showed that, even with our lower-performing models, the syntactic and semantic choices that a language makes in expressing aspect did affect the models' capabilities of classifying aspect.the differences in classification errors and successes that we observed, between the qualitative datasets of the two languages, may also indicate that there is a different way in which languages are semantically represented by transformer models, even with different model architectures. ## conclusion in this study, we conducted several experiments that test the capability of transformer models to grasp aspectual categories, viz. telicity and duration. we tested this capability using a binary classification setting. using two annotated datasets for telicity and duration @xcite , we fine-tuned transformer models of different architectures and in two languages and found that transformer models were very successful on the classification of aspect even when trained on small datasets. providing the verb position as additional information improved performance in both telicity and duration classification for english. the pretrained transformer models also possess knowledge of aspect even without finetuning (when looking at layerwise contextual word embeddings). however, our qualitative analysis also revealed weaknesses; for complex sentences, where the verbal aspect contradicted the temporal information in the context (e.g. telic verb with an atelic prepositional phrase, resulting in an overall atelic sentence), the models classified based on verb rather than context, meaning that they are able to distinguish the most important part of the sequence but not capture more fine-grained information when it is necessary.