Human Temporal Inferences Go Beyond Aspectual Class (2024)

Abstract: Past work in NLP has proposed the task of classifying English verb phrases into situation aspect categories, assuming that these categories play an important role in tasks requiring temporal reasoning. We investigate this assumption by gathering crowd-sourced judgements about aspectual entailments from non-expert, native English participants. The results suggest that aspectual class alone is not sufficient to explain the response patterns of the participants. We propose that looking at scenarios which can feasibly accompany an action description contributes towards a better explanation of the participants’ answers. A further experiment using GPT-3.5 shows that its outputs follow different patterns than human answers, suggesting that such conceivable scenarios cannot be fully accounted for in the language alone. We release our dataset to support further research.

## introduction aspect is a linguistic category concerned with how actions, as described by verb phrases, unravel over time. situation aspect foot_0 refers to the underlying semantic property of a verb phrase. for example, to sit is different from to pack in that there is a natural point at which packing is complete, but there is no such point for sitting. situation aspect is often defined in terms of three properties: stativity, telicity and durativity. the combination of values which a verb phrase takes for each of these properties is what decides its belonging to a particular aspectual class. for example, i love you is stative, atelic (has no pre-determined enpoint) and durative (spans across a period of time). by contrast, i caught the ball is dynamic, telic (has a clearly defined endpoint) and punctual (occurs instantly). work on automatic aspectual classification in english has been motivated as a pre-requisite for natural language understanding (nlu) in cases where temporal reasoning is required @xcite @xcite . consider the two examples: (1) i was listening to @xmath0. having said that, there is no empirical evidence that aspectual class is helpful for nlu tasks in practice, where more pragmatic inferences are favoured over strict logical entailments @xcite . this paper asks whether the role that aspectual classification is described to play on logical entailments is reflected in crowd-sourced data, where participants were allowed to take a less formal approach. we designed a survey with examples of verb phrases turned into sentence pairs: one in past progressive and one in past simple, like the pairs in examples (1) and (2) above. we gather participants' judgements on whether the past simple can be inferred from the past progressive. what our survey clearly shows is that aspectual classification does not best explain how non-expert participants reason about eventuality. instead, the results are better explained by considering possible scenarios which can accompany any given verb phrase, for example whether an an action is likely to be interrupted or not. we release the the anonymised survey responses to enable further research. 2 finally, with an experiment using gpt-3.5, we show that large language models (llms) do not capture the answer patterns seen in participant answers. we speculate that this is because people's strategies for reasoning about events are an example of 'understanding' that people gain through physical experiences of the world and cannot be modelled by linguistic material alone @xcite . ## background situation aspect is a semantic property of a situation description. in literature, we often see it characterised in terms of these three properties @xcite : telicity is a concept that describes whether a situation has a culmination point. a telic situation leads up to a necessary endpoint, beyond which it cannot continue, whether that point has been reached or not @xcite . conversely, an atelic situation lacks such a pre-defined finish, whether it's still in progress or not. for example, to drown is telic and to dance is atelic. durativity refers to the fact that certain situations span over a period of time regardless how long or short (durative), whilst others are instantaneous (punctual; @xcite . for example, to run is durative, whilst to die is punctual. valid combinations of aspectual features is what defines the distinction between different situation aspect categories, often referred to as aktionsarts @xcite , which offers alternative terminology for talking about aspectual classification (as summarised in ## human experiment in our experiment, we presented participants with pairs of sentences constructed from one base verb phrase: one in past progressive and one in past simple. we use the 'imperfective paradox as telicity figure 1 : the slider interface used to gather participant's answers. the closer to the edge of the slider, the more certainty in the answer is expressed. participants were instructed to use the 'does not make sense' checkbox if they deemed any or both of the sentences ungrammatical. ## discussion at first glance, it should not be surprising that the answers of the participants do not match with the category-based predictions, given our survey's setup. the nli literature observes that without providing people with explicit annotation instruc-tions on whether they are allowed to use their real world knowledge or consider any additional context from outside the text material, they tend to take on different strategies to resolving uncertainties @xcite @xcite . in our set-up, the participants were free to conjure up their own contexts and take on any approach they like, which is inline with how labels are gathered in most modern nli datasets @xcite . in this section we zoom in on particular examples to explore how such possible scenarios could explain the participants' answers distribution. ## llm experiment the training data used in the creation of any llm contains a vast number of examples of verb use in context. however, seeing a verb in textual context is not the same as experiencing an event described by that verb. therefore, we propose that by looking at outputs of llms, one can ask if the use of language alone is sufficient to account for the disagreement that certain predicates elicited amongst human participants. in other words: is the disagreement reflected in the way people talk about those events at large, or is there more to it? to investigate this, we ran an experiment with gpt-3.5 @xcite . the reason for choosing gpt-3.5 is twofold. first, it is one of the more recent llms featured prominently in current literature. second, unlike the newer gpt-4, it allows us to retrieve the probability values assigned to the top 5 candidates for the next predicted token. at this stage, we did not run experiments with other llms -our aim is not to provide a comprehensive model comparison, but rather use any llm as a large scale language resource that implicitly encodes a multitude of possible textual contexts. we used the text-davinci completion model. we set temperature to 0 and top_p to 1. 5 we used the following as a prompt template: answer the question with y for yes and n for no. question: if the sentence s1 is true, does it neces-knowledge about event structures can be captured by such a model without any experience of the physical world @xcite . having said this, the results presented here are based on one llm only, so our conclusions are not a definite answer, but an invitation for future research. we believe that by using llms as proxies for large-scale corpus analysis future research can ask interesting questions about the respective roles of textual information and physical experiences in building our 'understanding' of event structures. ## conclusion it is clear from our results that whilst aspectual class can be used as a rough guide as to which inferences some subjects may draw some of the time, it is far from being the main deciding factor. we also show that predictions of gpt-3.5 are not entirely aligned with the participants' responses. this opens the door to further research into what influences human understanding of event descriptions. the examples we collected for our experiments can also be used as a dataset to explore the role of pragmatics in nli as well as other nlu tasks.