World Knowledge Resolves Some Aspectual Ambiguity (2025)

Abstract: Annotating event descriptions with their aspectual features is often seen as a pre-requisite to temporal reasoning. However, a recent study by Pruś et al. (2024) has shown that non-experts’ annotations of the aspectual class of English verb phrases can disagree with both expert linguistic annotations and each another. They hypothesised that people use their world knowledge to tacitly conjure their own contexts, leading to disagreement between them. In this paper, we test that hypothesis by adding context to Pruś et al.’s examples and mirroring their experiment. Our results show that whilst their hypothesis explains some of the disagreement, some examples continue to yield divided responses even with the additional context. Finally, we show that outputs from GPT-4, despite to some degree capturing the aspectual class division, are not an accurate predictor of human answers.

## introduction aspect as a linguistic category refers to analysing situation and action descriptions in terms of their internal structure. for example, to unpack a bag differs structurally from to play with toys in that the former one culminates at a well defined endpoint whilst the latter one does not. aspectual properties of a predicate are consequential to the entailments one can make regarding the action it describes. in the english language specifically, aspectual class is told to determine whether a given predicate's past progressive (pp) form entails its past simple (ps) or not. for example, we note the following: in this paper we test that hypothesis. whilst pruś et al. kept their examples purposefully decontextualised, we expand the pp sentences to include context indicating that the action was interrupted or abandoned. for the boil an egg example, our pp sentence with an interruption was i was boiling an egg, but my camping stove ran out of fuel. we then mirror all of their study design details and compare the judgments collected between the two settings. we show that providing the additional context led to increased number of participants answering in accordance with theoretical predictions for some -but not all -predicate examples. whilst this supports the hypothesis of @xcite , it also reveals that there are more factors at play in shaping people's answers. the fact that certain predicates elicited mixed responses even in the setting with the additional context, tells us that aspectual class of certain predicates may be open to individual interpretation. finally, we present an experiment querying gpt-4 to generate responses for the same questions that were shown to the human participants. we observe that its response patterns might follows some general trends of human responses on the aspectual class level. however, looking at individual predicates we find that the distributions of gpt-4 outputs differ considerably from the distributions of human answers. vendler (1967) divides situations into 4 main categories: states, achievements, accomplishments and activities. those categories are known as aspectual classes. subsequent literature analysed these classes in terms of three different properties: stativity, durativity and telicity @xcite . stativity differentiates between stative and dynamic eventualities. broadly speaking, the term stative refers to situations that do not require effort to be sustained. all states are stative and they are contrasted with dynamic events. for example, to know the answer is a state, but to jump over the fence is an event. achievements, accomplishments and activities are all different types of events. durativity pertains to whether something occurs instantaneously (is punctual) or extends across time (is durative). for example, to dance is durative and to land is punctual. states are all durative, whilst events can be either durative or punctual. ## human experiment most of the experimental design details of this study directly follow those of @xcite (hereafter also referred to as the original study). this is to allow for a direct comparison of the results between the two studies. much like the original study, we present the participants with pairs of sentences constructed around a common base form verb phrase. one sentence contains the verb phrase in past progressive and the other one the same phrase in past simple. the participants are then asked to judge whether there is an entailment from the past progressive to past simple. whilst @xcite their examples devoid of context, we purposefully add context to the past progressive sentence signalling that the action at hand has been interrupted or abandoned. therefore, in the discussion and interpretation of the results, we will refer to the results from the original study as decontextualised setting and the results of this study as the interrupted setting. ## discussion in their experimental results, pruś et al. showed that where predicates are presented to participants devoid of context, there is room for disagreement on whether for a given predicate there exists an entailment from past progressive to past simple. they suggested a number of possible explanations for that disagreement. one explanation they proposed was that people tend to use a common-sense approach to reasoning. they speculated that people tended to answer 'yes' for some of the telic examples where the theoretically correct answer is 'no', because they simply did not conceive of a scenario where such action gets abandoned. our study addressed this suggestion by adding such a scenario as a context to the predicate. we observed that for some examples participants' answers significantly shifted towards 'no' after adding that context. the existence of examples where that shift occurred provides empirical backing to the above-mentioned suggestion of pruś et al.. having said that, the shift did not occur for all of the examples. we therefore note that ambiguity is harder to resolve in some cases than in others, and so the participants were still left to make a personal judgement. in this section we will discuss selected examples to suggest further insights on what factors may have influenced participants' answers. ## gpt-4 experiment the result of our experiments with human participants shows that judgements on the answer to the imperfective paradox are divided even with the additional context. those differences between judgements are at least in part driven by the differences in participants individual experiences in the real world. in this section we outline an experiment to test whether the distribution of event descriptions in the vast training dataset of a large language model (llm) reflects the common-sense inferences of the subjects to such an extent that its responses follow a similar distribution to theirs. we chose @xcite because it is one of the most widely used models at the time of writing. in order to query the model one sends an array of messages through the api. the messages can be assigned different roles. we use the developer role to provide instruction on the answer format and the user role to ask the actual question. the complete prompt looks as follows: developer: answer the question with yes or no. user: if the sentence s1 is true, does it necessarily mean that the sentence s2 is also true? for each predicate in each setting we queried the model 100 times with the same prompt. this is to approximate the underlying probabilities the model has assigned to either of the answers @xcite . the reason we used this approximation is because actual probabilities assigned to generated outputs are not available for gpt-4. we repeated this experiment for 3 different temperature values: 0.6, 0.8 and 1.0. for each predicate at each temperature setting we counted how many times the model answered 'yes' and confirmed that the remaining answers were 'no' (i.e. there was no nonsense answers that didn't follow the developer instructions). we then calculated the arithmetic mean and standard deviation across the three temperature settings. do the answers generated by gpt-4 then follow the similar patterns to people's answers? we observe that overall activities collect more 'yes' responses than accomplishments and achievements. this is in line with the theoretical predictions of the 'correct' answer as well as the overall picture of how people answered these questions. we also note that, similarly to human answers, cook at home was the only activity where a significant shift towards 'no' was observed. beyond that, however, our analysis revealed a number of differences between the human and the model responses. we observe a significant shift towards 'yes' for two examples of activities: enjoy your company and study history. neither of those examples elicited such a shift amongst people. we observe a shift towards 'no' for accomplishments and achievements. we observed a shift in the direction amongst participants too. however, amongst humans, we observed that the shift was more pronounces for achievements than accomplishments and the reverse is true for the model. in general, amongst accomplishments, human answers were often divided between 'yes' and 'no' even in the interrupted setting. consider boil a lobster as an example. amongst people it started with a 'yes'-skewed distribution in the decontextualised setting, whilst it did shift towards 'no', the human answered were still mixed. the model answers, however, usually followed one of two trajectories. they either started at unanimously 'no' in the decontextualised setting, in which case no shift occurred. alternatively, they shifted to unanimously 'no' in the interrupted setting. the intriguing outlier was read a book, where a very significant shift towards 'yes' occurred (from 0 to 84.75±7.5). the pattern of model answers for achievements was similar to that of accomplishments, with the difference that the model was less likely to assign any 'yes' answers even in the decontextualised setting. overall, the model responses paint distinctions between aspectual classes similarly to how people do. however, looking at individual predicates, we see that additional context has a much more extreme effect on the model's answers than it does on the human answers. recently, @xcite have shown the embeddings of gpt-2 do not reflect the theoretical predictions stemming from the imperfective paradox and speculated that the context of their examples could explain that. our results with gpt-4 are certainly in line with that observation. ## conclusions our results show that interruptions being less conceivable for some actions than others, was indeed a factor driving people to answer 'yes' in the decontextualised setting. with some telic examples still showing a strong mix of responses even with the explicit interruption, we can say that it was likely not the only factor at play. we suggest that judgements on which actions are activities and which are accomplishments can depend upon individual differences in experience and understanding in the real world. because a language model lacks such experience @xcite , the distribution of answers generated by gpt-4 differs from the distribution of human answers.