Evaluating Lexical Aspect with Large Language Models (2024)

Abstract: In this study, we explore the proficiency of large language models (LLMs) in understanding two key lexical aspects: duration (durative/stative) and telicity (telic/atelic). Through experiments on datasets featuring sentences, verbs, and verb positions, we prompt the LLMs to identify aspectual features of verbs in sentences. Our findings reveal that certain LLMs, particularly those closed-source ones, are able to capture information on duration and telicity, albeit with some performance variations and weaker results compared to the baseline. By employing prompts at three levels (sentence-only, sentence with verb, and sentence with verb and its position), we demonstrate that integrating verb information generally enhances performance in aspectual feature recognition, though it introduces instability. We call for future research to look deeper into methods aimed at optimizing LLMs for aspectual feature comprehension.

## introduction aspect is a verbal category that is closely linked to concepts such as tense, temporality, verbal semantics, and quantification. in linguistics, aspect refers to different perspectives on the internal temporal constitution of a situation @xcite @xcite . there is two main sub groups of aspect, the grammatical aspect which refers to the verbal flexion in languages such as slavic languages, and the lexical aspect which contains the semantics of the event or state of a verb phrase situated in time. in this paper, we focus on the lexical aspect with two important aspect features: duration and telicity. duration (durative/stative) is the property of a verb or verb phrase that presents a state or an action, regardless of their endpoints. durative aspect denotes the reading of an action, while stative aspect denotes the reading of a state. telicity (telic/atelic) distinguishes between verbs that describe an action label sentence durative the boxer is hitting his opponent. stative bread consists of flour, water and yeast. telic i ate a fish for lunch. atelic cork floats on water. or event as having a specific endpoint. telic aspect denotes the reading of the endpoint of an action or event, while atelic aspect denotes the reading of no endpoint. the identification of the aspectual features of the verbs in the sentence could be difficult as other verb categories or sentence elements such as tense, temporal adverbials, and context could affect the reading of the aspect @xcite . using computational models to identify the aspectual features could be therefore more challenging. there are various existing works on building datasets for lexical aspect and training models to classify the sentences in terms of their aspectual features @xcite @xcite @xcite . nowadays, the vast expanse of llms has also opened the chance to study linguistics using llms @xcite . therefore, it is interesting to probe the proficiency of llms on aspectual features. in this paper, based on a dataset on duration and telicity @xcite , we evaluate the ability of 6 different llms to identify the two aspectual features of sentences by zero-shot prompting the llms in three different levels: sentenceonly, sentence with verb, and sentence with verb and its position. our experimental results show that some llms are capable of capturing aspectual information, while there are some variations and weaker performance compared to the fine-tuning baseline. in addition, adding verb information generally improves the prediction performance of llms. overall, our study provides valuable insights into the challenges and opportunities in leveraging llms for evaluating lexical aspect. ## related work the evaluation and classification of aspectual features of verbs using nlp have been explored extensively in previous research. @xcite are the first to employ supervised machine learning methods for aspectual classification. friedrich and palmer (2014) introduced a semisupervised approach that combined linguistic and distributional features to predict a verb's stativity/duration, also providing two annotated datasets for stativity. furthermore, @xcite focused on classifying clauses based on their aspectual properties, and expanded the scope to include situation entity types in @xcite . @xcite contributed two english datasets with gold and silver annotations of telicity and duration, utilizing an l1regularized multi-class logistic regression model. @xcite computationally modeled vendler classes @xcite for 95 german verbs, combining distributional vectors with supervised classification. additionally, @xcite developed the first open-source tool for annotating morphosyntactic tense, mood, and voice for verbal complexes in multiple languages. kober et al. (2020) introduced a dataset for tense and aspect concepts using natural language inference and proposed modeling aspect of english verbs in context using compositional distributional models. in a more recent study by using a bunch of transformer-based models, @xcite conducted experiments on transformer models to identify aspectual features, revealing biases towards verb tense and word order. however, in the current era of the advances of llms, it is still unexplored whether the llms are able to capture the aspectual features. a more detailed introduction to aspect concepts and their computational approaches can be found in this survey @xcite . ## experiments dataset. we use the dataset with telicity and duration-annotated sentences created by @xcite . the dataset was built upon two previous datasets from @xcite and @xcite . it has two main subsets, one for duration and the other for telicity. each subset contains sentences with the main verbs and their positions in sentences, as well as binary labels for durative ('1') or stative aspect ('0') in the duration subset, and telic ('1') or atelic ('0') aspect in the telicity subset. the label distribution in the test sets is presented in §a.1. prompt. each question consists of a general instruction with a choice of answers (e.g. durative or stative) and the example sentence. we include the sentence, verb and verb information into the prompt. in addition, to test the robustness of the models as well as the ability of the models to comprehend the aspectual features both in the sentence level (without explicitly mentioning the verb) and the verb level (with explicitly mentioning the verb), we conduct the experiments in three different levels with different prompt formats. models. we evaluate the aspect tasks with the following close-and open-source instruction-tuned llms: gpt-3.5 @xcite and @xcite , llama-2-13b-chat-hf and meta-llama-3-8b-instruct (touvron et al., 2023), gemma-7b-it @xcite , and mixtral-8x7b-instruct-v0.1 @xcite . baseline. we compare our zero-shot prompting of llms with the baselines of fine-tuning bertbased models @xcite on the training data with and without adding information on the verb position as in @xcite . we select the best performing model bert-large-cased in their work for fine-tuning as baseline. ## discussion & conclusion this preliminary study evaluates the performance of various llms in recognizing lexical aspects, specifically duration and telicity, in zero-shot scenarios. we notice while llms, especially the closed-source ones (gpt-3.5 and gpt-4), are ca-pable of recognizing the lexical aspects of verbs in sentences, they lie behind the fine-tuned baselines, indicating the potential need for further adaptation to effectively probe deep linguistic features. we conduct experiments across three levels of prompting to assess the impact of including the verb and its position in the sentence. our results reveal that llms, particularly the closed-source ones, benefit from the additional context of verbs. however, this added complexity sometimes introduced regressions, indicating that while context aids comprehension, it can also pose challenges. the case analysis also introduces concerns about the complexity of hallucinations within the models. future research could explore methods to optimize llms for aspectual feature recognition, such as fine-tuning llms or incorporating additional linguistic knowledge into model training. currently, we only conduct the prompt in zero-shot settings, i.e. without context information. previous work showed that prompt-based methods may underestimate the linguistic knowledge of llms @xcite . therefore, we call for future exploration in different settings, such as few-shot prompting and chain-of-thought (cot, @xcite prompting. overall, our study offers valuable insights into the challenges and opportunities of utilizing llms for linguistic feature recognition.