Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognition (2024)

Abstract: Verbs form the backbone of language, providing the structure and meaning to sentences. Yet, their intricate semantic nuances pose a longstanding challenge. Understanding verb relations through the concept of lexical entailment is crucial for comprehending sentence meanings and grasping verb dynamics. This work investigates the capabilities of eight Large Language Models in recognizing lexical entailment relations among verbs through differently devised prompting strategies and zero-/few-shot settings over verb pairs from two lexical databases, namely WordNet and HyperLex. Our findings unveil that the models can tackle the lexical entailment recognition task with moderately good performance, although at varying degree of effectiveness and under different conditions. Also, utilizing few-shot prompting can enhance the models’ performance. However, perfectly solving the task arises as an unmet challenge for all examined LLMs, which raises an emergence for further research developments on this topic.

## introduction verbs hold a central role in language, serving as the foundational and semantic framework for conveying sentence meaning. understanding their semantic nuances has been a long challenge due to the considerable variability and relatively looser cohesion of verb meanings compared to nouns @xcite . exploring the relationships between verbs can be addressed by considering the concept of lexical entailment @xcite . analogous to logical entailment, which applies to propositions, lexical entailment describes the relationship between two verbs v 1 and v 2 , wherein the sentence ⟨subject⟩v 1 logically entails the sentence ⟨subject⟩v 2 , i.e., that one verb leads to another in the sentence. focusing on lexical entailment is paramount for several reasons. verbs are crucial for expressing actions and relationships between entities, making it essential to properly capture their nuances. grasping these relationships helps in deciphering sentence meanings and how verbs work together. verbs often have polysemous and contextdependent meanings, and capturing entailment relations among verbs involves addressing challenges like aspect, modality, tense, and fine-grained semantic differences. also, certain domains or specialized fields heavily rely on precise verb relations, such as in legal, scientific, or technical language. the study of entailment has long been recognized as a critical endeavor in nlp and related fields, as it is fundamental to several tasks such as classification, summarization, question answering, and machine translation. given their remarkable success in solving the aforementioned tasks, large language models (llms) have indeed reshaped the landscape of language understanding @xcite ; nonetheless, to achieve even more sophisticated capabilities in interpreting human communication through verbal nuances, unveiling entailment recognition into these models represents a demanding challenge for their development. this has a number of motivations, which can be summarized as follows. correctly inferring entailment relations between verbs is indeed essential for a llm to be robust in understanding nuanced meanings and logical connections within sentences. by exploring how llms interpret and handle entailment among verbs, we can also gain insights into their decision-making processes, unveiling their strengths and weaknesses, and also contributing to model interpretability and refinement. as previously mentioned, handling verb entailment is crucial in various nlp tasks: on the one hand, improving models' comprehension of these relations can enhance the performance of these applications, but on the other hand, identifying and addressing biases or misinterpretations in entailment relations is also important in refining models to handle diverse linguistic contexts and minimize errors in real-world applications. a related aspect, as recently noted in @xcite for textual entailment in general, is an emergence for developing evaluation datasets and benchmarks for assessing the performance of models in entailment tasks. in this regard, we recognize the primary role played by lexical databases, which are meaningful resources for semantically exploring verb relations. in this work, we will focus on two widely used resources, namely wordnet @xcite and hyperlex @xcite . our study aims to unveil the actual capabilities of llms in recognizing lexical entailment relations. by focusing on instances of verb pairs corresponding to entailment relations provided by lexical databases, we conduct a thorough evaluation based on eight llms, with emphasis on open llms. we define different prompting strategies with different context details, both under a zeroshot and a few-shot setting, for asking a llm to answer about a question relating to the entailment between any two verbs. our primary evaluation goal is to understand to what extent llms are able to recognize lexical entailment between verbs by measuring their compliance with well-grounded, manually curated linguistic resources. in the remainder of this paper, we discuss related work in sect. 2, the data sources and the selected llms in sect. 3. our defined methodology and experimental results are described in sects. 4 and 5. section 6 concludes the paper. ## related work we discuss recent studies involving llms, and more generally pre-trained language models (plms), to capture the meaning and connection between words. @xcite address word sense disambiguation in terms of textual entailment to understand if bert and roberta can discriminate between different senses in a variety of domains. @xcite train a mt5 model for generating chinese word glosses, and raise the need for models to rely on semantic vectors. dealing with negation represents an additional challenge. chen et al. (2023) assess negative commonsense knowledge of llms. experiments carried out on flan-t5 @xcite , @xcite , @xcite , instruct gpt @xcite , and chatgpt foot_0 reveal behavior inconsistency among the llms. @xcite test the commonsense knowledge of open source llms (t5 @xcite , llama @xcite , pythia @xcite , falcon @xcite , vicuna @xcite ) using both affirmative and negative sentences using various types of relations and negations. results have shown that the llms excel in classifying affirmative sentences but fail in dealing with negative ones. more generally, plms and llms have been tested over various types of semantic relations. @xcite analyze the ability of bert-based models and sentence-transformers to capture hierarchical semantic knowledge using wordnet-derived datasets. oliveira (2023) apply bert to capture synonyms, antonyms, hypernyms, and hyponyms in the portuguese language. hypernymy is also a focus of the study in @xcite , which builds a dataset on the wordnet hypernyms and test several plms and llms on hypernymy discovery, observing a consistent underperformance of llms when tasked with abstract concepts. @xcite assign words with a common wordnet hypernym into the same class, and train plms by gradually transitioning from predicting the class to predicting the token through a curriculum learning strategy. also, tikhomirov and loukachevitch (2024) evaluate the use of llms with various prompts for hypernym prediction. other studies investigated how to leverage the relationships and definitions provided by wordnet to enhance the representation through generated embeddings and data augmentation @xcite . our work uniquely provides an analysis of how currently used llms can recognize verb entailment relations, in contrast with works with broader scopes like @xcite and @xcite . our focus on verbs also differs from studies such as @xcite , which consider an organization of relations into domain-specific and high-level concepts. unlike @xcite , we do not focus on attempting to determine a specific sense for a sentence, instead our aim is to understand how llms answer to specific queries about semantic relations. our methodology differs from chen et al. (2023) and garcía-ferrero et al. (2023), since they consider negative relations and not necessarily on verbs. moreover, we focus on a representative set of recently developed open and commercially-licensed llms, while garcía-ferrero et al. (2023) consider open models only and chen et al. (2023) focus on models earlier than chatgpt. oliveira ( 2023 ) is limited to portuguese and uses bert to determine relations of hyponymy, hypernymy, synonymy, and antonymy. unlike @xcite , which train projection layers to learn wordnet hypernym relations, we approach the problem through a probing approach. compared to @xcite , which focuses on reducing model perplexity, our interest is understanding how models specifically handle verb entailment relations. ## results we organize the result presentation into two subsections, which correspond to evaluation on wordnet data, resp. hyperlex data. we shall first discuss results obtained in a zero-shot scenario, then those in the few-shot scenarios. in the tables, bold resp. underlined values will correspond to the highest resp. second-highest scores per column. ## conclusions we presented an investigation of the abilities of a representative body of llms in tackling the task of lexical entailment recognition. to the best of our knowledge, this is the first systematic study that aims to shed light on the actual skills of llms, with emphasis on open models, to recognize entailment relations between verbs, gauging their accuracy against well-grounded, manually curated lexical resources such as wordnet and hyperlex. to accomplish our research goal, we defined three prompt types, providing different levels of contextual information, in both zero-shot and few-shot learning scenarios. our results have shown evidence of both abilities and limitations that arise in the examined llms, which can be summarized as follows: (i) although at varying degree of effectiveness and under different conditions, the llms can tackle a task of lexical entailment recognition with moderately good results, however, perfectly solving the task remains an unmet challenge for all the examined llms; (ii) few-shot prompting can improve the models' performance in addressing the task; and we believe our study can advance our understanding of how llms grasp nuanced meanings and logical relationships among verbs, providing valuable insights into their interpretability and decision-making processes.