Assessing LLM s’ Understanding of Structural Contrasts in the Lexicon (2025)

Abstract: We present a new benchmark to evaluate the lexical competence of large language models (LLMs), built on a hierarchical classification of lexical functions (LFs) within the Meaning-Text Theory (MTT) framework. Based on a dataset called French Lexical Network (LN-fr), the benchmark employs contrastive tasks to probe the models’ sensitivity to fine-grained paradigmatic and syntagmatic distinctions. Our results show that performance varies significantly across different LFs and systematically declines with increased distinction granularity, highlighting current LLMs’ limitations in relational and structured lexical understanding.

## introduction large language models (llms) like @xcite , qwen @xcite , or llama @xcite do not merely generate coherent text. they can be prompted to solve a wide range of linguistic and cognitive tasks, such as question answering, information extraction, or machine translation, with remarkable performance @xcite . as a result, works on llms' evaluation have shifted focus away from grammaticality and coherence, towards reasoning capacities, factual consistency, bias, or other extra-linguistic properties @xcite . yet, there remain essential questions about the nature and depth of linguistic knowledge captured by these models and their ability to introspectively access and share this knowledge. while llms appear to "use language" fluently, the amount of linguistic structure they "understand" is not clearly circumscribed, nor is their ability to reason abstractly about linguistic objects. the lexicon is a case in point. a proper understanding of language necessarily entails a grasp of its lexicon-not as a mere inventory of words and their definitions, but as a structured system wherein lexical units are interconnected through a variety of relations (like synonymy, antonymy, morphological derivations, intensification, and others) that recur across most (if not all) languages. leveraging such relations to assess linguistic competence has long been an attractive idea: they are, for instance, at the heart of popular analogical benchmarks @xcite @xcite which have become a staple of the evaluation of distributional representations. however, these analogical datasets arguably lack both theoretical grounding and coverage in some areas. for instance, the bigger analogy test set @xcite , one of the most balanced, diverse and challenging benchmarks, covers very few syntagmatic (i.e. related to word combinations rather than word substitutions) lexical relations and leaves out many aspects related to meaning rather than strict morphology (like the analogy between the pairs continue:continuation::sell:sale). we therefore wish to ground an evaluation benchmark on a well-established lexicographic theory: the meaning-text theory (mtt) @xcite @xcite . mtt places the lexicon and its combinatorial properties at the core of linguistic modeling. to formally model the structure of the lexicon, mtt uses a system of lexical functions (lfs), which represent consistent and recurrent paradigmatic or syntagmatic relations between lexical units-that is, words taken in a specific sense. each lf encodes a specific semantic or syntactic relation between a lexical unit (its keyword) and a set of lexical units (its value). the following examples illustrate some of the most common lfs foot_0 : for instance, at a coarse level, we test whether the llm can tell apart instances of adjectival derivations (of any kind) from instances of other type of derivations (e.g. nominal, or verbal ones), and at a finer level, whether it can discriminate rather semantically neutral adjectival derivations (like destroy-destructive) from those involving a stronger meaning shift (like destroy-destructible). to this aim, we associate each target lf with a set of contrastive lfs, so that each contrastive lf both share a common property with the target (e.g. both correspond to some kind of adjectival derivation) and are distinguished by another property (e.g. they correspond to different degrees or types of meaning shifts), and ask llms to recognize the pairs of words obtained from the target and reject those obtained from its contrastive lfs. to automatically obtain the pairs of words, we leverage a high quality french lexicographic resource, the french lexical network @xcite ; atilf, 2024, henceforth, ln-fr), which offers extensive coverage and is closely aligned with the theoretical framework adopted here. although we use french data, the lexical relations we target are universal. we work from the assumption that if a model performs well on french, it should perform about as well on other languages similarly covered by its pretraining material. we thus contribute a hierarchy of lfs, wherein each intermediate level corresponds to some coarsegrained lexical relation (such as 'verbal collocation'), and immediate descendants correspond to distinct sub-relations of the former (such as 'support verbs' and 'semantically loaded verbal collocations'). we propose a benchmark of polar questions to test llms' ability to specifically recognize these contrasts, and assess several open-weights llms on this benchmark, as well as the effect of different prompting configurations. we also investigate the impact of surface cues on the llm's behavior. ## related work the semantic abilities of computational models have often been measured by their ability to recognize or perform analogies. analogical datasets such as sat @xcite , the google analogy test set @xcite , and bats @xcite have become popular benchmark of this capacity. they also have been applied to the evaluation of recent llms' semantic abilities: @xcite evaluate llms on well-established analogical benchmarks using prompts and their completion probabilities, and show, among many other things, that the lexical analogies of bats are more difficult for the models than the morphological or encyclopedic ones. @xcite show that automatically extracting analogies from a knowledge graph can be used to enhance llms performance on analogical benchmarks via fine-tuning or few-shot learning. some new benchmarks have also been developed: @xcite introduce a benchmark of analogies between longer texts, targeting concepts such as entailment or explanation, and @xcite introduce a benchmark of exam problems and associated analogical reasoning. while these resources are important tools to assess higher level linguistic and reasoning capabilities, they also steer away from evaluating the sheer lexical competence of language models. other approaches have taken inspiration from psycholinguistic methods like cloze completion tasks. some of the tasks considered in @xcite directly concerns lexical knowledge. they find that bert @xcite is better at recognizing hypernyms than distinguishing semantic roles. while models' mastery of paradigmatic relations such as synonymy or hyponymy is extensively tested in the aforementioned works, the type of knowledge underlying support or light verb constructions (like chance and take), or tied to the argument structure (doctor and patient) is more often overlooked. our work addresses this gap with a benchmark exclusively centered around the lexicon, allowing a systematic and granular exploration of llms' ability to recognize the whole range of lexical functions formally defined by meaning-text linguists. it is akin to the recent work of @xcite , who have also leveraged instances of lfs from ln-fr to diagnose lexical competence, but supplements theirs in several respects. @xcite designed a challenging analogy-based benchmark of 2,600 fine-grained lexical analogies using 25 common lfs (21 paradigmatic and 4 syntagmatic), and showed that moderately-sized llms achieve particularly strong performance on derivational morphology but struggle more with syntagmatic relations and distinguishing event-participant roles. in contrast, we organize relations in a system of hierarchical clusters, grouping specific relations into broader categories, and examine models' ability to make distinctions with variable levels of specificity. rather than directly requesting models to solve a given analogical equation (an open question), we use closed yes/no questions with more elaborate contexts. while this arguably makes the task less challenging, it also circumvents important shortcomings of bare analogical equations regarding the amount of information provided to llms, and makes it easier to avoid false negatives in the evaluation. in particular, it enables us to include information pertaining to word sense clarification and/or semantic roles indices in the prompts, and thereby study a wider and more balanced range of lexical relations. ## evaluation framework this section outlines the evaluation framework designed for our study, including our proposal of a hierarchical organization of lfs, the lexical dataset, and the construction of contrastive prompts. ## conclusion in this study, we introduce a structured benchmark for evaluating llms' lexical competence, grounded in a semantic-syntactic hierarchical classification of lfs. using contrastive prompts, we find that models can leverage lexical cues but struggle with deeper distinctions. they perform better on surface-level pos contrasts, while finer-grained or syntactically nuanced lfs pose greater challenges. moreover, model responses are partly driven by morphological similarity between word pairs, especially when such cues are amplified by the prompt design.