Evaluating Pretrained Causal Language Models for Synonymy (2025)

Abstract: The scaling of causal language models in size and training data enabled them to tackle increasingly complex tasks. Despite the development of sophisticated tests to reveal their new capabilities, the underlying basis of these complex skills remains unclear. We argue that complex skills might be explained using simpler ones, represented by linguistic concepts. As an initial step in exploring this hypothesis, we focus on the lexical-semantic concept of synonymy, laying the groundwork for research into its relationship with more complex skills. We develop a comprehensive test suite to assess various aspects of synonymy under different conditions, and evaluate causal open-source models ranging up to 10 billion parameters. We find that these models effectively recognize synonymy but struggle to generate synonyms when prompted with relevant context.

## introduction as language models (lms) have grown in size and have been trained on larger datasets, they have gained the ability to handle tasks that were previously out of reach. this advancement has led to the development of increasingly complex evaluation tasks to explore these new capabilities. while complex datasets can demonstrate lm overall progress @xcite , they assess multiple implicit skills simultaneously, often without offering a clear picture of what precise skills contribute to performance @xcite . for instance, a complex skill such as creative writing involves a combination of subjacent skills, including temporal coherence to ensure logical progression, commonsense reasoning to create plausible scenes, and the ability to use synonyms to avoid repetition and enrich the vocabulary. the interest in the connection between complex skills and their subjacent ones is twofold. first, @xcite propose that training first on subjacent skills such as synonymy may be more effective than training on complex skills like creative writing directly. second, if performance on subjacent tasks correlates with performance on complex ones, this could help explain why language models sometimes appear to acquire complex abilities suddenly, suggesting that the necessary building blocks were already in place. these subjacent skills might be defined in different ways, such as in terms of performanceimproving datasets @xcite or logical reasoning and rhetoric @xcite . another way to define them could be via linguistic concepts. although this approach might seem trivial, the majority of benchmarks do not test linguistic concepts, and the training of lms does not particularly take them into account. given that the skills of lms are not yet well defined, we propose linguistic concepts and their properties as a starting point to structure the exploration of lm skills. while a significant amount of work focused on syntax, either through benchmarks such as blimp @xcite or studies on specific phenomena @xcite @xcite , there has been comparatively less emphasis on evaluating lexical-semantic capabilities, particularly for causal lms. as a result, this area remains relatively underexplored. our study aims to examine the lexical-semantic concept of synonymy, by focusing on its different properties, including both implicit aspects, such as substituting a word with its synonym, and explicit aspects, such as expressing the relation between two words. in line with previous benchmarks on linguistic knowledge such as blimp @xcite and syntaxgym @xcite , we create minimal pairs that we evaluate using sentence perplexity. the advantage of this method is that it does not require a prompt in the form of an instruction and can be performed on small, pre-trained only models. however, while this method does indicate whether a model tends to choose the correct synonym over an alternative, it does not determine whether a synonym will actually be generated when the context constrains the model to do so. therefore, we develop a second set of evaluations, based on gap filling, to complete the study. our main contributions are: previous work focuses on masked language models like bert @xcite and shows that, while they capture some lexical meaning by successfully predicting standard lexical relations @xcite , they struggle with less typical constructions @xcite . focusing on synonyms, @xcite finds that bert's embeddings correctly identify a synonym as closer in representation to the target word than an homonym 75% of the time but fail to distinguish synonyms from words with different meanings in the same context. extending these findings, @xcite show that pre-trained-only masked lms seem to perform rather poorly in the synonym/antonym recognition task (sar), with performance approaching random guess in the case of bert. on the other hand, in synonym masked word retrieval (mwr) the results seem strong, with an whr (weighted hit rate) of only 1.4% (lower is better). these results show that the way the test is constructed can significantly affect performance. in line with these findings, our study aims to develop multiple tests that capture various aspects of the same concept for a comprehensive evaluation. while most lexical-semantics research focuses on masked lms, the recent prominence of causal lms led to a growing number of studies dedicated to them. @xcite probe causal models for their knowledge of antonyms through both cloze-style tasks and prompting. they conclude that in most cases the models do not effectively capture this relationship, underlining that the whr 1 code and datasets used in this study can be found at https://github.com/ioana-ivan/bm-semlex . metric, which seemingly indicates strong model performance in cloze tasks, may not be a reliable indicator. since it counts hits from a predetermined list of completions, it does not account for many possible irrelevant completions. to avoid this issue in our gap-filling experiments, we used a list of correct completions instead of incorrect ones. ## experimental settings this section outlines the experimental settings, including minimal pairs and gap-filling tests, subjacent data, and models. ## results figure 1 summarizes our main results. the first two graphs present the results for tests that include context (substitution, reference, and relation with context): on the left the results under the controlled condition, against the incorrect synonym (context, controlled), while on the right those under the random condition (context, random). the third graph displays the results of the only test without context, the relation test. this test assesses whether the language model correctly identifies both correct and incorrect synonyms as synonyms over random words (no context, random). finally, the fourth graph compares the results of the relation test with context with those of the same test without (ctx/no ctx, random). see appendix d for results in table form and appendix e for details on significance paired two-tailed t-tests. ## limitations resources. semcor is an outdated dataset that contains inappropriate language (from the authors' observations). although we attempted to mitigate this issue through manual selection, using newer datasets with similar annotations may be more advisable.