Which Model Mimics Human Mental Lexicon Better? A Comparative Study of Word Embedding and Generative Models (2025)

Abstract: Word associations are commonly applied in psycholinguistics to investigate the nature and structure of the human mental lexicon, and at the same time an important data source for measuring the alignment of language models with human semantic representations.Taking this view, we compare the capacities of different language models to model collective human association norms via five word association tasks (WATs), with predictions about associations driven by either word vector similarities for traditional embedding models or prompting large language models (LLMs).Our results demonstrate that neither approach could produce human-like performances in all five WATs. Hence, none of them can successfully model the human mental lexicon yet. Our detailed analysis shows that static word-type embeddings and prompted LLMs have overall better alignment with human norms compared to word-token embeddings from pretrained models like BERT. Further analysis suggests that the performance discrepancies may be due to different model architectures, especially in terms of approximating human-like associative reasoning through either semantic similarity or relatedness evaluation. Our codes and data are publicly available at: https://github.com/florethsong/word_association.

## introduction artificial intelligence, particularly large language models (llms), functionally emulates the way we humans perceive and conceptualize the physical reality, as well as how we understand and process multifaceted information @xcite . yet a pivotal open question remains unsolved: to what extent do llms align with the conceptual knowledge hierarchically encoded in human cognition as their capabilities advance? this is where the "machine psychology" comes into play to scrutinize llms' "behavioral traits" and "thinking patterns" through psychological tests adapted from interpretable research on human @xcite . successful modeling of the human mental lexicon can be viewed as an essential step in verifying human-like intelligence. human mental lexicon, in contrast to electronic lexica, is extremely versatile in supporting the association and generation of new concepts. indeed word association norms is a typical method of investigation: a stimulus word is presented to a human participant, who is simply required to produce the first word coming to mind @xcite . semantic similarities and relatedness that underlie the core of human mental lexicon is hereby quantified as collective linguistic norms. since distributional similarity between words is an important factor explaining associations, traditional studies extensively adopted distributional semantic models (dsms) and word embeddings to predict human word associations @xcite @xcite . on the other hand more recent studies, based on llms, proved that such systems can align, to a considerable extent, with human patterns of asso-ciating words @xcite @xcite ). an open question arisen is which one of these methods delivers better results in approximating human norms. besides the theoretical interest of the problem, the results are relevant to the problem of reverse dictionary, where a user tries to retrieve a word given a set of associates or a dictionary definition @xcite . reverse dictionary applications, which can be seen as the information retrieval modeling side of human lexical access (the so-called tip-of-the-tongue, anomia or dysnomia problem, see @xcite and @xcite ) can be helpful tools for writers and translators, and in this sense, generative llms show a lot of promise, as they could help a user by retrieving and generating a target word simply on the basis of a prompt with some lexical cues. from a psychological standpoint, word associations are also a fundamental indicator for human creativity and divergent thinking, as research indicates a consistent positive correlation between high levels of human creativity and the capacity to generate word associates that are distant in the lexical network @xcite @xcite . as illustrated by the task types in figure 1 focusing on semantic similarity and relatedness, this study designs a protocol of five-stage word association tasks (wats) to evaluate models against human norms. by taking the majority of human responses across various wats as a main proxy of human mental lexicon, this study compare the word association abilities of vectors from traditional static word-type embedding models (wems), mean-pooled word-token embeddings from representative pretrained language models (plms), and prompting strategies with mainstream llms. results show that although none of these models align fully with human mental lexicon and hence model effectively the versatility of the human cognitive ability, wems and llms can better mimic human associations than plms: llms outperform competitors in word retrieval tasks (with a focus on capturing semantic similarities, i.e., lexical interchangeability), while wems perform better in concept pairing (emphasizing the identification of semantic relatedness, that is, detecting mutual conceptual relations). while scaling-up and contextualization often helps embedding models, plms show more architecture-and task-dependent trade-offs. various human association norms originally designed to access preexisting word knowledge in the human mind and detect different aspects of cognitive development and competencies, such as eat (the edinburgh associative thesaurus, @xcite , usf (the university of south florida free association norms, @xcite , and swow (the small world of words, de deyne et al., 2019), can be applied in conjunction as a comprehensive benchmark for facilitating the measurement of the alignment between human internal semantic cognition and external word embeddings. wats with word embeddings wats have significantly contributed to benchmarking models' semantic representations and conceptual structures against human mental lexicon shown in diverse human-generated norms, both in theory and practice @xcite . they provide a powerful means to probe into two fundamental dimensions of distributional semantics: similarity (interchangeability of words, e.g., car/van) and relatedness (shared conceptual relations between words, e.g., car/wheel) @xcite . existing work @xcite @xcite has been extensively devoted to thorough comparisons across a wide spectrum of dsms from count (e.g., dissect ppmi, @xcite and predict models (e.g., word2vec, @xcite at early staticembedding generation to recent transformer-based contextual embedding models (e.g., @xcite . these studies consistently demonstrated the superior performance of static embeddings in out-of-context wats, while highlighting contextual embeddings' advantages in tasks requiring contextual sensitivity. collectively, they revealed the nuanced interplay between model design, task requirements, and cognitive plausibility of language representations. wats with llms recent work expanded the use of wats into dissecting the behaviors of llms as black-box systems to better understand their advantages and limitations in semantic-aware reasoning. @xcite established llmgenerated free association norms by prompting popular llms and found that llm-generated associations exhibit weaker concreteness effects and stronger societal biases compared to human norms. @xcite demonstrated gpt-series' ability to synchronize with human players in game-like free association interactions. beyond free association tasks, structured variants such as ontological classification @xcite , connection tasks @xcite , and similarity judgments on triads @xcite have assessed llms' ability to identify underlying internal relations or cluster words by shared characteristics. increasing interest has been in using wats to reveal both explicit and implicit societal biases encoded in llms. for example, studies by @xcite , @xcite , and @xcite presented how wats can uncover attitude disparities between model outputs and human responses, highlighting their utility in addressing ethical issues of language models. such studies stress wats' dual role in illuminating human and models' semantic networks; however, existing work mainly relied either on promptbased strategies with llms or on embedding similarity, without any systematic comparison between the two. also, previous studies were limited in scope, focusing only on one type of wat, therefore a more comprehensive evaluation is necessary. ## experimental settings according to @xcite , probing into the conceptual knowledge encoded within language models by examining the embedding space works well for traditional models, but it is less effective and practical for llms. this is due to the fact that embeddings from llms exhibit severe anisotropy in their vector spaces, which can significantly dis-tort similarity estimates (e.g., @xcite @xcite @xcite . therefore, a shift from the conventional approach of accessing the embedding space to a top-down approach in the context of llms was proposed, which means directly prompting llms with specific tasks and using their outputs to infer the knowledge in their vector spaces. therefore, we examine the capabilities of different models by employing two methodologies: embedding and prompting, which align with their default typical approaches to wats at hand. a basic assumption of embedding-based tests is that the strength of word associations increases with the cosine similarity of their embeddings @xcite , reflecting graded semantic relationships in vector spaces. for wems, we extracted static word-type embeddings and calculated the cosine similarities as the basis for their outputs. in terms of plms, both non-contextualized and contextualized word embeddings were meanpooled from the last hidden layers and cosine similarities were computed. regarding llms, we utilized zero-shot prompts to obtain direct responses. ## results and analysis this section reports the empirical results and findings obtained from operationalizing the series of tasks and metrics defined in section 3.1. the statistics corresponding to each task and significance test results are displayed in appendices b-f. ## conclusion this study systematically evaluates the intrinsic semantic capabilities of diverse language models, including wems, plms, and llms, by leveraging their typical operational modes (e.g., word embeddings vs. prompt-based generation). through the adaptation and integration of five kinds of classical psycholinguistic wats, we assess how well these models perform on cognitively motivated benchmarks. the results reveal distinct performance and limitations across architectures and configurations. first, wems and llms demonstrate better alignment with human association norms compared to plms, particularly in tasks requiring stable type-level semantic representations. notably, llms outperform the other models in word retrieval (tasks 1-2, similarity-dominant; task 3, considering both similarity and relatedness), while wems do better in concept pairing (tasks 4-5, relatedness-dominant), highlighting their complementary strengths across model architectures and the fact that human mental lexicon is good at synergizing similarity and relatedness, but not artificial systems. for wmes, increasing model size generally improves performance. however, plms exhibit architecture-dependent behaviors in terms of scaling and contextualization: encoder-only models like bert often degrade with larger scales and added contexts but decoder-only models (e.g., gpt-2) tend to benefit from both. for encoder-decoder models (e.g., t5), the impacts are task-specific. their performance notably improves in tasks 1 and 3 in these two settings but declines in task 2. llms' partial success in some wats by mimicking human semantic behaviors demystifies the claim of their human-like intelligence. yet they still struggle to fully replicate the versatility of the human mental lexicon, particularly in associating remote or abstract concepts. this suggests a tension between accuracy and creativity in language modeling, warranting deeper exploration. together, these findings provide comprehensive insights into the alignment between language models and human cognition and highlight the value of psycholinguistic data for diagnosing model capabilities and biases.