Psycholinguistic Word Features: a New Approach for the Evaluation of LLM s Alignment with Humans (2025)

Abstract: The evaluation of LLMs has so far focused primarily on how well they can perform different tasks such as reasoning, question-answering, paraphrasing, or translating. For most of these tasks, performance can be measured with objective metrics, such as the number of correct answers. However, other language features are not easily quantified. For example, arousal, concreteness, or gender associated with a given word, as well as the extent to which we experience words with senses and relate them to a specific sense. Those features have been studied for many years by psycholinguistics, conducting large-scale experiments with humans to produce ratings for thousands of words. This opens an opportunity to evaluate how well LLMs align with human ratings on these word features, taking advantage of existing studies that cover many different language features in a large number of words. In this paper, we evaluate the alignment of a representative group of LLMs with human ratings on two psycholinguistic datasets: the Glasgow and Lancaster norms. These datasets cover thirteen features over thousands of words. The results show that alignment is significantly better on the Glasgow norms evaluated (arousal, valence, dominance, concreteness, imageability, familiarity, and gender) than on the Lancaster norms evaluated (introceptive, gustatory, olfactory, haptic, auditory, and visual). This suggests a limitation of current LLMs in aligning with human sensory associations for words, which may be due to their lack of embodied cognition present in humans and illustrates the usefulness of evaluating LLMs with psycholinguistic datasets.

## introduction the evaluation of large language models (llms) poses significant challenges as they have to be evaluated on their performance on a large number of tasks and their answers are in natural lan-guage @xcite . one alternative is to have humans evaluate the llm responses. this, however, does not scale when an extensive evaluation with tens of thousands of questions has to be done for each model and new models appear every day. initiatives like the chatbot arena @xcite resort to the community to perform an evaluation of human preferences. in this case, the questions, answers, and participants are not controlled, so the results provide a comparative ranking of models but not a detailed analysis of their specific capabilities. another alternative is to use an llm to evaluate other llms @xcite . again, this method has limitations as the judging llm may have biases and inaccuracies, and someone has to evaluate this llm in the first place. the most widely used method to evaluate llms as of today is to run different benchmarks, mostly made of multiple-choice questions or tasks for which existing metrics can be used to provide a result. this enables the automation of the process and the evaluation of specific tasks, for example, maths @xcite , reasoning @xcite , or knowledge of many different topics @xcite . the results of those tests are then published on leaderboards @xcite and used to compare the performance of llms on a wide range of tasks. evaluating llms' ability to solve a math problem, a riddle, or answer a question about taxation is interesting but is not enough. llms interact with persons and generate text that is read by humans. therefore, we would like them to be aligned with human emotions, perceptions and preferences @xcite . to assess alignment, benchmarks for emotional alignment are also being developed, for example, by asking open questions to the llm and evaluating their responses using a second llm as a judge @xcite . this, as discussed before, relies on the judge llm and thus is limited by its capabilities. another option is to have humans rate the questions on a likert scale and then ask the llms to also answer on a likert scale @xcite . this requires new human studies which imply a significant effort @xcite . interestingly, human ratings have been used in psycholinguistics for decades and large datasets are available, for example, with ratings of words and expressions @xcite . although llms are entirely based on written language, they capture much of the meaning of words. for example, llm-based estimates of the valence and concreteness of words and expressions correlate very well with human ratings @xcite . at the same time, it is hard to deny that for humans word meaning is more than the occurrence of words together, which is from what llms learn. two aspects come into play here. the first is the symbol grounding problem @xcite . you cannot learn a language on the basis of words alone. some words must first be grounded in the world around us (at least 1% according to @xcite , or about 400 words). only then can they be used to accurately define the meaning of other words. the second aspect is that even though words can be defined from other words, in reality we have probably learned their full meaning through a mix of language and everyday experiences. the latter includes perception (our knowledge of the color purple is more than knowing it is a combination of red and blue), actions (our knowledge of a chair is based in part on having sat on chairs many times), emotions, social interactions, and so on. finally, theories like embodied cognition argue that the interactions of our body with the environment also shape our minds and are an essential part of our language learning process and influence word meaning @xcite , @xcite . therefore, it is interesting to study whether these fundamental differences between humans and llms limit their alignment and in which areas. in psycholinguistics, ratings of words and expressions are used to select stimuli for experiments that evaluate different aspects of language processing and learning, supporting the development and validation of theories of human cognition @xcite . features such as arousal, valence, concreteness, dominance and iconicity have been evaluated on thousands of words and expressions in many different languages @xcite . there are also studies with human ratings on different emotions such as happiness, disgust, anger, fear, or sadness @xcite which are useful in affective neurolinguistics studies (j. a. hinojosa and ferré, 2020). ratings of how humans associate words with the senses or parts of the body are also available for thousands of words @xcite and have been used to enrich language models @xcite . since all these datasets are available and have been used and validated in many studies, it is of interest to explore whether they can be used to evaluate llms. so, differently from existing studies @xcite that use llms to generate estimates of word features, use existing human ratings to evaluate llms. in this paper, we make the first contribution in this direction by presenting an initial study on the use of psycholinguistic datasets for llm evaluation and analyzing the results linking them to existing works in cognitive science. the rest of the paper is organized as follows. section 2 presents the motivation and objectives of the paper. section 3 presents the evaluation methodology including the selection of the datasets, the llms to evaluate and the procedures and metrics used. the results are presented in section 4 and discussed in section 5. the paper ends with the conclusion in section 6. ## motivation and objectives the main motivation of this work is to foster the evaluation of llms from a psycholinguistic perspective, reusing existing datasets and knowledge that have been gathered in human evaluations for decades. this would not only provide datasets for llm evaluation but also open new perspectives on how to evaluate llms and attract the psycholinguist community to llm evaluation research @xcite . for example, theories of language acquisition and processing that have been developed for humans can be used to better understand how llms process language. to achieve this main goal, in this paper we conduct an initial exploration to show the potential of putting together psycholinguistic word norms and llm evaluation with the following objectives: the following sections address each of these objectives in detail. ## methodology this section discusses the proposed methodology to evaluate the alignment of llm with humans using psycholinguistic word norms. the methodology includes the selection of psycholinguistic datasets, llm, and the metrics and procedures used in the evaluation. ## results all results and prompts used as well as the code to generate the plots are available in a public repository foot_0 . the results for the glasgow norms are presented first. as discussed in the previous section, in the following, only the estimate based on the llm estimated probabilities is used to present the results as, in general, it achieves better alignment with humans. ## discussion the results presented in the previous sections show that it is possible to use existing psycholinguistic norms to evaluate the alignment of llms with humans on different aspects of word meaning. the methodology proposed is in line with existing llm evaluation techniques and can be automated, allowing testing at scale. the results also show that the alignment of llms is currently limited to a few word norms and only for a few models. therefore, there is ample room for improvement that future llms should address. the alignment is in most cases worse for perceptual norms, in line with cognitive science results which show that perceptual information is not completely captured by text but also by embodied cognition @xcite . the use of psycholinguistic norms for evaluation has the additional advantage that it provides valuable insights on how to improve llms. now that we know that llms lack alignment on perceptual features and that this is probably linked to their lack of embodied cognition, we can start looking into how to train llms to acquire that knowledge. we can try generating synthetic text that covers that knowledge and using it in the post-training phase of the llms. we can also explore whether multimodal models have the same limitations, for example, for norms related to vision. we will then be able to use the benchmarks to assess the progress made in model alignment when those modifications are introduced. this would promote the participation of the psycholinguistic community in llm research. in fact, more broadly, psycholinguistics can contribute not only to the evaluation of llms but also to the understanding of their inner workings and explainability. psycholinguistics has studied how humans learn and process language for decades, developing theories and experiments to understand our mental processes. in this context, llms can be seen as another type of subject to study for which existing knowledge can be reused. for some models and features we obtained big differences between the pearson and the spearman correlations coefficients. to some extent, this is a nuisance as it is unclear which one to rely on. on the other hand, the difference is also informative. higher pearson coefficients indicate that observations outside the bulk of the distribution have the desired properties (i.e., the llm outliers agree with the human outliers). higher spearman correlations indicate that small differences around the mode of the distribution align between llms and humans. we recommend always computing both correlation coefficients to avoid drawing wrong conclusions (e.g., about quality differences between llms in leader boards). most of the time, rounding to the nearest integer did not make much difference. if it does, this indicates that much of the correlation is due to alignment between models and humans that are unlikely to have psychological significance because they are too small to be noticed by people (e.g., differences between likert values of 1.01 and 1.02). it is good to check for this possibility if a considerable difference is observed between the pearson and the spearman correlation. ## conclusion this paper proposes the use of psycholinguistic word norms for the evaluation of human and llm alignment. the initial results using thirteen word norms covering different aspects of word meaning indicate that current llms have limited alignment with humans, and more so for norms that are related to sensory experiences. this can be linked to the llms' lack of embodied cognition present in humans. the study and results show not only the potential of psycholinguistic word norms for evaluating llm alignment but also for analyzing the results through the lens of existing psycholinguistic theories. the methodology, metrics, datasets, and models used in our initial evaluation can be used and extended to define a comprehensive benchmark which can be included in leaderboards as part of the standard llm evaluation process. this will foster research to improve llms' alignment and the understanding of how models learn and process.