FAMWA : A new taxonomy for classifying word associations (which humans improve at but LLM s still struggle with) (2025)

Abstract: Word associations have a longstanding tradition of being instrumental for investigating the organization of the mental lexicon. Despite their wide application in psychology and psycholinguistics, analyzing word associations remains challenging due to their inherent heterogeneity and variability, shaped by linguistic and extralinguistic factors. Existing word-association taxonomies often suffer limitations due to a lack of comprehensive frameworks that capture their complexity.To address these limitations, we introduce a linguistically motivated taxonomy consisting of co-existing meaning-related and form-related relations, while accounting for the directionality of word associations.We applied the taxonomy to a dataset of 1,300 word associations (FAMWA) and assessed it using various LLMs, analyzing their ability to classify word associations.The results show an improved inter-annotator agreement for our taxonomies compared to previous studies ( \kappa = .60 for meaning and \kappa = .58 for form). However, models such as GPT-4o perform only modestly in relation labeling (with accuracies of 46.2% for meaning and 78.3% for form), which calls into question their ability to fully grasp the underlying principles of human word associations.

## introduction the word association task is a classic psychological experiment in which participants respond spontaneously with the first word(s) that come to mind (e.g., cat, bark, bone) when presented with a specific cue word (e.g., dog). for more than a century, word associations have been used by psychologists and psychiatrists to investigate cognitive processes, psychological behavior patterns, mental disorders, language acquisition, multilingualism, and the overall structure of the mental lexicon (see @xcite @xcite @xcite @xcite . experiments conducted across multiple languages have shown that word associations are characterized by both heterogeneity and variability. on the one hand, responses may be influenced by a wide range of relationships between cues and responses, depending on formal, semantic, and syntactic properties, as well as extralinguistic knowledge and cultural factors. on the other hand, there is considerable variation in responses across individuals, with greatly varying degrees of convergence depending on the cue word. this diversity poses challenges for linguistic analysis, and various taxonomies of wordassociation relations have been proposed to accurately account for word associations. although common elements of classification emerge from previous studies, there is no universally accepted framework for describing wordassociation relations, as existing taxonomies present various limitations. the categories used to analyze associations are not always explicitly defined, and some taxonomies focus only on a single aspect of word-association relations (e.g., semantic characteristics), while others merge semantic and formal relations into flat hierarchies, potentially leading to conflicting or incomplete descriptions. in addition, directionality is rarely considered, although the cue-response sequencing and the non-reciprocal nature of word associations call for a specific account of asymmetrical relations. finally, taxonomies are seldom evaluated through inter-annotator agreement or computational models, limiting their validation and reliability. in this study, we propose a linguistically motivated taxonomy of word-association relations based on a critical examination of previous classifications. the taxonomy includes two co-existing levels of linguistic analysis related to form and meaning, and takes into account the directionality of word associations. we apply this taxonomy on a dataset of 1,300 word associations in english using a well-defined annotation protocol, while assessing annotation quality through inter-annotator agreement. furthermore, we evaluate the ability of generative language models to classify word associations according to our taxonomy, exploring how well they capture the diversity of relations in word associations, and providing a detailed analysis of model performance. overall, the main contributions of the study include (i) a theoretical and methodological reflection on the analysis of wordassociation relations, (ii) the creation of a finely annotated dataset of word associations covering formal and semantic relations, and (iii) a discussion of how language models handle the heterogeneity of lexical relations within word associations. ## a taxonomy of cue-response relations this section presents the taxonomy we used to classify word-association relations, along with a labeled dataset of cue-response pairs. crucially, we employed a dual-level classification, where the relation between a cue and a response is annotated for both meaning and form. we also took the directionality of the relations into account to reflect the asymmetry of the associations. ## ability of language models to classify associations we now focus on examining the extent to which language-model-based systems are able to classify word associations, using human-designed linguistic classifications. previous works have provided abundant evidence that llms have linguistic knowledge, and in particular lexical knowledge (see, e.g., kello and bruna 2024 and hayashi 2025, who showed llms' ability to accurately detail lexical properties of words and to distinguish word senses in context). de deyne et al. ( 2024 ) reported that gpt-4 performed poorly in classifying word associations with the wax dataset and relation inventory, achieving an accuracy of only 47%. we hypothesize that a dataset with better iaa can lead to improved model performance. to test this hypothesis, we prompted various llms to label cue-response pairs and evaluated their performance on the famwa dataset. we also compared performance using the wax dataset and taxonomy, which highlights the impact of category distribution. ## conclusion our efforts to develop a linguistically motivated taxonomy of word-association relations proved effective, as we achieved higher inter-annotator agreement than comparable studies using different analytical frameworks. integrating a dual-level analysis of formal and semantic relations, while also accounting for directionality in associations, is not only more satisfactory from the perspective of linguistic description, but also ensures greater stability and consistency in annotation quality, at least with expert annotators. nevertheless, the observed agreement remains moderate, highlighting the inherent challenge of producing a metalinguistic analysis of word associations. this difficulty is frequently noted by researchers who attempt to classify word associations, and it contrasts with the naturalness of the word association task itself, which is effortless as it relies only on the existence of the mental lexicon. arguably, the basic task of generating word associations and the metalinguistic task of analyzing them involve fundamentally distinct cognitive processes and engage contrasting aspects of the language faculty. while expert human annotators achieve only moderate agreement, even advanced models like gpt-4o exhibit mediocre performance in analyzing semantic relations between cues and responses, despite their well-documented linguistic capacities. this is congruent with the previous conclusion that the word association task and its analysis leverage different types of capabilities. moreover, llms' ability to classify word associations is not improved by refinements in descriptive frameworks and varies considerably across relation classes. the limited performance of llms in labeling word associations should be analyzed in light of their ability to produce them. the inherent heterogeneity and variability of word associations pose challenges not only for their metalinguistic analysis, but also for their generation by language models. future research should explore this generative capacity in greater depth, for example through detailed analysis of the llm-generated association norms reported in lwow @xcite . short labels prompt labels definitions examples compo c+r compound cue response the sequence c r forms a compound hermit-crab compo r+c compound response cue the sequence r c forms a compound rights-human in mwe multiword expression r and c belong to the same multiword expression, containing other elements pedal-metal morpho morphological relation r and c belong to the same derivational or inflectional paradigm gave-gift similar similar c and r are similar in graphical or phonological form but not morphologically related hat-cat none no relation no formal relationship roof-house