Exploring Semantic Spaces for Detecting Clustering and Switching in Verbal Fluency (2022)

Abstract: In this work, we explore the fitness of various word/concept representations in analyzing an experimental verbal fluency dataset providing human responses to 10 different category enumeration tasks. Based on human annotations of so-called clusters and switches between sub-categories in the verbal fluency sequences, we analyze whether lexical semantic knowledge represented in word embedding spaces (GloVe, fastText, ConceptNet, BERT) is suitable for detecting these conceptual clusters and switches within and across different categories. Our results indicate that ConceptNet embeddings, a distributional semantics method enriched with taxonomical relations, outperforms other semantic representations by a large margin. Moreover, category-specific analysis suggests that individual thresholds per category are more suited for the analysis of clustering and switching in particular embedding sub-space instead of a one-fits-all cross-category solution. The results point to interesting directions for future work on probing word embedding models on the verbal fluency task.

## introduction the intrinsic evaluation of lexical knowledge represented in word embeddings has been of longstanding interest in distributional semantics @xcite , and remains an important topic in work on interpreting large-scale blackbox language models @xcite @xcite . while pretrained contextualized word representations have recently been evaluated in many novel, (psycho-) linguistically motivated probing tasks @xcite @xcite , the assessment of lexical semantics in word embeddings still commonly focuses on traditional benchmarks of human similarity annotations @xcite , datasets of analogies @xcite or taxonomic relations such as hypernymy @xcite . in this paper, we conduct an evaluation of word embeddings on the so-called verbal fluency task @xcite , where participants are asked to enumerate as many different words for a given category as possible within a given time interval (often 60 seconds), see figure 1 for an example response to the category hobby. the resulting production data are a rich source of participants' lexical-conceptual knowledge and typically show an interesting clustering-switching pattern where consecutive words either relate to the same subcategory (handball, football,horseback riding as a cluster for "sports" in figure 1 ) or switch between sub-categories (playing the guitar, model railway as a switch from "music" to "playing" in figure 1 ). verbal fluency is a very well-known and widely used cognitive performance test used in psychology, neuro-and psycholinguistics where robust and automatic methods for analyzing clusteringswitching patterns would be highly welcome @xcite . yet, to date, the analysis of verbal fluency data received little attention in research on computational semantics and word embeddings @xcite @xcite . the study by @xcite constitutes a noticeable exception, but is restricted to verbal fluency responses to a single category ("animals") and does not rely on human annota- we base our study on a recently collected dataset of german semantic verbal fluency responses that provides a much wider range of categories than previous studies and, additionally, has been analyzed in terms of clustering-switching patterns by human judges. 1 importantly, our verbal fluencybased evaluation of lexical knowledge in embedding spaces rests on human production and judgement data such that it may display different aspects of taxonomical-conceptual knowledge as compared to existing benchmarks for lexical relation prediction derived from standard lexical resources @xcite . for instance, the sub-categories for the "hobby" category illustrated in figure 1 could not be easily retrieved from wordnet @xcite which, curiously, lists "speleology" as the only direct hyponym for the most common synset of "hobby". in the following, we compare different word embeddings for german (bert-base, glove, fasttext, conceptnet), investigating to what extent distances in sub-spaces for categories like animals, body parts, clothes reflect conceptual switches found in verbal fluency data and to what extent it is possible to derive generic, cross-category distance thresholds for the accurate detection of clusters and switches. our results indicate that conceptnet embeddings @xcite , a distributional semantics method enriched with taxonomical relations, outperforms other semantic representations by a large margin. on the other hand, merely taxonomical relations (as represented in germanet @xcite ) have significant but weak correlations indicating that they are useful as accompanying modalities to word embeddings. moreover, category-specific analysis suggests that individual thresholds per category are more suited for the analysis of clustering and switching in particular embedding sub-spaces instead of a one-fitsall cross-category solution. a final experiment using simple clustering algorithms further corroborates the findings of the switching analyses. overall, the results point to interesting directions for future work on probing word embedding models on the verbal fluency task. ## background the verbal fluency task is a very well-known neuropsychological test that is used in clinical contexts for diagnosing, e.g., neurodegenerative diseases as well as in research on the cognitive processes underlying lexical knowledge, access, retrieval and executive control @xcite . participants' verbal fluency responses are typically scored in terms of the number of correct words produced for the category, whereas more fine-grained analyses measure the amount of switching and clustering in the word sequence @xcite . the semantic analysis of verbal fluency is commonly addressed by the manually defined subcategories for "animals" established in @xcite 's study, and extending these to other categories and languages is a notorious challenge in psychology @xcite . while there has been a lot of interest in psychology in using word embeddings for scoring semantic fluency @xcite @xcite @xcite , the task entered the radars of the nlp community only recently. one potential reason is that switching and clustering literature are restricted to an extremely limited number of categories (such as animals, groceries), although there are standardized tools that slightly extend the list of the categories -e.g. the popular rwt (regensburger wortflüssigkeitstest) @xcite includes five categories (animals, hobbies, occupations, groceries and first names). quantitative analyses of verbal fluency data have shown that enumeration speed and diversity within a category are very category-dependent and that categories can be more or less easy to enumerate. for automatic methods that measure clustering and switching, a first key step is to define appropriate thresholds that pinpoint switch boundaries for subcategory changes. @xcite investigate different ways of automatically scoring semantic fluency in english and korean. using a traditional word2vec model @xcite , they predict categorical switches in collected sequences if the predicted similarity between adjacent items drops below a defined threshold, similar to the approach in @xcite . complementary to this, the authors propose a model which aligns words from fluency sequences with wikipedia articles and predicts categorical switches when the intersection of articles linked to adjacent words drops below a certain threshold. despite well-known dif-ferences between verbal fluency categories, the robustness and quality of these threshold-based methods across categories has, to the best of our knowledge, not yet been analyzed. moreover, from an nlp perspective, traditional word2vec embeddings can be expected to achieve a lower performance in analyzing fine-grained conceptual relations as compared to various more recent embedding methods that capture global word distributions as in glove @xcite , subword representations as in fasttext @xcite or integrated taxonomical knowledge as conceptnet numberbatch @xcite . finally, contextualized embeddings from transformer language models such as bert @xcite constitute to be an obvious method to explore. however, to be used in such enumeration task, this dynamic embedding method needs to be transformed into static embeddings as detailed in section 3.3. in brief, the main contribution of this paper is to explore (i) state-of-the-art word embeddings for data collected in an ongoing verbal fluency study in wide categorical variety and (ii) the options for automatic scoring mechanisms for this broad range of categories. to our knowledge, this is the first (nlp-powered) study that systematically analyzes verbal fluency task across such categorical variety through various semantic representations and evaluation metrics together. ## clustering analysis the pairwise switch analysis revealed considerable differences between categories as well as favourable results for conceptnet as a semantic representation. in the following, we report results from an additional clustering-based analysis, in which we take a more global perspective: instead of investigating adjacent items in fluency sequences, we look at the global semantic organization of all lexical items in the respective categories. for each category c, we encode the assigned subcategories for individual items into sparse binary vectors. this transformation is necessary for k-means clustering. this gives us a feature vector with dimensions (n wordsc , n subcatsc ) for human annotations, where n wordsc is the number of lexical items and n subcatsc the number of subcategories in the respective category. as simplified, lets assume that we have only three subcategories. a word is represented by the vector [1,0,0] if the word has been assigned to subcategory 1 but not 2 and 3. we then retrieve the same lexical items from the semantic representations described in section 3.3, and use each of the feature vectors to fit a k-means clustering model. the k parameter is set depending on the number of annotated subcategories, determined as k @xmath0 for evaluation, we rely on the adjusted rand index (ari) @xcite for comparing the clusterings based on con-ceptnet, glove, fasttext and bert with the results for annotated subcategories. we use the scikitlearn foot_6 library for both clustering and evaluation. the scores reported in ## conclusion in this paper, we have explored a range of semantic spaces and switch detection methods for the analysis of the verbal fluency data. to the best of our knowledge, this is the first study that (i) incorporates the taxonomic relations using nlp techniques, (ii) explores a wide variety of semantic categories (10 categories), and (iii) explores the fitness of semantic representations in german for this task. nlp solutions so far are limited to typical/frequent categories like fruits and animals, leaving the annotation of other categories to laborious manual methods. to develop an automatic scoring mechanism, in-depth analysis for less frequent categories is necessary. our results revealed various category-specific characteristics. we showed that choosing individual threshold strategies to detect switch boundaries is essential and a "one-fits-all" solution (using a global threshold) results in less aligned sequences. still, it can be kept as an option since the degradation is not large for the easy to enumerate categories. in addition to providing an another perspective to analyze the verbal fluency data for psycholinguistic research, this study also prepares the ground for investigating interesting nlp tasks, like subcategory prediction/generation of the upcoming items. from that perspective, @xcite claims that transformer-based language models perform better on cognitive modeling (more specifically, on predicting the next items given a sequence) than the static approaches. however, their evaluation does not include a comparison to knowledge-enriched models. although their task differs from exploring semantic spaces to detect category switches, category-specific variations are observed from their results regarding 5 categories (fruits, vegetables, animals, supermarket items, tools, and foods). furthermore, applying mere taxonomic relations using a synset taxonomy falls behind embedding methods but proved to be instrumental as an accompanying information source, especially for the hard to enumerate categories. these results highlight that the task is more challenging than it seems, and we need to go beyond out-of-the-box nlp approaches by understanding the nature of these categories and the task. future studies aim to improve the integrated switch detection method around the taxonomy-enriched representations using additional modalities and knowledge-graph enriched bert models.