Conceptual structure coheres in human cognition but not in large language models (2023)

Abstract: Neural network models of language have long been used as a tool for developing hypotheses about conceptual representation in the mind and brain. For many years, such use involved extracting vector-space representations of words and using distances among these to predict or understand human behavior in various semantic tasks. In contemporary language models, however, it is possible to interrogate the latent structure of conceptual representations using methods nearly identical to those commonly used with human participants. The current work uses three common techniques borrowed from cognitive psychology to estimate and compare lexical-semantic structure in both humans and a well-known large language model, the DaVinci variant of GPT-3. In humans, we show that conceptual structure is robust to differences in culture, language, and method of estimation. Structures estimated from the LLM behavior, while individually fairly consistent with those estimated from human behavior, depend much more upon the particular task used to generate behavior responses–responses generated by the very same model in the three tasks yield estimates of conceptual structure that cohere less with one another than do human structure estimates. The results suggest one important way that knowledge inhering in contemporary LLMs can differ from human cognition.

## introduction since elman's pioneering work @xcite showcasing the ability of neural networks to capture many aspects of human language processing @xcite , such models have provided a useful tool, and sometimes a gadfly, for developing hypotheses about the cognitive and neural mechanisms that support language. when trained on a task that seems almost absurdly simplistic-continuous, sequential prediction of upcoming words in sentences-early models exhibited properties that upended received wisdom about what language is and how it works. they acquired internal representations that blended syntactic and semantic information, rather than keeping these separate as classic psycho-linguistics required. they handled grammatical dependencies, not by constructing syntactic structure trees, but by learning and exploiting temporal patterns in language. perhaps most surprisingly, they illustrated that statistical structure latent in natural language could go a long way toward explaining how we acquire knowledge of semantic similarity relations among words. because words with similar meanings tend to be encountered in similar linguistic contexts @xcite @xcite , models that exploit contextual similarity when representing words come to express semantic relations between them. this enterprise of learning by predicting has persisted into the modern era of transformer-based language models @xcite . though early work was limited in the nature and complexity of the language corpora used to train models @xcite , these ideas spurred a variety of computational approaches that could be applied to large corpora of written text. approaches such as latent semantic analysis @xcite and skipgram models @xcite , for instance, learn vector-space representations of words from overlap in their linguistic contexts, which turn out to capture a variety of semantic relationships amongst words, including some that are highly abstract @xcite @xcite . in all of this work, lexical-semantic representations are cast as static points in a highdimensional vector space, either computed directly from estimates of word co-occurrence in large text corpora @xcite , or instantiated as the learned activation patterns arising in a neural network model trained on such corpora. to evaluate whether a given approach expresses semantic structure similar to that discerned by human participants, the experimenter typically compares the similarities between word vectors learned by a model to decisions or behaviors exhibited by participants in semantic tasks. for instance, lsa models were tested on synonym-judgment tasks drawn from a common standardized test of english language comprehension by comparing the cosine distance between the vectors corresponding to a target word and each of several option words, and having the model "choose" the option with the smallest distance @xcite . the model was deemed successful because the choice computed in this way often aligned with the choices of native english speakers. such a procedure was not just a useful way for assessing whether model representations are human-like-it was just about the only way to do so for this class of models. in the era of large language models such as open ai's gpt3 @xcite ), meta's llama family @xcite , google's flan @xcite , and many others @xcite @xcite , this has changed. such models are many orders of magnitude larger than classical connectionist approaches, employ a range of architectural and training innovations, and are optimized on truly vast quantities of data-but nevertheless they operate on principles not dissimilar to those that elman and others pioneered. that is, they exploit patterns of word co-occurrence in natural language to learn distributed, context-sensitive representations of linguistic meaning at multiple levels, and from these representations they generate probabilistic predictions about likely upcoming words. current models generate plausible and grammatically well-formed responses, created by iteratively predicting what words are likely to come next and sampling from this distribution using various strategies @xcite @xcite . so plausible is the text that recent iterations like chatgpt @xcite can write essays sufficiently well to earn a a in an undergraduate setting @xcite , pass many text-based licensing exams in law and medicine @xcite @xcite produce working python code from a general description of the function (openai, year of the webpage, e.g., 2023), generate coherent explanations for a variety of phenomena, and answer factual questions with remarkable accuracy foot_0 . even in the realm of failures, work has shown that the kinds of reasoning problems llms struggle with are often the same ones that humans tend to find difficult @xcite . in short, if solely evaluated based on their generated text, such models appear to show several hallmarks of conceptual abilities that until recently were uniquely human. these innovations allow cognitive scientists, for the first time, to measure and evaluate conceptual structure in a non-human system using precisely the same natural-language-based methods that we use to study human participants. large language models can receive written instructions followed by a series of stimuli and generate interpretable, natural-language responses for each. the responses generated can be recorded and analyzed in precisely the same manner as responses generated by humans, and the results of such analyses can then be compared within and between humans and llms, as a means of understanding whether and how these intelligences differ. the current paper uses this approach to understand similarities and differences in the way that lexical semantic representations are structured in humans vs llms, focusing on one remarkable aspect of human concepts-specifically, their robustness. as rosch showed many years ago @xcite , the same conceptual relations underlie behavior in a variety of tasks, from naming and categorization to feature-listing to similarity judgments to sorting. similar conceptual relations can be observed across distinct languages and cultures @xcite . robustness is important because it allows for shared understanding and communication across cultures, over time, and through generations: homer still speaks to us despite the astonishing differences between his world and ours, because many of the concepts that organized his world cohere with those that organize ours. our goal was to assess whether conceptual structure in contemporary llms is also coherent when evaluated using methods comparable to those employed with human participants, or whether human and llm "mind" differ in this important regard. to answer this question, we first measured the robustness of conceptual structure in humans by comparing estimates of such structure for a controlled set of concepts using three distinct behavioral methods -feature-listing, pairwise similarity ratings, and triadic similarity judgements -across two distinct groups -dutch and north american -differing in culture and language. we then conducted the same behavioral experiments on llms, and evaluated (a) the degree to which estimated conceptual relations in the llm accord with those observed in humans, and (b) whether humans and llms differ in the apparent robustness of such structure. we further compared the structures estimated from the llm's overt patterns of behavior to those encoded in its internal representations, and also to semantic vectors extracted from two other common models in machine learning. in addition to simply demonstrating how methods from cognitive psychology can be used to better understand machine intelligence, the results point to an important difference between current state of the art llms and human conceptual representations. ## related work in addition to many of the studies highlighted in the previous section, here we note prior efforts to model human semantics using nlp models. many recent papers have evaluated ways in which llms are and are not humanlike in their patterns of behavior when performing tasks similar to those used in psychology-enough that, despite the relative youth of the technology, there has been a recent review summarising how llms can be used in psychology @xcite , along with work highlighting cases where llms can replicate classical findings in the social psychology literature @xcite . looking further back, several authors have evaluated the semantic structure of learned word embeddings in staticvector spaces @xcite , while others have examined the semantic structure of more fine-grained text descriptions of concepts in language models capable of embedding sequences @xcite . a few studies have used models to generate lists of features and estimated semantic structure from feature overlap @xcite @xcite , or have asked models to produce explicit pairwise similarity ratings @xcite . while such work often compares aspects of machine and human behaviors, to our knowledge no prior study has evaluated the coherence of elicited structures across tasks within a given model, or between structures elicited from humans and machines using the same set of tasks. ## measuring human conceptual structure for both human and llm experiments, we focused on a subset of 30 concepts (as shown in to measure the robustness of conceptual structure in humans, we estimated similarities amongst the 30 items using 3 different tasks: ## measuring machine conceptual structure in this section, we consider whether one of the most performant llms, openai's gpt 3, expresses coherence in the structural organization of concepts when tested using the same methods used in the human behavioral experiments. using the openai api, we conducted the feature listing and verification task, triadic comparison task, and the pairwise similarity rating task on gpt 3. given the recent deluge of open-source llms, we also tested flan-t5 xxl, and flan-u2 on the triadic comparison and pairwise ratings tasks to see how they perform relative to larger closed models. finally for completeness we also tested the similarity between embeddings extracted from gpt 3, word2vec, and the language component of clip. while word2vec embeddings are a staple of nlp research, relatively fewer works have explored the structure of the language models that are jointly trained in the clip procedure. after computing the similarity structure between concepts expressed by the nlp methods outlined above, we considered (a) how well these estimates aligned with structures estimated from human behaviors within each task, and (b) the structural coherence between the embeddings estimated via different methods from llm behavior. ## conclusion in this study, we compared the conceptual structures of humans and llms using three cognitive tasks: a semantic feature-listing task, a triplet similarity judgement task, and a pairwise rating task. our results showed that the conceptual representations generated from human judgments, despite being estimated from quite different tasks, in different languages, across different cultures, were remarkably coherent: similarities captured in one space accounted for 96% of the variance in the other. this suggests that the conceptual structures underlying human semantic cognition are remarkably robust to differences in language, cultural background, and the nature of the task at hand. in contrast, embeddings obtained from analogous behaviors in llms differed depending upon on the task. while embeddings estimated from verified feature lists aligned moderately well with those estimated from human feature norms, those estimated from triplet judgments or from the raw (unverified) feature lists did not, nor did the two embedding spaces from the llm cohere well with each other. embedding spaces extracted directly from model hidden representations or from other common neural network techniques did not fare better: in most comparisons, distances captured by one model-derived embedding space accounted for, at best, half the variance in any other. the sole exception was the space estimated from llmverified feature vectors, which cohered modestly well with embeddings taken directly from the gpt-3 embeddings obtained using the triplet task (72% of the variance) and the hidden layer (66% of variance)5. while recent advances in prompting techniques including chain-of-thought prompting @xcite , self-consistence @xcite , and tree-of-thoughts @xcite have been shown to improve performance in tasks with veridical solutions such as mathematical reasoning and knowledge retrieval, we highlight here through both direct and indirect tasks that the underlying conceptual structure learned by llms is brittle. we implemented chain-of-thought reasoning for some models and found that this led to llm model representations being more aligned with human conceptual structure (fig 6 ). however, the conceptual coherence within a model only increased for only some of the models but still nothing comparable to human conceptual robustness. together these results suggest an important difference between human cognition and current llm models. neuro-computational models of human semantic memory suggest that behavior across many different tasks is undergirded by a common conceptual "core" that is relatively insulated from variations arising from different contexts or tasks @xcite . in contrast, representations of word meanings in large language models depend essentially upon the broader linguistic context. indeed, in transformer architectures like gpt-3, each word vector is computed as a weighted average of vectors from surrounding text, so it is unclear whether any word possesses meaning outside or independent of context. because this is so, the latent structures organizing its overt behaviors may vary considerably depending upon the particular way the model's behavior is probed. that is, the llm may not have a coherent conceptual "core" driving its behaviors, and for this reason, may organize its internal representations quite differently with changes to the task instruction or prompt. context-sensitivity of this kind is precisely what grants such models their notable ability to simulate natural-seeming language, but this same capacity may render them ill-suited for understanding human conceptual representation.