Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models for Learning Conceptual Spaces (2023)

Abstract: The theory of Conceptual Spaces is an influential cognitive-linguistic framework for representing the meaning of concepts. Conceptual spaces are constructed from a set of quality dimensions, which essentially correspond to primitive perceptual features (e.g. hue or size). These quality dimensions are usually learned from human judgements, which means that applications of conceptual spaces tend to be limited to narrow domains (e.g. modelling colour or taste). Encouraged by recent findings about the ability of Large Language Models (LLMs) to learn perceptually grounded representations, we explore the potential of such models for learning conceptual spaces. Our experiments show that LLMs can indeed be used for learning meaningful representations to some extent. However, we also find that fine-tuned models of the BERT family are able to match or even outperform the largest GPT-3 model, despite being 2 to 3 orders of magnitude smaller.

## introduction conceptual spaces @xcite represent concepts in terms of cognitively meaningful features, called quality dimensions. for example, a conceptual space of colour is composed of three quality dimensions, representing hue, saturation and intensity. conceptual spaces provide an elegant framework for explaining various cognitive and linguistic phenomena @xcite . within artificial intelligence (ai), the role of conceptual spaces is essentially to act as an intermediate representation layer, in between neural and symbolic representations @xcite . as such, conceptual spaces could play a central role in the development of explainable ai systems. unfortunately, such representations are difficult to learn from data. most applications of conceptual spaces are thus limited to narrow domains, where meaningful representations can be learned from ratings provided by human participants @xcite @xcite . in this paper, we explore whether large language models (llms) could be used for learning conceptual spaces. this research question is closely related to the ongoing debate about the extent to which language models (lms) can learn perceptually grounded representations @xcite @xcite . recent work seems to suggest this might indeed be possible, at least for the colour domain. for instance, @xcite found that lms are able to learn representations of colour terms which are isomorphic to perceptual colour spaces. when it comes to predicting the typical colour of objectes, @xcite found that the predictions of lms are heavily skewed by surface co-occurrence statistics, which are unreliable for colours due to reporting bias @xcite , i.e. the fact that obvious colours are rarely mentioned in text. however, @xcite found the effects of reporting bias to largely disappear in recent llms. these findings suggest that it may now be possible to distill meaningful conceptual space representations from llms, as long as sufficiently large models are used. however, existing analyses are limited in two ways: we address the first limitation by including an evaluation in the taste domain. to address the second limitation, rather than considering discrete labels (e.g. sweet), we use llms to rank concepts according to the degree to which they have a particular feature (e.g. sweetness). ## datasets the primary focus of our experiments is on the taste domain, which has not yet been considered in this context, despite having a number of important advantages. for instance, the relevant quality dimensions are well-established and have a linear structure (unlike hue in the colour domain). this domain also seems particularly challenging, as the typical terms which are used to describe taste only apply to extreme cases. for instance, we can assert that grapefruit is bitter and bananas are sweet, but it is less clear how a language model would learn whether chicken is sweeter than cheese. as ground truth, we rely on the ratings that were collected by @xcite . they rated a total of 590 food items along six dimensions: sweet, salty, sour, bitter, umami and fat. the ratings were obtained by a panel of twelve assessors who were experienced in sensory profiling. they scored the food items during an eight month measurement phase, after having received 55 hours of training in a laboratory. we manually rephrased some of the names of the items in this dataset, to make them more natural. for instance, cherry (fresh fruit) was changed to cherry and hake (grilled with lemon juice) was changed to grilled hake with lemon juice. we complement our analysis in the taste domain with experiments on three basic physical domains: mass, size and height. these were found to be particularly challenging by @xcite , with llms often failing to outperform random guessing. as the ground truth for mass, we use the household dataset from @xcite , which specifies the mass of 56 household objects. the original dataset includes images of each object. we removed 7 items which were not meaningful without the image, namely big elephant, small elephant, ivan's phone, ollie the monkey, marshy the elephant, boy doll and dali clock, resulting in a dataset of 49 objects. we treat this problem as a ranking problem. @xcite also created a binary classification version of this dataset, which involves judging pairwise comparisons (e.g. is a red lego brick heavier than a hammer?). for size and height, we use the datasets created by @xcite . these size and height datasets each con-sist of 500 pairwise judgements (e.g. an ant is larger than a bird). note that unlike for the other datasets, no complete ranking is provided. ## methods we experiment with a number of different models. ## experiments taste domain the model is 2 to 3 orders of magnitude smaller. furthermore, we can see a large performance gap between the two variants of the bi-encoder model, with the model trained on chatgpt examples outperforming curie, and even davinci in two cases. ## conclusions we proposed the use of a dataset from the taste domain for evaluating the ability of llms to learn perceptually grounded representations. we found that llms can indeed make meaningful predic-tions about taste, but also showed that a fine-tuned deberta model, and in some cases even a finetuned bert-large bi-encoder, can outperform gpt-3. the performance of these smaller models crucially depends on the quality of the available training data. for this reason, we explored the idea of collecting training data from chatgpt, using a new prompting strategy. we complemented our experiments in the taste domain with an evaluation of physical properties, where we achieved considerably better results than those reported in the literature @xcite . whereas previous work was essentially aimed at understanding the limitations of language models, our focus was more practical, asking the question: can high-quality conceptual space representations be distilled from llms? our experiments suggest that the answer is essentially positive, but that new approaches may be needed to optimally take advantage of the knowledge that can be extracted from such models.