Is a Peeled Apple Still Red? Evaluating LLM s’ Ability for Conceptual Combination with Property Type (2025)

Abstract: Conceptual combination is a cognitive process that merges basic concepts, enabling the creation of complex expressions. During this process, the properties of combination (e.g., the whiteness of a peeled apple) can be inherited from basic concepts, newly emerge, or be canceled. However, previous studies have evaluated a limited set of properties and have not examined the generative process.To address this gap, we introduce the Conceptual Combination with Property Type dataset (CCPT), which consists of 12.3K annotated triplets of noun phrases, properties, and property types. Using CCPT, we establish three types of tasks to evaluate LLMs for conceptual combination thoroughly.Our key findings are threefold:(1) Our automatic metric grading property emergence and cancellation closely corresponds with human judgments.(2) LLMs, including OpenAI’s o1, struggle to generate noun phrases which possess given emergent properties.(3) Our proposed method, inspired by cognitive psychology model that explains how relationships between concepts are formed, improves performances in all generative tasks.The dataset and experimental code are available at https://github.com/seokwon99/CCPT.git.

## introduction the conceptual combination is a fundamental cognitive process that synthesizes multiple basic concepts into a novel concept @xcite @xcite . combining concepts is potentially limitless and plays an important role in various fields such as engineering, science @xcite , figurative language and literature @xcite . for example, 'a wilted flower' evokes our sadness because we internally compare a flower's former beauty and ⋆ authors equally contributed. ♯ work done during internship at seoul national university. current feeble state. a skilled writer knows it and kindles diverse imagery through conceptual combinations. the potential for such conceptual combinations is vast, so large language models (llms) should be able to interpret or generate meaningful insights even for rare or novel combinations. several works have explored concept knowledge of language models, as outlined in it is essential to understand not only properties that emerge from a combination, but also whether existing properties maintain or disappear @xcite @xcite . following @xcite , we identify three types of properties in conceptual combination, which are illustrated using the example of an "apple" in figure 1 : @xcite component property refers to a property inherent to individual concepts (e.g., "red" for "apple on a toothpick"), (2) emergent property is newly from the combination of concepts (e.g., "unstable" for "apple on a toothpick"), and (3) canceled property is negated due to the combination (e.g., "crisp" for "rotten apple"). these three types form a complete set of properties related to conceptual combination since they indicate already present, newly created, and newly disappeared. in this study, we aim to fill this gap by proposing the conceptual combination with property type (ccpt) dataset. our dataset contains 12.3k annotated triplets of noun phrase comprised of two basic concepts, property and property type. the basic concepts in noun phrases are comprised of 9k unique uni-gram concepts from conceptnet @xcite . to evaluate llms' ability on conceptual combination, we introduce two generative and one classification tasks, detailed in extensive experiments show that: (1) our automatic evaluation metric, based on the llm-as-ajudge @xcite ) method to detect property emergence and cancellation scores, closely matches human evaluations; (2) current llms, including o1 (openai, 2024c), struggle to create noun phrases with truly emergent properties, often defaulting to results where one component already has the target property; and (3) our proposed method, motivated by the spread activation model @xcite in cognitive psychology, improves performance across all generative tasks, indicating that considering diverse relationships between concepts enhances conceptual combination. ## benchmarking llms and humans we comprehensively assess 6 current llms with different architectures and sizes, including both open-source and closed-source models: llama-3.1-70b-instruct @xcite , qwen2.5-72b-instruct @xcite ## analysis 6.1 relevance between llm-as-a-judge metric and manual evaluation as shown in figure 4 , we compare the llm judge's metrics with human evaluations to verify the agreement between them. we randomly selected 300 pairs from ccpt, which consists of pairs from h-p, m-p, and n -p, covering both emergent properties (50 samples) and canceled properties (50 samples). human raters, recruited through amazon mechanical turk, as detailed in section b.5, are asked to rate the relevance of each pair using the same instructions provided to the llm judge. each problem is rated by three different raters. we calculated the pearson and spearman correlation coefficients between the llm judge's scores and human ratings, which are 0.85 and 0.83, respectively. these strong correlations indicate a high level of agreement, demonstrating the effectiveness of using llm for relevance scoring. ## conclusion in conclusion, our work introduces ccpt, a conceptual combination dataset designed to evaluate llms' ability to process conceptual combinations. ccpt comprises 12,315 annotated instances of noun phrases, properties, and property types. based on ccpt, we propose three downstream tasks: property induction, noun phrase completion, and property type prediction. to assess generative performance, we introduce two automatic evaluation metrics-emergence and cancel-task c 1 c 2 c 3 c 4 c 5 pi-eme. 45.1 45.4 45.2 45.9 45.8 pi-can. 60.5 61.1 61.1 62.7 63.3 npc-eme. 26.0 26.2 26.9 26.7 26.9