CR - LLM : A Dataset and Optimization for Concept Reasoning of Large Language Models (2024)

Abstract: Concept reasoning is an important capability for models to understand the world. However, the existing datasets, such as concept extraction and concept generation, suffer from modeledge leakage and context leakage. To address these limitations, we construct a dataset of concept reasoning for large language models (CR-LLM) with modeledge leakage prevention and context leakage prevention, which consists of 2,167 samples and covers different concept types. In addition, we propose a hybrid reasoning method, consisting of inductive reasoning, deductive reasoning and a controller. This method allows large language models to adaptively select the optimal reasoning method for each input sample. Finally, we conduct extensive experiments on CR-LLM using different models and methods. The results show that existing large language models and reasoning methods perform sub-optimally in the concept reasoning task. In contrast, our proposed method significantly improves the capabilities, achieving a 7% increase in accuracy compared to CoT and demonstrating better granularity. We release CR-LLM and code at https://github.com/Nianqi-Li/Concept-Reasoning-for-LLMs.

## introduction concept reasoning is an important ability for models to understand the world by producing appropriate entity concepts based on contextual information. this ability can enhance the model performance in several downstream tasks, such as entity linking @xcite ), text classification @xcite , probing @xcite , event plausibility @xcite , and recommendation systems @xcite . although previous studies highlight the reasoning abilities of large language models (llms), such as chain-ofthought @xcite and mathematical rea-* corresponding author. to evaluate llms' concept reasoning capability without the modeledge leakage and context leakage, we introduce a dataset of concept reasoning for llms (cr-llm). the input is a context with a masked entity, and the output is the concept for that masked entity in the context, as shown in figure 1 . cr-llm excludes target entities and concepts in input to prevent modeledge leakage and context leakage. the evaluation based on cr-llm shows that existing widely-used llms achieve limited performance on concept reasoning. intuitively, concept reasoning can benefit from the logical reasoning ability of llms, of which inductive and deductive reasoning are common forms @xcite , we thus propose a hybrid method that combines inductive and deductive reasoning to stimulate the concept reasoning capability of llms. inductive reasoning involves specific observation of patterns to draw a more general conclusion @xcite , which inspires us to retrieve similar samples with the input to conclude the possible concepts. on the contrary, deductive reasoning drives specific conclusions based on more general premises @xcite , which asks to retrieve relevant facts to infer. additionally, we design a controller based on llm, which allows the model to select the appropriate reasoning methods for different samples. contributions. the contributions of this paper are as follows: the results show that our proposed method significantly improves the model performance, achieving a 7% increase in accuracy compared to cot and demonstrating better granularity. ## related work concept related research previous studies on obtaining concepts can be divided into three categories: concept extraction, entity typing and concept generation. concept extraction focuses on extracting entity concepts from text using an extractive approach. inputs include the target entity and text with the concept, while outputs provide the concept of the target entity. common approaches encompass pattern matching @xcite , learning-based extraction @xcite @xcite , and knowledge-based extraction @xcite @xcite . entity typing aims to classify given entities into predefined concept sets. datasets and methods in this area vary depending on the granularity. coarse-grained entity typing methods such as bnn @xcite , finegrained entity classification datasets such as figer (gold) @xcite , and ultra-fine entity typing tasks @xcite are examples. a smaller body of research uses pre-trained language models to generate entity concepts from text @xcite , and most studies use datasets shared with entity typing. however, all concept extraction samples have context leakage as they only extract concepts in text. entity typing and concept generation also have context leakage: 21.5% of the text contains the output concept in ultra-fine @xcite , as do 40.7% of the gt-zh dataset @xcite . furthermore, due to the exposure of entity names, llms can directly answer questions using modeledge. therefore, their datasets and methods do not align with the requirements of concept reasoning in llms. reasoning related research reasoning is often considered a weakness in language models @xcite @xcite . recent research suggests that reasoning abilities may emerge in llms at a certain scale @xcite . consequently, stimulating the reasoning abilities of llms becomes a major research focus. in 2022, @xcite proposed chain-of-thought prompting, which encourages llms to engage in reasoning by articulating intermediate steps. subsequent researches extend this concept: @xcite introduced chain-of-thought based on self-consistency, replac-ing greedy decoding with multiple paths, and @xcite introduced tree-of-thought to increase the flexibility of reasoning paths. however, given the complexity and diversity of concept reasoning, these paradigms show mediocre performance in this task. therefore, there is still ample room for improvement and extension of concept reasoning. ## the cr-llm dataset in this section, we first define the task of concept reasoning. then, we describe the details of the dataset construction and analysis. finally, we present the evaluation methodology and provide evaluation results on multiple models. ## hybrid method as ## experiments in this section, we first conduct extensive experiments to verify the effectiveness of our hybrid method. then, we provide a detailed analysis and case study of our method and current model in concept reasoning. ## conclusions in this paper, we present the concept reasoning task for large language models for the first time. to evaluate the concept reasoning abilities of large language models, we construct a dataset, cr-llm, which is built by two steps: modeledge leakage prevention and context leakage prevention. the dataset consists of 2,167 samples, covering different entity types, such as "person" and "place". based on the dataset, we propose a hybrid reasoning method, consisting of inductive reasoning, deductive reasoning, and a controller. experimental results show that llms perform sub-optimally in concept reasoning. in contrast, our proposed method significantly improves the concept reasoning abilities of llms, achieving a 7% increase in accuracy compared to cot and demonstrating better conceptual granularity.