Inference Helps PLM s’ Conceptual Understanding: Improving the Abstract Inference Ability with Hierarchical Conceptual Entailment Graphs (2024)

Abstract: The abstract inference capability of the Language Model plays a pivotal role in boosting its generalization and reasoning prowess in Natural Language Inference (NLI). Entailment graphs are crafted precisely for this purpose, focusing on learning entailment relations among predicates. Yet, prevailing approaches overlook the *polysemy* and *hierarchical nature of concepts* during entity conceptualization. This oversight disregards how arguments might entail differently across various concept levels, thereby missing potential entailment connections. To tackle this hurdle, we introduce the *concept pyramid* and propose the HiCon-EG (Hierarchical Conceptual Entailment Graph) framework, which organizes arguments hierarchically, delving into entailment relations at diverse concept levels. By learning entailment relationships at different concept levels, the model is guided to better understand concepts so as to improve its abstract inference capabilities. Our method enhances scalability and efficiency in acquiring common-sense knowledge through leveraging statistical language distribution instead of manual labeling, Experimental results show that entailment relations derived from HiCon-EG significantly bolster abstract detection tasks. Our code is available at https://github.com/SXUCFN/HiCon-EG

## introduction cognitive research underscores abstract inference ability as the cornerstone of human cognition, empowering us to extrapolate and interpolate from past encounters, distill patterns, and adapt to novel scenarios @xcite . for instance, when humans comprehend "john presents his friend a book", they invariably perceive "john" and "his friend" as person, "book" as an entity, and abstract the event as "personx present persony entity". this event can be further abstracted as "per-sonx give persony entity". in natural language * contact authors figure 1 : after conceptualizing events, models can infer more information from conceptualized events. however, different levels of conceptualization may lead to different entailment relationships. for example, when pasta is conceptualized as f ood, we can infer that persony be. f ull. processing (nlp), "personx present persony entity" is defined as the given premise and "personx give persony entity" as the inferred hypothesis, constituting an textual entailment relationship. in the evolution of natural language inference (nli), numerous studies delve into abstract challenges across various domains such as common sense reasoning @xcite @xcite , question answering @xcite , knowledge base explanation @xcite , argumentation mining @xcite , machine translation @xcite , and beyond. in this paper, our goal is to enhance the capability of conceptual knowledge for pre-trained language models (plms) @xcite . within this landscape, a pivotal effort by @xcite introduced the abspyramid benchmark, aiming to comprehensively assess the abstraction prowess of plms through three entailment relationship types: nouns, verbs, and events. despite advancements, evaluations reveal that even state-of-the-art plms struggle with abstraction, trailing behind fine-tuned smaller models. hence, there's a pressing need for further research to better mine entailment relationships and bolster models' abstraction capabilities. entailment graphs, first proposed by @xcite , are graphs with verbs as nodes and entailment relations as edges, which can be seen as subproperty relations (in natural language form) in a schema of knowledge graphs @xcite . entailment graphs aim to globally discover textual entailment relationships between verbs which is different from logical entailment @xcite , and textual entailment has a more relaxed definition: "t entails h" (t ⊨ h) if, typically, a human reading t would infer that h is most likely true @xcite . early work explored the entailment relationships between bivalent verbs based on global transitivity constraints @xcite @xcite . @xcite extended the distributional inclusion hypothesis @xcite , allowing the discovery of entailment relationships between verbs of different valences. this evolution has enabled entailment graphs to discover more diverse entailment relationships (e.g., personx give persony entity ⊨ persony receive entity) beyond the entailment relationships between synonyms. in these works, to disambiguate polysemous verbs, the arguments of verbs are usually typed (conceptualised) @xcite , that is, these arguments are mapped to a limited finite number of basic types, such as person, location, time, etc. therefore, the nodes in the entailment graph are essentially events after abstraction, and the graph itself can be understood as a representation of abstract relations. these relationships include the abstract relations between vocabulary and their abstract concepts (present |= give) , as well as the conceptualized commonsense reasoning relations between abstract events (personx give persony entity |= persony receive entity). nevertheless, the limited argument types (ling and weld, 2021) used in the conceptualization of arguments often compromises the precision of events, resulting in inaccurate entailment relationships. a single instance can be understood through a spectrum of concepts with varying levels of granularity @xcite . for example, an apple can be seen as an object, food, fruit, etc. different granularity levels reveal distinct entailment relations. in figure 1 , consider the sentence "mrs. thompson gives her children some pasta." if pasta is conceptualized as an entity, the inference is "persony receives entity." viewing pasta as food allows for richer inferences, such as "personx f eeds persony entity . in this paper, we argue that entailment relationships at different concept levels can supplement richer verb entailment relationships and these relationships is helpful for the model to better understand the differences between noun concepts at different concept levels. based on the entailment relations across various levels of conceptual granularity. we create a hierarchical conceptual entailment graph (hicon-eg). in particular, we introduce a conceptual pyramid @xcite for hierarchically conceptualizing arguments. this approach enables us to uncover entailment relations under various conceptual constraints. to mitigate the sparsity issue stemming from the abundance of concepts, we propose a concept selection method grounded in entropy principles @xcite to identify the most representative concepts, thereby reducing unnecessary computations. our contributions can be summarized as follows: 1. we propose a novel complex-to-simple open information extraction method based on large language models (llms), which facilitates the extraction of multivalent arguments from lengthy texts. to mitigate the hallucination problem associated with llms, we further distill stable, smaller models. this method outperforms existing approaches on specific datasets, demonstrating superior performance. 2. we introduce the "conceptual pyramid" for the hierarchical conceptualization of arguments, enabling the mining of entailment relations under diverse conceptual constraints. to reduce computational costs, we propose an entropy-based concept selection method for identifying appropriate concepts for arguments under different predicates. experimental results demonstrate performance comparable to gpt-4, with lower error rates. 3. we evaluate the effectiveness of our method on abstraction detection and conceptualized commonsense reasoning tasks. results indicate significant performance enhancements on the abstraction detection task, with a slight edge over the baseline on conceptualized commonsense reasoning datasets. recently, @xcite introduced the abspyramid benchmark (including an abstract detection task) to evaluate models' abstraction capabilities, revealing that abstraction remains a challenge for llms. @xcite proposed absinstruct, built instructions with in-depth explanations to assist llms in capturing the underlying rationale of abstraction. @xcite introduced the product recovery benchmark, for entailment graphs in the e-commerce setting. ## our proposed approach the construction of the hierarchical entailment graph commences with the extraction of multivalent arguments for predicates from the multi-source newsspike corpus @xcite using our proposed c2s-oie method. subsequently, we engage in a multilevel conceptualization of the extracted predicate arguments, selecting the most appropriate concept for each argument governed by different predicates. finally, we compute the relevance score between pairs of predicates to construct the entailment graph @xcite . ## experiment due to lack of entailment graph datasets pertinent to problem, we construct data that conforms to hierarchical concept entailment based on existing datasets (section 4.1) and verify the effectiveness of our method in section 4.3. furthermore, to assess the efficacy of our open information extraction and hierarchical concept selection, we conduct ## conclusion in this paper, we propose a method for constructing a hierarchical conceptual entailment graph. this approach aids the model in identifying entailment relationships across diverse hierarchical concepts, thereby enhancing the abstract reasoning capabilities of existing models. we validate the value of our method across conceptualized commonsense reasoning and abstraction detection tasks, demonstrating the effectiveness of both the complex-to-simple open information extraction (c2s oie) method and the entropy-based concept selection method proposed in this paper. the experimental results show that the entailment relationships under different levels of concepts in hicon-eg can effectively help language models improve their understanding of concepts, thereby enhancing language models' performance on commonsense reasoning tasks.