A bs P yramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph (2024)

Abstract: Cognitive research indicates that abstraction ability is essential in human intelligence, which remains under-explored in language models. In this paper, we present AbsPyramid, a unified entailment graph of 221K textual descriptions of abstraction knowledge. While existing resources only touch nouns or verbs within simplified events or specific domains, AbsPyramid collects abstract knowledge for three components of diverse events to comprehensively evaluate the abstraction ability of language models in the open domain. Experimental results demonstrate that current LLMs face challenges comprehending abstraction knowledge in zero-shot and few-shot settings. By training on our rich abstraction knowledge, we find LLMs can acquire basic abstraction abilities and generalize to unseen events. In the meantime, we empirically show that our benchmark is comprehensive to enhance LLMs across two previous abstraction tasks.

## introduction abstraction is about finding common properties among different things and forming a broader concept, like the concept "furniture" subsuming "sofa" and "table," a key dimension of human cognition @xcite . with this ability, we can smoothly handle daily situations by learning from past experiences and generalizing to new circumstances @xcite . substantively, @xcite , in his k-theory, suggested that our minds organize past experiences in a hierarchical pyramid, with higher parts corresponding to greater abstraction. the nlp community has recently explored diverse, impressive abilities of llms, such as incontext learning @xcite , multi-step reasoning @xcite , and instruction following @xcite . meanwhile, the ability 1 the code and data are available at https://github.com/ hkust-knowcomp/abspyramid . a cat hunted a prey into its shelter. a cat chased a prey into its shelter. a cat chased a mouse into its shelter. a cat chased a mouse into its burrow. a cat hunted a mouse into its shelter. ## related work while the nlp community has studied various abilities of llms @xcite @xcite @xcite , the abstraction ability of llms remains insufficiently studied. unlike existing works that focus on entity-level abstraction @xcite @xcite @xcite , our research delves into event-level abstraction with only a few works investigating some restricted aspects: verb entailment graph: @xcite first proposed the task of entailment graph construction of verbs. following their work, various methods have been proposed to build better verb entailment graphs @xcite @xcite @xcite @xcite . nonetheless, those works consider verbs as binary relations with two arguments from a small set of entity types (e.g., 49 types in figer @xcite ), distorting the original semantics. abstractatomic: @xcite presented an annotated abstraction dataset. they recognized entities in head events from atomic @xcite and crowdsourced abstract concepts from the probase taxonomy @xcite for recognized entities and head events. even though they compiled a dataset comprising thousands of examples, it is specific to the social commonsense domain due to the base events sampled from atomic. textual and linguistic entailment: besides the entailment between verbs, recognizing textual entailment has long been a vital task in the realm of nlp @xcite , also known as natural language inference (nli). researchers have built many large-scale datasets of nli @xcite @xcite and its variants @xcite @xcite . while similar to our task, textual entailment employs a relaxed definition of whether a human reader would typically infer a hypothesis from a given premise @xcite head event: max is a playful puppy. ## abstraction knowledge structure abspyramid represents a large-scale abstraction repository of events in textual descriptions. this unified entailment graph contains 221k fiveelement tuples with the format of (head event, entailment relation, tail event, instance, abstract concept). in each tuple, we identify an instance in the head event and collect an abstract concept for it. particularly, instances are identified from three components of the head event: nouns, verbs, and head event as a whole. then, we replace the instance with its abstract concept to construct the tail event, resulting in the tail event being linguistically entailed by the head event. according to three kinds of instances, we define three types of entailment relation: noun-entail, verb-entail, and event-entail. we elaborate on each tuple element with a concrete example in figure 2 . ## data curation pipeline to build abspyramid, we create a crowdsourcing framework that allows for a scalable, broad collection of abstraction knowledge in the abovementioned format. ## abspyramid overview in this section, we carry out a thorough analysis of our benchmark abspyramid. ## abstraction detection experiment in this section, we conduct extensive experiments on the abspyramid @xcite dataset to evaluate an abundance of language models and provide comprehensive analyses. ## abstraction generation experiment in this section, we evaluate representative lms on the abspyramid @xcite . ## conclusion in this paper, we introduce abspyramid to evaluate llms' abstraction ability. a scalable pipeline is designed to curate abstraction knowledge for three components of events. we carry out extensive experiments to demonstrate the comprehensiveness of our benchmark and provide valuable insights into the abstraction abilities of llms.