A bs I nstruct: Eliciting Abstraction Ability from LLM s through Explanation Tuning with Plausibility Estimation (2024)

Abstract: Abstraction ability is crucial in human intelligence, which can also benefit various tasks in NLP study. Existing work shows that LLMs are deficient in abstract ability, and how to improve it remains unexplored. In this work, we design the framework AbsInstruct to enhance LLMs’ abstraction ability through instruction tuning. The framework builds instructions with in-depth explanations to assist LLMs in capturing the underlying rationale of abstraction. Meanwhile, we introduce a plausibility estimator to select instructions that are more consistent with the abstraction knowledge of LLMs to be aligned. Then, our framework combines abstraction instructions with general-purpose ones to build a hybrid dataset. Extensive experiments and analyses demonstrate that our framework can considerably enhance LLMs’ abstraction ability with strong generalization performance while maintaining their general instruction-following abilities.

## introduction abstraction ability is central to human cognition @xcite , which is identifying shared traits among items to build a broader concept, like deriving the concept of "beverage" from "coffee" and "tea." with this ability, we can derive general rules and principles from past experiences, which enables us to adeptly navigate new situations in our daily life @xcite . in nlp, building abstraction resources has long been a vital challenge to which the community has devoted many efforts @xcite . among them, @xcite built the first comprehensive benchmark, abspyramid, of abstract concepts for nouns, verbs, and events. in this benchmark, models are asked to detect the validity of an abstract concept, as shown in figure 1 . their 1 the code and data are available at https://github.com/ hkust-knowcomp/absinstruct question: in the sentence "ivy is a labrador retriever," @xmath0 explanation trace: the labrador retriever is a british breed of retriever gun dog. it was developed from… meanwhile, the term "dog" refers to a domesticated mammal… conclusion: yes, the meaning @xmath1 ## related work abstraction has long been widely applied across various tasks, including question answering @xcite , machine translation @xcite , and many others @xcite @xcite . while some works have studied entity abstraction @xcite @xcite without considering contexts, our work explores event abstraction with a few relevant fields: event abstraction: this field focuses on studying abstraction within an event as context. one line of works studied extracting entailment graphs for verbs with two arguments from large corpora @xcite @xcite @xcite @xcite . meanwhile, @xcite curated abstract concepts for nouns and events based on atomic @xcite . recently, @xcite compiled a large benchmark that unifies the scopes of the abovementioned works. they collected abstraction descriptions of events and hypernyms of nouns and verbs using chat-gpt (openai, 2022) and wordnet @xcite , which are then manually annotated. their studies suggest that llms still struggle with abstraction knowledge with various mistakes. thus, we present the first attempt to unlock the stronger abstraction abilities of llms. ## method eliciting abstraction knowledge from pre-trained llms can be challenging since it requires (1) underlying rationales in determining the validity of abstract concepts and ( 2 ) carefully curated instructions to better elicit the knowledge. here, we describe the process of absinstruct, which builds instructions with explanation traces and employs a plausibility estimator and several filters for data selection. this pipeline is depicted in figure 2 . ## abstraction instruction overview in this section, we apply absinstruct for inducing instruction data as a case study, with llama2 (7b) @xcite used to estimate plausibilities. our framework constructs 200 examples for each relation, derived from abspyramid. ## experiment we conduct extensive experiments and compare our framework with various baselines. ## main evaluation we present the results of each entailment relation and the average on the test set of abspyramid in our results unequivocally demonstrate that the "direct injection" baseline possesses limited efficacy in eliciting abstraction knowledge. for example, falcon (7b) only achieves performance slightly higher than a random guess. similarly, we observe that llms tuned with alpaca only capture limited generalization ability in abstraction detection, even with ten exemplars. for instance, falcon (7b) only achieves a macro f1-score of 52.13%, lagging behind our framework by about 10 points. ## conclusion abstraction knowledge is a critical kind of knowledge in human intelligence, as shown in previous cognitive studies @xcite . meanwhile, recent research @xcite shows that using llms' abstraction ability can better solve general nlp tasks, including stem questions @xcite , knowledge qa @xcite @xcite , and multi-hop reasoning @xcite . given this evidence, we can see that abstraction is also essential for a broad range of complex language understanding and reasoning tasks. in this paper, we propose absinstruct, which is the first attempt to elicit stronger abstraction abilities from pre-trained llms. our framework builds instructions for abstraction detection with explanation traces and a plausibility estimator. then, these abstraction instructions are combined with generaldomain ones from alpaca. in the experiments, we compare our framework with a lot of strong baselines to demonstrate the framework's effectiveness. we also provide comprehensive ablation studies of our framework and show its effectiveness on two out-of-domain datasets. what's more, evaluations on instruction datasets also show that our framework can improve the abstraction ability of llms without sacrificing llms' general instruction following ability. for future work, we can study how to equip llms with more abstraction knowledge during pre-training. more importantly, we also leave the study of using this enhanced abstraction knowledge in downstream tasks as future work.