Causality-aware Concept Extraction based on Knowledge-guided Prompting (2023)

Abstract: Concepts benefit natural language understanding but are far from complete in existing knowledge graphs (KGs). Recently, pre-trained language models (PLMs) have been widely used in text-based concept extraction (CE). However, PLMs tend to mine the co-occurrence associations from massive corpus as pre-trained knowledge rather than the real causal effect between tokens. As a result, the pre-trained knowledge confounds PLMs to extract biased concepts based on spurious co-occurrence correlations, inevitably resulting in low precision. In this paper, through the lens of a Structural Causal Model (SCM), we propose equipping the PLM-based extractor with a knowledge-guided prompt as an intervention to alleviate concept bias. The prompt adopts the topic of the given entity from the existing knowledge in KGs to mitigate the spurious co-occurrence correlations between entities and biased concepts. Our extensive experiments on representative multilingual KG datasets justify that our proposed prompt can effectively alleviate concept bias and improve the performance of PLM-based CE models.

## introduction the concepts in knowledge graphs (kgs) enable machines to understand natural languages better, and thus benefit many downstream tasks, such as question answering @xcite , commonsense reasoning @xcite and entity typing @xcite . however, the concepts, especially the fine-grained ones, in existing kgs still need to be completed. for example, in the widely used chinese kg cn-dbpedia @xcite , there are nearly 17 million entities but only 0.27 million concepts in total, and more than 20% entities even have no concepts. although probase (wu figure 1 : the example of concept bias. the plm-based ce models are biased to extract novel mistakenly as the concept of louisa may alcott from the text. et al., 2012) is a large-scale english kg, the finegrained concepts with two or more modifiers in it only account for 30% @xcite . we focus on extracting multi-grained concepts from texts to complete existing kgs. most of the existing text-based concept acquisition approaches adopt the extraction scheme, which can be divided into two categories: 1) patternmatching approaches @xcite @xcite , which can obtain high-quality concepts but only have low recall due to poor generalization; 2) learning-based approaches @xcite @xcite , which employ pre-trained language models (plms) fine-tuned with labeled data to extract concepts. however, an unignorable drawback of these learning-based approaches based on plms is concept bias. concept bias means the concepts are extracted based on their contextual (co-occurrence) associations rather than the real causal effect between the entities and concepts, resulting in low extraction precision. for example, in figure 1 , plms tend to extract novel and writer together as concepts for the entity louisa may alcott even if we explicitly input the entity louisa may alcott to the model. previous work demonstrates that causal inference is a promising technique for bias analy-sis @xcite . to analyze the reasons behind concept bias, we devise a structural causal model (scm) @xcite to investigate the causal effect in the plm-based concept extraction (ce) system, and show that pre-trained knowledge in plms confounds plms to extract biased concepts. during the pre-training, the entities and biased concepts (e.g., louisa may alcott and novel) often co-occur in many texts. thus, plms tend to mine statistical associations from a massive corpus rather than the real causal effect between them @xcite , which induces spurious co-occurrence correlations between entities (i.e., louisa may alcott) and biased concepts (i.e., novel). since we cannot directly observe the prior distribution of pre-trained knowledge, the backdoor adjustment is intractable for our problem @xcite . alternatively, the frontdoor adjustment @xcite can apply a mediator as an intervention to mitigate bias. in this paper, we adopt language prompting @xcite as a mediator for the frontdoor adjustment to handle concept bias. we propose a novel concept extraction framework with knowledge-guided prompt, namely kpce to extract concepts for given entities from text. specifically, we construct a knowledge-guided prompt by obtaining the topic of the given entity (e.g., person for louisa may alcott) from the knowledge in the existing kgs. our proposed knowledge-guided prompt is independent of pre-trained knowledge and fulfills the frontdoor criterion. thus, it can be used as a mediator to guide plms to focus on the right cause and alleviate spurious correlations. although adopting our knowledge-guided prompt to construct the mediator is straightforward, it has been proven effective in addressing concept bias and improving the extraction performance of plmbased extractors in the ce task. in summary, our contributions include: 1) to the best of our knowledge, we are the first to identify the concept bias problem in the plm- ## concept bias analysis in this section, we first formally define our task. then we investigate the concept bias issued by plms in empirical studies. finally, we devise a structural causal model (scm) to analyze the bias and alleviate it via causal inference. ## methodology in this section, we present our ce framework kpce and discuss how to perform prompting to alleviate concept bias. the overall framework of kpce is illustrated in figure 4 , which consists of two major modules: 1) prompt constructor: assigns the topic obtained from kgs for entities as a knowledge-guided prompt to estimate eq.1; louisa may alcott was an american novelist, short story writer and poet best known as the author of the novel little women and its sequels little men and jo's boys. ## conclusion in this paper, we identify the concept bias in the plm-based ce system and devise a structural causal model to analyze the bias. to alleviate concept bias, we propose a novel ce framework with knowledge-guided prompting to alleviate spurious co-occurrence correlation between entities and biased concepts. we conduct extensive experiments to justify that our prompt-based learning framework can significantly mitigate bias and has an excellent performance in concept acquisition.