Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models? (2023)

Abstract: In recent years, the injection of factual knowledge has been observed to have a significant positive correlation to the downstream task performance of pre-trained language models. However, existing work neither demonstrates that pre-trained models successfully learn the injected factual knowledge nor proves that there is a causal relation between injected factual knowledge and downstream performance improvements. In this paper, we introduce a counterfactual-based analysis framework to explore the causal effects of factual knowledge injection on the performance of language models within pretrain-finetune paradigm. Instead of directly probing the language model or exhaustively enumerating potential confounding factors, we analyze this issue by perturbing the factual knowledge sources at different scales and comparing the performance of pre-trained language models before and after the perturbation. Surprisingly, throughout our experiments, we find that although the knowledge seems to be successfully injected, the correctness of injected knowledge only has a very limited effect on the models’ downstream performance. This finding strongly challenges previous assumptions that the injected factual knowledge is the key for language models to achieve performance improvements on downstream tasks in pretrain-finetune paradigm.

## introduction in recent years, pre-trained language models (plms) have emerged as the dominant approach in natural language processing. through selfsupervised learning on large-scale text corpus, plms can acquire different kinds of knowledge automatically without additional manual guidance, which demonstrates significant generalizability and transferability improvements across tasks compared with previous architectures @xcite @xcite @xcite . some previous investigations contribute the superiors of pre-trained language models to their entailed various kinds of knowledge learned from the pre-training stage @xcite @xcite . among them, factual knowledge, which reveals the relationships between real-world entities (e.g., tim cook is the ceo of apple) and plays a critical role in human cognition @xcite , is regarded as a critical factor for a pre-trained language model to approach a trusted intelligent agent @xcite . consequently, how to improve the acquisition, modeling, and application of factual knowledge of pre-training language models has become a hot research topic. to this end, many studies have been devoted to further injecting factual knowledge to enhance the acquisition and modeling of factual knowledge in pre-trained language models @xcite @xcite , and have already reported successful performance improvements on specific downstream tasks. on the contrary, recent studies have found that pre-trained language models struggle with retaining factual knowledge, and the retained factual knowledge can exhibit inconsistencies with the original knowledge sources @xcite @xcite . furthermore, the indirect evaluations through downstream tasks only reflect that injecting factual knowledge is correlated to performance improvement, but can not establish the causality between them due to the existence of many additional potential confounding factors (training data domain, model parameter size, etc.). consequently, to identify the impact of factual knowledge injection on the downstream performance of pre-trained language models, the following two critical questions should be answered: unfortunately, it is infeasible to directly answer the above-two questions due to the lack of highly effective language model knowledge probing and confounding factor identification measurements. to this end, as shown in figure 1 , this paper introduces a counterfactual-based analysis framework @xcite to explore the causal effects of injecting factual knowledge in a "what-if" manner. moreover, figure 2 illustrates the applied framework. instead of directly probing the language model or exhaustively enumerating potential confounding factors, we analyze the above-two questions by perturbing the factual knowledge sources at different scales, then comparing the downstream performance of pretrained language models before and after the perturbation. the key motivation of our work is that: 1) if the knowledge injection approaches are ineffective, the performance of the model injected with the correct knowledge and perturbed knowledge should not perform very differently in the knowledge probing evaluation; 2) if the correctness of injected factual knowledge is indeed essential for downstream tasks, then injecting perturbed, wrong knowledge should cause significant performance decline. specifically, in order to observe the aspect from which factual knowledge affects plms, we conduct two kinds of perturbation on factual knowledge, including factual substitution, which replaces an entity in factual knowledge with another entity with the same type (e.g., substitute "tim cook is the ceo of apple" with "bill gates is the ceo of apple"), as well as ontological substitution that thoroughly perturb entities with their counterparts of another type (e.g., substitute "tim cook is the ceo of apple" with "microsoft is the ceo of apple"). in addition, to analyze the impact of perturbation, we investigate three knowledge injection approaches that acquire knowledge from two types of factual knowledge sources, including plain texts containing factual knowledge and structured factual knowledge bases. throughout empirical evaluations on a wide range of representative downstream nlp tasks, our findings surprisingly deviate from previous hypotheses. although the knowledge injection approaches seem to inject factual knowledge into plms successfully, we find that factual knowledge perturbation has only a very limited effect on plms' downstream task performance, i.e., the correctness of factual knowledge shows a very limited impact on all evaluated downstream tasks. furthermore, although the influence of ontological perturbation is slightly stronger than factual perturbation, it also does not cause statistically significant performance divergence in most downstream tasks. specifically, our experiments show that in most downstream tasks, the performance fluctuation caused by the above-two perturbation is not greater than the fluctuation caused by random seeds, and the results of t-test further demonstrate that there are no statistically significant differences between the performance before and after perturbation. through this counterfactual-based analysis, our findings demonstrate that injected factual knowledge is not the core reason for the performance improvements of previous knowledge injection approaches on pre-trained language models in the pretrain-finetune paradigm. the following part of this paper is organized as follows. in section 2, we briefly review the related work about factual knowledge probing and injection of plms. section 3 presents our proposed counterfactual-based analysis framework. the experimental results and the process leading to our conclusions are presented in section 4. in section 5, we provide a brief discussion and conclude our findings foot_0 . ## related work factual knowledge reveals the relationships between real-world entities and plays a crucial role in human cognitive @xcite ). therefore, lots of studies devoted to probing the factual knowledge entailed in plms on the one hand @xcite @xcite , and propose to enhance the factual knowledge in large-scale lms to benefit their performance on downstream tasks on the other @xcite @xcite . factual knowledge probing aims to evaluate how well current plms are entailing factual knowledge in parameters. currently, the most popular strategy is prompt-based probing @xcite @xcite , e.g., query bert with "tim cook is the ceo of [mask]" to determine whether it contains corresponding knowledge. and recent studies have found that prompt-based probing could be inaccurate @xcite , inconsistent @xcite @xcite , and unreliable @xcite . factual knowledge injection aims to inject factual knowledge into plms. currently, there are two main sources for injecting factual knowledge into language models, including plain text and structured knowledge base. for plain text, lms typically acquire factual knowledge through self-supervised learning on large-scale text corpus without any knowledge guided supervision. the popular objectives include casual language modeling @xcite @xcite , masked language modeling @xcite , denoising auto-encoder @xcite , etc. and such pre-trained lms have been shown to potentially entail a large scale of factual knowledge in the parameters @xcite . in addition, many studies intend to explicitly infuse the factual knowledge from structured knowledge base into lms @xcite . popular strategies include: 1) embedding combined methods @xcite @xcite , which encode structured knowledge via knowledge embedding algorithms and then enhance plms' text representation with knowledge graph embedding; 2) knowledge supervised methods @xcite , which utilize elements from structured knowledge as supervision signals and leverage base plms to learn their semantics. ## counterfactual-based knowledge analysis framework as mentioned above, this paper intends to investigate whether current factual knowledge injection approaches can inject factual knowledge into pretrained language models and whether the injected factual knowledge can benefit downstream nlp tasks in pretrain-finetune paradigm. however, it is currently not feasible to directly explore these questions due to the lack of effective knowledge probing and confounding factor discovery approaches for pre-trained language models. to this end, we propose to leverage a counterfactual-based knowledge analysis framework to answer these two questions in a "what-if" manner. specifically, we transform the problem of investigating the characteristics of a model with injected knowledge into comparing the behaviors between models injected with correct and incorrect knowledge, respectively. consequently, if the knowledge injection approaches work, models injected with incorrect knowledge should exhibit significantly inferior performance on the knowledge probing evaluation than models injected with correct knowledge. furthermore, if injecting factual knowledge is indeed helpful to downstream tasks, models injected with correct factual knowledge should perform significantly better than models injected with incorrect factual knowledge on downstream tasks. the overall counterfactual-based analysis framework is illustrated in figure 2 . specifically, given factual knowledge from text corpus or knowledge bases, we first conduct perturbation on the instances in them to obtain incorrect knowledge instances. then we pre-train language models with several representative factual knowledge injection approaches on both vanilla and perturbed knowledge sources. finally, we compare the performance of knowledge-injected models to reach conclusions about the above-two questions. in the following sections, we will first demonstrate how we conduct knowledge perturbation on different knowledge sources and then briefly introduce the investigated representative factual knowledge injection approaches in our experiments. ## experiments and findings based on the analysis framework presented in section 3, we conduct extensive experiments and intriguingly find that the correctness of factual knowledge shows a very limited impact on almost all downstream tasks. in this section, we will elaborate on our experimental procedures, provide a detailed account of our findings, and illustrate how we arrive at our conclusions. ## conclusions and discussions this paper proposes a counterfactual-based knowledge analysis framework to investigate the influence of injecting factual knowledge into lms. throughout our experiments, we find that even though current knowledge injection approaches do inject factual knowledge into models, there exist very limited causal relations between the correctness of factual knowledge and performance improvements. our findings strongly challenge previous assumptions that the injected factual knowledge is the core reason of previous factual injection approaches to achieve improvements on downstream tasks, but very likely due to other confounding factors. our conclusions can also shed light on future research directions. for example, current extremely large lms such as @xcite and llama @xcite can perform well on various downstream tasks but would still generate large amounts of factual-incorrect responses, and our conclusions indicate that we need to improve the evaluation paradigm for comprehensive plm evaluation on factual knowledge.