Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models (2023)

Abstract: While pre-trained language models (PLMs) have shown evidence of acquiring vast amounts of knowledge, it remains unclear how much of this parametric knowledge is actually usable in performing downstream tasks. We propose a systematic framework to measure parametric knowledge utilization in PLMs. Our framework first extracts knowledge from a PLM’s parameters and subsequently constructs a downstream task around this extracted knowledge. Performance on this task thus depends exclusively on utilizing the model’s possessed knowledge, avoiding confounding factors like insufficient signal. As an instantiation, we study factual knowledge of PLMs and measure utilization across 125M to 13B parameter PLMs. We observe that: (1) PLMs exhibit two gaps - in acquired vs. utilized knowledge, (2) they show limited robustness in utilizing knowledge under distribution shifts, and (3) larger models close the acquired knowledge gap but the utilized knowledge gap remains. Overall, our study provides insights into PLMs’ capabilities beyond their acquired knowledge.

## introduction recent research has demonstrated that language models pre-trained on vast amounts of internet data acquire a broad range of knowledge about linguistic structures @xcite , encyclopedic relations @xcite , levels of commonsense @xcite , and even coding and reasoning rules @xcite . recent studies on behavioral parametric probing and prompting @xcite @xcite has demonstrated that such knowledge, collectively referred to as "parametric knowledge," resides reliably within a subset of trained parameters in pre-trained models (plms). importantly, this knowledge can be identified without additional finetuning. for instance, ## framework 2.1 extract, train, and evaluate principles the primary objective of our evaluation framework is to measure how much of the knowledge present in the model's parameters is actually usable in downstream tasks. ideally, downstream tasks must be designed in a way that solely attributes any success to the model's knowledge being used, while ensuring that failure in performing the task is not due to a lack of pre-training knowledge. ## evaluating the knowledge utilization we separately report the fraction of facts (d) that can be extracted and the downstream performance of models in figure 3 . first, we find that, on par with previous work @xcite , there is a significant gap in the encyclopedic facts the models can correctly predict and the entire facts present in the diagnostic dataset d (figure 3a ). note that one can arbitrarily increase the number of correctly predicted by considering a prediction as correct if the gold entity is among the model's top-k predictions. however, we only consider @xmath0. nonetheless, we find that bert and roberta extract slightly more encyclopedic facts than gpt-neo and opt. critically, all models demonstrate a pronounced gap in downstream task performance, or knowledge utilization, (figure 3b ). this unexpected outcome occurs despite the downstream task being seemingly simple since (1) models are trained and evaluated on examples based on their accurate encyclopedic knowledge predictions, and (2) both k θ train and k θ test are sampled from the same distributions (i.i.d), so the models only encounter seen entities. notably, opt and gpt-neo manage to outperform bert and roberta by a small margin. this finding suggests that models struggle to utilize their entire parametric knowledge in downstream tasks. in the next sections, we investigate the potential causes of this gap. ## robustness of knowledge utilization we intentionally design the downstream task k θ to be straightforward and free of any distributional shift as we want to measure the maximum knowledge utilization of the model. however, in realworld applications, it is likely that the model encounter samples that are different from the training distribution. in this section, we investigate the robustness of knowledge application in the presence of such distributional shifts. ## effect of scaling law on the gaps recent nlp success has come from scaling up pretraining model parameters @xcite . with larger models and increased compute, capabilities such as in-context learning and chain-ofthought reasoning emerge @xcite . the expanded capacity allows these models to absorb more knowledge from pre-training data, improving their usefulness as knowledge sources. however, it remains uncertain if scaling boosts the proportion of pre-training knowledge applicable to downstream tasks. ideally, we like to see a narrowing gap in pre-training knowledge alongside superior knowledge utilization. to investigate this, we evaluate xtraeval on increasing sizes of opt and llama models. specifically, at each scale, we first extract the model's parametric knowledge and then create the downstream task based on it using the same procedure as described in section 2. figure 1 reports the results of this experiment. first, we confirm that a greater fraction of knowledge triples in d can be identified in larger models, the results suggest that while plms, even at small scales, pose considerable knowledge, extracting an equivalent amount of usable knowledge necessitates much larger models. for instance, opt-125m accurately predicts 34% of encyclopedic facts, but only opt-13b (approximately 100× larger) can reliably apply the same volume in downstream tasks. enhanced pre-training routines, including the use of more data or higher quality data, can bolster knowledge acquisition, as is clearly demonstrated by llama models. notably, llama-7b significantly outperforms opt-13b. while llama models possess a greater amount of usable knowledge due to superior initial knowledge, a gap in knowledge utilization remains discernible (figure 7 ). ## discussion lately, pre-trained language models with chatbot interfaces have increasingly been served as knowledge bases @xcite . these chatbots typically employ the model's parametric knowledge to respond to queries and offer information. our study examines the dependability of this knowledge and its impact on downstream task performance. we discover that, regardless of inductive biases, plms face difficulty utilizing their full knowledge in downstream tasks (section 3). this unreliability of parametric knowledge could constrain the concept of "plms as differentiable knowledge bases." additionally, our findings show that the utilization gap persists even with scaling (section 5). notably, while models at each scale capture more knowledge from pre-training data, obtaining the same amount of usable knowledge requires significantly larger models. this exposes a potential constraint in the recent trend of adopting mid-sized plms @xcite . lastly, we discover that knowledge utilization depends on the peculiarities of finetuning data for downstream tasks. specifically, as seen in section 4, plms struggle to apply their knowledge to relation types not encountered during finetuning, even if they accurately predicted such facts in step 1. this generalization gap could highlight challenges within the recent sft-rlhf paradigm @xcite . for instance, the model may only adhere to instructions and excel at tasks resembling the finetuning data. consequently, it might be necessary to meticulously craft finetuning data to activate and utilize all aspects of parametric knowledge in downstream tasks. however, it requires elaborate studies to establish the systematic issues in knowledge application beyond encyclopedic knowledge like procedural and task knowledge. using the parametric knowledge roberts et al. (2020) finetune a pre-trained t5 model for question answering in a closed-book setting and showed that it can perform on par or better than models that use explicit knowledge bases. @xcite made a similar observation for the bart model. more recently, plms are being used to generate facts and documents for knowledge-intensive tasks @xcite @xcite . in this paradigm, in order to answer factual questions, instead of retrieving relevant documents, the model has to first generate the facts and then answer the question with those facts as context. this paradigm shows that the models may not be able to use their parametric knowledge on their own and need explicit grounding to be able to use it. furthermore, there is a plethora of work that investigates whether the model employs its linguistic knowledge when solving downstream language understanding tasks. @xcite shows that roberta does not use its linguistic knowledge for solving nli. instead, it relies on shallow heuristics. @xcite 's observation aligns with this finding and shows the training data used for the downstream task needs to have enough evidence to trigger the model's linguistic knowledge. in our work, we use a more general notation of parametric knowledge and investigate utilization in cases where sufficient evidence is present in the finetuning data. ## conclusion in this study, we presented extract, train, and evaluate (xtraeval ), a framework designed to assess the parametric knowledge of pretrained language models. employing xtraeval we identified a previously unnoticed gap in what models know and how much of it they can actually use. our findings reveal that this gap exists not only in smaller models but also persists in larger ones. additionally, we demonstrate that a distributional shift in finetuning data can result in even larger gaps between the model's knowledge and its practical application in downstream tasks.