Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods (2024)

Abstract: Language Models (LMs) acquire parametric knowledge from their training process, embedding it within their weights. The increasing scalability of LMs, however, poses significant challenges for understanding a model’s inner workings and further for updating or correcting this embedded knowledge without the significant cost of retraining. This underscores the importance of unveiling exactly what knowledge is stored and its association with specific model components. Instance Attribution (IA) and Neuron Attribution (NA) offer insights into this training-acquired knowledge, though they have not been compared systematically. Our study introduces a novel evaluation framework to quantify and compare the knowledge revealed by IA and NA. To align the results of the methods we introduce the attribution method NA-Instances to apply NA for retrieving influential training instances, and IA-Neurons to discover important neurons of influential instances discovered by IA. We further propose a comprehensive list of faithfulness tests to evaluate the comprehensiveness and sufficiency of the explanations provided by both methods. Through extensive experiments and analysis, we demonstrate that NA generally reveals more diverse and comprehensive information regarding the LM’s parametric knowledge compared to IA. Nevertheless, IA provides unique and valuable insights into the LM’s parametric knowledge, which are not revealed by NA. Our findings further suggest the potential of a synergistic approach of combining the diverse findings of IA and NA for a more holistic understanding of an LM’s parametric knowledge.

## introduction language models encode the knowledge acquired during training as numeric values within the models' weights, transforming raw information from the training dataset into structured, internal representations. this embedding of knowledge, while and most important neurons n m , n t , n p , . . .. tests for sufficiency (activation of key neurons) and completeness (suppression of the activation of key neurons) -bottom left, alongside fine-tuning with influential training instances -bottom right, assess the attribution methods' fidelity to the model's mechanisms. research in this domain has explored lm's parametric knowledge with various attribution methods @xcite . commonly used among these are instance attribution @xcite ; @xcite ; @xcite ) and neuron attribution (na, @xcite ; @xcite ). ia identifies training instances that influence the model's parametric knowledge leveraged for its prediction on a test instance. however, despite their utility, ia methods have been criticized for their sensitivity to the choice of hyperparameters and for producing highly homogenous results @xcite . on the other hand, na locates specific neurons that hold the most important parametric knowledge for prediction on a test instance. while na has proven valuable in locating and editing the structured knowledge within a model @xcite , the granular nature of the neuron analysis presents challenges in the interpretation of the outcomes, necessitating manually defined human concepts for interpretation @xcite . while ia and na present different views on the parametric knowledge employed in an lm's prediction, no existing work has contrasted the two techniques to determine if they offer similar or complementary insights that could lead to a more comprehensive understanding of an lm's inner workings. this paper seeks to bridge this gap by establishing a unified evaluation framework, illustrated in figure 1 , that allows for the comparison of these disparate attribution methods, particularly focusing on their application for autoregressive lms. ## related work to understand how models encode and utilise parametric knowledge, researchers have employed a variety of approaches @xcite . among these, instance attribution and neuron attribution have emerged as two principal methodologies. ## an evaluation framework for attribution methods we next describe the proposed unified evaluation framework for attribution methods. to allow for the comparison of the two attribution methods ia and na, we first propose to align their results to common views over the lm's parametric knowledge ( §3.2). then, we describe the tests employed in the evaluation framework to assess whether the uncovered methods reflect the reasons employed by the models in their predictions ( §3.3, §3.4). ## analysis next, we investigate what are the characteristics of the group of influential training instances and the group of most important neurons. ## conclusion in this paper, we propose a unified evaluation framework for comparing and contrasting two different attribution methods, ia and na. to do so, we first introduce new attribution methods that align the neurons discovered by ia and na. we assess the sufficiency and comprehensiveness of the explanations for both methods with different faithfulness tests and conduct extensive analyses of their explanations to further investigate the distinct characteristics that yield different results. through the proposed evaluation framework, we confirm that ia and na result in different explanations about the knowledge responsible for the test prediction. this implies the potential advantage of combining ia and na to unveil a comprehensive view of the lm's parametric knowledge. in addition, the experimental results on the attribution methods' faithfulness suggest that the neurons are not sufficient nor comprehensive enough to fully explain the parametric knowledge used for the test prediction. to complement this drawback of the current attribution methods, we hypothesize that this is due to the importance of the attention weights for encoding knowledge, leaving this exploration for future work.