Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning (2022)

Abstract: Pretrained masked language models (PLMs) were shown to be inheriting a considerable amount of relational knowledge from the source corpora. In this paper, we present an in-depth and comprehensive study concerning specializing PLMs into relational models from the perspective of network pruning. We show that it is possible to find subnetworks capable of representing grounded commonsense relations at non-trivial sparsity while being more generalizable than original PLMs in scenarios requiring knowledge of single or multiple commonsense relations.

## introduction the past few years have witnessed the revolution of nlp methods with the advent of pretrained language models (plms) such as bert @xcite and roberta @xcite . they are first pretrained on vast amount of unlabeled text corpora using masked language modeling (mlm) objective and then fine-tuned on task-specific data, offering a surge of improvements on a wealth of nlp tasks. however, we know very little about what and how much knowledge embedded in plms actually contributes to the success. notable endeavors @xcite @xcite toward this understanding focus on probing linguistic knowledge therein. they demonstrated that pretraining did impart useful linguistic abstraction about syntax and semantics into plms. more recently, several works presented intriguing results examining relational knowledge within plms. relational knowledge @xcite is typically defined as describing the abstractive relationship between a pair of concepts or entities, which is crucial for facilitating language understanding. @xcite first posed the lama probe, an english benchmark comprising multiple sets of prompts. each prompt is a cloze-like sentence transformed from a relational knowledge triple: knowledge triple: <bus, hasa, ?> object label: seats. prompt: you are likely to find in a bus. by substituting with a special @xcite token and reusing the mlm head, prompt-based relational knowledge probing provides an estimated lower bound of what plms know without training an additional layer which was used in the previous linguistic probes. they showed that, even without grounded supervision, plms capture such relational knowledge at a level competitive to supervised alternatives. subsequent works further showed that some specific prompts, acquired either through heuristical mining @xcite or gradient-guided search @xcite , can better trigger the models to correctly predict the missing object. despite the mounting evidence for the existence of relational knowledge in plms, it remains unclear how such knowledge is represented internally. in light of this, we raise the core question in this paper: given the general language representation space modeled by a plm, can we extract its latent representation subspaces for different relations and specialize the plm into relation-specific knowledge models? these subspaces exclusively represent knowledge inherited from different subset of mlm data, expressing different relations between masked word and remaining context, thus can potentially benefit applications where knowledge of certain relations are explicitly required. we study this question by first drawing inspiration from recent findings @xcite @xcite : the more mlm pretraining simulates downstream task, the more successful the transfer will be. for example, filling like or hate into a cloze like "i [mask] this film, it's great." provide a clear way in which the model can implicitly learn to perform sentiment classification. similarly, we hypothesize that training on mlm data expressing certain relation r between masked word and remaining context would lead to effective transfer on knowledge probing that targets relation r. by exploiting this correlation conversely, we offer a new way for extracting representation subspaces responsible for different relational knowledge based on their transfer performance on knowledge probing task. instead of introducing additional parametric transformation upon the original space, we restrict these subspaces to have correspondence with subnetworks of plms and propose an end-to-end differentiable weights pruning framework. our experiments show that it is possible to find subnetworks capable of representing grounded commonsense relations at non-trivial sparsity while also exhibiting evident disentanglement. figure 1 exemplifies a cloze prompt where the identified subnetwork produces the valid answer seats by attending to relevant context, i.e., bus, while the original bert fails. we then examine the knowledge transfer ability of these subnetworks in scenarios requiring knowledge of single or multiple commonsense relations for reasoning. experiment on commonsense knowledge base completion show that the identified subnetworks even outperform strong supervised knowledge base completion methods. these subnetworks also outperform the original plms in both many-shot and zero-shot commonsense question answering tasks, when combined properly. code and all pruned subnetworks are opensourced at https://github.com/drsy/ lamp . ## methodology we first provide background on pretrained masked language models and the formulation of cloze prompt for querying these models, then we proceed to elaborate our proposed pruning procedure. ## experiments we present our pruning setup and detailed analysis in section 3.1. then we compare the knowledge transfer ability of pruned subnetworks against original plms in scenarios requiring knowledge of single or multiple relations for reasoning in section 3.2 and section 3.3 respectively. ## related work since the emergence of large pre-trained language models, much work has focused on understanding their internal contextual representations. most prior work @xcite pays attention to either using extraneous probing tasks to examine whether certain linguistic properties can be identified from those representations, or ablating the models to observe how behavior changes. more recently, some studies @xcite have shown the existence of linguistic knowledge (e.g., syntax) in various but generally lower layers of pre-trained transformers. to shed more light on how plms memorize abstract knowledge rather than statistical co-occurrence patterns, we extend previous probe @xcite on relational knowledge. specifically, we are concerned with commonsense knowledge that is grounded on conceptnet relations. our work differs in that we focus on not only probing but also bringing latent relational knowledge to the surface and unleashing more potential for better relation reasoning. another relevant line of research is network pruning @xcite and lottery ticket hypothesis @xcite @xcite . the former aims at reducing the size of model parameters without compromising accuracy and the latter reveals subnetworks whose initializations made them capable of being trained effectively comparable to the original model. in contrast, we seek to uncover subnetworks in over-parametrized plms that specializes on commonsense knowledge tailored for downstream tasks rather than focusing on good global initialization, and achieve good results. ## conclusion this study investigated specializing plms for better relational reasoning via network pruning. in the pilot experiment we find evidence of latent sparse subnetworks capable of representing grounded commonsense relations in various plms. further experiments revealed that such subnetworks possess stronger relational reasoning capability than original plms. our work provides a new vantage point about the internal mechanism as well as practical utilization of relational knowledge in plms, opening up avenues to better understanding and adapting pretrained language representations.