PALT : Parameter-Lite Transfer of Language Models for Knowledge Graph Completion (2022)

Abstract: This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a “fill-in-the-blank” task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-the-art approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters.

## introduction pretrained language models (lm) such as bert and gpt-3 have enabled downstream transfer @xcite . recent studies @xcite @xcite show that the implicit knowledge learned during pretraining is the key to success. among different transfer learning techniques @xcite @xcite , finetuning is the de facto paradigm to adapt the knowledge to downstream nlp tasks. knowledge graph (kg) completion is a typical knowledge-intensive application. for example, given a fact (chaplin, profession, __) missing an entity, it aims to predict the correct entity "screenwriter". this task provides a natural testbed to evaluate the knowledge transfer ability of different transfer learning approaches. finetuning @xcite ) has been recently adopted to advance the kg com-pletion performance. however, it presents two fundamental limitations. first, finetuning is computationally inefficient, requiring updating all parameters of the pretrained lms. this ends up with an entirely new model for each kg completion task. for example, storing a full copy of pretrained bert large (340m parameters) for each task is non-trivial, not to mention the billion parameter lms. second, the finetuning approaches often rely on task-specific architectures for various kg completion tasks. for instance, kg-bert @xcite designs different model architectures to adapt a pretrained bert to different tasks. this restricts its usability in more downstream tasks. in this work, we enable parameter-lite transfer of the pretrained lms to knowledge-intensive tasks, with a focus on kg completion. as an alternative to finetuning, our method, namely palt, tunes no existing lm parameters. we establish this by casting the kg completion into a "fill-in-the-blank" task. this formulation enables eliciting general knowledge about kg completion from pretrained lms. by introducing a parameter-lite encoder consisting of a few trainable parameters, we efficiently adapt the general model knowledge to downstream tasks. the parameters of the original lm network remain fixed during the adaptation process for different kg completion tasks. in contrast to finetuning which modifies all lm parameters, palt is lightweight. instead of designing task-specific model architectures, palt stays with the same model architecture for all kg completion tasks that we evaluate. the contributions are as follows: ## palt we propose parameter-lite transfer learning, called palt, as an alternative to finetuning for knowledge graph (kg) completion. instead of finetuning which modifies all the language model (lm) parameters and stores a new copy for each task, this method is lightweight for kg completion, which keeps original lm parameters frozen, but only tunes a small number of newly added parameters. the intuition is that lms have stored factual knowledge during the pretraining, and we need to properly elicit the relevant knowledge for downstream tasks without much modification to the original lms. to do so, palt first casts kg completion into a "fill-in-the-blank" task (sec. 2.1), and then introduces a parameter-lite encoder consisting of a few trainable parameters, while parameters of the original network remain fixed (sec. 2.2). the overall architecture of palt is shown in figure 1 . ## experiments in this section, we evaluate the parameter-lite transfer ability of palt on two kg completion tasks: triplet classification and link prediction. the details of the experimental setup, datasets, and comparison methods are described in appendix a. ## related work pretrained lms @xcite @xcite @xcite @xcite have achieved state-of-the-art results in many nlp tasks @xcite . some work also uses pretrained lms for knowledge-driven tasks @xcite @xcite by finetuning them on downstream tasks, which has been the de facto method to achieve superior results. however, finetuning modifies all parameters, which requires a large amount of computation and storage resources. recently, prompt-tuning @xcite , adaptors @xcite @xcite , and factual probing @xcite @xcite are developed to transfer pretrained lms to downstream tasks and show improvements on many nlp tasks. @xcite focus on prompt based finetuning, while @xcite follow standard prompt formulation for a single token. compared to the existing methods, there are several distinctive features of palt. first, instead of using the output of a single [mask] token, we leverage the next sentence prediction, which allows the answers with arbitrary lengths. second, we automatically acquire the template using the natural language descriptions of the relations available in the downstream kgs. besides, we also use the corresponding entity descriptions in the cloze statement, providing richer context. third, our method differs from prompt-tuning, which only inserts virtual tokens into the input. fourth, in contrast to the empirical calibration procedures that are highly customized for each task, our method automatically learns a few calibration parameters for each task. overall, compared to previous methods, our approach is lightweight. the parameter-lite encoder is unique and particularly useful for more nlp tasks. ## conclusion we propose palt, a parameter-lite transfer of pretrained language models (lm) for knowledge graph completion. to efficiently elicit general knowledge of lms learned about the task during pretraining, we reformulate kg completion as a "fill-in-theblank" task. we then develop a parameter-lite encoder including two groups of parameters. first, it contains a knowledge prompt encoder consisting of learnable continuous prompt tokens to better recall task-specific knowledge from pretrained lms. second, it calibrates pretrained lms representations and outputs for kg completion via two knowledge calibration encoders. as a result, our method achieves competitive or even better results than finetuning with far fewer tunable parameters. both the task formulation and parameterlite encoder can be inspiring for a wide range of knowledge-intensive tasks and deep lms. we hope this research can foster future research along the parameter-lite knowledge transfer direction in nlp.