ATAP : Automatic Template-Augmented Commonsense Knowledge Graph Completion via Pre-Trained Language Models (2024)

Abstract: The mission of commonsense knowledge graph completion (CKGC) is to infer missing facts from known commonsense knowledge. CKGC methods can be roughly divided into two categories: triple-based methods and text-based methods. Due to the imbalanced distribution of entities and limited structural information, triple-based methods struggle with long-tail entities. Text-based methods alleviate this issue, but require extensive training and fine-tuning of language models, which reduces efficiency. To alleviate these problems, we propose ATAP, the first CKGC framework that utilizes automatically generated continuous prompt templates combined with pre-trained language models (PLMs). Moreover, ATAP uses a carefully designed new prompt template training strategy, guiding PLMs to generate optimal prompt templates for CKGC tasks. Combining the rich knowledge of PLMs with the template automatic augmentation strategy, ATAP effectively mitigates the long-tail problem and enhances CKGC performance. Results on benchmark datasets show that ATAP achieves state-of-the-art performance overall.

## introduction commonsense knowledge is information that humans typically have that helps them make sense of everyday situations @xcite . commosense knowledge graphs (ckgs) are powerful representation of real-world commonsense knowledge. ckgs such as conceptnet @xcite , atomic @xcite , and dense-atomic @xcite provide a structured way to represent commonsense concepts based on triples, consisting of head nodes, tail nodes and relationship edges (i.e., <head entity, relation, tail @xmath0. unlike traditional knowledge figure 1 : the entity, prompt and @xcite are spliced together to obtain a template, which is input into a plm to complete the prediction task of @xcite . manual template are manually constructed from relations (e.g., atlocation). discrete templates is the optimal template for completing the task, which are selected through a large number of experiments on different templates. continuous template p i (@xmath1, . . . , k), unlike before, does not represent a specific word. it is a continuous, vectorized, trainable pseudo-label that is input into the plm in a special form. however, existing ckgs are typically constructed through manual or semi-automated methods, resulting in most ckgs are far from complete, which affects the performance. accordingly, commonsense knowledge graph completion (ckgc) is proposed to solve the inherent incompleteness problem of ckgs by predicting missing entities in incomplete triples (also called link prediction). existing ckgc work mainly focuses on triplebased methods (e.g., sgbc @xcite and inductive @xcite ), which utilize the structure of ckgs as the only source of infor-mation. but ckgs are usually sparse because there are a large number of entities with in-degree 1 in triples of the ckgs @xcite . these existing methods struggle with long-tail entities @xcite due to limited structural information (information scarcity) and imbalanced entity distributions (long-tail entities are prevalent). pre-trained language models (plms) (e.g., @xcite , @xcite ) have extensive knowledge bases. these huge training text corpus can be used as additional knowledge for many natural language processing tasks @xcite @xcite , and may also be used to alleviate information scarcity for long-tail entities. in particular, plms have proven to be very powerful in predicting @xcite in masked language model (mlm) learning @xcite . the work @xcite provides a reliable evaluation demonstrating the potential of plms to handle traditional knowledge graph completion tasks. they achieve this by converting the traditional knowledge graph completion task into an mlm task, and manually constructing prompt templates to guide the plm to complete the mlm task. therefore, a straightforward approach applied to ckgc might be to incorporate plms with manually constructed prompt templates. however, manual prompt templates have stochasticity in the performance of plms. as shown in figure 1 , the answer to the (oregon, atlocation, ?) using a manual template is incorrect. manually building a top-performing prompt template can be difficult, time-consuming and costly. the recent method in open-world knowledge graph completion domain @xcite proposes to automatically generate discrete prompt templates to complete the prediction of @xcite . however, neural networks are continuous, using discrete prompt templates may be not optimal @xcite . instead of manual or discrete prompt templates used in traditional close/open-world knowledge graph completion tasks, we propose an approach of automatically generating continuous prompt templates as depicted in figure 1 , and leverage it to the commonsense knowledge graph completion. in our study, for the first time, we propose a framework that enhances commonsense knowledge graph completion by automatically generating continuous prompt templates. this framework innovatively extends prompt learning (a strategy of in-context learning @xcite ) into ckgc to convert the triple form of commonsense knowledge in ckgs into the text form that plms are good at processing. compared with the existing triple-based methods, our framework leverages the knowledge in ckgs and plms simultaneously, effectively alleviating the information scarcity of long-tail entities. the main contributions can be summarized as follows: ## atap this section first introduces the definition of ckgc, and then we propose our atap framework, consisting of two key modules: continuous prompt automatic generation and prediction, as depicted in figure 2 and described in detail later. ## experiment in this section, we evaluate the ability of atap on the ckg completion task: link prediction. we compare atap with traditional and commonsense completion models. in addition, we conduct ablation experiments on the atap model. finally, we further analyze each module of atap. ## conclusion in this study, we propose a new framework atap to solve the ckgc task using automatically generated continuous prompts combined with plms. in atap, we propose a method to automatically generate continuous prompt templates to replace traditional manual and discrete templates. moreover, we propose a new prompt template training strategy specifically designed for relations in ckgc, wihch can guide the model to generate an optimal prompt template corresponding to each relation in the ckgc task. extensive experiments on ckgc datasets show that atap achieves state-of-the-art performance overall and reduces the cost of manual labor. experiments on traditional datasets also verify the robustness of our model. in our near future work, we intend to continue investigating the performance of llms on cskg tasks.