Multilingual Knowledge Graph Completion from Pretrained Language Models with Knowledge Constraints (2023)

Abstract: Multilingual Knowledge Graph Completion (mKGC) aim at solving queries in different languages by reasoning a tail entity thus improving multilingual knowledge graphs. Previous studies leverage multilingual pretrained language models (PLMs) and the generative paradigm to achieve mKGC. Although multilingual pretrained language models contain extensive knowledge of different languages, its pretraining tasks cannot be directly aligned with the mKGC tasks. Moreover, the majority of KGs and PLMs currently available exhibit a pronounced English-centric bias. This makes it difficult for mKGC to achieve good results, particularly in the context of low-resource languages. To overcome previous problems, this paper introduces global and local knowledge constraints for mKGC. The former is used to constrain the reasoning of answer entities , while the latter is used to enhance the representation of query contexts. The proposed method makes the pretrained model better adapt to the mKGC task. Experimental results on public datasets demonstrate that our method outperforms the previous SOTA on Hits@1 and Hits@10 by an average of 12.32% and 16.03%, which indicates that our proposed method has significant enhancement on mKGC.

## introduction knowledge graphs are collections of entities and facts, and utilized as a valuable resource in a variety of natural language processing (nlp) tasks, such as question answering and recommender systems @xcite @xcite . the language-specific nature of many nlp tasks necessitates to consider the knowledge our approach exhibits superior consistency and accuracy in generating answers. there is a significant amount of potential facts that have not been captured in current knowledge graphs, resulting in their incompleteness @xcite . to address this issue, various stud-ies have proposed for knowledge graph completion (kgc) to automatically discovery potential facts through observed facts @xcite , rules @xcite and language models @xcite . in fact, as shown in figure 1 , there is more english-centric knowledge than other languages, so that it is difficult to leverage knowledge graphs on non-english tasks. for example, english-centric commonsense reasoning tasks obtain better development and performance than other languages @xcite . and the knowledge coverage of non-english knowledge graphs is even worse, it will poses challenges for traditional kgc methods to achieve superior performance. nowadays, pretrained language models (plms) learn various knowledge modeling capabilities @xcite from massive unlabeled data. and most studies have demonstrated that the knowledge contained within plms can significantly improve the performance of downstream tasks @xcite . most recently, prix-lm @xcite approached mkgc as an end-to-end generative task using multilingual plms. for example, for predicting the missing entity of the query (86th all japan football championship, stadium, ?) (see figure 1 ), prix-lm converts the query into a sequence with pre-defined template, which is then processed by an encoder to generate a query representation. the decoder then uses this representation to generate the final answer axis bird stadium. despite the successes achieved through the combination of plms and the generative paradigm, there remain limitations for mkgc. on the one hand, the gap between the pretraining task and the kgc task may contribute to the limitations. it arise that the answers generated by prix-lm are ambiguous in type. on the other hand, languages and tokens that occur more frequently in the pretraining data have richer representations. linguistic bias for kgs and plms would arise that entities in low-resource languages are difficult to be represented, resulting answer incorrect. as illustrated in figure 1 , the query (86th all japan football championship, stadium, ?) expects a response of the type stadium, but the top-ranked answers from prix-lm are diverse, and the top answer is incorrect. we argue that the incorporation of knowledge constraints into the generation process can increase plms suitability for mkgc tasks. we categorize knowledge effective for mkgc into global and local knowledge. global knowledge limit the types of answers based on building the relationship of entity and relation representations. this helps to ensure that the generated answers are semantically and logically consistent with the intent of query. on the other hand, local knowledge in plms can enhance the ability to comprehend the interconnections between the sub-tokens within the query. this helps the model to better understand the context of query and generate more accurate answers. incorporating knowledge constraints into the generative process brings two advantages for mkgc: 1) it makes plms to better adapt to mkgc task. 2) it enables plms to learn more effective representations from low-resource data. in this paper, we propose to incorporate the global and local knowledge into the answer generation process through two knowledgeable tasks. to learn global knowledge, special tokens ( since the same special token is used in each triple in different languages, trained models are able to learn knowledge reasoning ability beyond language boundaries. to capture local knowledge, we consider the representation of answer and each word of query as two separate distributions p (h q ) and p (h [t ] ), and then use an estimator to estimate and maximize the mutual information between them i(h q ; h @xcite ). the local knowledge serves to augment the query representations for trained model through the utilization of minimal amounts of data. the experimental results on seven language knowledge graph from dbpedia show that our proposed method achieves significant improvement as compared to prix-lm and translated-based methods. we publicize the dataset and code of our work at https: //github.com/maxpa1n/gcplm-kgc . in short, our main contributions are as follows: ## basic model a knowledge graph @xmath0, e) is a collection of connected information about entities, often represented using triples (h, r, t) where @xmath1, @xmath2. prix-lm is an important work of mkgc and is also used as the basic model in this paper. prix-lm transfer link prediction from discriminative task to generative task for mkgc. the goal of mkgc is to generate the missing tail entity, which may contain multiple tokens, for the query (h, r, ?) of different languages. the use of template is employed as a means of transforming queries into textual sequences that can be encoded by plms. the template includes special tokens, which serve to identify the specific role of each element within the query triple: the training goal is to generate the tail entity x t by giving the sequence containing the head entity x h and relation x r . for example, for the query (le-bron james, team member of, ?), the constructed sequence @xmath3, and the target of mkgc is generate los angeles lakers @xcite . the process is as follows: the utilization of plms for generating answers directly can be subject to language bias, resulting in ambiguous and incorrect answers. the representation of the special token [t ] is a crucial factor in determining the quality of the generated answers. to improve the representation of the [t ] token, we have implemented two supplementary strategies aimed at incorporating additional knowledge into its representation. ## the proposed model in this section, we describe the components of our proposed approach. the architecture of the model is depicted in figure 2 . our approach comprises four key components: a query encoder, a global knowledge constraint, a local knowledge constraint, and an answer generation module. these components operate in tandem to generate accurate and coherent answers for given queries. ## experiments in this section, we evaluate the effectiveness of our approach on tasks related to mkgc and entity alignment for mkgs. to further understand the role of the various knowledge-gathering strategies in our method, we also conduct ablation experiments. additionally, we provide case studies to demonstrate the superior performance of our method on specific examples. these experiments and analyses provide insight into the strengths and limitations of our approach for addressing challenges in mkgc for sparse knowledge graphs. the embedding-based methods, including transe, complex, and rotate, were implemented using the openke framework @xcite . the results for these methods were obtained by training separate knowledge graphs for each language. the single make the monolingual version, which is trained independently for each language. the numbers in bold represent the best results among the methods and languages considered. ## conclusion our work improve the multilingual knowledge graph completion performance base on plm and generative paradigms. we propose two two knowledgeable tasks to integrate global and local knowledge into answer generation given a query. the global knowledge improves the type consistency of the generated answers. local knowledge enhances the accuracy of answer generation. we conducted experiments and the results showed that the proposed method is better than the previous model.