MEKER : Memory Efficient Knowledge Embedding Representation for Link Prediction and Question Answering (2022)

Abstract: Knowledge Graphs (KGs) are symbolically structured storages of facts. The KG embedding contains concise data used in NLP tasks requiring implicit information about the real world. Furthermore, the size of KGs that may be useful in actual NLP assignments is enormous, and creating embedding over it has memory cost issues. We represent KG as a 3rd-order binary tensor and move beyond the standard CP decomposition (CITATION) by using a data-specific generalized version of it (CITATION). The generalization of the standard CP-ALS algorithm allows obtaining optimization gradients without a backpropagation mechanism. It reduces the memory needed in training while providing computational benefits. We propose a MEKER, a memory-efficient KG embedding model, which yields SOTA-comparable performance on link prediction tasks and KG-based Question Answering.

## introduction natural language processing (nlp) models have taken a big step forward over the past few years. for instance, language models can generate fluent human-like text without any problems. however, some applications like question answering and recommendation systems need correct, precise, and trustworthy answers. for this goal, it is appropriate to leverage knowledge graphs (kg) @xcite a structured repository of essential facts about the real world. for convenience, the knowledge graph can be represented as a set of triples. a triple is two entities bound with relation and describes the fact. it takes the forms of ⟨e s , r, e o ⟩, where e s and e o represent objects and subject entities, respectively. for efficient use of information from kg, there is a need for the low-dimensional embedding of graph entities and relations. kg embedding models usually use a standard neural networks (nn) backward mechanism for parameter tuning, duplicating its memory consumption. hence, existing approaches to embedding learning have substantial memory requirements and can be deployed only on small datasets under a single gpu card. processing large kgs appropriate for the custom downstream task is a challenge. there are several libraries designed to solve this problem. framework libkge @xcite allows the processing of large datasets by using sparse embedding layers. despite the memory saving, sparse embedding has several limitations -for example, in the pytorch library, they are not compatible with several optimizers. pytorch-biggraph @xcite operates with large knowledge graphs by dividing them into partitions -distributed subgraphs. subgraphs need a place for storing, embedding models need modifications to work with partitions and perform poorly. the main contribution of our paper is a memory-efficient approach to learning knowledge graph embeddings meker (memory efficient knowledge embedding representation). it allows more efficient kg embedding learning, maintaining comparable performance to state-of-the-art models. meker leverages generalized canonical polyadic (cp) decomposition @xcite , which allows a better approximation of given data and analytical computation of the parameters' gradient. meker is evaluated on a link prediction task using several standard datasets and large datasets based on wikidata. experiments show that meker achieves highly competitive results on these two tasks. to demonstrate downstream usability, we create a knowledge base question answering system text2graph and use embeddings in it. the system with meker embeddings performs better as compared to other kg embeddings, such as ptbg @xcite . ## related work there are three types of approaches for learning kg embedding: distance-based, tensor-based, and deep learning-based models. the first group is based on the assumption of translation invariance in the embedding vector space. in model transe @xcite relations are represented as connection vectors between entity representations. transh @xcite implies relation as a hyperplane onto which entities are being projected. quate @xcite extends the idea with hypercomplex space and represents entities as embeddings with four imaginary components and relations as rotations in the space. tensor-based models usually represent triples as a binary tensor and look for embedding matrices as factorization products. rescal @xcite employs tensor factorization in the manner of dedicom @xcite , which decomposes each tensor slice along the relationship axis. distmult @xcite adapts this approach by restricting the relation embedding matrix to diagonal. on the one hand, it reduces the number of relation parameters, on the other hand, it losses the possibility of describing asymmetric relations. the complex @xcite represents the object and subject variants of a single entity as complex conjugates vectors. it combines tensor-based and translationbased approaches and solves the asymmetric problem. tucker @xcite uses tucker decomposition @xcite for finding representation of a knowledge graph elements. this work can also be considered a generalization of several previous link prediction methods. standard canonical polyadic (cp) @xcite decomposition in the link prediction task does not show outstanding performance @xcite . several papers address this problem by improving the cp decomposition approach. simplie (kazemi and poole, 2018) states that low performance is due to different representations of subject and object entity and deploys cp decomposition with dependently learning of subjects and objects matrices. cp-n3 @xcite highlights the statement that the frobenius norm regularizing is not fit for tensors of order more than 3 @xcite and proposes a nuclear p-norm instead of it. our approach also uses cp decomposition with enhancement. we consider remark from simplie and set the object and subject representations of one entity to be equals. at the same time, inside the local step of the cp decomposition algorithm, the matrices of subjects and objects consist of different elements and are different (see appendix). in contradistinction to cp-n3, we do not employ a regularizer to improve training but change the objective. instead of squared error, we use logistic loss, which is appropriate for one-hot data. we abandon the gradient calculation through the computational graph and count gradient analytically, which makes the training process less resource-demanding. approaches based on deep learning convolutions and attention mechanisms conve, gat, gaat @xcite @xcite achieve high performance in link prediction. besides, they have their disadvantages -it necessitate more time and memory resources than other types of models and usually needs pre-training. ## meker: memory efficient knowledge embedding representation our approach to entity embeddings relies on generalized cp tensor decomposition @xcite . namely, r-rank cp decomposition approximates an n-dimensional tensor as a sum of r outer products of n vectors. every product can also be viewed as a rank-1 tensor. this approximation is described by the following formula: x ≈ @xmath0, b, c|], where @xmath1. factors have the following shape @xmath2, @xmath3, @xmath4. the scheme of cp decomposition applied to the kg elements representation task is in figure 1 . we set matrix a equal to matrix c and simultaneously corresponding to subject and object entities. ## conclusion we propose meker, a linear knowledge embedding model based on generalized cp decomposition. this method allows for the calculation of gradient analytically, simplifying the training process under memory restriction. in comparison to previous kg embedding linear models @xcite , our approach achieves high efficiency while using less memory during training. on the standard link prediction datasets wn18rr and fb15k-237, meker shows quite competitive results. in addition, we created a text2graph -kbqa system based on the learned kb embeddings to demonstrate the model's effectiveness in nlp tasks. we obtained the required representations using meker on the wikipedia-based dataset wiki4m for questions in russian and on wiki-data5m for questions in english. text2graph outperforms baselines for english and russian, while using meker's embeddings provides additional performance gain compared to ptbg embeddings. furthermore, our model's link prediction scores on wiki4m and wikidata5m outperform the baseline results. meker can be helpful in question-answering systems over specific kg, in other words, in systems that need to embed large sets of facts with acceptable quality. all codes to reproduce our experiments are available online. 4