Highly Efficient Knowledge Graph Embedding Learning with O rthogonal P rocrustes A nalysis (2021)

Abstract: Knowledge Graph Embeddings (KGEs) have been intensively explored in recent years due to their promise for a wide range of applications. However, existing studies focus on improving the final model performance without acknowledging the computational cost of the proposed approaches, in terms of execution time and environmental impact. This paper proposes a simple yet effective KGE framework which can reduce the training time and carbon footprint by orders of magnitudes compared with state-of-the-art approaches, while producing competitive performance. We highlight three technical innovations: full batch learning via relational matrices, closed-form Orthogonal Procrustes Analysis for KGEs, and non-negative-sampling training. In addition, as the first KGE method whose entity embeddings also store full relation information, our trained models encode rich semantics and are highly interpretable. Comprehensive experiments and ablation studies involving 13 strong baselines and two standard datasets verify the effectiveness and efficiency of our algorithm.

## introduction the recent growth in energy requirements for natural language processing (nlp) algorithms has led to the recognition of the importance of computationally cheap and eco-friendly approaches @xcite . the increase in computational requirements can, to a large extent, be attributed to the popularity of massive pre-trained models, such as language models (e.g., bert @xcite and gpt-3 @xcite ) and knowledge graph embeddings (kges, e.g., sacn @xcite ), that require significant resources to train. a number of solutions have been proposed such as reducing the number of parameters the model contains. for instance, @xcite introduced a distilled version of bert and * chenghua lin is the corresponding author. @xcite decreased the parameters used for training kges with the help of the quaternion. in contrast with previous work, this paper explores algorithmic approaches to the development of efficient kge techniques. knowledge graphs are core to many nlp tasks and downstream applications, such as question answering @xcite , dialogue agents @xcite , search engines @xcite and recommendation systems @xcite . facts stored in a knowledge graph are always in the format of tuples consisting of one head entity, one tail entity (both are nodes in knowledge graphs) and a relation (an edge in knowledge graphs) between them. kges learn representations of relations and entities in a knowledge graph, which are then utilised in downstream tasks like predicting missing relations @xcite @xcite . the application of deep learning has led to significant advances in kge @xcite . nonetheless, such approaches are computationally expensive with associated environmental costs. for example, training the sacn model @xcite can lead to emissions of more than 5.3kg co 2 (for more data of other algorithms, see tab. 2 ). to alleviate the computational cost we introduce procrustes, a lightweight, fast, and eco-friendly kge training technique. procrustes is built upon three novel techniques. first, to reduce the batch-wise computational overhead, we propose to parallelise batches by grouping tuples according to their relations, which ultimately enables efficient full batch learning. second, we turn to a closedform solution for orthogonal procrustes problem to boost the embedding training, which has never been explored in the context of kges. third, to break though the bandwidth bottleneck, our algorithm is allowed to be trained without negative samples. our contribution is three-fold: (1) we introduce three novel approaches to substantially reduce computational overhead of embedding large and complex knowledge graphs: full batch learning based on relational matrices, closed-form orthogonal procrustes analysis for kges, and non-negativesampling training. (2) we systemically benchmark the proposed algorithm against 13 strong baselines on two standard datasets, demonstrating that it retains highly competitive performance with just order-of-minute training time and emissions of less than making two cups of coffee. (3) we successfully encode both entity and relation information in a single vector space for the first time, thereby enriching the expressiveness of entity embeddings and producing new insights into interpretability. ## methodology we propose a highly efficient and lightweight method for training kges called procrustes, which is more efficient in terms of time consumption and co 2 emissions than previous counterparts by orders of magnitude while retaining strong performance. this is achieved by introducing three novel optimisation strategies, namely, relational mini-batch, closed-form orthogonal procrustes analysis, and non-negative sampling training. ## related work kge techniques. in recent years, a growing body of studies has been conducted on the matter of training kges. roughly speaking, these kge methods fall into two categories: distance-based models and semantic matching models. the line of researches regarding distance-based models, which measures plausibility of tuples by calculating distance between entities with additive functions, was initialised the kge technique proposed by @xcite , namely, transe. after that, a battery of follow-ups have been proposed, including example models like transh @xcite , transr @xcite , and transd @xcite . these algorithms have enhanced ability on modelling complex relations by means of projecting entities into different (more complex) spaces or hyper-planes. more recently, a number of studies attempt to further boost the quality of kges through a way of adding orthogonality constraints @xcite for maintaining the relation embedding matrix being orthogonal, which is also the paradigm we follow in the present paper (see § 2). in contrast, semantic matching models measure the plausibility of tuples by computing the similarities between entities with multiplicative functions. such an similarity function could be realised using, for example, a bilinear function or a neural network. typical models in this line includes dist-mult @xcite , complex @xcite , conve @xcite , tucker @xcite , and quate @xcite . ## conclusion in this paper, we proposed a novel kge training framework, namely procrustes, which is eco-friendly, time-efficient and can yield very competitive or even near-state-of-the-art performance. extensive experiments show that our method is valuable especially considering its significant and substantial reduction on training time and carbon footprint.