Sharing Parameter by Conjugation for Knowledge Graph Embeddings in Complex Space (2022)

Abstract: A Knowledge Graph (KG) is the directed graphical representation of entities and relations in the real world. KG can be applied in diverse Natural Language Processing (NLP) tasks where knowledge is required. The need to scale up and complete KG automatically yields Knowledge Graph Embedding (KGE), a shallow machine learning model that is suffering from memory and training time consumption issues. To mitigate the computational load, we propose a parameter-sharing method, i.e., using conjugate parameters for complex numbers employed in KGE models. Our method improves memory efficiency by 2x in relation embedding while achieving comparable performance to the state-of-the-art non-conjugate models, with faster, or at least comparable, training time. We demonstrated the generalizability of our method on two best-performing KGE models 5^{\bigstar}\mathrm{E} (CITATION) and \mathrm{ComplEx} (CITATION) on five benchmark datasets.

## introduction a knowledge graph (kg) is a representation of confident information in the real world and employed in diverse natural language processing (nlp) applications, e.g., recommender system, question answering, and text generation. a triple in the form of (head, relation, tail) is widely used as the representation of elements in the kg instead of raw text for scalability. cite (clinician, synset_domain_topic_of, psychology) as an example, clinician and psychology is the head and tail entity respectively, and synset_domain_topic_of is the relation of the head entity pointing to the tail entity. knowledge graph embedding (kge) models are designed for automatic link prediction. relations in kg have multiple categories, e.g., symmetry, antisymmetry, inversion, and hierarchical. missing links indicate incomplete ties between en-tities and are a common phenomenon as finding the missed connections is labor-intensive work. the theoretical space complexity of kge models are often o(n e d e +n r d r ), which is proportional to the number of kg elements, i.e. entities n e and relations n r , and embedding dimension d e , d r respectively. scaling a kg is problematic as n e , n r can go up to millions; also because kge models are often shallow machine learning models composed of simple operations, e.g., matrix multiplication. caution that a shallow model needs a large dimension size d to depict the data feature, yielding the issue of the drastic increase of embedding parameters @xcite . kge models represented using complex numbers have state-of-the-art performance, while they demand high memory costs. e.g., if using one of the best models complex @xcite to create embedding for the benchmark dataset fb15k whose n @xmath0, 951, n @xmath1, 345, and the best-performing dimensionalities d @xmath2, d @xmath3, will result in the parameter size of 65, 184, 000. considering the data type 64-bit integer (signed), who has a size of 8 bytes in pytorch, the memory cost will be 65, 184, 000 × 8 ≈ 497 mb. a kg for real-world application could have a much larger size, e.g., ibm's kg contains @xmath4, which is actively in use and continually growing @xcite , would @xmath5. inspired by the improved performance of complex number representation and non-euclidean models where transformation parameters attempt to interact rather than be independent, we intuited the idea of sharing parameters for memory efficiency. we demonstrate a parameter-sharing method for complex numbers employed in kge models. specifically, our method formulates conjugate parameters in appropriate dimensions of the transformation functions to reduce relation parameters. by using our method, models can reduce their space complexity to o(n e d e + n r d r /2), which means the relation embedding size is half the original model. in the second place, using conjugate parameters may help save training time, especially on the datasets who have more parameter patterns. further, our method can be easily applied to various complex number represented models. we verified our method on two best-performing kge models, i.e., complex @xcite and 5 ⋆ e @xcite . the experiments were conducted on five benchmark datasets, i.e., fb15k-237, wn18rr, yago3-10, fb15k, and wn18, by which we empirically show that our method reserves the models' ability to achieve state-of-the-art results. we also see 31% training time saved on average for 5 ⋆ e in addition to the memory. our method is implemented in pytorch foot_0 and the code with hyperparameter settings foot_1 are available online. ## related works we describe the categorizations of kge models according to the representation method and the vector space that inspired our idea. representation method real and complex number representations are used to quantify entities and relations. translation approaches including transe (bordes et al., 2013) and its variants @xcite describe embeddings using real number representation. although these simple models cost fewer parameters, they can only encode two or three relation patterns, e.g., transe cannot encode symmetric relations. complex @xcite creates embedding with complex number representation, which can handle a wider variety of relations than using only real numbers, among them symmetric and antisymmetric relations @xcite . 5 ⋆ e @xcite utilizes möbius transformation, a projective geometric function that supports multiple simultaneous transformations in complex number representation and can embed entities in much lower ranks. vector space euclidean and non-euclidean spaces are practiced for the calculation of triple plausibility. factorization models such as rescal @xcite and distmult @xcite employ element-wise multiplication in euclidean space. correspondingly, the plausibility of a triple is measured according to the angle of transformed head and tail entities. murp @xcite minimizes hyperbolic distances other than euclidean. it needs fewer parameters than its euclidean analog. atth @xcite leverages trainable hyperbolic curvatures for each relation to simultaneously capture logical patterns and hierarchies. compared with euclidean, the hyperbolic models can save more structures using variational curvatures in different areas to depict hierarchical relations. relational constrain on parameters replacing real number with complex number representation enables the imaginary part to have an effect on the real part parameters, the boosted performance of which indicates the hidden relation among parameter. using hyperbolic space other than euclidean enables the distances or angles at different positions to vary, the increased accuracy hints us to add various constraints on parameters. learning from the work by @xcite , the potential of improving representations through conjugate symmetric constraint is revealed. therefore, we hypothesize the efficiency of relational parameters and propose a parameter-sharing method using conjugate numbers. ## method complex number employed in current kge models enforces multiplicative constraint on representations; our method further adds conjugate constraint within the parameters. note that we don't reduce the dimensions of the parameters, instead, we share the dimensions. we economize 50% of the memory in relation embedding by sharing half of the parameters in the conjugate form. our approach is at least comparable in accuracy to the baselines. in addition, our method reduces calculation in the regularization process, e.g., for the 5 ⋆ ϵ model, 31% of training time is saved on average for five benchmark datasets. ## conclusions we propose using shared conjugate parameters for transformations, which suffices to accurately represent the structures of the kg. our method can help scaling up kg with less carbon footprints easily: first, it reduces parameter size and consumes less or at least comparable training time while achieving consistent accuracy as the non-conjugate model, including reaching state-of-the-art results; second, it is easily generalizable across various complex number represented models.