D y ERNIE : D ynamic E volution of R iemannian M anifold E mbeddings for T emporal K nowledge G raph C ompletion (2020)

Abstract: There has recently been increasing interest in learning representations of temporal knowledge graphs (KGs), which record the dynamic relationships between entities over time. Temporal KGs often exhibit multiple simultaneous non-Euclidean structures, such as hierarchical and cyclic structures. However, existing embedding approaches for temporal KGs typically learn entity representations and their dynamic evolution in the Euclidean space, which might not capture such intrinsic structures very well. To this end, we propose DyERNIE, a non-Euclidean embedding approach that learns evolving entity representations in a product of Riemannian manifolds, where the composed spaces are estimated from the sectional curvatures of underlying data. Product manifolds enable our approach to better reflect a wide variety of geometric structures on temporal KGs. Besides, to capture the evolutionary dynamics of temporal KGs, we let the entity representations evolve according to a velocity vector defined in the tangent space at each timestamp. We analyze in detail the contribution of geometric spaces to representation learning of temporal KGs and evaluate our model on temporal knowledge graph completion tasks. Extensive experiments on three real-world datasets demonstrate significantly improved performance, indicating that the dynamics of multi-relational graph data can be more properly modeled by the evolution of embeddings on Riemannian manifolds.

## introduction learning from relational data has long been considered as a key challenge in artificial intelligence. in recent years, several sizable knowledge graphs (kgs), e.g. freebase @xcite and wikidata @xcite , have ˚corresponding author. many facts in temporal knowledge graphs induce geometric structures over time. for instance, increasing trade exchanges and economic cooperation between two major economies might promote the trade exports and economic growths of a series of countries in the downstream supply chain, which exhibits a tree-like structure over time. moreover, an establishment of diplomatic relations between two countries might lead to regular official visits between these two countries, which produces a cyclic structure over time. embedding methods in euclidean space have limitations and suffer from large distortion when representing large-scale hier-archical data. recently, hyperbolic geometry has been exploited in several works @xcite as an effective method for learning representations of hierarchical data, where the exponential growth of distance on the boundary of the hyperbolic space naturally allows representing hierarchical structures in a compact form. while most graph-structured data has a wide variety of inherent geometric structures, e.g. partially tree-like and partially cyclical, the above studies model the latent structures in a single geometry with a constant curvature, limiting the flexibility of the model to match the hypothetical intrinsic manifold. thus, using a product of different constant curvature spaces @xcite might be helpful to match the underlying geometries of temporal knowledge graphs and provide high-quality representations. existing non-euclidean approaches for knowledge graph embeddings @xcite lack the ability to capture temporal dynamics available in underlying data represented by temporal kgs. the difficulty with representing the evolution of temporal kgs in non-euclidean spaces lies in finding a way to integrate temporal information to the geometric representations of entities. in this work, we propose the dynamic evolution of riemannian manifold embeddings (dyernie), a theoretically founded approach to embed multi-relational data with dynamic relationships on a product of riemannian manifolds with different curvatures. to capture both the stationary and dynamic characteristics of temporal kgs, we characterize the time-dependent representation of an entity as movements on manifolds. for each entity, we define an initial embedding (at t 0 ) on each manifold and a velocity vector residing in the tangent space of the initial embedding to generate a temporal representation at each timestamp. in particular, the initial embeddings represent the stationary structural dependencies across facts, while the velocity vectors capture the time-varying properties of entities. our contributions are the following: (i) we introduce riemannian manifolds as embedding spaces to capture geometric features of temporal kgs. (ii) we characterize the dynamics of temporal kgs as movements of entity embeddings on riemannian manifolds guided by velocity vectors defined in the tangent space. (iii) we show how the product space can be approximately identified from sectional cur-vatures of temporal kgs and how to choose the dimensionality of component spaces as well as their curvatures accordingly. (iv) our approach significantly outperforms current benchmarks on a link prediction task on temporal kgs in low-and highdimensional settings. @xcite we analyze our model's properties, i.e. the influence of embedding dimensionality and the correlation between node degrees and the norm of velocity vectors. ## conclusion in this paper, we propose an embedding approach for temporal knowledge graphs on a product of riemannian manifolds with heterogeneous curvatures. to capture the temporal evolution of temporal kgs, we use velocity vectors defined in tangent spaces to learn time-dependent entity representations. we show that our model significantly outperforms its euclidean counterpart and other state-of-the-art ap-proaches on three benchmark datasets of temporal kgs, which demonstrates the significance of geometrical spaces for the temporal knowledge graph completion task.