Fully Hyperbolic Neural Networks (2022)

Abstract: Hyperbolic neural networks have shown great potential for modeling complex data. However, existing hyperbolic networks are not completely hyperbolic, as they encode features in the hyperbolic space yet formalize most of their operations in the tangent space (a Euclidean subspace) at the origin of the hyperbolic model. This hybrid method greatly limits the modeling ability of networks. In this paper, we propose a fully hyperbolic framework to build hyperbolic networks based on the Lorentz model by adapting the Lorentz transformations (including boost and rotation) to formalize essential operations of neural networks. Moreover, we also prove that linear transformation in tangent spaces used by existing hyperbolic networks is a relaxation of the Lorentz rotation and does not include the boost, implicitly limiting the capabilities of existing hyperbolic networks. The experimental results on four NLP tasks show that our method has better performance for building both shallow and deep networks. Our code will be released to facilitate follow-up research.

## preliminaries hyperbolic geometry is a non-euclidean geometry with constant negative curvature k. several hyperbolic geometric models have been applied in previous studies: the poincaré ball (poincaré disk) model @xcite , the poincaré half-plane model @xcite , the klein model @xcite and the lorentz (hyperboloid) model @xcite . all these hyperbolic models are isometrically equivalent, i.e., any point in one of these models can be transformed to a point of others with distance-preserving transformations @xcite . we select the lorentz model as the framework cornerstone, considering the numerical stability and calculation simplicity of its exponential/logarithm maps and distance function. ## experiments to verify our proposed framework, we conduct experiments on both shallow and deep neural networks. for shallow neural networks, we present results on knowledge graph completion. for deep neural networks, we propose a lorentz transformer and present results on machine translation. dependency tree probing is also done on both lorentz and euclidean transformers to compare their capabilities of representing structured information. due to space limitations, we report the results of network embedding and fine-grained entity typing experiments in the appendix a. for knowledge graph completion and network embedding, we use our fully hyperbolic linear layer, and for other tasks, we use the general formula given in §3.1, which is a relaxation of our fully hyperbolic linear layer. in the following sections, we denote the models built with our proposed framework as hy-bonet. we demonstrate that hybonet not only outperforms euclidean and poincaré models on the majority of tasks, but also converges better than its poincaré counterpart. all models in §4.1 are trained with 1 nvidia 2080ti, models in §4.2 are trained with 1 nvidia 40gb a100 gpu. we optimize our model with riemannian adam @xcite . for pre-processing and hyper-parameters of each experiment, please refer to appendix b. ## related work hyperbolic geometry has been widely investigated in representation learning in recent years, due to its great expression capacity in modeling complex data with non-euclidean properties. previous works have shown that when handling data with hierarchy, hyperbolic embedding has better representation capacity and generalization ability @xcite @xcite @xcite @xcite . moreover, @xcite and @xcite introduce the basic operations of neural networks in the poincaré ball and the lorentz model respectively. after that, researchers further introduce various types of neural models in hyperbolic space including hyperbolic attention networks @xcite , hyperbolic graph neural networks @xcite , hyperbolic prototypical networks @xcite and hyperbolic capsule networks @xcite . recently, with the rapid development of hyperbolic neural networks, people attempt to utilize them in various downstream tasks such as word embeddings @xcite , knowledge graph embeddings @xcite , entity typing @xcite ), text classification @xcite , question answering @xcite and machine translation @xcite , to handle their non-euclidean properties, and have achieved significant and consistent improvement. our work not only focus on the improvement in the downstream tasks that hyperbolic space offers, but also show that hyperbolic linear transformation used in previous work is just a relaxation of lorentz rotation, giving a different theoretical interpretation for the hyperbolic linear transformation. ## conclusion and future work in this work, we propose a novel fully hyperbolic framework based on the lorentz transformations to overcome the problem that hybrid architectures of existing hyperbolic neural networks relied on the tangent space limit network capabilities. the experimental results on several representative nlp tasks show that compared with other hyperbolic networks, hybonet has faster speed, better convergence, and higher performance. in addition, we also observe that some challenging problems require further efforts: (1) though we have verified the effectiveness of fully hyperbolic models in nlp, exploring its applications in computer vision is still a valuable direction. (2) though hybonet has better performance on many tasks, it is slower than euclidean networks. also, because of the floatingpoint error, hybonet cannot be sped up with half precision training. we hope more efforts can be devoted into this promising field. decoder @xcite : results following chami et al. (2019), we report roc auc results for link prediction and f1 scores for node classification on four different network embedding datasets. the description of the datasets can be found in our appendix. chami et al. (2019) compute gromovs δ-hyperbolicity(jonckheere et al., 2008; adcock et al., 2013; narayan and saniee, 2011) for these four datasets. the lower the δ is, the more hyperbolic the graph is. the results are reported in appendix a.1. hy-bonet outperforms other baselines in those highly hyperbolic datasets. for disease dataset, hy-bonet even achieves a 12% (absolute) improvement on node classification over previous hyperbolic gcns. on the less hyperbolic datasets such as pubmed and cora, hybonet still performs well on link prediction, and remains competitive for node classification. although hybonet does not significantly better than lgcn on all datasets, we observe that hybonet is far more stable than lgcn. out of 128 link prediction experiments in grid search, there are 89 times that lgcn generates nan and fails to finish training, while hy-bonet remains stable and is faster than lgcn.