Neural Networks in a Product of Hyperbolic Spaces (2022)

Abstract: Machine learning in hyperbolic spaces has attracted much attention in natural language processing and many other fields. In particular, Hyperbolic Neural Networks (HNNs) have improved a wide variety of tasks, from machine translation to knowledge graph embedding. Although some studies have reported the effectiveness of embedding into the product of multiple hyperbolic spaces, HNNs have mainly been constructed in a single hyperbolic space, and their extension to product spaces has not been sufficiently studied. Therefore, we propose a novel method to extend a given HNN in a single space to a product of hyperbolic spaces. We apply our method to Hyperbolic Graph Convolutional Networks (HGCNs), extending several HNNs. Our model improved the graph node classification accuracy especially on datasets with tree-like structures. The results suggest that neural networks in a product of hyperbolic spaces can be more effective than in a single space in representing structural data.

## introduction machine learning that utilizes the properties of noneuclidean spaces has attracted much attention in recent years @xcite . in representation learning on natural language and graphs, where hierarchical data appear, hyperbolic spaces have recently been shown to be effective. in natural language processing (nlp), hyperbolic spaces have been applied to a variety of tasks such as word embedding @xcite , document embedding @xcite , natural language inference @xcite , and machine translation @xcite . hyperbolic spaces have also been shown to be effective in graph embedding @xcite @xcite , which is helpful for nlp models to utilize external knowledge graphs @xcite . recent progress in the use of hyperbolic space is supported by the development of hyperbolic neural networks (hnns) @xcite , which consist of components such as linear and attention layers that are appropriately extended to hyperbolic spaces. how to define linear operations in hyperbolic space is non-trivial, and several different methods have been proposed @xcite . unlike euclidean spaces, a product space of non-euclidean spaces is geometrically different from a single space of the same dimension, and some studies have reported that using the product of small hyperbolic spaces improves the performance in graph and word embedding @xcite . in addition, @xcite observed that the superiority of their hyperbolic machine translation model over the euclidean counterpart is lost as the dimensionality of word features increases, and they proposed using a product of multiple small hyperbolic spaces as a possible solution. however, existing hnn frameworks are defined in a single hyperbolic space, and how to extend hnns to product spaces is still an open question. therefore, this paper proposes a novel method to extend a given hnn in a single space to a product of hyperbolic spaces. more specifically, we construct a general method to extend a hyperbolic matrix-vector multiplication, a major factor that distinguishes hnn variants, to a product space. we apply our method to hyperbolic graph convolutional networks (hgcns) @xcite and show that our method outperforms the baselines especially on datasets with tree-like structures, suggesting that neural networks in a product of hyperbolic spaces are more effective for representing structural data than neural networks in a single hyperbolic space. ## proposed method neural network layers can be considered to be composed of basic operations: vector addition, scalarvector multiplication, matrix-vector multiplication, and nonlinear activation functions. in this section, we introduce how to extend the basic operations in hyperbolic neural networks to the product space of m hyperbolic spaces, @xmath0. here we will treat the case where h is b (poincaré ball model), and all the curvatures are the same. ## conclusion we proposed a general method to extend existing single-space hnn architectures to a product space. we applied our method to hgcn and conducted experiments across several graph datasets and hnn architectures. the results show that models using a product of hyperbolic spaces perform better on treelike datasets than models using a single hyperbolic space especially in the node classification task.