A Rotated Hyperbolic Wrapped Normal Distribution for Hierarchical Representation Learning
Abstract
We present a rotated hyperbolic wrapped normal distribution (RoWN), a simpleyet effective alteration of a hyperbolic wrapped normal distribution (HWN).
The
HWN expands the domain of probabilistic modeling from Euclidean to hyperbolicspace, where a tree can be embedded with arbitrary low distortion in theory.
In thiswork, we analyze the geometric properties of the diagonal HWN, a standard choiceof distribution in probabilistic modeling.
The analysis shows that the distributionis inappropriate to represent the data points at the same hierarchy level throughtheir angular distance with the same norm in the Poincaré disk model.
We thenempirically verify the presence of limitations of HWN, and show how RoWN, theproposed distribution, can alleviate the limitations on various hierarchical datasets,including noisy synthetic binary tree, WordNet, and Atari 2600 Breakout.
Introduction
Hyperbolic space has served as an effective medium to learn parsimonious representations of hierarchical data, including vocabulary with relationships (Nickel and Kiela, 2017, 2018; Tifrea et al.
,2019), knowledge graphs (Chami et al., 2020; Sun et al., 2020), and social networks (Zhao et al., 2011;
Shavitt and Tankel, 2008).
Recent studies reveal that the underlying anatomy in much complex datais non-Euclidean, supporting the success of representation learning in hyperbolic space (Bronsteinet al., 2017).
However, due to the absence of well-defined distribution that is easy to sample and hasanalytic density function in hyperbolic space, earlier studies have focused on the non-probabilisticlearning framework.
Hyperbolic wrapped normal distribution (HWN) has been recently proposed as an alternative to thecelebrated normal distribution in Euclidean space.
With an analytic density function and easiness ofsampling, the HWN expands the domain of probabilistic modeling from Euclidean to hyperbolic space.
The HWN has been successfully applied in various probabilistic models, including VAE (Kingma and
Welling, 2014) and probabilistic word embeddings (Vilnis and McCallum, 2015).
However, unlikethe normal distribution in Euclidean space, the geometric characteristics of the HWN have not beenfully understood so far.
Therefore, figuring out what can be done or not with the HWN is difficult.
In this work, we analyze the geometric properties of a diagonal HWN, a standard choice of distributionin many probabilistic models (Mathieu et al., 2019; Nagano et al., 2019).
Based on the observationthat the principal axes of the diagonal normal distribution in Euclidean space have a parallel structurewith the standard bases, we also focus on the structure of the principal axes of HWN in the hyperbolicspace to delve into a deeper understanding.
Our analysis shows that the principal axes of the diagonal
HWN are locally parallel to the standard bases in the Poincaré disk model.
[Figure]
Figure 1 (a) shows a common understanding on learned representation of hierarchical data in Poincarédisk.
The hierarchical structure spreads out like the spokes of a wheel.
The local variation in ahierarchy can then be represented as an angular difference between nodes at the same level of thenorm, i.e., the principal axis representing local variation is orthogonal to the radial axis.
However,our analysis of the geometric property reveals that the local variation can only be modeled along withthe standard bases with the diagonal HWN.
To fix the structure of the principal axes in the diagonal HWN, we propose a simple yet effectivealteration of HWN, named a rotated hyperbolic wrapped normal distribution (RoWN).
By rotating thediagonal covariance matrix before parallel transportation of HWN, we could resolve the limitationsin local variation structure while keeping the valuable properties, such as easy sampling and tractabledensity of the original HWN.
We verify the representation learned with RoWN agrees with the common characteristic of representation in the hyperbolic space, which is barely observable with the diagonal and the full covariance
HWN, by using synthetic noisy binary tree dataset.
We demonstrate the usefulness of RoWN on thebenchmark datasets: WordNet and Atari 2600 breakout.
We summarize our contributions as follows:
We provide an analysis of geometric properties of HWN and its potential limitations in
representation learning.
We propose a novel and efficient method of using hyperbolic distribution, namely RoWN,and apply it to probabilistic
models.3.
We demonstrate the performance of RoWN through the comparison with the Euclideannormal distribution, diagonal covariance HWN, and full covariance HWN on one syntheticdataset and two benchmark datasets.
Preliminaries
This section reviews the wrapped normal distribution defined on the Lorentz model.
We first introducethe Lorentz model of the hyperbolic space and necessary concepts to understand the wrapped normaldistribution.
2.1 The Lorentz model
Hyperbolic space is a non-Euclidean space, having a constant negative Gaussian curvature.
Figure 1illustrates two models for the hyperbolic space, among four standard equivalent models: (1) Klein model, (2) Poincaré disk model, (3) Lorentz (Hyperboloid) model, and (4) Poincaré half-plane model.
In particular, the Lorentz model is famous for its numerical stability in computing the distance andcomes with a simpler closed form of geodesics (Nickel and Kiela, 2018).
The Lorentz model L nis the Riemannian manifold consisting of the set of points z ∈ R n +1 satisfying ⟨ z , z ⟩ L = − 1 andz 0 > 0 , where the Lorentizan product is defined as:
[Equation]
which also works as the metric tensor on the hyperbolic space, i.e., the metric tensor g of Lorentzmodel is g ( x ) =
diag[ − 1, 1, · · · , 1]
2.2 Tangent space of the Lorentz model
Let T x L n be a tangent space of x at L n .
T x L n is a set of points satisfying the orthogonality relationwith x in terms of the Lorentzian product: T x L n := { u : ⟨ u , x ⟩ L = 0 } .
The metric tensor g inducesan inner product of two tangent vectors from a tangent space.
Geodesic γ : [0, 1] → L n generalizesstraight lines in Riemannian manifold which is the shortest curve between two points.
The exponentialmap exp x : T x L n → L n maps a tangent vector u ∈ T x L n onto L n as
[Equation]
such that exp x ( u ) =
y , γ (0) =
x , γ (1) =
y .
The log map, inverse of the exponential map, isdefined as log u ( x ) :
= exp − 1 x ( u ) .
Parallel transport is an operation that transports a tangent vector in the tangent space at x to anothervector in the tangent space at y along the geodesic from x to y without changing its metric tensor.
The parallel transport in the Lorentz model is given by:
[Equation]
where α
= −⟨ x , y ⟩ L .
2.3 Hyperbolic wrapped normal distribution
One of the key challenges in adopting hyperbolic space to probabilistic models is finding a distribution on hyperbolic space that is easy to sample and has a closed-form density function.
Thetwo most common distributions of hyperbolic space used in previous work are Riemannian normaldistribution (Mathieu et al., 2019; Said et al., 2014) and hyperbolic wrapped normal distribution(HWN) (Nagano et al., 2019).
Our work is mainly based on the HWN because sampling in Riemannian normal distribution is limited with only a unit variance.
The sampling process with the HWN follows:
Sample v ∈ R n from a Gaussian distribution N ( 0 , Σ) in Euclidean space.2.
Parallel transport the vector [0, v ] ∈ T
L L n to the tangent space.3.
Project the transported tangent vector to L n using exponential mapping.
The density of a sample can be measured via the change of variable method.
For convenience, wedenote the procedure 2 and 3 as a single operation:
[Equation]
with given µ ∈ L n and v ∈ R n , and 0 L = [1, . . .
, 0] stands for the origin of L n .
[Figure]
Rotated Hyperbolic Wrapped Normal Distribution
This section introduces several observations on the geometric properties of the HWN distributiontransformation from the Euclidean space to the hyperbolic space.
We show the limitations of thea diagonal HWN for representation learning of hierarchical data.
Then, we propose a simple yeteffective modification of the HWN, called a rotated hyperbolic wrapped normal distribution.
3.1 Observations on the hyperbolic wrapped normal distribution
First, we investigate the changes in the normal distribution during the transformation from Euclideanto hyperbolic space by Equation 3.
As the principal axes characterize the covariance structure of thenormal distribution in the Euclidean space, we investigate how the principal axes are transformedin the hyperbolic space.
Before deriving our main proposition, we first show that the straight linesthat pass through the origin in the Euclidean space are transformed to the geodesics in the hyperbolicspace by Equation 3.
Proposition 1.
Suppose ℓ s ( t ) =
t s ∈ R n be a line passing through the origin, where s ∈ R n is adirectional vector.
Then the curve f µ ( ℓ s ( t )) in the Lorentz model L n becomes a geodesic.
The proof of the proposition is provided in Appendix A.
The proposition indicates that every straightline that passes through the origin including the principal axes, is transformed into a geodesic in thehyperbolic space.
Based on the first proposition, we provide our main proposition, which fully characterizes the structureof principal axes when projected to the Poincaré disk model:
Proposition 2.
Let Proj( u ) be the projection function from Lorentz model to Poincaré model, i.e.,
Proj( u ) = x 1: ( u ) x 0 ( u )+1 , ∀ u ∈ L n .
Let ℓ s be a principal axis of the normal distribution defined in R nand µ the mean of HWN in L n , then s is the tangent vector of Proj( f µ ( ℓ s )) on Proj( µ ) .
The proof of the proposition is provided in Appendix B.
The proposition reveals that the principalaxes of the HWN are locally parallel to the standard bases in the Poincaré disk model.
To visualize
[Figure]
the proposition, we plot the contour line and the principal axes of the two-dimensional diagonalnormal distribution before and after the transformation in Figure 2.
We observe that the tangent linesof the transformed principal axes are parallel to the standard bases in the hyperbolic space.
When a popular diagonal normal distribution is employed as a variational distribution, the locallyparallel principal axes might be problematic in learning hierarchical representations.
For example,suppose one tries to represent the variability along the radial direction or in angular differences.
In thecase, both the major (red) and minor (blue) axes in Figure 2b cannot model the variability properly.
3.2 Rotated hyperbolic wrapped normal distribution
Based on the observation, we propose a simple yet effective alternative to the diagonal HWN, arotated hyperbolic wrapped normal distribution (RoWN).
Rotating covariance matrix to the directionof µ enables aligning the major axis of the normal distribution in the Euclidean space to the radialaxis in the hyperbolic space as visualized in the Figure 2.
To construct the distribution, we start with a mean vector µ ∈ L n and a diagonal covariance matrix Σas in the standard HWN.
We change the covariance matrix of the normal distribution as follows:
Compute the rotation matrix R that rotates the x-axis ( [ ± 1, . . .
, 0] ∈ R n ) to µ 1: .
Input Mean µ ∈ L n , diagonal covariance matrix Σ ∈ R n × n Output Sample z ∈ L n 1: x = [ ± 1, . . . , 0] ∈ R n , y = µ 1: / ∥ µ 1: ∥
▷ ± is determined by the sign
Substitute the covariance matrix of Gaussian normal with R Σ R T .
Thus, the rotation matrix R , which rotates a unit vector from x to y , can be computed as:
[Equation]
Note that the construction is straightforward but still keeps the following benefits of the HWN: 1)
Thesampling can be done efficiently, and 2) the computation of the probability density of the samples istractable.
As the HWN provides a tractable probability density function for any kind of covariancematrix, we can easily compute the probability density of a given sample from RoWN.
See Appendix Cfor more details.
We empirically demonstrate the influence of different variational distributions in Figure 3.
Wevisualize the variational distributions of the synthetic binary tree dataset of depth two after traininga hyperbolic VAE.
As the result shows, the model with the diagonal HWN represents the variationin a child parallel to the standard bases, whereas the model with RoWN represents the variation inangular difference.
Please check the detailed description on the synthetic dataset in Section 4.1.
Experiments
In this section, we exhibit three different experiments, where each experiment presents differentbehavior of RoWN in diverse applications.
For all applications, we compare the performance of
RoWN with three baselines, including the normal distribution in the Euclidean space and the diagonal
HWN (Nagano et al., 2019), and the full covariance HWN.
4.1 Noisy synthetic binary tree
A synthetic binary tree dataset is first used to show the performance of representing hierarchy in
Nagano et al.
(2019), where each node in a tree corresponds to a sequence of binary values.
Figure 4ashows an example of the depth three binary tree, where a parent and child only differ in one digit.
Weadd spherical noises to the nodes in the same level of hierarchy as described in Figure 4a as the noisysamples.
With the noisy samples, we can create a dataset containing a local-level variation in thehierarchy.
For the experiments, we uniformly sample the spherical noise from [0, π/ 4] .
We train hyperbolic VAE on noisy synthetic binary tree with varying depths.
The detailed modeldescription is available in Appendix E.1.
We set the latent dimension the same as the depth.
Wereport 1) the correlation between the hamming distance and the embedding distance and 2) thecorrelation between the depth and the Poincaré norm of the embeddings.
The first correlation is [Figure] [Table] computed over all possible pairs of test points.
As Table 1 shows, the full covariance HWN and
RoWN improve the diagonal HWN except depth six, outperforming the Euclidean model in everysetting.
RoWN preserves the depth information better than the other distributions in general.
Weadditionally visualize the variational mean obtained by training the tree of depth three in Figure 4b,where the hierarchical structure is well preserved in the hyperbolic embedding space [Table]
4.2 WordNet
We train a probabilistic word embedding model with WordNet dataset (Fellbaum, 1998).
The datasetconsists of 82,115 nouns and 743,241 hypernymy relationships, where the hypernymy relationshipsexpress the explicit hierarchy between the nouns.
The probabilistic hyperbolic embedding model(Nagano et al., 2019) is employed to learn the representations with the objective of as follows:
[Equation]
where m is a margin, θ i represents the mean and variance parameters of normal distribution
q i definedfor noun i , and s ∼ t and s ̸∼ t denote the presence and absence of hypernymy relation betweenword pair s and t , respectively.
We evaluate the learned representations by computing the average rank of all the hypernymies.
Therank of a given pair of words s and t is computed among the distances between all possible pairs ofthe words s and t ′ without hypernymy.
Table 2 shows the empirical performances of representing [Table] the word data.
We report the performance with the mean rank (MR) and the mean average precision(mAP).
RoWN preserves the hierarchical structure better than the other distributions, while the fullcovariance HWN fails due to the unstable optimization.
4.3 Atari 2600 Breakout
Trajectories of some Atari 2600 games can be structured as a tree-like hierarchy along the timehorizon as Nagano et al.
(2019) points out.
For example, given Atari 2600 Breakout, the root nodecan be the starting state of the game where no blocks have been broken.
As the game progresses thestates of the blocks form hierarchical structure depending on which blocks have been broken.
Tolearn the implicit hierarchy that can be observed from the trajectories of Breakout, we train the VAEmodels with the Atari 2600 Breakout images.
The images of Breakout are collected by using a pre-trained Deep Q-network (Mnih et al.
, 2015)and divided into training set and test set with 90,000 and 10,000 images respectively.
We label eachimage with the score obtained from the game environment.
So the labels are correlated to the numberof broken blocks.
To train VAE, we use a DCGAN-based architecture, which was originally usedto evaluate HWN in Nagano et al.
The detailed architecture is provided in Appendix E.4.
To evaluate the models, we measure the correlation between the Poincaré norm of the test imagesand the labeled scores.
The evaluation results and the generated images from the trained modelsare reported in Table 3 and Figure 5, respectively.
In learning Breakout images, RoWN and the fullcovariance HWN outperform the diagonal HWN.
Especially, RoWN aligns the hierarchical structuresbetter with respect to the norm in low latent dimensions.
Throughout the experiments, we assume the root node to be placed near the origin.
Indeed, theroot node is likely to be located near the origin regardless of the models in our experiments.
In
Appendix D, we discuss what makes the root node to be placed near the origin in detail.
[Figure]
Breakout images.
Every five images are generated from the randomly sampled latent vectors of dimensionality two having Poincaré norm 0.1, 0.9, 0.95.
The larger norm of the sample latent vectors, the more blocks are broken out in the generated images.
[Table]
Related work
Hierarchical representation learning with hyperbolic space.
Earlier studies on learning representation of hierarchical data have focused on embedding the hierarchical structure into Euclideanspace (Nickel et al., 2011; Grover and Leskovec, 2016; Nguyen et al., 2017), which are later shown torequire excessive number of dimensions to embed the original structure without any distortion (Linialet al., 1994).
The hyperbolic space has been proposed as an alternative medium to embed the hierarchical data.
Theoretically, any tree-structured data can be embedded to the hyperbolic space witharbitrarily low distortion (Sarkar, 2011).
Based on the theory, learning representation of hierarchicalstructure in the hyperbolic space has been shown great success in various datasets including WordNethierarchy, graph-structured data, and social network data (Nickel and Kiela, 2017, 2018; Chami et al.
,2020; Sun et al., 2020;
Zhao et al., 2011; Shavitt and Tankel, 2008).
However, these studies are onlylimited to modeling the explicit hierarchy, which means that the dataset contains an explicit relationbetween data points.
Furthermore, the learning frameworks are limited in the non-probabilistic settingbecause of the absence of well-behaved distribution in hyperbolic space.
Distributions in the hyperbolic space.
The probabilistic model enables measuring the uncertaintyand provides a principled way of learning.
To extend the probabilistic learning framework from
Euclidean to hyperbolic space or Riemannian manifold in general, one needs to define a well-behaveddistribution that has a tractable density and is easy to sample from.
Recently, a few studies hasproposed probabilistic learning schemes in the hyperbolic space (Mathieu et al., 2019; Nagano et al.,2019).
Mathieu et al.
(2019) introduces reparametrizable sampling schemes for the two canonical
Gaussian generalizations defined on the Poincaré disk model.
The scheme is used to train a hyperbolic
VAE and show an improved generalization performance with high interpretability.
Nagano et al.(2019) suggests a method of integrating the Bayesian framework with the hyperbolic space in the
Lorentz model where the simpler closed form of geodesics is defined.
Based on Nagano et al.
(2019),we analyze the geometric properties of the distribution lying on the Lorentz model and show thelimitations of the existing method.
Limitations
We explore a better method of representing hierarchical data in the hyperbolic space.
To this end, wepropose a simple yet effective alternative of hyperbolic wrapped normal distribution.
However, theproposed distribution is limited only to the hyperbolic space and no generalization method toward
Rimannian space is studied yet.
To explore the usefulness of alternative Riemannian spaces, finding acommon distribution that can work well in any Riemannian space will be necessary.
RoWN is a subset of the full covariance HWN.
However, in many cases RoWN outperforms the fullcovariance HWN in our experiments.
In general, optimizing the covariance matrix requires to learnthe quadratic number of parameters with respect to the dimensionality.
We conjecture the hardness ofoptimization leads to the poor performance of the full covariance HWN.
To overcome this limitation,a search for a better optimization algorithm in hyperbolic space needs to be explored.
Conclusions
In this work, we propose a novel method of using RoWN for representing the data with a hierarchicalstructure.
With an in-depth analysis of the geometric properties of HWN, we demonstrate why thecommon choice of the diagonal covariance matrix for HWN may be inappropriate but the rotatedcovariance matrix.
Our empirical results present that RoWN exhibits better representation ability,both qualitatively and quantitatively, compared to all the baselines:
Euclidean normal distribution,diagonal HWN, and full covariance HWN.
We hope that our method helps better understanding theanatomy of hyperbolic space and be a promising technique for efficient representation learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] The code isprovided with the supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]