L APLACIAN E IGENSPACES , H OROCYCLES AND N EURON M ODELS ON H YPERBOLIC S PACES
Anonymous authors
Paper under double-blind review
A BSTRACT
We use hyperbolic Poisson kernel to construct the horocycle neuron model on hyperbolic spaces, which is a spectral generalization of the classical neuron model.
We prove a universal approximation theorem for horocycle neurons.
As a corollary, we obtain a state-of-the-art result on the expressivity of f 1 a,p , which is used in the hyperbolic multiple linear regression.
Our experiments get state-of-the-art results on the Poincare-embedding subtree classiﬁcation task and the classiﬁcation accuracy of the two-dimensional visualization of images
I NTRODUCTION
Conventional deep network techniques attempt to use architecture based on compositions of simple functions to learn representations of Euclidean data (LeCun et al., 2015).
They have achieved remarkable successes in a wide range of applications (Hinton et al., 2012;
He et al., 2016).
Geometric deep learning, a niche ﬁeld that has caught the attention of many authors, attempts to generalize conventional learning techniques to non-Euclidean spaces (Bronstein et al., 2017; Monti et al., 2017).
There has been growing interest in using hyperbolic spaces in machine learning tasks because they are well-suited for tree-like data representation (Ontrup & Ritter, 2005; Alanis-Lobato et al., 2016;
Nickel & Kiela, 2017; Chamberlain et al., 2018; Nickel & Kiela, 2018; Sala et al., 2018; Ganea et al., 2018b; Tifrea et al., 2019; Chami et al., 2019; Liu et al., 2019; Balazevic et al., 2019; Yu & Sa, 2019; Gulcehre et al., 2019; Law et al., 2019).
Many authors have introduced hyperbolic analogs of classical learning tools (Ganea et al., 2018a; Cho et al., 2019; Nagano et al., 2019; Grattarola et al., 2019; Mathieu et al., 2019; Ovinnikov, 2020;
Khrulkov et al., 2020;
Shimizu et al., 2020).
Spectral methods are successful in machine learning, from nonlinear dimensionality reduction (Belkin & Partha, 2002) to clustering (Shi & Malik, 2000; Ng et al., 2002) to hashing (Weiss et al., 2009) to graph CNNs (Bruna et al., 2014) to spherical CNNs (Cohen et al., 2018) and to inference networks (Pfau et al., 2019).
Spectral methods have been applied to learning tasks on spheres (Cohen et al., 2018) and graphs (Bruna et al., 2014), but not yet on hyperbolic spaces.
This paper studies a spectral generalization of the FC (afﬁne) layer on hyperbolic spaces.
Before presenting the spectral generalization of the afﬁne layer, we introduce some notations.
Let ( · , · ) E be the inner product, | · | the Euclidean norm, and ρ an activation function.
The Poincaré ball model of the hyperbolic space H n ( n ≥ 2) is a manifold { x ∈ R n : | x | < 1 } equipped with a Riemannian metric ds 2 H n = (cid:80)
ni
=1 4(
−| x | 2 ) − 2 dx 2 i .
The boundary of H n under its canonical embedding in R n is the unit sphere S n − 1 .
The classical neuron y =
ρ (( x, w ) E + b ) is of input x ∈ R n , output y ∈ R , with trainable parameters w ∈ R n , b ∈ R .
An afﬁne layer R n → R m is a concatenation of m neurons.
An alternative representation of the neuron x (cid:55)→ ρ (( x, w ) E + b ) is given by
[Equation]
This neuron is constant over any hyperplane that is perpendicular to a ﬁxed direction ω .
In H n , a horocycle is a n − 1 dimensional sphere (one point deleted) that is tangential to S n − 1 .
Horocycles are hyperbolic counterparts of hyperplanes (Bonola, 2012).
Horocyclic waves (cid:104) x, ω
[Equation]
H
[Equation]
= 12 log
[Equation]
−|
[Equation]
2 |
[Equation]
ω |are constant over any horocycle that is tangential to S n − 1 at ω .
Therefore,
[Equation]
generalizes the classical neuron model (1), and a concatenation of ﬁnitely many (2) generalizes the
FC (afﬁne) layer.
We call (2) a horocycle neuron.
Figure 1 (middle) is an example on H 2 .
The neuron models in (1, 2) are related to spectral theory because ( · , ω ) E (respectively (cid:104)· , ω (cid:105) H ) are building blocks of the Euclidean (respectively hyperbolic) Laplacian eigenspace.
Moreover, many
L 2 spaces have a basis given by Laplacian eigenfunctions (Einsiedler & Ward, 2017).
On one side, all Euclidean (respectively hyperbolic) eigenfunctions are some kind of “superposition” of ( · , ω ) E (respectively (cid:104)· , ω (cid:105) H ).
On the other side, neural networks based on (1) (respectively (2)) represent functions that are another kind of “superposition” of ( · , ω ) E (respectively (cid:104)· , ω (cid:105) H ).
They heuristically explain why the universal approximation property is likely to hold for networks constructed by (1) and (2).
By using the Hahn Banach theorem, an injectivity theorem of Helgason, and integral formula, we prove that ﬁnite sums of horocycle neurons (2) are universal approximators (Theorem 2).
Let p ∈ H n , T p ( H n ) be the tangent space of H n at p , a ∈ T p ( H n ) , ⊕ be the Möbius addition (Ungar, 2008).
We remind the reader that the following functions
[Equation]
are building blocks of many hyperbolic learning tools (Ganea et al., 2018a; Mathieu et al., 2019;
Shimizu et al., 2020).
Figure 1 illustrates examples of different neuron models (1, 2, 3) on H 2 .
In Lemma 1, we shall present a close relationship between (2) and (3).
Using this relationship and
Theorem 2, we obtain a novel result on the expressivity of f 1 a,p (Corollary 1).
This article contributes to hyperbolic learning.
We ﬁrst apply spectral methods, such as the horocycle, to hyperbolic deep learning.
We prove results on the expressivity of horocycle neurons (2) and f 1 a,p (3).
In experiments, we obtain state-of-the-art results on the Poincaré-embedding subtree classiﬁcation task and the classiﬁcation accuracy of the 2-D visualization of images.
R ELATED WORK
Universal approximation There is a vast literature on universal approximation (Cybenko, 1989;
Hornik et al., 1989; Funahashi, 1989; Leshno et al., 1993).
Cybenko (1989)’s existential approach uses the Hahn Banach theorem and Fourier transform of Radon measures.
To prove Theorem 2, we also use the Hahn Banach theorem, and additionally an integral formula (7) and an injectivity Theorem 1 of Helgason.
Generalizing integral formulas and injectivity theorems is easier than generalizing
Fourier transform of Radon measures on most non-Euclidean spaces.
(Carroll & Dickinson, 1989) uses the inverse Radon transform to prove universal approximation theorems.
This method relates to ours, as injectivity theorems are akin to inverse Radon transforms.
However, using the injectivity theorem is an existential approach while using the inverse Radon transform is a constructive one.
Spectral methods Spectral methods in Bronstein et al.
; Bruna et al.
; Cohen et al.
(2018) use a basis of L 2 ( X ) given by eigenfunctions, where X is a ﬁnite graph or the sphere.
Because
L 2 ( H n ) has no eigenfunctions as a basis, our approach is different from theirs.
Hyperbolic deep learning One part of the hyperbolic learning concerns embedding data into the hyperbolic space (Nickel & Kiela, 2017; Sala et al., 2018).
Another part concerns learning architectures with hyperbolic data as the input (Ganea et al.
(2018a);
Cho et al.
Ganea et al.
(2018a) proposes two ways to generalize the afﬁne layer on hyperbolic spaces: one by replacing the linear and bias part of an afﬁne map with (25, 26) of their paper; another one by using a concatenation of f 1 a,p in their hyperbolic multiple linear regression (MLR).
The latter seems more relevant to ours.
A level set of f 1 a,p is a hypercycle that has the same distance to a chosen geodesic hypersurface, while a level set of a horocycle neuron is a horocycle that has the same “spectral” distance to an ideal point at inﬁnity.
Based on functions similar to f 1 a,p , Mathieu et al.
; Shimizu et al.
(2020) build the gyroplane layer and Poincaré FC layer.
Ganea et al.
(2018a);
Cho et al.
(2019) take geodesics as decision hyperplanes, while we (initially) take horocycles.
We shall construct the horocycle multiple linear regression (MLR), where decision hypersurfaces are geodesics.
Geodesics decision hyperplanes (Ganea et al., 2018a; Cho et al., 2019) and geodesic decision hypersurfaces here arise from different methods.
Khrulkov et al.
(2020) investigates the hyperbolic image embedding, where prototypes (or models) of each class are center-based.
We study a different one, and we shall call our prototypes end-based.
H YPERBOLIC SPACES
This section reviews facts from hyperbolic geometry that are used in the proof of Theorem 2.
For the reader who is not interested in the proof, (4) is enough for the implementation.
Hyperbolic metric We use the Poincaré model.
The hyperbolic space H n ( n ≥ 2) is the manifold { x ∈ R n : | x | < 1 } equipped with a Riemannian metric ds 2 = (cid:80) n i
=1 4(
−| x | 2 ) − 2 dx 2 i .
Let o be the origin of H n .
The distance function d H n satisﬁes d H n ( o, x )=2 arctanh | x | .
Geodesics, horocycles and corresponding points Geodesics in H n are precisely circular arcs that are orthogonal to S n − 1 .
Horocycles in H n are precisely ( n − 1) -dimensional spheres that are tangential to S n − 1 (Helgason, 1970).
Horocycles are hyperbolic analogs of hyperplanes.
Figureillustrates geodesics and horocycles on H 2 .
Hyperbolic Poisson kernel The Poisson kernel for H n is
[Equation]
where x ∈ H n , ω ∈ S n − 1 (Helgason (1970)[p.108]).
The function (cid:104)· , ω (cid:105) H deﬁned by
[Equation]
is constant over any horocycle that is tangential to S n − 1 at ω (Figure 1 (middle), (6)).
Riemannian volume The Riemannian volume induced by the metric ds 2 on H n is
[Equation]
Horocycles Let Ξ be the set of horocycles of H n , and let Ξ ω be the set of all horocycles that are tangential to S n − 1 at ω .
Given λ ∈ R , we let ξ λ,ω be the unique horocycle that connects ω and tanh ( λ/ 2) · ω .
We have Ξ ω = ∪ λ ∈ R { ξ λ,ω } and Ξ = ∪ ω ∈ S n − 1 Ξ ω .
The length of any geodesic (that ends at ω ) line segment cut by ξ λ 1 ,ω and ξ λ 2 ,ω equals | λ 1 − λ 2 | (A.2).
Therefore | λ 1 − λ 2 | is a natural distance function deﬁned on Ξ ω , and the map λ → ξ λ,ω is an isometry between R and Ξ ω .
This isometry is closely related to (cid:104)· , ω (cid:105) H (A.3): for any x ∈ ξ λ,ω ,
[Equation]
The annoying / 2 in (6) is a tradeoff that the metric here is different from that in Helgason (2000).
Integral formula For ﬁxed ω ∈ S n − 1 , H n = ∪ λ ∈ R ξ λ,ω .
Let d Vol ξ λ,ω be the measure induced by ds 2 on ξ λ,ω .
Let L be a family of geodesics that end at ω , δ > 0 , and U =
L ∩ ( ∪ λ ≤ α ≤ λ + δ ξ α,ω ) .
For l ∈ L , d H ( l ∩ ξ λ,ω , l ∩ ξ λ + δ,ω )
=
δ (A.2), hence d Vol ( U ) =
δ · d Vol ξ λ,ω ( U ∩ ξ λ,ω ) and therefore
[Equation]
The above proof (for H n ) is essentially the same as that in (Helgason, 2000)[p.37]
(for H 2 ).
To further convince the reader that (7) holds for all n , we give another simple proof in A.4.
Injectivity theorem With respect to the canonical measure on Ξ , Helgason (1970)[p.13] proved
Theorem 1 (Helgason) .
If f ∈ L 1 ( H n ) and (cid:82) ξ f ( z ) d Vol ξ ( z ) = 0 for a.e ξ ∈ Ξ , then f = 0 a.e..
Theorem 1 demonstrates that if the integral of f ∈ L 1 ( H n ) over almost every horocycle is zero then f is also zero.
This theorem and the integral formula (7) are essential for the proof of Theorem 2.
L EARNING ARCHITECTURES AND EIGENFUNCTIONS OF THE L APLACIAN
In this section, we discuss a heuristic connection between the representation properties of eigenfunctions and classical neurons, and then we deﬁne some horocycle-related learning tools.
4. E IGENSPACES AND NEURON MODELS
On a Riemannian manifold X , the Laplace-Beltrami L X is the divergence of the gradient, and its representation property is well-known (Einsiedler & Ward, 2017): if X is a compact Riemannian manifold or bounded domain in R n , then L 2 ( X ) has a basis given by eigenfunctions.
This statement is false if X is R n or H n (Hislop, 1994).
Eigenspaces of on R n and H n
Our work is motivated by the theory of eigenspaces, in which Euclidean (respectively hyperbolic) eigenfunctions are obtained from ( x, ω ) E (respectively (cid:104) x, ω (cid:105) H ) by some kind of superposition.
For example, all smooth eigenfunctions of L R n are precisely the functions (M. Hashizume & Okamoto, 1972)[p.543]
[Equation]
and eigenfunctions of L H n are precisely the functions (Helgason, 1970)[Theorem 1.7, p.139]
[Equation] [Equation]
) where T in (8) and (9) are some technical linear forms of suitable functional spaces on S n − 1 .
Neuron models By (8) and (1), Euclidean eigenfunctions (respectively classical neurons) are superpositions of ( · , ω ) E and exp (respectively ρ ), with homogeneity and additivity.
By (9) and (2), hyperbolic eigenfunctions (respectively horocycle neurons) are superpositions of (cid:104)· , ω (cid:105) H and exp (respectively ρ ).
The representation property of eigenfunctions on compact manifolds suggests that the universal approximation property is likely to hold for networks constructed by ( · , ω ) E or (cid:104)· , ω (cid:105) H .
However, this heuristic is not proof (A.5).
4. H OROCYCLE BASED LEARNING ARCHITECTURES
Horocycle neuron
In the implementation of the horocycle neuron (2), we take 12 log (cid:16) 1 −| x | 2 | x − ω | 2 + (cid:15) + (cid:15) (cid:17) for (cid:104) x, ω (cid:105) H , where (cid:15) is a small constant to ensure numerical stability.
For updating ω , we use the sphere optimization algorithm (Absil et al., 2008; Bonnabel, 2013) (A.6).
Horocycle feature and horocycle decision hypersurface Given a non-origin point x ∈ H n , for y ∈ H n we deﬁne h x ( y )
=
(cid:104) y,
x/
| x |(cid:105) H and call it the horocycle feature attached to x .
This feature is useful in the Poincaré embedding subtree classiﬁcation task (see the experiment and Figure 3[left]).
The horocycle is the hyperbolic analog of the Euclidean hyperplane, and therefore it could be a possible choice of decision hypersurface, which may arise from a level set of a horocycle feature.
End-based clustering and end prototypes Natural clustering is a topic in representation learning (Bengio et al., 2013), and the common prototype-based clusters are center-based (Tan et al., 2005).
We propose a type of clustering that embeds high-dimensional data in H n and places prototypes in S n − 1 .
Figure 3[right] is an example for n = 2 .
For ω ∈ S n − 1 and any b ∈ R , the function x ∈ H n (cid:55)→ − log (cid:16) 1 −| x | 2 | x − ω | 2 (cid:17) + b measures the relative distance of H n from ω in Gromov’s bordiﬁcation theory (Bridson & Haeﬂiger, 2009)[II.8].
Moreover, we deﬁne Dist : H n × S n − 1 × R → R by
[Equation]
It is a relative distance function, and this is why Dist may assume negative values and why there is a bias term b in (10).
Consider classes
Cls = { C 1 , C 2 , . . .
, C M } and labeled training examples { ( X 1 , Y 1 ) , . . .
, ( X N , Y N ) } , where X i ∈ R D are D -dimensional input features and
Y i ∈ { 1 , 2 , . .
, M } .
Each example X i belongs to the class C Y i .
In light of (10), our goal is to ﬁnd a neural network NN θ : R D → H n that is parameterized by θ , prototypes ω 1 , . . .
, ω M ∈ S n − 1 , and real numbers b 1 , . . .
, b M ∈ R such that
[Equation]
is maximized.
We call { NN θ ( X j ) : 1 ≤ j ≤ N } the end-based clustering and ω i end prototypes (in hyperbolic geometry, the end is an equivalence class of parallel lines in Figure 2[left]).
In experiments, we take NN θ
=
Exp ◦ NN (cid:48)
θ , where NN (cid:48) θ : R D
→ R n is a standard neural network parameterized by θ and Exp : R n → H n is the exponential map of the hyperbolic space.
Horocycle layer, horocycle multiple linear regression (MLR) and geodesic decision hypersur- faces We call a concatenation of (2) a horocycle layer, and we shall carefully describe a prototypical learning framework for end-based clusterings.
Using the same notions as in the previous paragraph, the classiﬁcation task has M classes, and NN θ =
Exp ◦ NN (cid:48) θ : R D
→ H n is a deep network.
For prototypes ω 1 , . . .
, ω M ∈ S n − 1 , real numbers b 1 , . . .
, b M ∈ R , and any example X , our feedforward for prediction will be
[Equation]
The goal is to maximize the accuracy (11), and then we need a loss function for the backpropagation.
Following the convention of prototypical networks (Snell et al., 2017; Yang et al., 2018), we choose an increasing function ρ (in our experiments, ρ ( x ) = x or ρ = tanh . 2 ) and let the distribution over classes for an input X (with label Y ) be
[Equation]
Therefore, given a batch of training examples, the loss function is
[Equation]
The training proceeds by minimizing L , and we call this framework a horocycle MLR.
The set of parameters of the framework is { θ } ∪ { ω 1 , . . .
, ω M } ∪ { b 1 , . . .
, b M } .
It is worth mentioning that decision boundaries of the horocycle MLR are geodesics, which follows from
[Equation]
and the theorem of Apollonian circles (A.7).
Poisson neuron and Poisson multiple linear regression (MLR)
Although (cid:104) x, ω (cid:105) H (4) is well- motivated by the theory of eigenspaces (9) and ﬁts naturally into metric learning (see 10 or also
Corollary 1), it is only deﬁned on H n .
Some reader might not be convinced that the neuron has to be deﬁned on hyperbolic spaces.
Therefore, we try to remove the log in (4) and deﬁne the Poisson neuron model by P ρw,λ,b ( x ) =
ρ (cid:16) λ | w | 2 −| x | 2 | x − w | 2 + b (cid:17) for w ∈ R n , λ, b ∈ R , which is well-deﬁned on R n \{ w } .
Notice that if | x | < | w | then | w | 2 −| x | 2 | x − w | 2 = [Equation] [Equation] x
[Equation] | w | [Equation] w/ | w |
[Equation]
H .
In A.8, Figure 7 illustrates an example of Poisson neuron on R 2 .
In the implementation, we take | w | 2 −| x | 2 | x − w | 2 + (cid:15) for | w | 2 −| x | 2 | x − w | 2 , where (cid:15) is a small constant for numerical stability.
We call a concatenation of Poisson neurons a Poisson layer,
Suppose the classiﬁcation task hasclasses.
Let NN θ : R D → R 2 be the feature descriptor, X the input, and x =NN θ ( X ) .
Let w i =( cos
( i
5) , sin
Left when score functions are and we use it with a deep neural network NN θ : R D
→ R n to construct the Poisson MLR, which is similar to the horocycle MLR.
Let w 1 , . .
. , w M ∈ R n and b 1 , . .
, b M ∈ R , the feedforward for prediction of our framework is
[Equation]
We let the p θ ( Y = C j | X ) ∝ e SC j ( X ) and take the loss as in (12).
This framework is called a Poisson
MLR.
We use the usual optimization algorithms to update parameters in the Poisson neuron.
The
BatchNorm(Ioffe & Szegedy, 2015) seems crucial for (13) in the experiment.
Figure 4 illustrates that high-conﬁdence prediction regions (deep red areas) of the Poisson MLR are compact sets, in contrast to classical classiﬁers Hein et al.
(2019)[Theorem 3.1].
We shall use this ﬁgure to explain an experiment in Section 6.4.
R EPRESENTATIONAL POWER
In this section, ρ is a continuous sigmoidal function (Cybenko, 1989), ReLU(Nair & Hinton, 2010),
ELU(Clevert et al., 2016), or Softplus(Dugas et al., 2001).
We remind the reader that ρ is sigmoidal if lim t →∞ ρ ( t )
= 1 and lim t →−∞ ρ ( t ) = 0 .
The following theorem justiﬁes the representational power of horocycle neurons.
Theorem 2.
Let K be a compact set in H n , and 1 ≤ p< ∞ .
Then ﬁnite sums of the form
[Equation]
are dense in L p ( K, µ ) , where µ is either d Vol (5) or the induced Euclidean volume.
We provide a sketch of the proof here and go through the details in A.9.
It sufﬁces to prove the theorem for a sigmoidal function ρ and µ =
d
Vol , as other cases follow from this one.
Assume that these ﬁnite sums are not dense in L p ( K, d Vol ) .
By the Hahn-Banach theorem, there exists
some nonzero
h ∈ L q ( K, d Vol ) , where q = p/ ( p − 1) if p> 1 and q = ∞ if p =1 , such that (cid:82) K F ( x ) h ( x ) d Vol ( x ) =for all ﬁnite sums of the form (14).
Extend h to be a function H that is deﬁned on H n by assigning
H ( x )= h ( x ) if x ∈ K and H ( x )=
if x ∈ H n \ K .
Using the property of sigmoidal functions, the bounded convergence theorem, and the integral formula (7), we prove that the integration of H on almost every horocycle is zero.
By the injectivity Theorem 1, H is almost everywhere zero, which contradicts our assumption and completes the proof.
In A.10, we shall prove the same result for Poisson neurons.
In A.11, we prove the following lemma, which demonstrates a close relationship between horocycle neurons and the widely used f 1 a,p (3).
Lemma 1.
Let K be a compact set in H n , ω ∈ S n − 1 , and (cid:15) > 0 .
There exists c, d ∈ R , p ∈ H n , and a ∈ T p ( H n ) such that the function D ( x ) =
cf 1 a,p ( x )+ d −(cid:104) x, ω (cid:105) H satisﬁes || D
|| L p ( K,d Vol ) < (cid:15) .
This lemma suggests that horocycle neurons are boundary points of some “compactiﬁcation” of the space of f 1 a,p .
The above lemma together with Theorem 2 implies
Corollary 1.
Let K be a compact set in H n and 1 ≤ p< ∞ .
Finite sums of the form
F ( x ) =
N (cid:88) i
= α
i ρ ( c i f 1 a i ,p i ( x ) + d i ) , p i ∈ H n , a i ∈ T p i ( H n ) , α i , c i , d i ∈ R , are dense in L p ( K, µ ) , where µ = d Vol or µ is the induced Euclidean volume.
This result provides novel insights into the hyperbolic neural network (Ganea et al., 2018a), gyroplane layer (Mathieu et al., 2019), and Poincaré FC layer (Shimizu et al., 2020).
Although level sets of f 1 a,p are hypercycles, our proof of Lemma 1 relies on the theory of horocycles.
It would be interesting to have more natural approaches to treat the expressivity of f 1 a,p .
E XPERIMENTS
In this section, we ﬁrst play with the MNIST toy.
Next, we apply a horocycle feature to the Poincaré embedding subtree classiﬁcation task.
After that, we construct 2-D clusterings of image datasets by using the horocycle MLR.
Finally, we provide evidence for further possible applications of the Poisson MLR.
We use the framework or some functions of Tensorﬂow, Keras, and scikit-learn (Abadi et al., 2015; Chollet et al., 2015; Pedregosa et al., 2011).
6. MNIST
The MNIST (LeCun et al., 1998) task is popular for testing hyperbolic learning tools (Ontrup &
Ritter, 2005; Nagano et al., 2019; Mathieu et al., 2019; Grattarola et al., 2019; Ovinnikov, 2020;
Khrulkov et al., 2020).
We train two different classiﬁers.
A.12, A.14 and code contain details.
The ﬁrst one is given by a single horocycle layer followed by the softmax classiﬁer.
The average test error rates after 600 epochs is 1 .
96 % , and Theorem 2 provides the rationale for this experiment (A.13).
The second one is a Poisson MLR.
It is the best hyperbolic geometry related MNIST classiﬁer (Table 1), in which Ontrup & Ritter (2005)’s uses the hyperbolic SOM, Grattarola et al.
(2019) uses the adversarial autoencoder, and Khrulkov et al.
(2020) uses the hyperbolic MLR.
Our experiment performs well on MNIST suggests that horocycle and Poisson neurons are computationally efﬁcient and easily coordinate with classical learning tools (such as the convolutional layer and the softmax).
[Table]
This paper
[Table]
6. P OINCARÉ EMBEDDING SUBTREE CLASSIFICATION
Given a Poincaré embedding (Nickel & Kiela, 2017) PE : { WordNet noun } → H D ofnouns and given a node x ∈ { WordNet noun } , the task is to classify all other nodes as being part of the subtree rooted at x (Ganea et al., 2018a).
Our model is a logistic regression, where the horocycle feature p ∈ { WordNet noun } (cid:55)→
h PE ( x ) ( PE ( p ) /s )
( s is a hyperparameter lying in [1 , 1 . 5] ) is the only predictor, and the dependent variable is whether p is in the subtree rooted at x .
The decision hypersurface is a horocycle, as illustrated in Figure 3 (left).
In the experiment, we pre-train three different Poincaré embeddings 3 in each of H 2 , H 3 , H 5 , H 10 .
For each x ∈ { animal, group, location, mammal, worker } and D ∈ { 2 , 3 , 5 , 10 } , we randomly select one of three pre-trained Poincaré embedding PE : { WordNet noun } → H D and then test the model.
[Table]
Table 2 reports the F1 classiﬁcation scores and two standard deviations of 100 trials for each { x, D } .
Different Poincaré embeddings account for the most variance of the performance.
Our model is different from the existing ones.
Firstly, we take the horocycle as the decision hypersurface, while others take the geodesic.
Secondly, we train a logistic regression on top of the horocycle feature attached to PE ( x ) , which is efﬁciently calculated, while others train the hyperbolic MLR with different parametrizations.
On the number of parameters, we have three (independent of D ), Ganea et al.
(2018a) has 2 D , and Shimizu et al.
(2020) has D + 1 .
The number of parameters explains why our model is prominent in low dimensions.
6. E ND - BASED CLUSTERING FOR 2D DIMENSION REDUCTION
In this experiment, we use the horocycle MLR (Section 4.2) to construct end-based clusterings
NN θ : R D → H 2 for MNIST, Fashion-MNIST(Xiao et al., 2017), and CIFAR-10(Krizhevsky, 2012).
We take NN θ =
Exp ◦ NN (cid:48)
θ , where Exp is the exponential map of H 2 and NN (cid:48)
θ : R D
→ R 2 is a network with four convolutional blocks for MNIST/Fashion-MNIST or a ResNet-32 structure for
CIFAR-
A.16 and code contain details.
Figure 5 illustrates end-based clusterings for MNIST, Fashion-MNIST, and CIFAR-10, with performance reported in the caption.
Our accuracy for Fashion-MNIST is 8% higher than all numbers presented in McInnes et al.
Moreover, Table 3 compares numbers of Yang et al.
Ghosh & Kirby (2020) and ours for MNIST, as our methods are similar.
We all use convolutional networks as the (Feature descriptor) and prototype-based functions as the loss.
However, Yang et al.
(2018); Ghosh & Kirby (2020) use the center-based prototype loss, while we use the end-based (12).
Yang et al.
(2018)[Figure 1] points out that the traditional CNN is good at linearly separating feature representations, but the learned features are of large intra-class variations.
The horocycle MLR leads to the inter-class separability in the same way (angle accounts for label difference) a traditional CNN does.
At the same time, it also obtains intra-class compactness (Figure 5).
[Table] [Table]
P OISSON MLR
Using a Poisson MLR whose feature descriptor is a ResNet-32 structure, we obtain a classiﬁer with test error rate 6 .
46 % on CIFAR-10.
It is on par with other methods with similar network structures (Yang et al., 2018).
Moreover, we apply Poisson MLR to the classiﬁcation task of ﬂowers (Tensorﬂow),
Poisson on the task of ﬂowers
Figure 4 explains this experiment.
In the earlier epochs of the training, feature vectors NN θ ( X ) of the Poisson model are not even close to its compact high-conﬁdence prediction regions (deep red areas), and therefore on the test data, it is not much better than a random guess.
In the end, feature vectors lying in these compact regions are of small intra-class variations, which is good for generalization (Yang et al., 2018).
which is a typical example of overﬁtting.
Replacing the MLR part of the Keras model (Tensorﬂow) by a Poisson MLR, the new Poisson model shows better generalization performance (Figure 6).
A. and code contain the details.
This subsection provides evidence of further applications of horocycles.
C ONCLUSION
Based on the spectral theory of hyperbolic spaces, we introduced several horocycle-related tools.
They ﬁnd applications in the hyperbolic neural networks, the Poincaré embedding subtree classiﬁcation task, and the visualization and classiﬁcation of image datasets.
We give an existential proof of a universal approximation theorem for shallow networks constructed by horocycle neurons or f 1 a,p .
Hopefully, it will trigger further research on the expressivity problems, such as constructive approaches, quantitative results, and beneﬁt of depth (Mhaskar & Poggio, 2016), on horocycle neurons, f 1 a,p , and similar functions on more general manifolds.
A A PPENDIX A. N OTATIONS AND S YMBOLS
[Table]
metric
[Table]
n
[Table]
curvature
[Table]
as a set,
[Table] [Table]
∈
[Table]
|
[Table]
|
[Table] [Table]
x ∈ H n , ω ∈ S n − 1 , P ( x, ω ) = (cid:16) 1 −| x | 2 | x − ω | 2 (cid:17) n −
[Table]
Model
[Table]
f 1 a,p ( x ) = 2 | a | 1 −| p | 2 sinh − 1 (cid:16) 2( − p ⊕ x,a ) E (1 −|− p ⊕ x
| 2 ) | a | (cid:17)
[Table]
The
[Table]
set of horocycles
[Table]
are tangential
[Table]
Laplace
[Table]
Beltrami operator on
[Table]
h x ( y ) =
(cid:104) y,
x/ | x |(cid:105) H
[Table]
Dist( x, ω, b ) = − 2 (cid:104) x, ω (cid:105) H + b
[Table]
Cls = { C 1 , C 2 , . . .
, C M }
[Table]
θ
[Table]
P ρw,λ,b ( x ) =
ρ (cid:16) λ | w | 2 −| x | 2 | x − w | 2 + b (cid:17)
Conventional symbols
Symbol
In most cases it refers n, m, i integers x, y, w points in R n or H n , or real numbers o the origin of R n or H
n b, c, d, α, δ real numbers λ real or complex number t real number, represent the timestamp in optimization ω point in S n −ρ an activation function f, g functions
K a compact set
X a manifold p a point in H n or on a manifold a an element in T p ( H n ) ξ a horocycle µ a measure
L a family of geodesics lines l a geodesics line
U a set in H n
F, h, H functions
M number of classes
D dimension
A. P ROOF OF THE ISOMETRY
Given ω ∈ S n − 1 and λ ∈ R , we let ξ λ,ω the unique horocycle that connects ω and tanh ( λ/ 2) · ω .
The length of any geodesic (that ends at ω ) line segment cut by ξ λ 1 ,ω and ξ λ 2 ,ω equals | λ 1 − λ 2 | .
This fact is obvious in the half-space model.
There is a Riemannian isometry F : { z ∈ R n : | z | < 1 } → { ( x 1 , · · · , x n ) : x 1 > 0 } (the latter is with the metric ds 2 = dx 21 + ··· + dx 2 n x) such that F ( ω ) = ∞ and F ( o ) =
Using d H n ( o, tanh( λ i / 2) ω )
=
| λ i | , d { ( x 1 , ··· ,x n ): x 1 > 0 } ((1 , 0 , . . . , 0) , ( e ± λ i , 0 , . . . , 0))
=
| λ i | , F ( ω ) = ∞ and F ( o )
= (1 , 0 , . . .
, 0) , we have F (tanh( λ i / 2) ω ) =
( e λ i , 0 , . . . , 0) .
Therefore, F maps ξ λ
i ,ω to { ( x 1 , x 2 , . . .
, x n ) : x 1 = e λ i } .
Any geodesic (that ends at ω ) line segment cut by ξ λ 1 ,ω and ξ λ 2 ,ω is mapped by F to { ( t, α 2 , . .
. , α n ) : ( t − e λ 1 )( t − e λ 2 ) < 0 } for some ﬁxed α j .
It is easy to check the length of this segment with respect to dx 21 + ··· + dx 2 n x(as α
i are constants, the metric reduces to dx 21 /xon
this segment) is | λ 1 − λ 2 | .
A. P ROOF OF (6)
Because x is on ξ λ which is a sphere with center 1+tanh λ/ 2ω and radius 1 − tanh λ/ 2, we have (cid:12)(cid:12)(cid:12) x − 1+tanh λ/ 2ω
[Equation]
(cid:12)(cid:12)(cid:12) 1 −
[Equation]
2(cid:12)(cid:12)(cid:12) 2 , which leads to | x | 2 − (1+tanh λ/ 2)( x, ω ) E +tanh
λ/ 2 | ω | 2 =
0 , and then 1+tanh λ/ 2| x − ω | 2 = 1 − tanh λ/ 2( | ω 2 | − | x | 2 ) , and ﬁnally (cid:104) x, ω (cid:105)
H
=
12 log | ω | 2 −| x | 2 | x − ω | 2 = 12 log 1+tanh λ/ 2 1 − tanh λ/ 2 = λ/ 2 .
A. A NOTHER PROOF OF THE INTEGRAL FORMULA (7)
We use H n for the upper half space model { ( x 1 , · · · , x n ) : x 1 > 0 } with the Riemannian volume dx 1 ··· dx n x n. Let ω =
( ∞ , 0 , . . . , 0) and o be (1 , 0 , . . . , 0) as in (A.2), then ξ λ,ω
= { ( x 1 , x 2 , . . .
, x n ) : x 1 = e λ } .
The induced Riemannian metric on ξ λ,ω (respectively volume d Vol ξ λ,ω ) is dx 22 + ··· + dx 2 n e 2 λ (respectively dx 2 ··· dx n e ( n − 1) λ ).
For any integral function f on H n , using change of variable x 1 = e λ
[Equation]
d Vol
[Equation]
λ
[Equation]
dλ.
The above identity is equivalent to the integral formula
[Equation]
R (cid:16)(cid:82) ξ λ
[Equation]
ω f
[Equation]
z )
[Equation]
z )
[Equation]
dλ.
presented in (7), according to the Riemannian isometry in (A.2).
A. T HE HEURISTIC IS NOT A PROOF
The spectral theory does not directly lead to universal approximation theorems because of the following: 1, superpositions in (1, 2) and (8, 9) are different (similarly, although another kind of superposition in Hilbert’s 13th problem (Hilbert, 1935; Arnold, 2009) was a driving force for universal approximation theorems (Nielsen, 1987), the former is hardly relevant for networks (Girosi & Poggio, 1989)); 2, desired representation properties of hyperbolic eigenfunctions are unknown, partially because H n is non-compact; 3, results in spectral theory favor Hilbert spaces, while universal approximation theorems embrace more than L 2 space.
A. O PTIMIZATION
The parameters update for the horocycle unit (2) involves the optimization problem on the sphere (for ω ) and the hyperbolic space (for x ).
We use a standard algorithm of sphere optimization (Absil et al., 2008) to update ω , and in the supplement we present an optimization approach based on the geodesic polar-coordinates to update x .
In the implementation of a horocycle layer, the forward propagation is trivial, while the backpropagation involves optimization on the sphere and hyperbolic space.
In the following, η is the learning rate, α t is the value of α ( α may be η , s , z , ω, . . . )
at the t -th step, T p X is the tangent ﬁber at p , ∇ is the gradient, and ∇ H is the hyperbolic gradient.
It sufﬁces to consider the layer s = (cid:104) z, ω (cid:105) .
Optimization on the sphere The parameter update of ω in s = (cid:104) z, ω (cid:105) involves the optimization on the sphere.
The projection of ∂ L θ ∂s ∇ s ( ω t ) =
∂
L θ ∂s z t − ω t | z t − ω t | 2 ∈ T ω t R n onto T ω t S n − 1 is given by Absil et al.
(2008)[p.48]
[Equation]
Two well-known update algorithms of w t Absil et al.
(2008)[p.76] are:
[Equation]
A. A PROOF OF A POLLONIUS THEOREM
Theorem 3 (Apollonius) .
Given distinct ω 1 , ω 2 ∈ S n − 1 and a positive number λ , the locus { x : | x − ω 1 | = λ | x − ω 2 |} is a sphere orthogonal to S n − 1 .
Proof.
If λ is one then it is trivial.
We assume now λ is not one.
By | x − ω 1 | = λ | x − ω 2 | , we can have
[Equation]
The locus is a sphere with center O = ω 1 − λω 2 1 − λ and radius R = (cid:113) | ω 1 − λω 2 | 2 | 1 − λ |− 1 .
The theorem of
Apollonius (in all dimension) claims that this sphere is orthogonal to S n − 1 .
To prove this, it sufﬁces to prove | oO | 2 = 1 + R 2 (recall o is the origin of H n ), which follows from
[Equation]
A. I NVERSION
∪ { given the sphere { x : | x − w 0 | = r } , the corresponding inversion is given by
[Equation]
For x ∈ R n ∪ {∞} , Iv ( x ) is called the inverse of x with respect to { x : | x − w 0 | = r } .
A. P ROOF OF T HEOREM
Theorem 2 Let K be a compact set in H n , and 1 ≤ p< ∞ .
Then ﬁnite sums of the form
[Equation]
are dense in L p ( K, µ ) , where µ is either d Vol (5) or the induced Euclidean volume.
Proof.
We ﬁrst treat the case ρ is sigmoidal and µ = d Vol .
Assume that these ﬁnite sums are not dense in L p ( K, d Vol ) .
By the Hahn-Banach theorem, there exists
some nonzero
h ∈ L q ( K, d Vol ) , where q = p/ ( p − 1) if p> 1 and q = ∞ if p =1 , such that (cid:82) K F ( x ) h ( x ) d Vol ( x ) =
for all ﬁnite sums of the form (14).
As K is a compact set, by Hölder’s inequality, (cid:82) K | h ( x ) | d Vol ≤ ( (cid:82) K d Vol ) 1 /p
||
h
||
L q ( K,d Vol ) , which leads to h ∈ L 1 ( K, d Vol ) .
Extend h to be a function H that is deﬁned on H n by assigning H ( x )=
h ( x ) if x ∈ K and H ( x )=
if x ∈ H n \ K .
Then
H ∈ L 1 ( H n , d Vol ) ∩
L q ( H n , d Vol ) and
[Equation]
for all ﬁnite sums of the form (14).
For any ω ∈ S n − 1 and λ, b ∈ R , we set F ω,λ,b ( x ) =
ρ ( λ ( (cid:104) x, ω (cid:105) H − b )) .
These functions are uniformly bounded, as | F ω,λ,b ( x ) |≤ 1 .
Moreover,
[Equation]
According to (15), for all ω, λ, b , we have (cid:82) H n F ω,λ,b ( x ) H ( x )
d Vol ( x ) = 0 .
Functions { F ω,λ,b } λ ∈ R converge pointwise as λ →∞ , and they are uniformly bounded by | H |∈ L 1 ( H n , d Vol ) .
By the bounded convergence theorem, for all ω ∈ S n − 1 , b ∈ R , we have
[Equation]
By the integral formula (7) (with notations deﬁned there), (6) and (17), for all b ∈ R , (cid:90) ∞
2 b (cid:32)(cid:90)
[Equation]
Taking the derivative of (cid:82) ∞ 2 b (cid:16)(cid:82) ξ t,ω H ( z ) d Vol ξ t,ω ( z ) (cid:17) dt with respect to b , we deduce from (18) that (cid:82) ξ 2 b,
ω H ( z ) d
Vol ξ 2 b,ω
( z ) = 0 for a.e. b ∈ R .
In other words, the integration of H on a.e. ξ ∈ Ξ ω is zero.
This fact is valid for all ω ∈ S n − 1 .
Therefore, the integration of H on a.e. ξ ∈ Ξ is zero.
By the injectivity Theorem 1, H = 0 a.e., which contradicts our assumption.
Therefore, ﬁnite sums of the form (14) are dense in L p ( K, d Vol ) .
The case ρ is ReLU, ELU or Softplus and µ = d Vol follows from the above case and the fact that x (cid:55)→ ρ ( x + 1) −
ρ ( x ) is sigmoidal.
The case µ is the Euclidean volume follows from previous cases and the fact that the Euclidean volume on compact K is bounded from above by λd Vol for some constant λ .
A. U NIVERSAL APPROXIMATION THEOREM FOR P OISSON NEURONS .
In this section, ρ is a continuous sigmoidal function (Cybenko, 1989), ReLU(Nair & Hinton, 2010),
ELU(Clevert et al., 2016), or Softplus(Dugas et al., 2001).
We also recall the Poisson neuron:
[Equation]
Theorem 4.
Let K be a compact set in H n , and 1 ≤ p< ∞ .
Then ﬁnite sums of the form
F
[Equation]
(19) are dense in L p ( K, µ ) , where µ is either d Vol (5) or the induced Euclidean volume.
Proof.
We ﬁrst treat the case ρ is sigmoidal and µ = d Vol .
Assume that these ﬁnite sums are not dense in L p ( K, d Vol ) .
By the Hahn-Banach theorem, there exists
some nonzero
h ∈ L q ( K, d Vol ) , where q = p/ ( p − 1) if p> 1 and q = ∞ if p =1 , such that (cid:82) K F ( x ) h ( x ) d Vol ( x ) =
for all ﬁnite sums of the form (19).
As K is a compact set, by Hölder’s inequality, (cid:82) K | h ( x ) | d Vol ≤ ( (cid:82) K d Vol ) 1 /p
||
h
||
L q ( K,d Vol ) , which leads to h ∈ L 1 ( K, d Vol ) .
Extend h to be a function H that is deﬁned on H n by assigning H ( x )=
h ( x ) if x ∈ K and H ( x )=
if x ∈ H n \ K .
Then
H ∈ L 1 ( H n , d Vol ) ∩
L q ( H n , d Vol ) and (cid:90)
[Equation]
for all ﬁnite sums of the form (19).
For any ω ∈ S n − 1 , λ ∈ R , and b > 0 , we set
[Equation]
These functions are uniformly bounded, as | F ω,λ,b ( x ) |≤ 1 .
Moreover,
[Equation]
According to (20), for all ω, λ, b , we have (cid:82) H n F ω,λ,b ( x ) H ( x )
d Vol ( x ) = 0 .
Functions { F ω,λ,b } λ ∈ R converge pointwise as λ →∞ , and they are uniformly bounded by | H |∈ L 1 ( H n , d Vol ) .
By the bounded convergence theorem, for all ω ∈ S n − 1 , b ∈ R , we have
[Equation]
By the integral formula (7) (with notations deﬁned there), (6) and (22), for all b ∈ R , (cid:90) ∞
[Equation]
Taking the derivative of (cid:82)
[Equation]
ω H ( z ) d
Vol ξ t,ω ( z ) (cid:17) dt with respect to b , we deduce from (23) that (cid:82) ξ log b,
ω H ( z ) d
Vol ξ log b,ω ( z ) = 0 for a.e. b> 0 .
In other words, the integration of H on a.e. ξ ∈ Ξ ω is zero.
This fact is valid for all ω ∈ S n − 1 .
Therefore, the integration of H on a.e. ξ ∈ Ξ is zero.
By the injectivity Theorem 1, H = 0 a.e., which contradicts our assumption.
Therefore, ﬁnite sums of the form (19) are dense in L p ( K, d Vol ) .
The case ρ is ReLU, ELU or Softplus and µ = d Vol follows from the above case and the fact that x (cid:55)→ ρ ( x + 1) −
ρ ( x ) is sigmoidal.
The case µ is the Euclidean volume follows from previous cases and the fact that the Euclidean volume on compact K is bounded from above by λd Vol for some constant λ .
We refere the reader to the difference of (16) and (21), (17) and (22), and (18) and (23).
However, basically the proofs are the same.
The points are the integral formula (7), the injectivity Theoremand the fact that level sets of horocycle/Poisson neurons are horocycles.
Moreover, as a corollary of
Theorem 4, we have
Corollary 2.
Let K be a compact set in R n , and 1 ≤ p< ∞ .
Then ﬁnite sums of the form
[Equation]
are dense in L p ( K, µ ) , where µ is the Euclidean volume.
Proof.
Because K is compact, there exists a positive number R such that K ⊂ { x ∈ R n : | x | < R } .
By the above theorem, ﬁnite sums of the form
[Equation]
are dense in L p ( K/R, µ ) .
Then the corollary follows from
[Equation]
A. P ROOF OF THE L EMMA
Recall
[Equation]
The proof of Lemma 1 follows from the following direct computation.
Proof.
Let t ∈ (0 , 1) .
Take p t =
tω and a t = − ω , then we have
[Equation]
Let F t ( x ) = 2( − p t ⊕ x,a t ) E (1 −|− p t ⊕ x
| 2 ) | a t | , then
F t ( x ) =
[Equation]
x, a t ) E
[Equation]
=
[Equation] [Equation]
A
[Equation] [Equation]
B
[Equation]
( x ) , where A t , B t are deﬁned as the corresponding numerator and denominator.
We have
A t ( x ) | t = =
[Equation]
|
[Equation]
B t ( x ) | t =
[Equation]
∂B t ( x ) /∂t | t = = 2 | x − ω | 2 ( | x | 2 − 1) .
Let G t ( x ) = sinh − 1 ( F t ( x ))
+ log 1 − t 1+ t , then
G t ( x
[Equation]
=
[Equation]
By L’Hôpital’s rule, lim
[Equation]
Therefore, lim
[Equation]
−
[Equation]
|
[Equation]
For t < 1 , we take p t
=
tω, a t = − ω, c t = t 2 − 1 4 , d t = 12 log 1+ t 1 − t , then for all x ∈ K , lim t< 1 ,t → 1 c t f 1 a t ,p t ( x ) + d t =
lim t< 1 ,t →
[Equation]
If there exists c 1 , c 2 such that | c t f 1 a t , p t ( x ) + d t | (= | G t ( x ) | / 2) ≤ c 2 for all t ∈ ( c 1 , 1) , x ∈
K , then by the dominated convergence theorem, there exists t such that || c t f 1 a t ,p t ( x ) + d t − (cid:104) x, ω (cid:105)
H
||
L p ( K,m ) < (cid:15) , which proves the lemma.
Note that
[Equation]
where U j and L l are continuous functions deﬁned on K × { ω } .
There exist positive numbere c 3 , cand c 1 ∈ (0 , 1) such that for all x ∈ K and t ∈ ( c 1 , 1) ,
[Equation]
Therefore, for x ∈ K and t ∈ ( c 1 , 1) , we have
[Equation]
This implies that for t ∈ ( c 1 , 1) , G t | K and therefore | c t f 1 a t ,p t + d t || K are uniformly bounded, which ﬁnishes the proof of the lemma.
A. T HE FIRST MNIST CLASSIFIER IN 6.
At the preprocessing stage, we compute the projection of the 28 × 28 input pattern on the 40 principal components and then scale them so that the scaled 40-dimensional PCA features are within the unit ball.
In our network,
Input layer: scaled 40-dimensional PCA features; 2.
First layer: 40 inputs/1000 outputs horocycle layer (tanh activation); 3.
Last layer: 1000 inputs/10 outputs afﬁne layer; 4.
Loss: cross entroy loss.
Take learning rate = 1 , learning rate decay = 0 . 999 , and batch size = 128 , and run it three times.
The average test error rates after 600 epochs is 1 .
PCA follows LeCun et al.
(1998)(C.3), where 40 PCA is used for the quadratic network.
Quadratic network has a similar structure to ours, because our neuron are contructed by quotient of quadratic functions followed by log.
A. H OROCYCLE LAYER FOLLOWED BY MLR CAN APPROXIMATE THE CLASSFICATION FUNCTION
Suppose the MNIST classiﬁcation function M is deﬁned on ∪ 9 j =
K j ⊂
H 40 , where
K i are relatively compact and M|
K
j = j .
By Theorem 2, for 0 ≤ j ≤ 9 , there exist F j ( x ) = (cid:80) N j i =
α j,i ρ ( λ j,i (cid:104) x, ω
j,i (cid:105) H + b j,i ) such that F j approximates I K j , where I is the indicator function.
Therefore, a network with the ﬁrst (horocycle) layer given by ρ ( λ j,i (cid:104) x, ω
j,i (cid:105) H + b j,i )(0 ≤ j ≤ 9 , 1 ≤ i ≤ N j ) followed by a classical MLR with parameters given by α j,i (0 ≤ j ≤ 9 , 1 ≤ i ≤ N j ) (with arg max for prediction) approximates M .
A. T HE SECOND MNIST CLASSIFIER IN 6.
At the preprocessing stage, we do data augmentation by letting each image 1 step toward each of itscorners, so that our traning set has 300000 examples.
In our network,
Input layer: (28,28, 1); 2.
First block: 32-ﬁlters 3 × 3 convolution, ReLU, 2 × 2 max-pooling, BatchNorm; 3.
Second block: 64-ﬁlters 3 × 3 convolution, ReLU, BatchNorm; 4.
Thrid block: 64-ﬁlters 3 × 3 convolution,ReLU, 2 × 2 max-pooling, BatchNorm;
Fourth block: 128-ﬁlters 3 × 3 convolution, ReLU, 2 × 2 max-pooling, BatchNorm; 6.
Fifth block: FC 1000, ReLU, BatchNorm; 7.
Last block: 1000 input/10 output Poisson layer, sigmoid, BatchNorm; 8.
Loss: cross entroy loss.
In optimization, we take Adam(Kingma & Ba, 2015).
The batch size is 128 in the ﬁrst 5 epochs, and 1024 in the next 15 epochs.
After 5 epochs, we set ω
i in the Poisson layer to be non-trainable.
We train our network ﬁve times, the average test error rate after 20 epochs is 0 .
The (cid:15) in | w | 2 −| x | 2 | x − w | 2 + (cid:15) is an important hyperparameter for the numerical stability.
We train this MNIST model with (cid:15) ∈ { 10 − 1 , 10 − 2 , 10 − 4 , 10 − 6 , 10 − 8 , 10 − 10 , 10 − 20 } .
They all show robust performance.
A. E XPERIMENT OF P OINCARE TREE CLASSIFICATION TASK
Given a Poincaré embedding (Nickel & Kiela, 2017) PE : { WordNet noun } → H D of the
WordNet noun nodes and given a node x , the task is to classify all other nodes as being part of the subtree rooted at x (Ganea et al., 2018a).
Our model is a logistic regression, where the horocycle feature p ∈ { WordNet noun } (cid:55)→
h PE ( x ) ( PE ( p ) /s )
( s is a hyperparameter lying in [1 , 1 . 5] ) is the only predictor, and the dependent variable is whether p is in the subtree rooted at x .
Let P be the set of all nodes in the Poincare embedding, and let p range from P .
Input: h PE ( x ) ( PE ( p ) /s )
(s is a hyperparameter.)
Only layer: 1 input/1 output afﬁne layer.
(two parameters: one for input, one for bias.) 3.
Loss: Logistic.
(with respect to 1 if p in the tree rooted at x ; 0 else.)
In each training, x is one of {animal, group, location, mammal, worker}, dim is one of {2,3,5,10}, and Poincaré embeddings are from the animation_train.py of Ganea et al.
(2018b) 4 (with tree=
wordnet_full, model=poincare, dim=dim, seed randomly ∈ { 7 , 8 , 9 } ).
All nodes in the subtree rooted at x are divided into training nodes (80%) and test nodes (20%).
The same splitting procedure applies for the rest nodes.
We choose s that has the best training F1, and then record the corresponding test F1.
For each x and dim, we do the training 100 times.
The average test F1 classiﬁcation scores are recorded in Table 2.
The horocycle feature performs well here because it is compatible with the Poincaré embedding algorithm.
Let x be a node that is not at the origin.
It seems that the Poincaré embedding algorithm tends to pull all nodes that are from the subtree rooted at x towards the direction of x | x | , therefore y → (cid:68) y, x | x | (cid:69) H is a suitable feature for this task.
A. E ND - BASED CLUSTERING IN H
For MNIST, at the preprocessing stage, we do data augmentation by letting each image 1 step toward each of its 4 corners, so that our traning set has 300000 examples.
Our network for H 2 embedding of
MNIST dataset is
Input layer: (28,28, 1); 2.
First block: 32-ﬁlters 3 × 3 convolution, ReLU, 2 × 2 max-pooling, BatchNorm; 3.
Second block: 64-ﬁlters 3 × 3 convolution, ReLU, BatchNorm; 4.
Thrid block: 64-ﬁlters 3 × 3 convolution,ReLU, 2 × 2 max-pooling, BatchNorm; 5.
Fourth block: 128-ﬁlters 3 × 3 convolution, ReLU, 2 × 2 max-pooling, BatchNorm; 6.
Fifth block: FC 1000, ReLU, BatchNorm; 7.
Sixth block: FC 2, ReLU, BatchNorm, Exp ; 8.
Last block: 2 input/10 output horocycle layer, sigmoid;
where Exp is the exponential map
T
o H 2 (= R 2 )
→ H 2 .
We apply the data augmentation as in A.14.
In optimization, learning rate is 0.1, learning rate decay is 0.99, batch size is 128, epochs is 50.
Our network, data augmentation and optimization for H 2 embedding of Fashion-MNIST dataset is completely the same as that for MNIST.
For MNIST and Fashion-MNIST we use sphere optimization.
We would like to remark that there are interesting new features in sphere optimization.
Because the S 1 is compact, for any continuous function f , there exists x = argmax S 1 f .
The derivative of f at x vanish, so the usual optimization algorithm to ﬁnd the minimum will fail in the general case.
In our experiments, we solve this problem by adding the following tricks:
Observation: if the class C α are all close to ω ∈ S 1 , and the end prototype ω
α for the class
C α is around − ω , then ω
α is a maximum point of the loss function and therefore can not be improved through normal SGD.
We solve this problem by adopting an idea(supervised variation) of k-means clustering.
In each early epochs, optimization consists of two parts.
In the ﬁrst part, the normal SGD applies.
In the second part, we move end prototypes ( ω i ) to the average direction of the class (using training data).
Observation: if the class C α and class C β are all close to ω ∈ S 1 , and the end prototype ω α , ω β are also both around ω , then all points in class C α and class C β , end prototypes
ω
α , ω β will all be pulling to ω by the SGD, and ﬁnally the network can not distinguish class C α and class C β .
We solve this problem by adding a loss if two prototypes are close.
With these small tricks, our 2D end-based clustering algorithm is very stable for MNIST and FashionMNIST.
We run it on MNIST 10 times, and they all get a test acc around 99% within 20 epochs.
Suppose the classiﬁcation task has M classes and the prototype of the i -th class is ω i .
We write down the additional loss function for the second observation as follows
[Equation]
10 × ReLU
[Equation]
− 0 .
9 − (cid:15) ))
[Equation]
where (cid:15) is a small constant for numerical stability.
For CIFAR-10, our network for H 2 embedding of CIFAR-10 dataset is
Input layer: (32,32, 3); 2.
First block: ResNet-32/128 output; 3.
Second block: FC 2, ReLU, BatchNorm, Exp ; 4.
Last block: 2 input/10 output horocycle layer; 5.
Loss: cross entroy loss.
In the data augmentation, we apply horizontal/vertical shifts and horizontal ﬂip.
We use Adam.
The batch size is 32 in the ﬁrst 100 epochs, or 1024 in the next 50 epochs.
The weights of the horocycle layer are ﬁxed at the beginning of the training and are non-trainable, which follows an idea of Mettes et al.
A. P OISSON MLR
We train the network that is the same as in A.14 on Fashion-MNIST, with the same data augmentation as in A.14.
In optimization, we also use Adam.
The batch size is 128 in the ﬁrst 10 epochs, orin the next 10 epochs.
After 10 epochs, we set ω
i in the Poisson layer to be non-trainable.
We train our network ﬁve times, the average test error rate after 20 epochs is 5 .
For CIFAR-10, we try a model similar to that in MNIST and Fashion-MNIST, but with more ﬁlters and one more afﬁne layer.
In the data augmentation, we apply horizontal/vertical shifts and horizontal ﬂip.
In our network,
Input layer: (32,32, 3); 2.
First block: ResNet-32/128 output; 3.
Second block: FC 128, ReLU, BatchNorm; 4.
Last block: 128 input/10 output Poisson layer, BatchNorm; 5.
Loss: cross entroy loss.
We use Adam.
The batch size is 32 in the ﬁrst 80 epochs, or 1024 in the next 20 epochs.
Test acc greater than 93 .
For the classiﬁcation task of ﬂowers (Tensorﬂow), The dataset of 3670 photos of ﬂowers containsclasses: daisy, dandelion, roses, sunﬂowers and tulips.
The keras model is
Input layer: (180,180, 3); 2.
First block: 16-ﬁlters 3 × 3 convolution, ReLU, 2 × 2 max-pooling; 3.
Second block: 32-ﬁlters 3 × 3 convolution, ReLU, 2 × 2 max-pooling; 4.
Thrid block: 64-ﬁlters 3 × 3 convolution,ReLU, 2 × 2 max-pooling; 5.
Fourth block: FC 128, ReLU; 6.
Last block: 128 input/10 output FC layer; 7.
Loss: cross entroy loss.
Our Poisson model is
Input layer: (180,180, 3); 2.
First block: 16-ﬁlters 3 × 3 convolution, ReLU, 2 × 2 max-pooling; 3.
Second block: 32-ﬁlters 3 × 3 convolution, ReLU, 2 × 2 max-pooling; 4.
Thrid block: 64-ﬁlters 3 × 3 convolution,ReLU, 2 × 2 max-pooling; 5.
Fourth block: FC 128, ReLU; 6.
Last block: BatchNorm, 128 input/10 output Poisson layer, sigmoid, BatchNorm; 7.
Loss: cross entroy loss.
We use 2936 photos for training and the rest 734 for testing.
We train two models 5 times, and Figure 6 records the average of training and test accuracies in 20 epochs.
Although the test accuracy of the
Poisson model is bad in the beginning, it is fairly higher (around 4% ) than the test accuracy of the keras model in the end.