M ANIFOLD N ET :
A D EEP N EURAL N ETWORK
FOR
M ANIFOLD - VALUED D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Developing deep neural networks (DNNs) for manifold-valued data sets has gained much interest of late in the deep learning research community.
Examples of manifold-valued data include data from omnidirectional cameras on automobiles, drones etc., diffusion magnetic resonance imaging, elastography and others.
In this paper, we present a novel theoretical framework for DNNs to cope with manifold-valued data inputs.
In doing this generalization, we draw parallels to the widely popular convolutional neural networks (CNNs).
We call our network the
ManifoldNet.
As in vector spaces where convolutions are equivalent to computing the weighted mean of functions, an analogous deﬁnition for manifold-valued data can be constructed involving the computation of the weighted Fréchet Mean (wFM).
To this end, we present a provably convergent recursive computation of the wFM of the given data, where the weights makeup the convolution mask, to be learned.
Further, we prove that the proposed wFM layer achieves a contraction mapping and hence the ManifoldNet does not need the additional non-linear ReLU unit used in standard CNNs.
Operations such as pooling in traditional CNN are no longer necessary in this setting since wFM is already a pooling type operation.
Analogous to the equivariance of convolution in Euclidean space to translations, we prove that the wFM is equivariant to the action of the group of isometries admitted by the Riemannian manifold on which the data reside.
This equivariance property facilitates weight sharing within the network.
We present experiments, using the
ManifoldNet framework, to achieve video classiﬁcation and image reconstruction using an auto-encoder+decoder setting.
Experimental results demonstrate the efﬁcacy of ManifoldNet in the context of classiﬁcation and reconstruction accuracy.
I NTRODUCTION
Convolutional neural networks (CNNs) have attracted enormous attention in the past decade due to their signiﬁcant success in Computer Vision, Speech Analysis and other ﬁelds.
CNNs were pioneered by LeCun et al.
and gained much popularity ever since their signiﬁcant success on Imagenet data reported in Krizhevsky et al.
CNNs have traditionally been restricted to dealing with data residing in vector spaces.
However, in the past few years, there is growing interest in generalizing the CNNs and deep networks in general to data that reside on smooth non-Euclidean spaces.
In this context, at the outset, it would be useful to categorize problems into 1) those that involve data as samples of real-valued functions deﬁned on a manifold and 2) those that are simply manifold-valued and hence are sample points on a manifold.
In the context of input data being samples of functions deﬁned on smooth manifolds, recently there has been a ﬂurry of activity in developing methods that can cope speciﬁcally with samples of functions on a sphere that are encountered in many applications such as, omnidirectional cameras on drones, robots etc., meteorological data and many others.
The key property that allows learned weight sharing in CNNs is the equivariance to translations.
The simplest technique to achieve equivariance is via data augmentation (Krizhevsky et al., 2012;
Dieleman et al., 2015).
Cascade of wavelet transforms to achieve equivariance was shown in Bruna & Mallat (2013); Oyallon & Mallat (2015).
In Gens (2014), authors describe ‘Symnet’, which achieves invariance to symmetry group actions.
Equivariance to discrete group actions was achieved through parameter sharing in Ravanbakhsh et al.
For the case of data on a spherical domain, one considers exploiting equivariance to the rotation group.
Several research groups recently reported spherical-CNNs (SCNNs) to accommodate such an equivariance in deﬁning the convolution of functions (Worrall et al., 2017; Cohen & Welling, 2016;
Cohen et al., 2018).
In another recent work Esteves et al.
, authors describe a polar transformer network, which is equivariant to rotations and scaling transformations.
By combining this with a spatial transformer (Jaderberg et al., 2015), they achieve equivariance to translations as well.
More generally, equivariance of convolution operations to group actions admitted by the underlying manifold is what is needed to this end and most recent work reported in Chakraborty et al.
Kondor & Trivedi (2018) achieves this for Riemannian homogeneous spaces.
In this paper we will consider the second problem, namely, when the input data are sample points on known Riemannian manifolds for example, the manifold of symmetric positive deﬁnite matrices,
SPD ( n ) , the special orthogonal group, SO ( n ) , the n -sphere, S n , the Grassmannian, Gr ( p, n ) , and others.
To be precise, the domain of interest is an n -dimensional ﬁeld of points sampled from a
Riemannian manifold. There is very little prior work that we are aware of on DNNs that can cope with input data samples residing on these manifolds with the exception of Huang et al.
Huang & Van Gool (2017).
In Huang et al.
, authors presented a deep network architecture for classiﬁcation of hand-crafted features residing on a Grassmann manifold that form the input to the network.
In Huang & Van Gool (2017), authors presented a deep network architecture for data on SPD ( n ) .
In both of these works, the architecture does not involve the use of any convolution or equivalent operations on Gr ( p, n ) or SPD ( n ) .
Further, it does not use the natural invariant metric or intrinsic operations on the Grassmannian or the SPD ( n ) in the network blocks.
Using intrinsic operations within the layers guarantees that the result remains on the manifold and hence does not require any projection (extrinsic) operations to ensure the result lies in the same space.
Further, using extrinsic operations can yield results that are susceptible to signiﬁcant inaccuracies when the data variance is large (Salehian et al., 2015).
Moreover, since there are no convolution type operations deﬁned for data on these manifolds in their network, it can not be considered a generalization to the CNN and as a consequence does not consider equivariance property to the action of the group of isometries denoted by I ( M ) , admitted by the manifold M .
There are several deep networks reported in literature to deal with cases when data reside on 2manifolds encountered in Computer Vision and Graphics for modeling shapes of objects.
Some of these are based on graph-based representations of points on the surfaces in 3D and a generalization of
CNNs to graphs (Henaff et al., 2015; Defferrard et al., 2016).
There is also recent work in Masci et al.
(2015) where the authors presented a deep network called geodesic CNN (GCNN), where convolutions are performed in local geodesic polar charts constructed on the manifold.
For more literature on deep networks for data on 2-manifolds, we refer the interested reader to a recent survey paper Bronstein et al.
(2017) and references therein.
In this paper, we present a novel DNN framework called the ManifoldNet.
This is a potential analog of a CNN and can cope with input data sampled from a Riemannian manifold.
The intuition in deﬁning the analog relies on the equivariance property.
Note that convolution of functions in vector spaces are equivariant to translations in the spatial domain and in the pixel domain. I.e. if the input pixels are all translated by a ﬁxed amount, then the output pixels are equally translated. Further, it is easy to show that traditional convolutions are equivalent to computing the weighted mean (Goh et al., 2011).
Hence, for the case of manifold-valued data, we can deﬁne the analogous operation of a weighted Fréchet mean (wFM) and prove that it is equivariant to the action of I ( M ) .
This will be achieved in a subsequent section.
Our key contributions in this work are: [ presented in section 2 ](i) we deﬁne the analog of convolution operations for manifold-valued data to be one of estimating the wFM for which we present a provably convergent, efﬁcient and recursive estimator.
(ii)
A proof of equivariance of wFM to the action of I ( M ) .
This equivariance allows the network to share weights within the layers.
(iii)
A novel deep architecture involving the Riemannian counterparts to the conventional CNN units.
[ presented in section 3 ] (iv) Several real data experiments on classiﬁcation and reconstruction demonstrating the performance of the ManifoldNet.
G ROUP ACTION EQUIVARIANT NETWORK FOR MANIFOLD - VALUED DATA
In this section, we will deﬁne the equivalent of a convolution operation on Riemannian manifolds.
As mentioned in the introduction, the domain of interest is an n -dimensional ﬁeld of manifold valued points.
Before formally deﬁning such an operation and building the DNN for the manifold-valued data, dubbed a ManifoldNet, we ﬁrst present some relevant concepts from differential geometry that will be used in the rest of the paper.
Preliminaries.
Let ( M , g M ) be a orientable complete Riemannian manifold with a Riemannian metric g M , i.e., ( ∀ x ∈ M ) g M x : T x M × T x M
→ R is a bi-linear symmetric positive deﬁnite map, where T x M is the tangent space of M at x ∈ M .
Let d : M × M → [0 , ∞ ) be the metric (distance) induced by the Riemannian metric g M .
With a slight abuse of notation we will denote a
Riemannian manifold ( M , g M ) by M unless speciﬁed otherwise.
Let ∆ be the supremum of the sectional curvatures of M .
Deﬁnition 1.
Let p ∈ M , r > 0 .
Deﬁne B r ( p ) = { q ∈ M| d ( p, q ) < r } to be a open ball at p of radius r .
Deﬁnition 2.
(Groisser, 2004)
The local injectivity radius at p ∈ M , r inj ( p ) , is deﬁned as r inj ( p ) =
sup (cid:8) r | Exp p : ( B r ( 0 ) ⊂
T p M ) → M is deﬁned and is a diffeomorphism onto its image } .
The injectivity radius Manton (2004) of M is deﬁned as r inj ( M ) =
inf p ∈M { r inj ( p ) } .
Within B r ( p ) , where r ≤ r inj ( M ) , the mapping Exp − 1 p : B r ( p ) → U ⊂ T p M , is called the inverse
Exponential/ Log map.
Deﬁnition 3.
(Kendall, 1990)
An open ball B r ( p ) is a regular geodesic ball if r < r inj ( p ) and r < π/ (cid:0) 2∆ 1 / 2 (cid:1) .
In Deﬁnition 3 and below, we interpret 1 / ∆ 1 / 2 as ∞ if ∆ ≤ 0 .
It is well known that, if p and q are two points in a regular geodesic ball B r ( p ) , then they are joined by a unique geodesic within B r ( p ) (Kendall, 1990).
Deﬁnition 4.
(Chavel, 2006)
U ⊂ M is strongly convex if for all p, q ∈ U , there exists a unique length minimizing geodesic segment between p and q and the geodesic segment lies entirely in U .
Deﬁnition 5.
(Groisser, 2004) Let p ∈ M .
The local convexity radius at p , r cvx ( p ) , is deﬁned as r cvx ( p ) =
sup { r ≤ r inj ( p ) |B
r ( p ) is strongly convex } .
The convexity radius of M is deﬁned as r cvx ( M ) =
inf p
[Equation]
{ r cvx ( p ) } .
For the rest of the paper, we will assume that the samples on M lie inside an open ball U =
B r ( p ) where r = min { r cvx ( M ) , r inj ( M ) } , for some p ∈ M , unless mentioned otherwise.
Now, we are ready to deﬁne the operations necessary to develop the ManifoldNet.
2. W FM ON M AS A GENERALIZATION OF CONVOLUTION
We will now deﬁne a convolution type operation on points sampled from M .
This convolution operation will perform an averaging operation over a moving window, but will replace weighted sums with weighted intrinsic averages.
Let { X i } Ni =1 be the manifold-valued samples on M .
We deﬁne the convolution type operation on M as the weighted Fréchet mean (wFM) (Maurice Fréchet, 1948) of the samples { X i } N i =1 .
Also, by the aforementioned condition on the samples, the existence and uniqueness of FM is guaranteed (Afsari, 2011).
As mentioned earlier, it is easy to show (see Goh et al.
that convolution ψ ∗ =
b (cid:63) a of two functions a : X ⊂ R n → R and b : X ⊂ R n → R can be formulated as computation of the weighted mean ψ ∗ = argmin ψ (cid:82) a ( u )( ψ − (cid:101) b u ) 2 d u , where, ∀ x ∈ X, (cid:101)
b u ( x ) =
b ( u + x ) and (cid:82) a ( x ) d x = 1 .
Here, f 2 for any function f is deﬁned pointwise.
Further, the deﬁning property of convolutions in vector spaces is the linear translation equivariance.
Since weighted mean in vector spaces can be generalized to wFM on manifolds and further, wFM can be shown (see below) to be equivariant to group actions admitted by the manifold, we claim that wFM is a generalization of convolution operations to manifold-valued data.
Let { w i } Ni =1 be the weights such that they satisfy the convexity constraint, i.e., ∀ i, w i > 0 and (cid:80) i
w i
= 1 , then wFM, wFM ( { X i } , { w i } ) is deﬁned as:
[Equation]
Analogous to the equivariance property of convolution translations in vector spaces, we will now proceed to show that the wFM is equivariant under the action of the group of isometries of M .
We will ﬁrst formally deﬁne the group of isometries of M (let us denote it by G ) and then deﬁne the equivariance property and show that wFM is G -equivariant.
Deﬁnition 6 ( Group of isometries of M ( I ( M ) )) .
A diffeomorphism φ : M → M is an isometry if it preserves distance, i.e., d ( φ ( x ) , φ ( y ))
=
d ( x, y ) .
The set I ( M ) of all isometries of M forms a group with respect to function composition.
Rather than write an isometry as a function φ , we will write it as a group action.
Henceforth, let G denote the group
I ( M ) , and for g ∈ G , and x ∈ M , let g.x denote the result of applying the isometry g to point x .
Clearly M is a G set (see Dummit & Foote (2004) for the deﬁnition of a G set).
We will now deﬁne equivariance and show that wFM , is G -equivariant.
Deﬁnition 7 ( Equivariance).
Let X
and Y be G sets.
Then, F : X → Y is said to be G -equivariant if ∀ g ∈ G , ∀ x ∈ X , F ( g.x ) = g.F ( x ) .
Let U ⊂ M be an open ball inside which FM exists and is unique, let P be the set consists of all possible ﬁnite subsets of U .
Theorem 1.
Given { w i } satisfying the convex constraint, let F : P → U be a function deﬁned by { X i } (cid:55)→ wFM ( { X i } , { w i } ) .
Then, F is G -equivariant.
Proof.
Let g ∈ G and { X i } Ni =1 ∈ P , now, let M ∗ =
wFM
( { X i } , { w i } ) , as g.F ( { X i } ) =
g.M ∗ , it sufﬁces to show g.M ∗ is wFM ( { g.X i } , { w i } ) (assuming the existence and uniqueness of wFM ( { g.X i } , { w i } ) which is stated in the following claim).
Claim: Let U
= B r ( p ) for some r > 0 and p ∈ M .
Then, { g.X i } ⊂ B r ( g.p ) and hence wFM ( { g.X i } , { w i } ) exists and is unique.
Let (cid:102)
M be wFM ( { g.X i } , { w i } ) .
Then, (cid:80) Ni =
w i d 2 (cid:16) g.
X
i , (cid:102) M (cid:17) =
(cid:80) Ni
=
w i d 2 (cid:16) X
i , g − 1 .
(cid:102)
M (cid:17) .
Since, M ∗ = wFM ( { X i } , { w i } ) , hence, M ∗ = g − 1 .
(cid:102)
M , i.e., (cid:102)
M = g.M ∗ .
Thus, g.M ∗ = wFM ( { g.X i } , { w i } ) , which implies F is G -equivariant.
(cid:4)
Now we give some examples of M with the corresponding group of isometries G .
Let M = SPD ( n ) (the space of n × n symmetric positive-deﬁnite matrices).
Let d be the Stein metric on SPD ( n ) .
Then, the group of isometries G is O ( n ) (the space of n × n orthogonal matrices).
A class of Riemannian manifolds on which G acts transitively are called Riemannian homogeneous spaces.
We can see that on a Riemannian homogeneous space M , wFM is G -equivariant.
Equipped with a G -equivariant operator on M , we can claim that the wFM (deﬁned above) is a valid convolution operator since group equivariance is a unique deﬁning property of a convolution operator.
The rest of this subsection will be devoted to developing an efﬁcient way to compute wFM .
Let ω M > 0 be the Riemannian volume form.
Let p
X
be the probability density of a U -valued random variable X with respect to ω M on U ⊂ M , so that Pr ( X ∈ A ) = (cid:82)
A p X ( Y ) ω
M ( Y ) for any Borel-measurable subset A of
U. Let Y ∈ U , we can deﬁne the expectation of the real valued random variable d 2 ( , Y ) :
U → R by E (cid:2) d 2 ( , Y ) (cid:3)
= (cid:82) U d 2 ( X, Y )
p X ( X
) ω
M ( X ) .
Now, let w : U → (0 , ∞ ) be an integrable function and
(cid:82) U w ( X ) ω
M ( X ) = 1 .
Then, observe that, E w (cid:2) d 2 ( , Y ) (cid:3) :
= (cid:82)
U w ( X ) d 2 ( X, Y )
p X ( X
) ω
M ( X ) =
C (cid:82) U d 2 ( X, Y ) (cid:101)
p X ( X
) ω
M ( X ) =
C (cid:101) E (cid:2)
d 2 ( , Y ) (cid:3) .
Here, (cid:101) p X is the probability density corresponding to the probability measure (cid:101)
Pr deﬁned by, (cid:101)
Pr ( X ∈ X ) =
(cid:82) X (cid:101)
p X ( Y ) ω
M ( Y ) :
= (cid:82) X 1 C p
X
( Y ) w ( Y ) ω
M ( Y ) , where, X lies in the Borel σ -algebra over U and C = (cid:82) U p X
( Y )
w ( Y ) ω
M ( Y ) .
Note that the constant C > 0 , since p X is a probability density, w > 0 and M is orientable.
Thus, E w (cid:2)
d 2 ( , Y ) (cid:3) with respect to p X is proportional to (cid:101) E (cid:2) d 2 ( , Y ) (cid:3) with respect to (cid:101) p X .
Now, we will state the following proposition (the proof is in the appendix section).
Proposition 2.
[Equation]
Let { X i } Ni =1 be samples drawn from p X and (cid:110) (cid:101)
X i (cid:111) N i =1 be samples drawn from (cid:101) p X .
In order to compute wFM , we will now present an online algorithm (inductive FM Estimator – dubbed iFME ).
Given, { X i } Ni =1 ⊂ U and { w i := w ( X i ) } Ni =1 such that ∀ i, w i > 0 , the n th estimate, M n of wFM ( { X i } , { w i } ) is given by the following recursion:
[Equation]
In the above equation, Γ YX : [0 , 1] → U is the shortest geodesic curve from X to Y .
Observe that, in general wFM is deﬁned with (cid:80) Ni
=
w i = 1 , but in above deﬁnition, (cid:80) Ni
=
w i (cid:54) = 1 .
We can normalize { w i } to get { (cid:101) w i } by (cid:101) w i =
w i / ( (cid:80) i w i ) , but then Eq.
2 will not change as (cid:101) w n / (cid:16)(cid:80) nj =1 (cid:101) w j (cid:17) =
w n / (cid:16)(cid:80) nj
=
w
j (cid:17) .
This gives us an efﬁcient inductive/recursive way to deﬁne convolution operation on M .
Now, we state that the proposed wFM estimator is consistent (the proof is in the appendix).
Proposition 3.
Using the above notations and assumptions, let { X i } Ni
=1 be i.i.d.
samples drawn from p X on M .
Let the wFE be ﬁnite.
Then, M N converges a.s. to wFE as N → ∞ .
2. N ONLINEAR OPERATION BETWEEN W FM - LAYERS FOR M - VALUED D ATA
In the traditional CNN model, we need a nonlinear function between two convolutional layers similar to ReLU and softmax.
As argued in Mallat (2016), any nonlinear function used in CNN is basically a contraction mapping.
Formally, let F be a nonlinear mapping from U to V .
Let assume, U and V are metric spaces equipped with metric d U and d V respectively.
Then, F is a contraction mapping iff ∃ c < 1 such that, d V ( F ( x ) , F ( y ))
≤ c d U ( x, y ) .
F is a non-expansive mapping (Mallat, 2016)
iff d V ( F ( x ) , F ( y ))
≤
d U ( x, y ) .
X 1 X
XwFM
M
Z
Z d
M u y c y
Invariant layer
F C l a y e r X
Xw F M l a y e
[Figure]
s
[Figure]
One can easily see that the popular choices for nonlinear operations like ReLU, sigmoid are in fact non-expansive mappings.
Now, we will show that the function wFM as deﬁned in 1, is a contraction mapping for non-trivial choices of weights.
Let { X i } Ni =1 and { Y j } Mj =1 be the two set of samples on M .
Without any loss of generality, assume N ≤ M .
We consider the set
[Equation]
Clearly { Y j } Mj =1 ∈ U M
and we embed { X i } Ni =1 in U M as follows: we construct (cid:110) (cid:101)
X i (cid:111)
M i =1 from { X i } Ni =1 by deﬁning (cid:101)
X i
= X ( i − 1) mod N +1 .
Let us denote the embedding by ι .
Now, deﬁne the distance on U M as d (cid:18)(cid:110) (cid:101)
X i (cid:111)
M i =1 , { Y j } Mj = (cid:19) =
max i,j d ( X i , Y j ) .
The choice of weights for wFM is said to be trivial if one of the weights is 1 and hence the rest are 0 .
Proposition 4.
For all nontrivial choices of { α i } Ni =1 and { β j } Mj =1 satisfying the convexity constraint , ∃ c < 1 such that,
[Equation]
2. T HE INVARIANT ( LAST ) LAYER
We will form a deep network by cascading multiple wFM blocks each of which acts as a convolutiontype layer.
Each convolutional-type layer is equivariant to the group action, and hence at the end of the cascaded convolutional layers, the output is equivariant to the group action applied to the input of the network.
Let d be the number of output channels each of which outputs a wFM , hence each of the channels is equivariant to the group action.
However, in order to build a network that yields an output which is invariant to the group action, we now seek the last layer (i.e., the analogue to a linear classiﬁer) to be invariant to the group action.
The last layer is thus constructed as follows: Let { Z 1 , · · · , Z d } ⊂ M be the output of d channels and M u = FM (cid:16) { Z i } di =1 (cid:17) = wFM (cid:16) { Z i } di =1 , { 1 / d } d 1 (cid:17) be the unweighted FM of the outputs { Z i } di =1 .
Then, we construct a layer with d outputs whose i th output o i =
d ( M u , Z i ) .
Let c be the number of classes for the classiﬁcation task, then, a fully connected (FC) layer with inputs { o i } and c output nodes is build.
A softmax operation is then used at the c output nodes to obtain the outputs { y i } ci =1 .
In the following proposition we claim that this last layer with { Z i } di =1 inputs and { y i } ci =1 outputs is group invariant.
Proposition 5.
The last layer with { Z i } di =1 inputs and { y i } ci =1 outputs is group invariant.
In Fig.
1 we present a schematic of ManifoldNet depicting the different layers of processing the manifold-valued data as described above in Sections 2.1-2.3.
E XPERIMENTS
In this section we present performance of the ManifoldNet framework on several standard computer vision problems.
The breadth of application coverage here includes classiﬁcation and reconstruction problems.
We begin with a video classiﬁcation problem and then present a reconstruction problem using an auto-encoder-decoder set up.
3. V IDEO C LASSIFICATION
We start by using the method in Yu & Salzmann (2017) which we summarize here.
Given a video with dimensions F × 3 × H × W of F frames, 3 color channels and a frame size of H × W , we can apply a convolution layer to obtain an output of size F × C × H (cid:48)
× W (cid:48) consisting of C channels of size H (cid:48)
× W (cid:48) .
We compute the covariance matrix of the channels to obtain a sequence of F symmetric positive (semi) deﬁnite matrices of size C × C . Covariance Matrices C × C C × C
C × C C × C ... [ , , … , ] ∈ y 1 yy n R n = d ( , FM ({ }))
y i x i x i wFM ({ },{ }) x i w i . Text C × C C × C
C × C
C × C
C × C From here we can apply a series of temporal ManifoldNet wFM s to transform the F × C × C input to a temporally shorter F (cid:48)
× K × C × C output, where K is the number of wFM channels.
Within the temporal ManifoldNet wFM s we use a simple weight normalization to ensure that the weights are within [0 , 1] , and for the weights w i of any output channel we add a weight penalty of the form ( (cid:80) w i − 1) 2 to the loss function to ensure that we obtain a proper wFM .
We then reshape this to F (cid:48)
K × C × C and pass it through an invariant ﬁnal layer (section 2.3) to obtain a vector of size F (cid:48) K .
Finally, a single FC+SoftMax layer is applied to produce a classiﬁed output.
We call this the SPD temporal convolutional architecture network (SPD-TCN).
Figure 2 illustrates the network architecture described above.
In general, the SPD-TCN tends to perform very well on video classiﬁcation tasks while using very few parameters , and runs efﬁciently due to the wFM structure.
[Table]
We tested the SPD-TCN on the
Moving MNIST dataset (Srivastava et al., 2015).
Recently, in
Chakraborty et al.
(2018) authors developed a manifold valued recurrent network architecture, dubbed SPD-SRU, which produced state-of-the-art classiﬁcation results on Moving
MNIST dataset in comparison to LSTM (Hochreiter & Schmidhuber, 1997), SRU (Oliva et al., 2017), TT-LSTM and TT-GRU (Yang et al., 2017) networks.
For the LSTM and SRU networks, convolution layers are also used before the recurrent unit.
We will compare directly with the results presented in Chakraborty et al.
For details of the various architectures used please see section 5 of Chakraborty et al.
We will also compare with Huang & Van Gool (2017) , which proposes a set of layers for learning SPD matrices, dubbed SPDNet. When using SPDNet we ﬁrst downsample the videos to 20 × 20 frames and compute the covariance matrix of the frames, which is then fed to an architecture similar to that of the emotion recognition experiment of Huang & Van Gool (2017) Speciﬁcally, we use three blocks of BiMap/ReEig layers, since such a conﬁguration gave the best validation performance. The transformation sizes for these blocks are set to 400 × 200 , 200 ×and 100 × 50 respectively.
The Moving MNIST data generated in Srivastava et al. (2015) consists of 1000 samples, each of 20 frames.
Each sample shows two randomly chosen MNIST digits moving within a 64 × 64 frame, with the direction and speed of movement ﬁxed across all samples in a class.
The speed is kept the same across different classes, but the digit orientation will differ across two different classes.
We summarize the 10 -fold cross validation results for several orientation differences between classes in Table 1.
For this experiment the SPD-TCN will consist of a single wFM layer with kernel size 5 and stride 3 returning 8 channels, making for an 8 × 8 covariance matrix.
We then apply three temporal SPD wFM layers of kernel size 3 and stride 2 , with the following channels 1 → 4 → 8 → 16 , i.e. after these three temporal SPD wFM s we have 16 temporal channels.
This 16 × 8 × 8 is used as an input to the invariant ﬁnal layer to get a 16 dimensional output vector, which is transformed by a FC+SoftMax layer to obtain the output.
3. D IMENSIONALITY R EDUCTION
[Figure]
Here we present experiments demonstrating the applicability of the theory layed out in
Section 2 to the case of linear dimensionality reduction, speciﬁcally principal component analysis (PCA), which is the workhorse of many machine learning algorithms.
In
Chakraborty et al.
, authors presented an online subspace averaging algorithm for construction of principal components via intrinsic averaging on the Grassmannian.
In this section, we achieve the intrinsic Grassmann averaging process in the framework of ManifoldNet to compute the principal subspaces and achieve the dimensionality reduction.
In the context of DNNs, dimensionality reduction is commonly achieved via an autoencoder architecture.
More recently, DNNs have shown promising results when the data manifold is intrinsically non-linear, as in the case of natural images.
In the deep learning community this has become a ﬁeld in its own right, known as representation learning or feature learning (Bengio et al., 2013) with works including Vincent et al.
(2010), Kingma & Welling (2013)
Oord et al.
(2016), Van
Den Oord et al.
, Kingma & Dhariwal (2018).
Many of these architectures are modiﬁcations of the traditional autoencoder network, which involves learning an identity map through a small latent space.
In our application, we modify the traditional autoencoder model by adding a ManifoldNet layer to perform a learned linear dimensionality reduction in the latent space, although in principal, our techniques can be applied to most autoencoder based models such as the variational autoencoders.
To compute a linear subspace in the ManifoldNet framework we use an intrinsic averaging scheme on the Grassmannian.
A point on the Grassmannian Gr ( k, n ) corresponds to k -dimensional subspace of R n and thus can be speciﬁed by an orthonormal basis X .
Chakraborty et al.
(2017) proposed an efﬁcient intrinsic averaging scheme on Gr ( k, n ) that converges to the k -dimensional principal subspace of a normally distributed dataset in R n .
In the ManifoldNet framework we can modify this technique to learn a wFM of points on the Grassmannian that corresponds to a subspace of the latent space which minimizes the reconstruction error by using a Grassmannian averaging layer that learns the weights in the wFM .
This essentially will give us a lower dimensional representation of the samples after projecting them on to the learned subspace.
Note that combining the convergence proof in Chakraborty et al.
(2017) and
Proposition 2 (ii), we claim that the wFM learned using the ManifoldNet asymptotically converges to the principal subspace.
Now, we give a detailed description of our experimental setup to show the applicability of ManifoldNet to dimensionality reduction.
3.2. V IDEO R ECONSTRUCTION E XPERIMENT
A traditional convolutional autoencoder performs non-linear dimensionality reduction by learning an identity function through a small latent space.
A common technique used when the desired latent space is smaller than the output of the encoder is to apply a fully connected layer to match dimensions.
We replace this fully connected layer by a weighted subspace averaging and projection block, called the Grassmann averaging layer.
Speciﬁcally, we compute the wFM of the output of the encoder to get a subspace in the encoder output space.
We then project the encoder output onto this space to obtain a reduced dimensionality latent space.
In general this offers a signiﬁcant parameter reduction while also increasing the reconstruction error performance of the autoencoder and giving realistic reconstructions.
We call an autoencoder with the Grassmann averaging block
[Figure]
autoencoder as shown in Fig.
In the experiments we compare this to other dimensionality reduction techniques, including regular autoencoders that use fully connected layers to match encoder and latent space dimensions.
We begin by testing on aframe color sample of video from theﬁlm “Santa Clause Conquers the Martians” of frame size 320 × 240 .
Here we use an 8 layer encoding-decoding architecture with Conv → ELU → Batchnorm layers, with the ﬁnal layer applying a sigmoid activation to normalize pixel values.
The encoder returns a feature video consisting of 128 channels of size 120 for a dimension of 1000 × 15360 .
We compare a fully connected layer to a Grassmann averaging layer, both mapping to a desired latent space of dimension 1000 × 20 .
The per pixel average reconstruction error for the Grassmann block network is 0 .
0110 , compared to 0 .
for the fully connected network
[Figure]
We also observe a parameter reduction of 46% , which can be attributed to the number of parameters in the large fully connected layer.
In Fig.
5, the computation time is plotted against error tolerance for the autoencoder and the iFME+autoencoder.
We can see that iFME+autoencoder achieves faster convergence than autoencoder.
It is possible to obtain a low reconstruction error on autoencoding tasks and still observe low visual quality reconstructions.
To ensure this is not the case we run the same experiment on 300 frames of the 1280 ×short ﬁlm 1 , with a latent space frame dimension of 300x50.
In Fig.
4 we compare the visual quality of our autoencoder to that of PCA with 50 principcal components, i.e., we reduce the dimension from 1280 × 720 × 3 to 50 .
The entire sample reconstruction is shown in 2 (in the same order as in Fig. 4).
C ONCLUSIONS
In this paper, we presented a novel deep network called ManifoldNet suited for processing manifoldvalued data sets.
Inputs to the ManifoldNet are manifold-valued and not real or complex-valued functions deﬁned on non-Euclidean domains.
Our key contributions are: (i)
A novel deep network to be perceived as a generalization of the CNN to the case when the input data are manifold-valued using purely intrinsic operations on the manifold where the data reside.
(ii) Analogous to convolutions in vector spaces – which can be computed using the weighted mean – we present wFM operations on the manifold and prove the equivariance of the wFM to natural group operations admitted by the manifold.
This equivariance allows us to share the learned weights within a layer of the ManifoldNet.
(iii)
An efﬁcient recursive wFM estimator that is provably convergent is presented.
(iv) Experimental results demonstrating the efﬁcacy of the ManifoldNet for, (a) video classiﬁcation and (b) principal component computation from videos and reconstruction are also presented.
A PPENDIX
Proposition 2.
(i) supp ( p X )
=
supp ( (cid:101) p X ) .
(ii) wFE ( X , w ) =
FE (cid:16)(cid:101) X (cid:17) .
Proof.
Let X ∈ supp ( p X ) , then, p X ( X ) > 0 .
Since, w ( X ) > 0 , hence, (cid:101) p X ( X ) > 0 and thus,
X ∈ supp ( (cid:101) p X ) .
On the other hand, assume (cid:101) X to be a sample drawn from (cid:101) p X .
Then, either p X (cid:16) (cid:101) X (cid:17) = 0 or p X (cid:16) (cid:101) X (cid:17) > 0 .
If, p X (cid:16) (cid:101) X (cid:17) = 0 , then, (cid:101) p X (cid:16) (cid:101) X (cid:17) = 0 which contradicts our assumption.
Hence, p X (cid:16) (cid:101) X (cid:17) > 0 , i.e., (cid:101) X ∈ supp
p X ) .
This concludes the proof of part (i).
Let X and (cid:101)
X be the M valued random variable following p X and (cid:101) p X respectively.
We deﬁne the weighted Fréchet expectation (wFE) of X as:
[Equation]
Observe,
[Equation]
Hence, we get FE (cid:16)(cid:101) X (cid:17) =
wFE ( X , w ) , as C is independent of the choice of Y , which concludes the proof of part (ii).
[Equation]
Proposition 3.
Using the notations and assumptions used in the paper, let { X i } Ni
=1 be i.i.d.
samples drawn from p X on M .
Let the wFE be ﬁnite.
Then, M N converges a.s. to wFE as N → ∞ .
Proof.
Using Proposition 2, we know that ∃ (cid:101)
p
X such that, wFE ( X , w )
=
FE (cid:16)(cid:101) X (cid:17) .
Thus, it is enough to show the consistency of our proposed estimator when weights are uniform.
In order to prove the consistency, we will split the proof into two cases namely, manifolds with (i) non-positive sectional curvature and (ii) non-negative sectional curvature.
The reason for doing this split is so that we can use existing theorems in literature for proving the result.
We will use the theorems proved in Sturm (2003) and Bonnabel (2013) for manifolds with non-positive and non-negative sectional curvatures respectively.
Note that the proof holds only for manifolds with a uniform sign of sectional curvatures.
Theorem 6 ( M has non-negative sectional curvature) .
Using the above notations, if ∃ A > 0 such that, d ( M n , X n +1 ) ≤
A for all n .
Then, M N converges a.s. to wFE as N → ∞ (see Bonnabel (2013) for the proof).
Theorem 7 ( M has non-positive sectional curvature) .
Using the above notations M N converges a.s. to wFE as N → ∞ (see Sturm (2003) for the proof). [Equation]
Proposition 5. The last layer with { Z i } di =1 inputs and { y i } ci =1 outputs is group invariant. Proof. Using the above construction, let W ∈ R c × d and b ∈ R c be the weight matrix and bias respectively of the FC layer.
Then,
[Equation]
where, F is the softmax function.
In the above equation, we treat d ( M u , Z ) as the vector [ d ( M u , Z 1 ) , · · · , d ( M u , Z d )] t .
Observe that, g.M u = FM (cid:16) { g.Z i } di =1 (cid:17) .
As each of the d channels is group equivariant, Z i becomes g.Z i .
Because of the property of the distance under group action, d ( g.M u , g.Z i ) =
d ( M u , Z i ) .
Hence, one can see that if we change the inputs { Z i } to { g.Z i } , the output y will remain invariant.
(cid:4)