The Gyro-Structure of Some Matrix Manifolds
Abstract
In this paper, we study the gyrovector space structure ( gyro-structure ) of matrix manifolds.
Our work is motivated by the success of hyperbolic neural networks (HNNs) that have demonstrated impressive performance in a variety of applications.
At the heart of HNNs is the theory of gyrovector spaces that provides a powerful tool for studying hyperbolic geometry.
Here we focus on two matrix manifolds, i.e., Symmetric Positive Deﬁnite (SPD) and Grassmann manifolds, and consider connecting the Riemannian geometry of these manifolds with the basic operations, i.e., the binary operation and scalar multiplication on gyrovector spaces.
Our work reveals some interesting facts about SPD and Grassmann manifolds.
First,
SPD matrices with the Afﬁne-Invariant (AI) and Log-Euclidean (LE) geometries have rich structure with strong connection to hyperbolic geometry.
Second, linear subspaces, when equipped with our proposed basic operations, form what we call gyrocommutative and gyrononreductive gyrogroups .
Furthermore, they share remarkable analogies with gyrovector spaces.
We demonstrate the applicability of our approach for human activity understanding and question answering.
Introduction
Data lying on matrix manifolds are commonly encountered in various applied areas such as medical imaging [3, 37], shape analysis [41], drone classiﬁcation [6], image recognition [11] and human behavior analysis [10, 15, 16, 20, 21, 22, 31, 40].
These data arise from constraint sets of the problem, for which there is a natural representation of elements in the form of matrix arrays [2].
Due to the non-Euclidean nature of these data, traditional optimization algorithms usually fail to obtain good results in the matrix manifold setting.
While a large body of works [6, 7, 8, 10, 20, 21, 22, 31, 33, 48] has been developed to generalize traditional optimization algorithms to this setting, there is still a lack of works that translate the language of differential geometry to basic operations on matrix manifolds so that they can be used in computational building blocks of neural network models on these manifolds just as basic operations on Euclidean spaces, e.g., matrix-matrix addition and scalar-matrix multiplication are used in deep neural networks (DNNs).
To address the above issue, we propose a novel framework based on the theory of gyrovector spaces [44, 45, 46] that has been successfully applied in the context of HNNs [12, 39].
Our aim is to uncover hidden analogies between the target manifolds and Euclidean spaces in the same way that we uncover hidden analogies between hyperbolic and Euclidean spaces [12, 39, 44, 45, 46].
Although there are some works [1, 17, 24, 25, 26, 27, 29] showing the gyro-structure of SPD manifolds with the
AI geometry, none of them provides a rigorous mathematical formulation for the connection between the basic operations of [1, 17, 18, 24, 25, 26, 27] and the AI geometry of SPD manifolds.
In this paper, we show how the basic operations can be constructed from the Riemannian geometry of matrix manifolds, and derive their compact expressions for SPD and Grassmann manifolds.
In the case of
SPD manifolds with the LE geometry [3], one recovers precisely the operations of [3] that give these manifolds a vector space structure.
In the case of Grassmann manifolds, we obtain gyrocommutative and gyrononreductive gyrogroups that share remarkable analogies with gyrovector spaces.
To the 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
best of our knowledge, our work is the ﬁrst that studies the structure of Grassmann manifolds under the framework of gyrovector spaces.
Our main contributions are (1) We propose a method for constructing some basic operations, i.e., matrix-matrix addition and scalar-matrix multiplication on SPD and Grassmann manifolds; (2) We derive compact expressions of these operations for the considered manifolds; (3) We verify the gyro-structure of SPD manifolds, and some axioms of gyrovector spaces for Grassmann manifolds; (4) We showcase our approach on the tasks of human activity understanding and question answering.
Background 2.1 Gyrovector Spaces
Gyrovector spaces form the setting for hyperbolic geometry in the same way that vector spaces form the setting for Euclidean geometry [44, 45, 46].
We ﬁrst recap the deﬁnitions of gyrogroups and gyrocommutative gyrogroups proposed in [44, 45, 46].
For greater mathematical detail and in-depth discussion, we refer the interested reader to these papers.
Deﬁnition 2.1 ( Gyrogroups [46]).
A pair ( G, ⊕ ) is a groupoid in the sense that it is a nonempty set,
G , with a binary operation, ⊕ .
A groupoid ( G, ⊕ ) is a gyrogroup if its binary operation satisﬁes the following axioms for a, b, c ∈ G :
(G1)
There is at least one element e ∈ G called a left identity such that e ⊕ a = a .
(G2)
There is an element (cid:9) a ∈ G called a left inverse of a such that (cid:9) a ⊕ a = e . (G3)
There is an automorphism gyr[ a, b ] : G → G for each a, b ∈ G such that
a
[Equation] [Equation]
[Equation]
gyr[ a, b ] c (Left Gyroassociative Law) .
The automorphism gyr[ a, b ] is called the gyroautomorphism, or the gyration of G generated by a, b .
(G4) gyr
[ a, b ] = gyr
[ a ⊕ b, b ] (Left Reduction Property).
Deﬁnition 2.2 ( Gyrocommutative Gyrogroups [46]).
A gyrogroup ( G, ⊕ ) is gyrocommutative if it satisﬁes
[Equation]
The following deﬁnition of gyrovector spaces is slightly different from Deﬁnition 3.2 in [46].
Deﬁnition 2.3 ( Gyrovector Spaces).
A gyrocommutative gyrogroup ( G, ⊕ ) equipped with a scalar multiplication
[Equation]
is called a gyrovector space if it satisﬁes the following axioms for s, t ∈ R and a, b, c ∈ G :
(V1) 1 (cid:12) a = a, 0 (cid:12) a = t (cid:12) e = e, and ( − 1) (cid:12) a = (cid:9) a .
(V2) ( s + t ) (cid:12) a = s (cid:12) a ⊕ t (cid:12) a . (V3) ( st ) (cid:12) a = s (cid:12) ( t (cid:12) a ) .
(V4) gyr
[ a, b ]( t (cid:12) c ) =
t (cid:12) gyr[ a, b ] c . (V5) gyr
[ s (cid:12) a, t (cid:12) a ] = Id , where Id is the identity map.
Note that the axioms of gyrovector spaces considered in our work are more strict than those proposed in [24, 25, 26, 27].
Thus many results proved in [24, 25, 26, 27] can be applied to our case, which gives rise to interesting applications.
Proposed Approach
For simplicity of exposition, we will concentrate on real matrices.
Denote by M n,m the space of n × m matrices, Sym + n the space of n × n SPD matrices, Sym n the space of n × n symmetric matrices, O n the space of n × n orthogonal matrices, Gr n,p the p -dimensional subspaces of R n .
Let
M be a Riemannian homogeneous space, T P M be the tangent space of M at P ∈ M .
Denote by exp( P ) and log( P ) the usual matrix exponential and logarithm of P , Exp P ( W ) the exponential map at P that associates to a tangent vector W ∈ T P M a point of M , Log P ( Q ) the logarithmic map of Q ∈ M at P .
Let T P → Q ( W ) be the parallel transport of W from P to Q along geodesics connecting P and Q .
We will use superscripts for the exponential and logarithmic maps, and the parallel transport to indicate their associated Riemannian metric (in the case of SPD manifolds) or the target manifolds (in the case of Grassmann manifolds).
The following deﬁnitions generalize those of [12] to the matrix manifold setting.
Deﬁnition 3.1.
Let r be a positive real number.
The binary operation P ⊕ r Q where P , Q ∈ M is obtained by projecting Q r in the tangent space at the identity element I ∈ M with the logarithmic map, computing the parallel transport of this projection from I to P r along geodesics connecting I and P r , and then projecting it back on the manifold with the exponential map, i.e.,
[Equation]
Deﬁnition 3.2.
The scalar multiplication t ⊗ P where t ∈ R and P ∈ M is obtained by projecting
P in the tangent space at the identity element I ∈ M with the logarithmic map, multiplying this projection by the scalar t in T I M , and then projecting it back on the manifold with the exponential map, i.e.,
[Equation]
In addition to the basic operations deﬁned above, we need to determine an automorphism in order to verify the gyro-structure of a given matrix manifold.
In the following, we will provide such automorphisms and derive compact expressions of the basic operations for SPD and Grassmann manifolds.
The obtained expressions will ease the task of verifying the axioms of gyrovector spaces.
Also, they often lead to simple and efﬁcient implementations of neural networks on the considered manifolds.
Furthermore, they enable effective generalizations of some operations on Euclidean spaces, e.g., matrix scaling to these manifolds (see Sections 4.1.2 and 4.2.2).
3.1 Gyrovector Spaces of SPD Matrices
We investigate the gyro-structure of SPD manifolds with the AI and LE geometries in Sections 3.1. and 3.1.2, respectively.
The AI and LE frameworks are reviewed in the supplemental material.
We refer the interested reader to that document for our theoretical results that reveal hidden analogies between SPD manifolds with the AI and LE geometries and Euclidean spaces.
3.1.1 AI Gyrovector Spaces
We ﬁrst examine SPD manifolds with the AI geometry.
Lemma 3.3 gives a compact expression of the binary operation (matrix-matrix addition).
Lemma 3.3.
For P , Q ∈ Sym + n , the binary operation P ⊕ rai Q is given as
[Equation]
Proof See the supplemental material.
An implicit assumption in Eq.
(3) is that positive deﬁnite matrices are taken from the computations related to matrix powers.
This assumption is used in all computations in Sections 3.1.1 and 3.1.2.
The identity element of Sym + n is the n × n identity matrix I n .
Then, from Eq.
(3), the inverse of P is given by
[Equation]
Lemma 3.4.
For P ∈
Sym + n and t ∈ R , the scalar multiplication t ⊗ ai P is given as
[Equation]
Proof See the supplemental material.
Deﬁnition 3.5 ( AI Gyrovector Spaces).
Deﬁne a binary operation ⊕ rai and a scalar multiplication ⊗ ai by Eqs.
(3) and (4), respectively.
Deﬁne a gyroautomorphism generated by P and Q as
[Equation] [Equation]
where F rai ( P , Q )
=
( P r 2 Q r P
[Equation]
Theorem 3.6.
Gyrogroups (Sym + n , ⊕ rai ) with the scalar multiplication ⊗ ai form gyrovector spaces (Sym + n , ⊕ rai , ⊗ ai ) .
Proof See the supplemental material.
The expressions of the binary operation, scalar multiplication, and gyroautomorphism given in
Eqs.
(3), (4), and (5) have appeared in [1, 17, 18].
However, these works concern with algebraic structures referred to as generalized gyrovector spaces where their considered set of axioms is different from the one in Section 2.1.
The expressions given in Eq.
(5) with r = 1 and r = 2 have also appeared in [24, 25, 26, 27].
However, no general form of the gyroautomorphism is given in these works.
3.1.2 LE Gyrovector Spaces
The LE metrics have proven to yield similar results as the AI metrics in practice, but with much simpler and faster computations [3].
This motivates us to study the gyro-structure of SPD manifolds with the LE geometry.
As far as we know, this topic has not been investigated in previous works.
The result given in the following lemma is not trivial as closed formulae do not exist for the parallel transport associated with the LE geometry.
Lemma 3.7.
For P , Q ∈
Sym + n , the binary operation P ⊕ rle Q is given as
[Equation]
Proof See the supplemental material.
From Lemma 3.7, the inverse of P is given by
[Equation]
Similarly to the scalar multiplication ⊗ ai , the scalar multiplication ⊗ le is constructed from Eq.
Lemma 3.8 gives an expression of this operation that is straightforward.
Lemma 3.8.
For P ∈
Sym + n and t ∈ R , the scalar multiplication t ⊗ le P is given by
[Equation]
Deﬁnition 3.9 ( LE Gyrovector Spaces).
Deﬁne a binary operation ⊕ rle and a scalar multiplication ⊗ le by Eqs.
(6) and (7), respectively.
Deﬁne a gyroautomorphism generated by P and Q as gyr
[Equation]
Theorem 3.10.
Gyrogroups (Sym + n , ⊕ rle ) with the scalar multiplication ⊗ le form gyrovector spaces (Sym + n , ⊕ rle , ⊗ le ) .
Proof See the supplemental material.
The conclusion of Theorem 3.10 agrees with [3] which shows that the space of SPD matrices with the LE geometry has a vector space structure.
This vector space structure is given by the operations deﬁned in [3] that turn out to be the binary operation and scalar multiplication on LE gyrovector spaces in the speciﬁc case where r = 1 .
Indeed, it can be shown that the mapping exp : (Sym n , + , . )
→ (Sym + n , ⊕ rle , ⊗ le ) is a vector space isomorphism.
Thus, for any r , the operations deﬁned above also turn the space of SPD matrices with the LE geometry into a vector space.
This generalizes the result of [3].
3.2 Grassmann Gyrocommutative and Gyrononreductive Gyrogroups
In the previous sections, we concern with the non-positively curved spaces of SPD matrices.
In this section, we focus on another family of matrix manifolds, i.e., the non-negatively curved Grassmann manifolds.
We adopt the following deﬁnition for Grassmann manifolds:
[Equation]
The Riemannian geometry of Grassmann manifolds is reviewed in the supplemental material.
Let I n,p = (cid:20)
I p   (cid:21) be the identity element of Gr n,p .
For the sake of convenience, we denote
P = Log gr I n,p ( P ) .
For P , Q ∈ Gr n,p , assuming that I n,p and P are not in each other’s cut locus (see the supplemental material for a deﬁnition of cut locus of Grassmann manifolds), and I n,p and Q are not in each other’s cut locus, the binary operation is deﬁned as
[Equation]
Lemma 3.11.
For P , Q ∈ Gr n,p , the binary operation P ⊕ gr Q is given as
[Equation]
where [ ., . ] denotes the matrix commutator.
Proof See the supplemental material.
Lemma 3.12 gives a closed-form expression of the binary operation in terms of P and Q .
Lemma 3.12.
For P , Q ∈ Gr n,p , the binary operation P ⊕ gr Q is given as
[Equation]
Proof See the supplemental material.
For P ∈ Gr n,p such that I n,p and P are not in each other’s cut locus, the inverse (cid:9) gr P of P is deﬁned as
[Equation]
The scalar multiplication is deﬁned as in Eq.
(2) (subject to the assumption stated above).
Lemma 3.13.
For P ∈ Gr n,p and t ∈ R , the scalar multiplication t ⊗ gr P is given by
[Equation]
Proof See the supplemental material.
Lemma 3.14 gives a closed-form expression of the scalar multiplication in terms of t and P .
Lemma 3.14.
For P ∈ Gr n,p and t ∈ R , the scalar multiplication t ⊗ gr P is given by
[Equation]
Proof See the supplemental material.
Grassmann manifolds, when equipped with the above operations, do not form gyrovector spaces.
However, as we will show later, they still form spaces that verify most of the axioms of gyrovector spaces.
Note that since these operations can only be deﬁned under the assumptions stated at the beginning of this section, all the axioms in Section 2.1 must be considered under these assumptions.
We thus make them implicitly in the following.
In order to specify the structure of Grassmann manifolds, we need to deﬁne some new algebraic structures that are generalizations of groups (Deﬁnitions 3.15 and 3.16) and vector spaces (Deﬁnition 3.17).
Deﬁnition 3.15 ( Nonreductive Gyrogroups).
A groupoid ( G, ⊕ ) is a nonreductive gyrogroup if its binary operation satisﬁes axioms (G1), (G2), and (G3).
Deﬁnition 3.16 ( Gyrocommutative and Gyrononreductive Gyrogroups).
A nonreductive gyrogroup ( G, ⊕ ) is gyrocommutative and gyrononreductive if it satisﬁes the Gyrocommutative Law.
Deﬁnition 3.17 ( Nonreductive Gyrovector Spaces).
A gyrocommutative and gyrononreductive gyrogroup ( G, ⊕ ) equipped with a scalar multiplication (cid:12) is called a nonreductive gyrovector space if it satisﬁes axioms (V1), (V2), (V3), (V4), and (V5).
Deﬁnition 3.18 gives the expression of a gyroautomorphism in Grassmann manifolds.
Deﬁnition 3.18 ( Grassmann Gyrocommutative and Gyrononreductive Gyrogroups).
Deﬁne a binary operation ⊕ gr and a scalar multiplication ⊗ gr by Eqs.
(8) and (9), respectively.
For
P , Q ∈ Gr n,p , assuming that I n,p and P are not in each other’s cut locus, I n,p and Q are not in each other’s cut locus, I n,p and P ⊕ gr Q are not in each other’s cut locus, then a gyroautomorphism generated by P and Q can be deﬁned as
[Equation]
where F gr ( P , Q ) is given by
[Equation] [Equation]
The following identities are required for the veriﬁcation of some axioms of nonreductive gyrovector spaces for Grassmann manifolds subject to certain conditions (Theorem 3.20).
These identities are new to the best of our knowledge.
Lemma 3.19.
For P , F ∈ Gr n,p , O ∈
O n , ∆ ∈ T P Gr n,p , m ∈ N , and s ∈ R , the following identities hold:
Log gr OFO T ( OPO T )
=
O Log gr F ( P ) O T ; 2.
Exp gr OPO T ( O ∆ O T ) =
O Exp gr P (∆)
O T ; 3.
[∆ , P ] m =
( I n − 2 P )( − [∆ , P ]) m
( I n − 2 P ) ; 4.
exp( s [∆ , P ])
=
( I n − 2 P ) exp( − s [∆ , P ])
( I n − 2 P ) .
Proof See the supplemental material.
We are now ready to state the main results of this section.
Theorem 3.20.
Groupoids (Gr n,p , ⊕ gr ) form gyrocommutative and gyrononreductive gyrogroups.
Furthermore, gyrocommutative and gyrononreductive gyrogroups (Gr n,p , ⊕ gr ) with the scalar multiplication ⊗ gr satisfy axioms (V1) and (V4), and axioms (V2), (V3), and (V5) under the following conditions:
[Equation]
p
[Equation]
R , and s
[Equation]
R such that:
[Equation]
where λ i is an eigenvalue of [ P , I n,p ] , and Im( λ i ) is the imaginary part of λ i .
(V3) ( st ) ⊗ gr P = s ⊗ gr ( t ⊗ gr P ) for P ∈ Gr n,p , s ∈ R , and t ∈ R such that:
[Equation]
⊗
[Equation]
Id for P ∈ Gr n,p , and s, t ∈ R such that:
[Equation]
Proof See the supplemental material.
The following corollaries are useful in practice for the veriﬁcation of axioms (V2), (V3), and (V5).
Corollary 3.21.
Gyrocommutative and gyrononreductive gyrogroups (Gr n,p , ⊕ gr ) with the scalar multiplication ⊗ gr satisfy axioms (V2), (V3), and (V5)
if the right-hand sides of the inequalities stated in Theorem 3.20 are replaced with
[Equation]
where (cid:107) .
(cid:107) denotes the Hilbert-Schmidt norm.
Proof See the supplemental material.
Corollary 3.22.
Gyrocommutative and gyrononreductive gyrogroups (Gr n,p , ⊕ gr ) with the scalar multiplication ⊗ gr satisfy axioms (V2), (V3), and (V5) under the following conditions: (V2) | s | ≤ 1 ;
[Equation]
≤
[Equation]
(V
[Equation]
max {| s | , | t | , | s + t |} ≤ 1 .
Proof See the supplemental material.
It follows from Corollary 3.22 that if max {| s | , | t | , | s + t |} ≤ 1 , then Grassmann manifolds, when equipped with the basic operations deﬁned in Eqs.
(8) and (9), verify all the axioms of nonreductive gyrovector spaces.
Applications
To showcase our approach, we propose new methods for human activity understanding and question answering.
We refer the interested reader to the supplemental material for more applications.
4.1 Human Activity Understanding
In this section, we develop a class of RNNs on SPD manifolds for human activity understanding.
It is worth mentioning that the operations deﬁned in Section 3.1 as well as those constructed in
Section 4.1.2 can be used to build any type of neural networks on SPD manifolds, e.g., convolutional neural networks.
However, since RNNs are based on update equations that involve the basic operations on matrices, they are well suited for validating our approach.
4.1.1 Problem Formulation
Human activities can be recognized from low/mid level features [35, 36] or high-level poses [23, 32, 33, 34].
We use 3D skeleton data as they have shown to outperform low/mid level features for the considered task [23].
The goal is to build a model that for each sequence of 3D positions of body (hand) joints identiﬁes the action performed by the person (or group of persons) in the sequence.
4.1.2 Proposed Method
We will make use of the basic operations on AI and LE gyrovector spaces and the concept of gyroderivative in these spaces (see the supplemental material) that is similar to hyperbolic derivative [4] in Möbius gyrovector spaces.
We also need to generalize some operations of Euclidean RNNs to the
SPD manifold setting.
Here we focus on 2 operations, i.e., matrix scaling and pointwise nonlinearity.
Matrix Scaling If P ∈ Sym + n , W ∈ R n , W > 0 , then the matrix scaling W ⊗ vspd P is given by
[Equation]
where U diag( V ) U T is the eigenvalue decomposition of P , and W ∗ V is the element-wise multiplication.
Pointwise nonlinearity If ϕ is a pointwise nonlinear activation function, then the pointwise nonlinearity ϕ ⊗ a ( P ) is given by
[Equation]
where (cid:15) > 0 is a rectiﬁcation threshold, and U diag( V ) U T is the eigenvalue decomposition of P .
By adapting a class of models that are invariant to time rescaling [42] to the SPD manifold setting (see the supplemental material), we obtain the following update equations for our models:
[Equation]
where X t ∈ Sym + n is the input at frame t , H t − 1 , H t ∈
Sym + n are the hidden states at frames t −and t , respectively, W h , W x ∈ R n , and α ∈ R is a learnable parameter.
4.1.3 Implementation Details
In order to retain the correlation of neighboring joints [5, 49] and to increase feature interactions encoded by covariance matrices, we ﬁrst identify a closest left (right) neighbor of every joint based on their distance to the hip (wrist) joint 1 , and then combine the 3D coordinates of each joint and those of its left (right) neighbor to create a feature vector for the joint.
For a given frame t , a mean vector µµµ t and a covariance matrix ΣΣΣ t are computed from the set of feature vectors of the frame and then combined [30] to create a SPD matrix as
[Equation]
The lower part of matrix log( Y t ) is ﬂattened to obtain a vector ˜ v t .
All vectors ˜ v t within a time window [ t, t + c − 1] where c is a constant are used to compute a covariance matrix as Z t =c (cid:80) t
+ c − 1 i =
t (˜ v
i − ¯ v t )
(˜ v i − ¯ v t ) T , where ¯ v t = 1 c (cid:80) t + c − 1 i =
t ˜ v i .
Matrix Z t is then the input data at frame t of the networks.
Our network GyroAI-HAUNet is illustrated in Fig.
1a.
For classiﬁcation, the network output is projected to the tangent space at the identity matrix using the logarithmic map.
The lower part of the resulting matrix is ﬂattened and then is fed to a fully-connected layer.
[Figure]
[Figure] [Figure]
[Table]
4.1.4 Results
We use three datasets, i
[Table]
e., HDM05 [16], FPHA [13]
[Table]
and NTU RGB+D 60 (NTU60) [38].
These datasets include three different types of activities: body actions (HDM05), hand actions (FPHA), and interaction actions (NTU60).
In the same spirit of previous works [6, 20, 40], we are interested in comparing manifold networks with a focus on SPD manifolds, and do not necessarily seek state-ofthe-art performance for the target task.
We use a temporal pyramid representation for each sequence.
At temporal pyramid M , a given sequence is partitioned into M subsequences of equal size.
Each subsequence is then fed to a model with its own parameter set.
The outputs from all the models are concatenated to create a ﬁnal representation of the sequence.
We run each model three times and report the best accuracy from these three runs [12].
The experimental settings can be found in the supplemental material.
This document also reports the mean accuracies and standard deviations of some representative methods, an ablation study, and a comparison of our networks against Euclidean
RNNs, transformers, HNNs, and graph neural networks (GNNs).
Comparison against SPD Neural Networks Our networks, referred to as GyroAI-HAUNet and
GyroLE-HAUNet, are compared against SPDNet [20], SPDNetBN [6], and SPD-SRU [8].
Results of these networks are obtained using their ofﬁcial code 2,3,4 with default parameter settings.
Results for M = 1 and M = 3 5 are given in Tab. 1.
On HDM05 dataset, GyroAI-HAUNet outperforms
SPDNet and SPD-SRU, and performs worse than SPDNetBN when M = 1 .
However, when M = 3 ,
GyroAI-HAUNet gives the best result among the competing networks.
On FPHA and NTU datasets, GyroAI-HAUNet gives the best results for both M = 1 and M = 3 .
In terms of model size,
GyroAI-HAUNet and GyroLE-HAUNet require far fewer parameters than SPDNet and SPDNetBN in all cases.
For example, on FPHA dataset, when M = 3 , GyroAI-HAUNet outperforms SPDNetBN by 4.17% with 2.8 times fewer parameters.
Also, on NTU60 dataset, when M = 3 , GyroAI-HAUNet outperforms SPDNetBN by 15.20% with 10.7 times fewer parameters.
4.2 Question Answering
In this section, we consider learning word embeddings in Gr n,p .
We also investigate the use of product manifolds
Gr n 1 ,p ×
Sym + n 2 for training word embeddings.
This idea is mainly motivated by the work of [14] that shows the efﬁcacy of mixed-curvature representations for graph embeddings.
4.2.1 Problem Formulation
Let Q be a list of questions, A be a list of answers to the questions in Q .
Each question q ∈ Q has a list of candidate answers in A .
The candidate set comes with their relevancy judgements, where answers that are correct (positive) have labels equal to 1 and 0 otherwise.
The goal is to build a model that for each query q and its list of candidate answers generates an optimal ranking such that correct answers appear at top of the list [29, 43].
4.2.2 Proposed Method
The core idea is to learn a scoring function [29, 43] given as
[Equation]
where Q and A are embeddings of question q and answer a , respectively, w f , w b ∈ R are parameters of the model, and d ( . ) is a distance function in the target manifolds.
We deﬁne a matrix scaling operation as follows.
Matrix Scaling We parameterize a point P ∈ Gr n,p by a matrix B ∈ M p,n − p such that
[Equation]
Then point P can be computed (see Eq.
(8) and the supplemental material) by
P = exp([ P , I n,p ]) I n,p exp( − [ P , I n,p ]) = exp (cid:18) (cid:20)
B − B T (cid:21) (cid:19)
I n,p exp (cid:18) − (cid:20)
B − B T (cid:21) (cid:19) .
The matrix scaling ⊗ mgr is deﬁned as
[Equation]
[Equation] [Equation]
where A ∈ M p,n − p .
The embedding of question q is computed from those of its tokens as
[Equation]
where T i ∈
Gr n,p , i = 1 , . . .
, l are embeddings of the tokens in question q , S ∈ M p,n − p
and
B ∈ Gr n,p are parameters of the model.
The embedding of answer a is the summation of those of its tokens using operation ⊕ gr .
For P , Q ∈ Gr n,p , the distance function d ( ., . ) is deﬁned as
[Equation]
We also train word embeddings in product manifold Gr n 1 ,p ×
Sym + n 2 using the distance function
[Equation]
where P gr , Q gr ∈
Gr n 1 ,p , P spd , Q spd ∈
Sym + n 2 , d F 1 spd ( ., . ) is the Finsler distance function proposed in [29], and τ is a constant.
This results in three models based on the scaling, rotation, and reﬂection transformations of [29].
Our Grassmann model is shown in Fig.
1b.
4.2.3 Results
We use two datasets, i.e., TrecQA [47] (clean version) and WikiQA [50].
The mean average precision (MAP) and mean reciprocal rank (MRR) are used as evaluation metrics [43].
We compare our models 6 against the SPD models of [29].
Since the codes of these models are not publicly available,
[Table]
we implemented them by following closely the instructions in [29].
The implementation details and experimental settings can be found in the supplemental material.
Tab. 2 reports the means and standard deviations of MAP and MRR from three runs (DOF stands for degrees of freedom).
Our Grassmann model GyroGR-QANet performs favorably against most of the
SPD models on TrecQA dataset.
Also, results of our models based on the product manifold show the efﬁcacy of mixed-curvature representations [14] in question answering.
Note that the numbers of DOF of our models are smaller than those of the SPD models.
We also note that techniques in [19, 28] could potentially improve the performance of our models.
Limitations of Our Work
To develop our SPD models, we only construct the basic operations and two other operations, i.e., matrix scaling and pointwise nonlinearity (see Section 4.1.1).
Other operations [39] should also be designed in order to improve the representation power of our networks.
Also, the question of how to generalize a broader class of DNNs to the SPD manifold setting should be addressed in future work.
Our experiments for question answering have shown the usefulness of product manifolds
Gr n 1 ,p ×
Sym + n 2 for learning word, entity, and relation embeddings.
More investigation is needed to see if a product of multiple smaller-dimension SPD and Grassmann manifolds will improve performance [14].
It would also be interesting to ﬁnd out patterns that show the relationship between the performance of the embeddings and the theoretical curvature of the manifolds in our problems [9].
Conclusion
We have shown that the AI and LE geometries of SPD manifolds have strong connection to hyperbolic geometry, and Grassmann manifolds share very similar properties with gyrovector spaces.
We have presented new methods for generalizing Euclidean neural networks to the SPD and Grassmann manifold settings.
Our experimental results on human activity understanding and question answering have demonstrated the effectiveness of the proposed approach.
Acknowledgments
We are grateful for the constructive comments and feedback from the anonymous reviewers.
We thank Amir Shahroudy, Jun Liu, Tian-Tsong Ng, Gang Wang, Guillermo Garcia-Hernando, Shanxin
Yuan, Seungryul Baek, and Tae-Kyun Kim for giving us access to the NTU RGB+D and FPHA datasets.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See the supplemental material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We included the codes, data, and instructions for a part of our experiments as a URL and in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the supplemental material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We included new codes as a URL and in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] We clearly cited the authors of the codes and data used in our experiments.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]