Original: N EURAL A RCHITECTURE S EARCH OF SPD M ANIFOLD N ETWORKS
Revised: N EURAL A RCHITECTURE S EARCH OF SPD M ANIFOLD N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we propose a new neural architecture search (NAS) problem of
Revised: In this paper, we propose a new neural architecture search (NAS) problem of

Original: Symmetric Positive Deﬁnite (SPD) manifold networks.
Revised: Symmetric Positive Deﬁnite (SPD) manifold networks.

Original: Unlike the conventional
Revised: Unlike the conventional

Original: NAS problem, our problem requires to search for a unique computational cell called the SPD cell.
Revised: NAS problem, our problem requires to search for a unique computational cell called the SPD cell.

Original: This SPD cell serves as a basic building block of SPD neural architectures.
Revised: This SPD cell serves as a basic building block of SPD neural architectures.

Original: An efﬁcient solution to our problem is important to minimize the extraneous manual effort in the SPD neural architecture design.
Revised: An efﬁcient solution to our problem is important to minimize the extraneous manual effort in the SPD neural architecture design.

Original: To accomplish this goal, we ﬁrst introduce a geometrically rich and diverse SPD neural architecture search space for an efﬁcient SPD cell design.
Revised: To accomplish this goal, we ﬁrst introduce a geometrically rich and diverse SPD neural architecture search space for an efﬁcient SPD cell design.

Original: Further, we model our new NAS problem using the supernet strategy which models the architecture search problem as a one-shot training process of a single supernet.
Revised: Further, we model our new NAS problem using the supernet strategy, which models the architecture search problem as a one-shot training process of a single supernet.

Original: Based on the supernet modeling, we exploit a differentiable NAS algorithm on our relaxed continuous search space for SPD neural architecture search.
Revised: Based on the supernet modeling, we exploit a differentiable NAS algorithm on our relaxed continuous search space for SPD neural architecture search.

Original: Statistical evaluation of our method on drone, action, and emotion recognition tasks mostly provides better results than the stateof-the-art SPD networks and NAS algorithms.
Revised: Statistical evaluation of our method on drone, action, and emotion recognition tasks mostly provides better results than the stateof-the-art SPD networks and NAS algorithms.

Original: Empirical results show that our algorithm excels in discovering better SPD network design, and providing models that are more than 3 times lighter than searched by state-of-the-art NAS algorithms.
Revised: Empirical results show that our algorithm excels in discovering better SPD network design and providing models that are more than 3 times lighter than searched by state-of-the-art NAS algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Designing a favorable neural network architecture for a given application requires a lot of time, effort, and domain expertise.
Revised: Designing a favorable neural network architecture for a given application requires a lot of time, effort, and domain expertise.

Original: To mitigate this issue, researchers in the recent years have started developing algorithms to automate the design process of neural network architectures (Zoph & Le, 2016; Zoph et al., 2018; Liu et al., 2017; 2018a;
Revised: To mitigate this issue, researchers in the recent years have started developing algorithms to automate the design process of neural network architectures (Zoph & Le, 2016; Zoph et al., 2018; Liu et al., 2017; 2018a;

Original: Real et al., 2019; Liu et al., 2018b).
Revised: Real et al., 2019; Liu et al., 2018b; Tian et al., 2020).

Original: Although these neural architecture search (NAS) algorithms have shown great potential to provide an optimal architecture for a given application, it is limited to handle architectures with Euclidean operations and representation.
Revised: Although these neural architecture search (NAS) algorithms have shown great potential to provide an optimal architecture for a given application, it is limited to handle architectures with Euclidean operations and representation.

Original: To deal with non-euclidean data representation and corresponding set of operations, researchers have barely proposed any NAS algorithms —to the best of our knowledge.
Revised: To deal with non-euclidean data representation and corresponding set of operations, researchers have barely proposed any NAS algorithms —to the best of our knowledge.

Original: It is well-known that manifold-valued data representation such as symmetric positive deﬁnite (SPD) matrices have shown overwhelming accomplishments in many real-world applications such as pedestrian detection (Tuzel et al., 2006; 2008), magnetic resonance imaging analysis (Pennec et al., 2006), action recognition (Harandi et al., 2014), face recognition (Huang et al., 2014; 2015), braincomputer interfaces (Barachant et al., 2011), structure from motion (Kumar et al., 2018; Kumar, 2019), etc. 
Revised: It is well-known that manifold-valued data representation such as symmetric positive deﬁnite (SPD) matrices have shown overwhelming accomplishments in many real-world applications such as pedestrian detection (Tuzel et al., 2006; 2008), magnetic resonance imaging analysis (Pennec et al., 2006), action recognition (Harandi et al., 2014), face recognition (Huang et al., 2014; 2015), braincomputer interfaces (Barachant et al., 2011), structure from motion (Kumar et al., 2018; Kumar, 2019), etc. Also, in applications like diffusion tensor imaging of the brain, drone imaging, samples are collected directly as SPD’s.

Revised: As a result, neural network usage based on Euclidean data representation becomes inefﬁcient for those applications.

Original: Consequently, this has led to the development of  SPD neural network (SPDNet) architectures for further improvements in these area of research (Huang & Van Gool, 2017; Brooks et al., 2019).
Revised: Consequently, this has led to the development of the SPD neural network (SPDNet) architectures for further improvements in these areas of research (Huang & Van Gool, 2017; Brooks et al., 2019).

Original: However, these architectures are handcrafted, so, the operations or the parameters deﬁned for these networks generally changes as per the application.
Revised: However, these architectures are handcrafted, so the operations or the parameters deﬁned for these networks generally change as per the application.

Original: This motivated us to propose a new NAS problem of SPD manifold networks.
Revised: This motivated us to propose a new NAS problem of SPD manifold networks.

Original: A solution to this problem can reducethe unwanted efforts in SPDNet design.
Revised: A solution to this problem can reduce unwanted efforts in SPDNet design.

Original: Compared to the traditional NAS problem, our NAS problem requires a new deﬁnition of computation cell and proposal for diverseset of SPD candidate operation.
Revised: Compared to the traditional NAS problem, our NAS problem requires a new deﬁnition of computation cell and proposal for diverse SPD candidate operation set.

Original: In particular, we model the basic architecture cell with a speciﬁc directed acyclic graph (DAG), where each node is a latent SPD representation and each edge corresponds to a SPD candidate operation.
Revised: In particular, we model the basic architecture cell with a speciﬁc directed acyclic graph (DAG), where each node is a latent SPD representation, and each edge corresponds to a SPD candidate operation.

Original: Here, the intermediate transformations between nodes respects the geometry of the SPD manifolds.
Revised: Here, the intermediate transformations between nodes respect the geometry of the SPD manifolds.

Original: For solving the suggested NAS problem, we exploit a supernet search strategy which models the architecture search problem as a one-shot training process of a supernet that comprises of a mixture of SPD neural architectures.
Revised: For solving the suggested NAS problem, we exploit a supernet search strategy which models the architecture search problem as a one-shot training process of a supernet that comprises of a mixture of SPD neural architectures.

Original: The supernet modeling enables us to perform a differential architecture search on a continuous relaxation of SPD neural architecture search space, and therefore, can be solved using a gradient descent approach.
Revised: The supernet modeling enables us to perform a differential architecture search on a continuous relaxation of SPD neural architecture search space, and therefore, can be solved using a gradient descent approach.

Original: Our evaluation validates that the proposed method can build a reliable SPD network from scratch.
Revised: Our evaluation validates that the proposed method can build a reliable SPD network from scratch.

Original: We show the results of our method on benchmark datasets that clearly show results better than handcrafted SPDNet
Revised: We show the results of our method on benchmark datasets that clearly show results better than handcrafted SPDNet.

Original: Our work makes the following contributions:
Revised: Our work makes the following contributions

Original:  • We introduce a NAS problem of SPD manifold networks that opens up a new direction of research in automated machine learning and SPD manifold learning.
Revised: : • We introduce a NAS problem of SPD manifold networks that opens up a new direction of research in automated machine learning and SPD manifold learning.

Original: Based on a supernet modeling, we propose a  differentiable NAS algorithm for SPD neural architecture search.
Revised: Based on a supernet modeling, we propose a novel differentiable NAS algorithm for SPD neural architecture search.

Original: Concretely, we exploit  Fr´echet mixture of SPD operationswith sparsemax constraint to introduce sparsity, and bi-level optimization with SPD manifold update for joint architecture search and network training.
Revised: Concretely, we exploit a sparsemax-based Fr´echet mixture of SPD operations to introduce sparsity that is essential for an effective diffentiable search, and bi-level optimization with manifold-based update and convexity-based update to jointly optimize architecture parameters and network kernel weights.

Original: • Besides well-studied operations from exiting SPDNets (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020), we follow Liu et al.
Revised: • Besides well-studied operations from exiting SPDNets (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020), we follow Liu et al.

Original: (2018b) to further introduce some new SPD layers, i.e., skip connection, none operation, max pooling and averaging pooling.
Revised: (2018b) to further introduce some new SPD layers, i.e., skip connection, none operation, max pooling and averaging pooling.

Original: Our  additional set of SPD operations make the search space more diverse for the  search algorithm to obtain more generalized SPD neural network architecture.
Revised: Our introduced additional set of SPD operations make the search space more diverse for the neural architecture search algorithm to obtain more generalized SPD neural network architectures.

Original: • Evaluation on  benchmark datasets shows that our searched SPD neural architectures canclearly outperform the existing handcrafted SPDNets (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020) and the state-of-the-art NAS methods (Liu et al., 2018b; Chu et al., 2020).
Revised: • Evaluation on three benchmark datasets shows that our searched SPD neural architectures can outperform the existing handcrafted SPDNets (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020) and the state-of-the-art NAS methods (Liu et al., 2018b; Chu et al., 2020).

Original: Notably, our searched architecture is more than 3 times lighter than  searched by the  NAS algoithms.
Revised: Notably, our searched architecture is more than 3 times lighter than those searched by the traditional NAS algorithms.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In recent years, plenty of research work has been published in the area of NAS.
Revised: In recent years, plenty of research work has been published in the area of NAS (Gong et al., 2019;

Revised: Liu et al., 2019; Nayman et al., 2019; Guo et al., 2020).

Original: This is probably due to the success of deep learning for several applications which has eventually led to the automation of neural architecture design.
Revised: This is probably due to the success of deep learning for several applications which has eventually led to the automation of neural architecture design.

Original: Also, improvements in the processing capabilities of machines has inﬂuenced the researchers to work out this computationally expensive yet an important problem.
Revised: Also, improvements in the processing capabilities of machines has inﬂuenced the researchers to work out this computationally expensive yet an important problem.

Original: Computational cost for some of the well-known NAS algorithms is in thousands of GPU days which has resulted in the development of several computationally efﬁcient methods (Zoph et al., 2018;
Revised: Computational cost for some of the well-known NAS algorithms is in thousands of GPU days which has resulted in the development of several computationally efﬁcient methods (Zoph et al., 2018;

Original: Real et al., 2019;
Revised: Real et al., 2019;

Original: Liu et al., 2018a; 2017; Baker et al., 2017; Brock et al., 2017; Bender, 2019; Elsken et al., 2017; Cai et al., 2018; Pham et al., 2018; Negrinho & Gordon, 2017; Kandasamy et al., 2018; Chu et al., 2020).
Revised: Liu et al., 2018a; 2017; Baker et al., 2017; Brock et al., 2017; Bender, 2019; Elsken et al., 2017; Cai et al., 2018; Pham et al., 2018; Negrinho & Gordon, 2017; Kandasamy et al., 2018; Chu et al., 2020).

Original: In this work, we propose a new NAS problem of SPD networks.
Revised: In this work, we propose a new NAS problem of SPD networks.

Original: We solve this problem using a supernet modeling methodology with a one-shot differentiable training process of an overparameterized supernet.
Revised: We solve this problem using a supernet modeling methodology with a one-shot differentiable training process of an overparameterized supernet.

Original: Our modeling is driven by the recent progress in supernet methodology.
Revised: Our modeling is driven by the recent progress in supernet methodology.

Original: Supernet methodology has shown a great potential than other NAS methodologies in terms of search efﬁciency.
Revised: Supernet methodology has shown a great potential than other NAS methodologies in terms of search efﬁciency.

Original: Since our work is directed towards solving a new NAS problem, we conﬁne our discussion to the work that have greatly inﬂuenced our method i.e., one-shot NAS methods and SPD networks.
Revised: Since our work is directed towards solving a new NAS problem, we conﬁne our discussion to the work that have greatly inﬂuenced our method i.e., one-shot NAS methods and SPD networks.

Original: To the best of our knowledge, there are mainly two types of one-shot NAS methods based on the architecture modeling (Elsken et al., 2018) (a) parameterized architecture (Liu et al., 2018b; Zheng et al., 2019; Wu et al., 2019; Chu et al., 2020), and (b) sampled architecture (Deb et al., 2002; Chu et al., 2019).
Revised: To the best of our knowledge, there are mainly two types of one-shot NAS methods based on the architecture modeling (Elsken et al., 2018) (a) parameterized architecture (Liu et al., 2018b; Zheng et al., 2019; Wu et al., 2019; Chu et al., 2020), and (b) sampled architecture (Deb et al., 2002; Chu et al., 2019).

Original: In this paper, we adhere to the parametric modeling due to its promising results on conventional neural architectures.
Revised: In this paper, we adhere to the parametric modeling due to its promising results on conventional neural architectures.

Original: A majority of the previous work on NAS with continuous search space ﬁne-tunes the explicit feature of speciﬁc architectures (Saxena & Verbeek, 2016; Veniat &
Revised: A majority of the previous work on NAS with continuous search space ﬁne-tunes the explicit feature of speciﬁc architectures (Saxena & Verbeek, 2016; Veniat &

Original: Denoyer, 2018; Ahmed & Torresani, 2017; Shin et al., 2018).
Revised: Denoyer, 2018; Ahmed & Torresani, 2017; Shin et al., 2018).

Original: On the contrary, Liu et al. (2018b);
Revised: On the contrary, Liu et al. (2018b);

Original: Liang et al.
Revised: Liang et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: Chu et al.
Revised: Chu et al.

Original: (2020) provides architectural diversity for NAS with highly competitive performances.
Revised: (2020) provides architectural diversity for NAS with highly competitive performances.

Original: The other part of our work focuses on SPD network architectures.
Revised: The other part of our work focuses on SPD network architectures.

Original: There exist algorithms to develop handcrafted SPDNet (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020).
Revised: There exist algorithms to develop handcrafted SPDNet (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020).

Original: To automate the process of SPD network design, in this work, we take the best of both these ﬁelds (NAS, SPDNets) and propose aSPD network NAS algorithm.
Revised: To automate the process of SPD network design, in this work, we choose the most promising approaches from these ﬁelds (NAS (Liu et al., 2018b), SPD networks (Huang & Van Gool, 2017)) and propose a NAS algorithm for

Original:  Next, we provides a short summary on the essential notions of Riemannian geometry of SPD manifolds, followed by an introduction of some basic SPDNet operations and layers.
Revised: SPD inputs. Next, we summarize the essential notions of Riemannian geometry of SPD manifolds, followed by an introduction of some basic SPDNet operations and layers.

Original: As some of the introduced operations and layers have been well-studied by the existing literature, we applied them directly to deﬁne oursearch space of SPD neural architectures.
Revised: As some of the introduced operations and layers have been well-studied by the existing literature, we applied them directly to deﬁne our SPD neural architectures’ search space.

Original: Representation and Operation: We denote n × n real SPD as X ∈ S n ++ .
Revised: Representation and Operation: We denote n × n real SPD as X ∈ S n ++ .

Original: A real SPD matrix
Revised: A real SPD matrix

Original: X ∈ S n ++ satisﬁes the property that for any non-zero z ∈ R n , z T X z > 0 .
Revised: X ∈ S n ++ satisﬁes the property that for any non-zero z ∈ R n , z T X z > 0 (Harandi et al., 2017).

Original: We denote T X M as the tangent space of the manifold M at X ∈ S n ++ .
Revised: We denote T X M as the tangent space of the manifold M at X ∈ S n ++ and log corresponds to matrix logarithm.

Original: Let X 1 , X 2 be any two points on the SPD manifold then the distance between them is given by
Revised: Let X 1 , X 2 be any two points on the SPD manifold then the distance between them is given by

Original: [Equation]
Revised: [Equation]

Original: There are other efﬁcient methods to compute distance between two points on the SPD manifold, however, their discussion is beyond the scope of our work.
Revised: There are other efﬁcient methods to compute distance between two points on the SPD manifold (Gao et al., 2019; Dong et al., 2017b), however, their discussion is beyond the scope of our work.

Original: Other property of the Riemannian manifold of our interest is local diffeomorphism of geodesics which is a one-to-one mapping from the point on the tangent space of the manifold to the manifold (Lackenby, 2020).
Revised: Other property of the Riemannian manifold of our interest is local diffeomorphism of geodesics which is a one-to-one mapping from the point on the tangent space of the manifold to the manifold (Pennec, 2020; Lackenby, 2020).

Original: To deﬁne such notions, let X ∈ S n ++ be the reference point and, Y ∈ T X S n ++ , then Eq:(2) associates Y ∈ T X S n ++ to a point on the manifold.
Revised: To deﬁne such notions, let X ∈ S n ++ be the base point and, Y ∈ T X S n ++ , then Eq:(2) associates Y ∈ T X S n ++ to a point on the manifold (Pennec, 2020).

Original: [Equation]
Revised: [Equation]

Original: Similarly, an inverse map is deﬁned as 
Revised: Similarly, an inverse map is deﬁned as log X ( Z ) =

Revised: X 12 log

Original: [Equation]
Revised: [Equation]

Original:  1) Basic operations of SPD Network: It is well-known that operations such as mean centralization, normalization, and adding bias to a batch of data are inherent performance booster for most neural networks.
Revised: X , ∀ Z ∈ S n ++ [Equation] 1) Basic operations of SPD Network: It is well-known that operations such as mean centralization, normalization, and adding bias to a batch of data are inherent performance booster for most neural networks.

Original: In the same spirit, existing works like Brooks et al.
Revised: In the same spirit, existing works like Brooks et al.

Original: ; Chakraborty (2020) use the notion of these operations for the SPD or general manifold data to deﬁne analogous operations on manifolds.
Revised: ; Chakraborty (2020) use the notion of these operations for the SPD or general manifold data to deﬁne analogous operations on manifolds.

Original: Below we introduce them following the work of Brooks et al.
Revised: Below we introduce them following the work of Brooks et al.

Original: • Batch mean, centering and bias : Given a batch of N SPD matrices { X i } Ni =1 , we can compute its Riemannian barycenter ( B ) as B = argmin X[Equation] Ni
Revised: • Batch mean, centering and bias : Given a batch of N SPD matrices { X i } Ni =1 , we can compute its Riemannian barycenter ( B ) as B = argmin X

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: X

Original: [Equation]
Revised: [Equation]

Original: M (

Original: X
Revised: X

Original: i ,
Revised: µ

Original: X µ ) .

Original: It is sometime referred as Fr ´ echet mean (Moakher, 2005; Bhatia & Holbrook, 
Revised: It is sometimes referred as Fr ´ echet mean (Moakher, 2005; Bhatia & Holbrook, 2006).

Original: This deﬁnition is trivially extended to compute the weighted Riemannian Barycenter  also known as weighted Fr ´ echet Mean (wFM) 
Revised: This deﬁnition can be extended to compute the weighted Riemannian Barycenter 1 also known as weighted Fr ´ echet Mean (wFM) .

Original: [Equation]
Revised: [Equation]

Original: X i , X µ ); s.t.

Original: [-w i ≥ 
-]
Revised: [-w i ≥ 
-]

Original: and N (cid:88) i = w i =(3) Eq:(3) can be approximated using Karcher ﬂow (Karcher, 1977; Bonnabel, 2013; Brooks et al., 2019) or recursive geodesic mean (Cheng et al., 2016; Chakraborty et al., 2020).
Revised: Eq:(3) can be approximated using Karcher ﬂow (Karcher, 1977; Bonnabel, 2013; Brooks et al., 2019) or recursive geodesic mean (Cheng et al., 2016; Chakraborty et al., 2020).

Original: 2) Basic layers of SPD Network: Analogous to standard CNN, methods like Huang & Van Gool (2017); Brooks et al. 
Revised: Basic layers of SPD Network: Analogous to standard CNN, methods like Huang & Van Gool ); Brooks et al. [Equation]

Original: ; Chakraborty et al.
Revised: ; Chakraborty et al.

Original: (2020) designed SPD layers to perform operations that respect SPD manifold constraints.
Revised: ) designed SPD layers to perform operations that respect SPD manifold constraints.

Original: Assuming X
Revised: Assuming X

Original: k − 1 ∈ S n ++ be the input SPD matix to the k th layer, the SPD network layers are deﬁned as follows: • BiMap layer : This layer corresponds to a dense layer for SPD data.
Revised: k − 1 ∈ S n ++ be the input SPD matix to the k th layer, the SPD network layers are deﬁned as follows: • BiMap layer : This layer corresponds to a dense layer for SPD data.

Original: The BiMap layer reduces the dimension of a input SPD matrix via a transformation matrix W k as X k
Revised: The BiMap layer reduces the dimension of a input SPD matrix via a transformation matrix W k as X k

Original: = W k X k − 1 W Tk .
Revised: = W k X k − 1 W Tk .

Original: To ensure the matrix X k to be a valid SPD matrix, the W k matrix must be of full row-rank.
Revised: To ensure the matrix X k to be an SPD matrix, the W k matrix must be of full row-rank.

Original: • Batch normalization layer: To perform batch normalization after each BiMap layer, we ﬁrst compute the Riemannian barycenter of the batch of SPD matrices followed by a running mean update step, which is Riemannian weighted average between the batch mean and the current running mean, with the weights (1 − θ ) and ( θ ) respectively.
Revised: • Batch normalization layer: To perform batch normalization after each BiMap layer, we ﬁrst compute the Riemannian barycenter of the batch of SPD matrices followed by a running mean update step, which is Riemannian weighted average between the batch mean and the current running mean, with the weights (1 − θ ) and ( θ ) respectively.

Original: Once mean is calculated, we centralize and add bias to each SPD sample of the batch using Eq:(4) (Brooks et al., 2019): Batch centering : Centering the B :X ci = P B → I ( X i ) =
Revised: Once mean is calculated, we centralize and add bias to each SPD sample of the batch using Eq:(4) (Brooks et al., 2019), where P is the notation used for parallel transport :

Original: B − 12 X i B − 12 , I is the identity matrix Bias the batch : Bias towards G :

Original: X bi = P I

Original: → G ( X ci ) =

Original: G 12 X ci G 12 , I is the identity matrix

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: • ReEig layer : The ReEig layer is analogous to ReLU like layers present in the classical ConvNets.
Revised: • ReEig layer : The ReEig layer is analogous to ReLU like layers present in the classical ConvNets.

Original: It aims to introduce non-linearity to SPD network.
Revised: It aims to introduce non-linearity to SPD network.

Original: The ReEig for the k th layer is deﬁned as: X k
Revised: The ReEig for the k th layer is deﬁned as: [Equation]

Original: = U k − 1 max( (cid:15) I , Σ k − 1 ) U Tk − 1 where , X k − 1 =
Revised: I , [Equation] [Equation] 1 where , X k − 1 =

Original: U k − 1 Σ k − 1 U Tk − 1 , I is the identity matrix, and (cid:15)  is a rectiﬁcation threshold value.
Revised: U k − 1 Σ k − 1 U Tk − 1 , I is the identity matrix, and (cid:15) > 0 is a rectiﬁcation threshold value.

Original: U k − 1 , Σ k − 1 are the orthonormal matrix and singular-value matrix respectively which are obtained via matrix factorization of X k − 1 . 
Revised: U k − 1 , Σ k − 1 are the orthonormal matrix and singular-value matrix respectively which are obtained via matrix factorization of X k − 1 . [Figure] ou t

Original:  • LogEig layer : To map the manifold representation of SPD to ﬂat space so that a Euclidean operation can be performed, LogEig layer is introduced.
Revised: [Figure] • LogEig layer : To map the manifold representation of SPD to ﬂat space so that a Euclidean operation can be performed, LogEig layer is introduced.

Original: The LogEig layer is deﬁned as: X k
Revised: The LogEig layer is deﬁned as: X k

Original: = U k − 1 log(Σ k − 1 ) U
Revised: = U k − 1 log(Σ k − 1 ) U

Original: Tk − 1 where , X k − 1 =
Revised: Tk − 1 where , X k − 1 =

Original: U k − 1 Σ k − 1 U Tk − 1 .
Revised: U k − 1 Σ k − 1 U Tk − 1 .

Original: The LogEig layer is used with fully connected layers to solve tasks with SPD representation.
Revised: The LogEig layer is used with fully connected layers to solve tasks with SPD representation.

Original: • ExpEig layer : This layer maps the corresponding SPD representation from ﬂat space back to SPD manifold space.
Revised: • ExpEig layer : This layer maps the corresponding SPD representation from ﬂat space back to SPD manifold space.

Original: It is deﬁned as X k =
Revised: It is deﬁned as X k =

Original: U k − 1 exp(Σ k − 1 ) U Tk − 1 where , X k − 1 =
Revised: U k − 1 exp(Σ k − 1 ) U Tk − 1 where , X k − 1 =

Original: U k − 1 Σ k − 1 U Tk − 1 .
Revised: U k − 1 Σ k − 1 U Tk − 1 .

Original: Weighted Riemannian pooling layer : It uses wFM deﬁnition to compute the output of the layer.
Revised: Weighted Riemannian pooling layer : It uses wFM deﬁnition to compute the output of the layer.

Original: Recent method use recursive geodesic mean algorithm to calculate the mean (Chakraborty et al., 2020), in contrast, we use Karcher ﬂow algorithm to compute it (Karcher, 1977) as it is simple and widely used in practise. [Figure]
Revised: Recent method use recursive geodesic mean algorithm to calculate the mean (Chakraborty et al., 2020), in contrast, we use Karcher ﬂow algorithm to compute it (Karcher, 1977) as it is simple and widely used in practice.

Original:  N EURAL A RCHITECTURE S EARCH OF SPD M ANIFOLD N ETWORK
Revised:  N EURAL A RCHITECTURE S EARCH OF SPD M ANIFOLD N ETWORK

Original: As alluded before, to solve the suggested problem, there are a few key changes that must be introduced.
Revised: As alluded before, to solve the suggested problem, there are a few key changes that must be introduced.

Original: Firstly, a new deﬁnition of the computation cell is required.
Revised: Firstly, a new deﬁnition of the computation cell is required.

Original: In contrast to thepopular computational cell, our computational cell —which we call as SPD cell, must incorporate the notion of SPD manifold geometry while performing any operation.
Revised: In contrast to the computational cells designed by regular NAS algorithms like Liu et al. (2018b); Chu et al. (2020), our computational cell —which we call as SPD cell, additionally incorporate the notion of SPD manifold geometry so that SPD representations can be treated properly.

Original: Similar to the basic NAS cell design, our SPD cell can either be a normal cell that returns SPD feature maps of the same width and height or, a reduction cell in which the SPD feature maps are reduced by a certain factor in width and height.
Revised: On the other hand, like the regular NAS cell design, our SPD cell can either be a normal cell that returns SPD feature maps of the same width and height or, a reduction cell in which the SPD feature maps are reduced by a certain factor in width and height.

Original: Secondly, to solve our new NAS problem will require an appropriate and diverse SPD search space that can help NAS method to optimize for an effective SPD cell, which can then be stacked and trained to build an efﬁcient SPD neural network architecture.
Revised: Secondly, to solve our new NAS problem will require an appropriate and diverse SPD search space that can help NAS method to optimize for an effective SPD cell, which can then be stacked and trained to build an efﬁcient SPD neural network architecture.

Original: Concretely, a SPD cell is modeled by a directed asyclic graph (DAG) which is composed of nodes and edges.
Revised: Concretely, a SPD cell is modeled by a directed asyclic graph (DAG) which is composed of nodes and edges.

Original: Each node is a latent representation of the SPD manifold valued data  and, each edge corresponds to a valid candidate operation on SPD manifold (see Fig.1(a)).
Revised: In our DAG each node is an latent representation of the SPD manifold valued data i.e. an intermediate SPD feature map and, each edge corresponds to a valid candidate operation on SPD manifold (see Fig.1(a)).

Original: Each edge of a SPD cell is associated with a set of candidate SPD manifold operations ( O M ) that transforms the SPD valued latent representation from the source node (say X ( i ) M ) to the target node (say X ( j ) M ).
Revised: Each edge of a SPD cell is associated with a set of candidate SPD manifold operations ( O M ) that transforms the SPD valued latent representation from the source node (say X ( i ) M ) to the target node (say X ( j ) M ).

Original: We deﬁne the intermediate transformation between the nodes in our SPD cell as: 
Revised: We deﬁne the intermediate transformation between the nodes in our SPD cell as: X ( j ) M = argmin

Original: [Equation]
Revised: [Equation]

Original: <j δ 2 M (cid:16) O ( i,j ) M (cid:0) X ( i ) M (cid:1) , X ( j ) M (cid:17)
Revised: , X [Equation]

Original: , where δ M denotes the geodesic distance Eq:
Revised: , where δ M denotes the geodesic distance Eq:(1).

Original: Generally, this transformation result corresponds to the unweighted Fr ´ echet mean of the operations based on the predecessors, such that the mixture of all operations still reside on SPD manifolds.
Revised: Generally, this transformation result corresponds to the unweighted Fr´echet mean of the operations based on the predecessors, such that the mixture of all operations still reside on SPD manifolds.

Original: Note that our deﬁnition of SPD cell ensures that each computational graph preserves the appropriate geometric structure of the SPD manifold.
Revised: Note that our deﬁnition of SPD cell ensures that each computational graph preserves the appropriate geometric structure of the SPD manifold.

Original: Equipped with the notion of SPD cell and its intermediate transformation, we are prepared to propose our search space (§3.1) followed by the solution to our SPDNet NAS problem (§3.2) and its results (§4).
Revised: Equipped with the notion of SPD cell and its intermediate transformation, we are prepared to propose our search space ( § 3.1) followed by the solution to our SPDNet NAS problem ( § 3.2) and its results ( § 4).

Original: 3. S EARCH S PACE
Revised: 3. S EARCH S PACE

Original: Our search space consists of a set of valid SPD network operations which is deﬁned for the supernet search.
Revised: Our search space consists of a set of valid SPD network operations which is deﬁned for the supernet search.

Original: First of all, the search space includes some existing SPD operations 1 , e.g., BiMap, batch normalization, ReEig, LogEig, ExpEig and weighted Riemannian pooling layers, all of which are  introduced in Sec.2.
Revised: First of all, the search space includes some existing SPD operations 2 , e.g., BiMap, batch normalization, ReEig, LogEig, ExpEig and weighted Riemannian pooling layers, all of which are [Table] introduced in Sec.2.

Revised: Though those individual operations (e.g., BiMap, LogEig, ExpEig) have been explored well by existing works, different aggregations on them are still understudied, which are essential to enrich our search space.

Original: To be speciﬁc, following Liu et al.
Revised: To be speciﬁc to enrich the search space, following Liu et al.

Original: (2018b)
Revised: (2018b)

Original: ; Gong et al.
Revised: ; Gong et al.

Original: (2019) traditional NAS methods, we apply the SPD batch normalization to every SPD convolution operation (i.e., BiMap), and design three variants of convolution blocks including the one without activation (i.e., ReEig), the one using post-activation and the one using pre-activation (see Table 1).
Revised: (2019) traditional NAS methods, we apply the SPD batch normalization to every SPD convolution operation (i.e., BiMap), and design three variants of convolution blocks including the one without activation (i.e., ReEig), the one using post-activation and the one using pre-activation (see Table 1).

Original: In addition, we introduce ﬁve new operations analogous to DARTS
Revised: In addition, we introduce ﬁve new operations analogous to DARTS

Original: (Liu et al., 2018b) to enrich the search space in the context of SPD networks.
Revised: (Liu et al., 2018b) to enrich the search space in the context of SPD networks.

Original: These are skip normal, none normal, average pooling, max pooling and skip reduced.
Revised: These are skip normal, none normal, average pooling, max pooling and skip reduced.

Original: Most of these operations are not  fully explored for SPD networks.
Revised: The effect of such diverse operation choices have not been fully explored for SPD networks.

Original: All the candidate operations are illustrated in Table (1), and the deﬁnitions of the new operations are detailed as follows:
Revised: All the candidate operations are illustrated in Table (1), and the deﬁnitions of the new operations are detailed as follows:

Original: (a) Skip normal : It preserves the input representation and is similar to skip connection.
Revised: (a) Skip normal : It preserves the input representation and is similar to skip connection.

Revised: (b)

Original: (b) None normal : It corresponds to the operation that returns identity as the output i. e, the notion of zero in the
Revised: None normal : It corresponds to the operation that returns identity as the output i.e, the notion of zero in the

Original: SPD space.
Revised: SPD space.

Original: (c) Max pooling :
Revised: (c) Max pooling :

Original: Given a set of SPD matrices, max pooling operation ﬁrst projects these samples to a ﬂat space via a LogEig operation, where a standard max pooling operation is performed.
Revised: Given a set of SPD matrices, max pooling operation ﬁrst projects these samples to a ﬂat space via a LogEig operation, where a standard max pooling operation is performed.

Original: [Table]

Original: Finally, an ExpEig operation is used to map the sample back to the SPD manifold.
Revised: Finally, an ExpEig operation is used to map the sample back to the SPD manifold.

Original: (d) Average pooling : Similar to Max pooling, the average pooling operation ﬁrst projects the samples to the ﬂat space using a LogEig operation, where a standard average pooling is employed.
Revised: (d) Average pooling : Similar to Max pooling, the average pooling operation ﬁrst projects the samples to the ﬂat space using a LogEig operation, where a standard average pooling is employed.

Original: To map the sample back to SPD manifold, an ExpEig operation is used.
Revised: To map the sample back to SPD manifold, an ExpEig operation is used.

Original: (e) Skip reduced : It is similar to ‘skip normal’ but in contrast, it decomposes the input into small matrices to reduces the inter-dependency between channels.
Revised: (e) Skip reduced : It is similar to ‘skip normal’ but in contrast, it decomposes the input into small matrices to reduces the inter-dependency between channels.

Original: Our deﬁnition of reduce operation is in line with the work of Liu et al.
Revised: Our deﬁnition of reduce operation is in line with the work of Liu et al.

Original: (2018b).
Revised: (2018b).

Revised: The newly introduced operations allow us to generate a more diverse discrete search space.

Revised: As presented in Table 2, the randomly selected architecture (generally consisting of the newly introduced

Revised: SPD operations) shows some improvement over SPDNet and SPDNetBN, both of which only contain conventional SPD operations.

Revised: This establishes the effectiveness of the introduced rich search space.

Original: 3. S UPERNET S EARCH
Revised: 3. S UPERNET S EARCH

Original: To solve the suggested new NAS problem, one of the most promising NAS methodologies is supernet modeling.
Revised: To solve the suggested new NAS problem, one of the most promising NAS methodologies is supernet modeling.

Original: While we can resort to some other NAS methods to solve the problem like reinforcement learning based method (Zoph & Le, 2016) or evolution based algorithm
Revised: While we can resort to some other NAS methods to solve the problem like reinforcement learning based method (Zoph & Le, 2016) or evolution based algorithm

Original: (Real et al., 2019), in general, the supernet method models the architecture search problem as a one-shot training process of a single supernet that consists of all architectures.
Revised: (Real et al., 2019), in general, the supernet method models the architecture search problem as a one-shot training process of a single supernet that consists of all architectures.

Original: Based on the supernet modeling , we can search for the optimal SPD neural architecture either using parameterization of architectures or sampling of single-path architectures.
Revised: Based on the supernet modeling , we can search for the optimal SPD neural architecture either using parameterization of architectures or sampling of single-path architectures.

Original: In this paper, we focus on the parameterization approach that is based on the continuous relaxation of the SPD neural architecture representation .
Revised: In this paper, we focus on the parameterization approach that is based on the continuous relaxation of the SPD neural architecture representation .

Original: Such an approach allows for an efﬁcient search of architecture using the gradient descent approach.
Revised: Such an approach allows for an efﬁcient search of architecture using the gradient descent approach.

Original: Next, we introduce our supernet search method, followed by a solution to our proposed bi-level optimization problem.
Revised: Next, we introduce our supernet search method, followed by a solution to our proposed bi-level optimization problem.

Original: Fig.1(b) and
Revised: Fig.1(b) and

Original: Fig.1(c) illustrates an overview of our proposed method.
Revised: Fig.1(c) illustrates an overview of our proposed method.

Original: To search for an optimal SPD architecture ( α ), we optimize the over parameterized supernet.
Revised: To search for an optimal SPD architecture ( α ), we optimize the over parameterized supernet.

Original: In essence, it stacks the basic computation cells with the parameterized candidate operations from our search space in a one-shot search manner.
Revised: In essence, it stacks the basic computation cells with the parameterized candidate operations from our search space in a one-shot search manner.

Original: The contribution of speciﬁc subnets to the supernet helps in deriving the optimal architecture from the supernet.
Revised: The contribution of speciﬁc subnets to the supernet helps in deriving the optimal architecture from the supernet.

Original: Since the proposed operation search space is discrete in nature, we relax the explicit choice of an operation to make the search space continuous.
Revised: Since the proposed operation search space is discrete in nature, we relax the explicit choice of an operation to make the search space continuous.

Original: To do so, we use wFM over all possible candidate operations.
Revised: To do so, we use wFM over all possible candidate operations.

Original: Mathematically,
Revised: Mathematically,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where, O k M is the k th candidate operation between nodes, X µ is the intermediate SPD manifold mean (Eq.3) and, N e denotes number of edges.
Revised: where, O k M is the k th candidate operation between nodes, X µ is the intermediate SPD manifold mean (Eq.3) and, N e denotes number of edges.

Original: We can compute wFM solution either using Karcher ﬂow (Karcher, 1977) or recursive geodesic mean (Chakraborty et al., 2020) algorithm.
Revised: We can compute wFM solution either using Karcher ﬂow (Karcher, 1977) or recursive geodesic mean (Chakraborty et al., 2020) algorithm.

Original: Nonetheless, we adhere to Karcher ﬂow algorithm as it is widely used to calculate wFM 2 .
Revised: Nonetheless, we adhere to Karcher ﬂow algorithm as it is widely used to calculate wFM 3 .

Original: To impose the explicit convex constraint on ˜ α , we project the solution onto the probability simplex as 
Revised: To impose the explicit convex constraint on ˜ α , we project the solution onto the probability simplex as minimize α

Original: [Equation]
Revised: [Equation]

Original:  Eq:(6) enforces the explicit constraint on the weights to supply α for our task and can easily be added as a convex layer in the framework (Agrawal et al., 2019).
Revised: subject to [Equation] Eq:(6) enforces the explicit constraint on the weights to supply α for our task and can easily be added as a convex layer in the framework (Agrawal et al., 2019).

Original: This projection is likely to reach the boundary of the simplex, in which case α becomes sparse (Martins & Astudillo, 2016).
Revised: This projection is likely to reach the boundary of the simplex, in which case α becomes sparse (Martins & Astudillo, 2016).

Original: Optionally, softmax, sigmoid and other regularization methods can be employed to satisfy the convex constraint.
Revised: Optionally, softmax, sigmoid and other regularization methods can be employed to satisfy the convex constraint.

Original: However, (Chu et al., 2020) has observed that the use of softmax can cause performance collapse and may lead to aggregation of skip connections.
Revised: However, Chu et al. (2020) has observed that the use of softmax can cause performance collapse and may lead to aggregation of skip connections.

Original: While Chu et al.
Revised: While Chu et al.

Original: (2020) suggested sigmoid can overcome the unfairness problem with softmax, it may output smoothly changed values which is hard to threshold for dropping redundant operations with non-marginal contributions to the supernet.
Revised: (2020) suggested sigmoid can overcome the unfairness problem with softmax, it may output smoothly changed values which is hard to threshold for dropping redundant operations with non-marginal contributions to the supernet.

Original: Also,
Revised: Also,

Original: FairDARTS (Chu et al., 2020) regularization, may not preserve the summation equal to 1 constraint.
Revised: FairDARTS (Chu et al., 2020) regularization, may not preserve the summation equal to 1 constraint.

Original: Besides, Chakraborty et al.
Revised: Besides, Chakraborty et al.

Original: (2020) proposes recursive statistical approach to solve wFM with convex constraint, however, the deﬁnition proposed do not explicitly preserve the equality constraint and it requires re-normalization of the solution.
Revised: (2020) proposes recursive statistical approach to solve wFM with convex constraint, however, the deﬁnition proposed do not explicitly preserve the equality constraint and it requires re-normalization of the solution.

Original: In contrast, our approach composes of the sparsemax transformation for convex Fr´echet mixture of SPD operations with the following two advantages: 1)
Revised: In contrast, our approach composes of the sparsemax transformation for convex Fr ´ echet mixture of SPD operations with the following two advantages: 1)

Original: It can preserves most of the important properties of softmax such as, it is simple to evaluate, cheaper to differentiate (Martins & Astudillo, 2016).
Revised: It can preserve most of the important properties of softmax such as, it is simple to evaluate, cheaper to differentiate (Martins & Astudillo, 2016).

Original: 2) It is able to produce sparse distributions such that the best operation associated with each edge is more likely to make more dominant contributions to the supernet, and thus more optimal architecture can be derived (refer Figure 2(a),2(b) and §4).
Revised: 2) It is able to produce sparse distributions such that the best operation associated with each edge is more likely to make more dominant contributions to the supernet, and thus more optimal architecture can be derived (refer Figure 2(a),2(b) and § 4).

Original: From Eq:(5–6), the mixing of operations between nodes is determined by the weighted combination of alpha’s ( α k ) and the set of operations.
Revised: From Eq:(5–6), the mixing of operations between nodes is determined by the weighted combination of alpha’s ( α k ) and the set of operations.

Original: This relaxation makes the search space continuous and therefore, architecture search can be achieved by learning a set of alpha ( α = { α k , ∀ k ∈ N e } ).
Revised: This relaxation makes the search space continuous and therefore, architecture search can be achieved by learning a set of alpha ( α = { α k , ∀ k ∈ N e } ).

Original: To achieve our goal, we must simultaneously learn the contribution of several possible operation within all the mixed operations ( w ) and the corresponding architecture α .
Revised: To achieve our goal, we must simultaneously learn the contribution of several possible operation within all the mixed operations ( w ) and the corresponding architecture α .

Original: Consequently, for a given w , we can ﬁnd α and vice-versa resulting in the following bi-level optimization problem.
Revised: Consequently, for a given w , we can ﬁnd α and vice-versa resulting in the following bi-level optimization problem.

Original: minimize
Revised: minimize

Original: [Equation]
Revised: [Equation]

Original: The lower-level optimization E Ltrain corresponds to the optimal weight variable learned for a given α i.e., w opt ( α ) using a training loss.
Revised: The lower-level optimization E Ltrain corresponds to the optimal weight variable learned for a given α i.e., w opt ( α ) using a training loss.

Original: The upper-level optimization E Uval solves for the variable α given the optimal w using a validation loss.
Revised: The upper-level optimization E Uval solves for the variable α given the optimal w using a validation loss.

Original: This bi-level search method gives optimal mixture of multiple small architectures.
Revised: This bi-level search method gives optimal mixture of multiple small architectures.

Original: To derive each node in the discrete architecture, we maintain top- k operations i.e, with the k th highest weight among all the candidate operations associated with all the previous nodes.
Revised: To derive each node in the discrete architecture, we maintain top- k operations i.e, with the k th highest weight among all the candidate operations associated with all the previous nodes.

Original: Bi-level Optimization: The bi-level optimization problem proposed in Eq:(7) is difﬁcult to solve.
Revised: Bi-level Optimization: The bi-level optimization problem proposed in Eq:(7) is difﬁcult to solve.

Original: Following Liu et al.
Revised: Following Liu et al.

Original: (2018b) work, we approximate w opt ( α ) in the upper- optimization problem to skip inner-optimization as follows:
Revised: (2018b) work, we approximate w opt ( α ) in the upper- optimization problem to skip inner-optimization as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, η is the learning rate and ∇ is the gradient operator.

Original: Note that the gradient based optimization for w must follow the geometry of SPD manifold to update the structured connection weight, and its corresponding SPD matrix data.

Original: Applying the chain rule to Eq:(8) gives

Original: [Equation]
Revised: [Equation]

Original: where, ˜ w = Ψ r (cid:0) w − η ˜ ∇ w E Ltrain ( w, α ) (cid:1) denotes the weight update on the SPD manifold for the forward model.

Original: ˜ ∇ w , Ψ r symbolizes the Riemannian gradient and the retraction operator respectively.

Original: The second term in the Eq:(9) involves second order differentials with very high computational complexity, hence, using the ﬁnite approximation method the second term of Eq:(9) reduces to:

Original: [Equation] E [Equation] where, w ± =
Revised: [Figure]

Original: Ψ r ( w ± δ ˜ ∇ ˜ w E Uval ( ˜ w, α )) and δ is a small number set to 0 .

Original: 01 / (cid:107)∇ ˜ w E Uval ( ˜ w, α ) (cid:107) 2 .

Original: For training SPD network, one must use the back-propagation in the context of Riemannian geometry of the SPD manifold.

Original: For concrete derivations on back-propagation for SPD network layers, refer to

Original: Huang & Van Gool (2017) work.

Original: The pseudo code of our method is outlined in Algorithm ( 1 ).

Original: E XPERIMENTS AND R ESULTS

Original: To keep the experimental evaluation consistent with the previously proposed SPD networks (Huang & Van Gool, 2017; Brooks et al., 2019), we used RADAR (Chen et al., 2006),

Original: HDM05 (M ¨ uller et al.,

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Softmax
Revised: Softmax

Original: Sigmoid
Revised: Sigmoid

Original: Sparsemax
Revised: Sparsemax

Original: Operations
Revised: Operations

Original: E dg e s
Revised: E dg e s

Original: E dg e s E dg e s
Revised: E dg e s E dg e s

Original: Operations
Revised: Operations

Original: Operations (a) Distribution of edge weights for  1 c_{k-2} c_{k-1} S k i p_no
Revised: Operations (a) Distribution of edge weights for operation selection 1 c_{k-2} c_{k-1} S k i p_no

Original: r m a l
Revised: r m a l

Original: S k i p_no
Revised: S k i p_no

Original: r m a l
Revised: r m a l

Original: B i _ M a p_2_no r m a l B
Revised: B i _ M a p_2_no r m a l B

Original: i M a p_2_no r m a l c_{k} c_{k-2} c_{k-1} c_{k} 

Revised: i M a p_2_no r m a l c_{k} c_{k-2} c_{k-1} c_{k}

Original: B i M a p_ 1_
Revised: B i M a p_ 1_

Original: r e du ce d B
Revised: r e du ce d B

Original: i M a p_1_ r e du ce d
Revised: i M a p_1_ r e du ce d

Original: B
Revised: B

Original: i M a p_1_ r e du ce d
Revised: i M a p_1_ r e du ce d

Original: B i M a p_2_ r e du ce d 1 c_{k-2} c_{k-1}
Revised: B i M a p_2_ r e du ce d 1 c_{k-2} c_{k-1}

Original: B
Revised: B

Original: i M a p_1_no r m a l S k i p_no
Revised: i M a p_1_no r m a l S k i p_no

Original: r m a l B
Revised: r m a l B

Original: i M
Revised: i M

Original: a p_0_no r m a l
Revised: a p_0_no r m a l

Original: S k i p _n o r m a l c_{k} c_{k-2} c_{k-1} c_{k} 

Revised: S k i p _n o r m a l c_{k} c_{k-2} c_{k-1} c_{k}

Original: B
Revised: B

Original: i M a p_1_ r e du ce d B i M a p _ 2_ r e du ce d
Revised: i M a p_1_ r e du ce d B i M a p _ 2_ r e du ce d

Original: B i M a p_2_ r
Revised: B i M a p_2_ r

Original: [Figure]
Revised: [Figure]

Original: du ce d
Revised: du ce d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: du ced
Revised: du ce

Original: [Figure]
Revised: [Figure]

Revised: Here, η is the learning rate and ∇ is the gradient operator.

Revised: Note that the gradient based optimization for w must follow the geometry of SPD manifold to update the structured connection weight, and its corresponding SPD matrix data.

Revised: Applying the chain rule to Eq:(8) gives

Revised: [Equation]

Revised: where, ˜ w = Ψ r (cid:0) w − η ˜ ∇ w E Ltrain ( w, α ) (cid:1) denotes the weight update on the SPD manifold for the forward model.

Revised: ˜ ∇ w , Ψ r symbolizes the Riemannian gradient and the retraction operator respectively.

Revised: The second term in the Eq:(9) involves second order differentials with very high computational complexity, hence, using the ﬁnite approximation method the second term of Eq:(9) reduces to:

Revised: [Equation]

Revised: where, w ± =

Revised: Ψ r ( w ± δ ˜ ∇ ˜ w E Uval ( ˜ w, α )) and δ is a small number set to 0 .

Revised: 01 / (cid:107)∇ ˜ w E Uval ( ˜ w, α ) (cid:107) 2 .

Revised: Though the structure of bi-level optimization the same as the DARTS Liu et al.

Revised: (2018b), there are some key differences.

Revised: Firstly, the updates on the manifold-valued kernel weights are constrained on manifolds, which ensures that the feature maps at every intermediate layer are SPDs.

Revised: For concrete derivations on back-propagation for SPD network layers, refer to Huang & Van Gool (2017) work.

Revised: Secondly, the update on the aggregation weights of the involved SPD operations needs to satisfy an additional strict convex constraint, which is enforced as part of the optimization problem.

Original:  and AFEW (Dhall et al., 2011) datasets.
Revised: The pseudo code of our method is outlined in Algorithm ( 1 ).  E XPERIMENTS AND R ESULTS To keep the experimental evaluation consistent with the previously proposed SPD networks (Huang & Van Gool, 2017; Brooks et al., 2019), we used RADAR (Chen et al., 2006), HDM05 (M ¨ uller et al., 2007), and AFEW (Dhall et al., 2014) datasets.

Original: For SPDNetNAS, we ﬁrst optimize the supernet on the training/validation sets, and then prune it with the best operation for each edge.
Revised: For SPDNetNAS, we ﬁrst optimize the supernet on the training/validation sets, and then prune it with the best operation for each edge.

Original: Finally, we train the optimized architecture from scratch to document the results.
Revised: Finally, we train the optimized architecture from scratch to document the results.

Original: For both these stages, we consider the same normal and reduction cells.
Revised: For both these stages, we consider the same normal and reduction cells.

Original: A cell receives preprocessed inputs which is performed using ﬁxed BiMap 2 to make the input of same initial dimension.
Revised: A cell receives preprocessed inputs which is performed using ﬁxed BiMap 2 to make the input of same initial dimension.

Original: All architectures are trained with a batch size of 30.
Revised: All architectures are trained with a batch size of 30.

Original: Learning rate ( η ) for RADAR, HDM05, and AFEW dataset is set to 0.025, 0.025 and 0.
Revised: Learning rate ( η ) for RADAR, HDM05, and AFEW dataset is set to 0.025, 0.025 and 0.

Original: respectively.
Revised: respectively.

Original: Besides, we conducted experiments where we select architecture using a random search path (SPDNetNAS (R)), to justify whether our search space with the introduced SPD operations can derive meaningful architectures.
Revised: Besides, we conducted experiments where we select architecture using a random search path (SPDNetNAS (R)), to justify whether our search space with the introduced SPD operations can derive meaningful architectures.

Original: We refer to SPDNet (Huang & Van Gool, 2017), SPDNetBN (Brooks et al., 2019), and ManifoldNet (Chakraborty et al., 2020) for comparison against handcrafted
Revised: We refer to SPDNet (Huang & Van Gool, 2017), SPDNetBN (Brooks et al., 2019), and ManifoldNet (Chakraborty et al., 2020) for comparison against handcrafted

Original: SPD networks.
Revised: SPD networks.

Original: SPDNet and SPDNetBN are evaluated using their original implementations.
Revised: SPDNet and SPDNetBN are evaluated using their original implementations.

Original: We follow the video classiﬁcation setup of (Chakraborty et al., 2020) to evaluate ManifoldNet on AFEW.
Revised: We follow the video classiﬁcation setup of (Chakraborty et al., 2020) to evaluate ManifoldNet on AFEW.

Original: It is non-trivial to adapt ManifoldNet to RADAR and HDM05, as ManifoldNet requires SPD features with multiple channels and both of the two datasets can hardly obtain them.
Revised: It is non-trivial to adapt ManifoldNet to RADAR and HDM05, as ManifoldNet requires SPD features with multiple channels and both of the two datasets can hardly obtain them.

Original: For comparing against
Revised: For comparing against

Original: Euclidean NAS methods, we used DARTS (Liu et al., 2018b) and FairDARTS (Chu et al., 2020) by treating SPD’s logarithm maps as Euclidean data in their ofﬁcial implementation with default setup.
Revised: Euclidean NAS methods, we used DARTS (Liu et al., 2018b) and FairDARTS (Chu et al., 2020) by treating SPD’s logarithm maps as Euclidean data in their ofﬁcial implementation with default setup.

Original: We observed that using raw SPD’s as input to Euclidean NAS algorithms degrades its performance.
Revised: We observed that using raw SPD’s as input to Euclidean NAS algorithms degrades its performance.

Original: a) Drone Recognition: For this task, we used the RADAR dataset from (Chen et al., 2006).
Revised: a) Drone Recognition: For this task, we used the RADAR dataset from (Chen et al., 2006).

Original: The synthetic setting for this dataset is composed of radar signals, where each signal is split into windows of length 20 resulting in a 20x20 covariance matrix for each window (one radar data point).
Revised: The synthetic setting for this dataset is composed of radar signals, where each signal is split into windows of length 20 resulting in a 20x20 covariance matrix for each window (one radar data point).

Original: The synthesized dataset consists of 1000 data points per class.
Revised: The synthesized dataset consists of 1000 data points per class.

Original: Given 20 × 20 input covariance matrices, our reduction cell reduces them to 10 × 10 matrices followed by normal cell to provide complexity to our network.
Revised: Given 20 × 20 input covariance matrices, our reduction cell reduces them to 10 × 10 matrices followed by normal cell to provide complexity to our network.

Original: Following Brooks et al.
Revised: Following Brooks et al.

Original: (2019), we assign 50%, 25%, and 25% of the dataset for training, validation, and test set respectively.
Revised: (2019), we assign 50%, 25%, and 25% of the dataset for training, validation, and test set respectively.

Revised: The Euclidean NAS algorithms are evaluated on the euclidean map of the input.

Revised: For direct SPD input the performance of darts(95.86%) and fairdarts (92.26%) are worse as expected.

Original: For this dataset, our algorithm takes 1 CPU day of search time to provide the SPD architecture.
Revised: For this dataset, our algorithm takes 1 CPU day of search time to provide the SPD architecture.

Original: Training and validation take 9 CPU hours forepochs 3 .
Revised: Training and validation take 9 CPU hours for 200 epochs 4 .

Original: Test results on this dataset are provided in Table (2) which clearly shows the beneﬁt of our method.
Revised: Test results on this dataset are provided in Table (2) which clearly shows the beneﬁt of our method.

Original: Statistical performance show that our NAS algorithm provides an efﬁcient architecture with much fewer parameters (more than 140 times) than state-of-the-art Euclidean NAS on the SPD manifold valued data.
Revised: Statistical performance show that our NAS algorithm provides an efﬁcient architecture with much fewer parameters (more than 140 times) than state-of-the-art Euclidean NAS on the SPD manifold valued data.

Original: The normal and reduction cells obtained on this dataset are shown in Fig. 2(b).
Revised: The normal and reduction cells obtained on this dataset are shown in Fig. 2(b).

Original: b) Action Recognition: For this task, we used the HDM05 dataset (M¨uller et al., 2007) which contains 130 action classes, yet, for consistency with previous work (Brooks et al., 2019), we used 117 class for performance comparison.
Revised: b) Action Recognition: For this task, we used the HDM05 dataset (M¨uller et al., 2007) which contains 130 action classes, yet, for consistency with previous work (Brooks et al., 2019), we used 117 class for performance comparison.

Original: This dataset has 3D coordinates of 31 joints per frame.
Revised: This dataset has 3D coordinates of 31 joints per frame.

Original: Following the previous works (Harandi et al., 2017; Huang & Van Gool, 2017), we model an action for a sequence using 93 × 93 joint covariance matrix.
Revised: Following the previous works (Harandi et al., 2017; Huang & Van Gool, 2017), we model an action for a sequence using 93 × 93 joint covariance matrix.

Original: The dataset has 2083 SPD matrices distributed among all 117 classes.
Revised: The dataset has 2083 SPD matrices distributed among all 117 classes.

Original: Similar to the previous task, we split the dataset into 50%, 25%, and 25% for training, validation, and testing.
Revised: Similar to the previous task, we split the dataset into 50%, 25%, and 25% for training, validation, and testing.

Original: Here, our reduction cell is designed to reduce the matrices dimensions from 93 to 30 for legitimate comparison against Brooks et al.
Revised: Here, our reduction cell is designed to reduce the matrices dimensions from 93 to 30 for legitimate comparison against Brooks et al.

Original: To search for the best architecture, we ran our algorithm for 50 epoch (3 CPU days).
Revised: To search for the best architecture, we ran our algorithm for 50 epoch (3 CPU days).

Original: Figure 2(b) show the ﬁnal cell architecture that got selected based on the validation performance.
Revised: Figure 2(b) show the ﬁnal cell architecture that got selected based on the validation performance.

Original: The optimal architecture is trained from scratch for 100 epochs which took approximately 16 CPU hours.
Revised: The optimal architecture is trained from scratch for 100 epochs which took approximately 16 CPU hours.

Original: The test accuracy achieved on this dataset is provided in Table (2).
Revised: The test accuracy achieved on this dataset is provided in Table (2).

Original: Statistics clearly show that our models despite being lighter performs better than the NAS models and the handcrafted SPDNets.
Revised: Statistics clearly show that our models despite being lighter performs better than the NAS models and the handcrafted SPDNets.

Original: The NAS models’ inferior results show that the use of SPD layers for respecting SPD geometries is crucial for SPD data analysis.
Revised: The NAS models’ inferior results show that the use of SPD layers for respecting SPD geometries is crucial for SPD data analysis.

Original: [Table]
Revised: [Table]

Original: c) Emotion Recognition: We used AFEW dataset (Dhall et al., 2011) to evaluate the transferability of our searched architecture for emotion recognition.
Revised: c) Emotion Recognition: We used AFEW dataset (Dhall et al., 2014) to evaluate the transferability of our searched architecture for emotion recognition.

Original: This dataset has 1345 videos of facial expressions classiﬁed into 7 distinct classes.
Revised: This dataset has 1345 videos of facial expressions classiﬁed into 7 distinct classes.

Original: To train on the video frames directly, we stack all the handcrafted
Revised: To train on the video frames directly, we stack all the handcrafted

Original: SPDNets and our searched SPDNet on top of a covolutional network Meng et al.
Revised: SPDNets and our searched SPDNet on top of a covolutional network Meng et al.

Original: with its ofﬁcial implementation.
Revised: with its ofﬁcial implementation.

Original: For ManifoldNet, we compute a 64 × 64 spatial covariance matrix for each frame on the intermediate CNN features of 64 × 56 × 56 (channels, height, width).
Revised: For ManifoldNet, we compute a 64 × 64 spatial covariance matrix for each frame on the intermediate CNN features of 64 × 56 × 56 (channels, height, width).

Original: We follow the reported setup of Chakraborty et al.
Revised: We follow the reported setup of Chakraborty et al.

Original: (2020) to ﬁrst apply a single wFM layer with kernel size 5, stride 3 and 8 channels, followed by three temporal wFM layers of kernel size 3 and stride 2, with the channels being 1, 4, 8 respectively.
Revised: (2020) to ﬁrst apply a single wFM layer with kernel size 5, strideand 8 channels, followed by three temporal wFM layers of kernel size 3 and stride 2, with the channels being 1, 4, 8 respectively.

Revised: We closely follow the ofﬁcial implementation of manifoldnet 5 for the wFM layers and adapt the code to our speciﬁc task.

Original: Since SPDNet, SPDNetBN and our SPDNetNAS require a single channel SPD matrix as input, we use the ﬁnal 512 dimensional vector extracted from the covolutional network, project it using a dense layer to a 100 dimensional feature vector and compute a 100 × 100 temporal covariance matrix.
Revised: Since SPDNet, SPDNetBN and our SPDNetNAS require a single channel SPD matrix as input, we use the ﬁnal 512 dimensional vector extracted from the covolutional network, project it using a dense layer to a 100 dimensional feature vector and compute a 100 × 100 temporal covariance matrix.

Original: To study the transferability of our algorithm, we evaluate its searched architecture on RADAR and HDM05.
Revised: To study the transferability of our algorithm, we evaluate its searched architecture on RADAR and HDM05.

Original: In addition, we evaluate DARTS and FairDARTS directly on the video frames of AFEW.
Revised: In addition, we evaluate DARTS and FairDARTS directly on the video frames of AFEW.

Original: Table (3) reports the evaluations results.
Revised: Table (3) reports the evaluations results.

Original: As we can observe, the transferred architectures can handle the new dataset quite convincingly, and their test accuracies are better than those of the existing SPDNets and the Euclidean NAS algorithms.
Revised: As we can observe, the transferred architectures can handle the new dataset quite convincingly, and their test accuracies are better than those of the existing SPDNets and the Euclidean NAS algorithms.

Original: In Appendix, we present results of competing methods and our searched models on the raw SPD features of AFEW.
Revised: In Appendix, we present results of competing methods and our searched models on the raw SPD features of AFEW.

Original: [Table]
Revised: [Table]

Original:  Ablation study:
Revised: d) Ablation study:

Original: [Table]
Revised: [Table]

Original: Lastly, we conducted some ablation study to realize the effect of probability simplex constraint (sparsemax) on our suggested Fr ´ echet mixture of SPD operations.
Revised: Lastly, we conducted some ablation study to realize the effect of probability simplex constraint (sparsemax) on our suggested Fr´echet mixture of SPD operations.

Original: Although in Fig.
Revised: Although in Fig.

Original: 2(a) we show better probability weight distribution with sparsemax, Table(4) shows that it performs better empirically as well on both RADAR and HDM05 compared to the softmax and the sigmoid.
Revised: 2(a) we show better probability weight distribution with sparsemax, Table(4) shows that it performs better empirically as well on both RADAR and HDM05 compared to the softmax and the sigmoid.

Original: Therefore, SPD architectures derived using the sparsemax is observed to be better.
Revised: Therefore, SPD architectures derived using the sparsemax is observed to be better.

Original: C ONCLUSION AND F UTURE D IRECTION
Revised: e) Statistical comparison under same model complexity: We compare the statistical performance of our method against the other competing methods under similar model sizes. Table 5 show the results obtained on the RADAR dataset. One key point to note here is that when we increase the number of parameters in SPDNet and SPDNetBN, we observe a very severe degradation in the performance accuracy —mainly because the network starts overﬁtting rapidly. The performance degradation is far more severe for the HDM05 dataset with SPDNet (1.047MB) performing 0.7619% and SPDNetBN (1.082MB) performing 1.45% and hence, is not reported in the table below.

Revised: That further indicates the ability of SPDNetNAS to generalize better and avoid overﬁtting despite the larger model size.

Revised: Similarly, we experimented on the AFEW dataset.

Revised: To have a fair comparison against the related method like ManifoldNet, whose model size is about (76MB), we must reduce the model size accordingly.

Revised: ManifoldNet model size is large mainly due to multiple ﬁnal dense fully connected layers.

Revised: Hence, to reduce the model size, we decreased the number of FC layers.

Revised: The performance result with comparable model sizes on the AFEW dataset is shown in Table 5.

Revised: Again, we can infer that our SPDNetNAS achieves a signiﬁcant performance improvement over the others.

Revised: [Table]

Revised:  C ONCLUSION AND F UTURE D IRECTION

Original: In this work, we present a neural architecture search problem of SPD manifold networks.
Revised: In this work, we present a neural architecture search problem of SPD manifold networks.

Original: To solve it, a SPD cell representation and corresponding candidate operation search space is introduced.
Revised: To solve it, a SPD cell representation and corresponding candidate operation search space is introduced.

Original: A parameterized supernet search method is employed to explore the relaxed continuous SPD search space following a bi-level optimization problem with probability simplex constraint for effective
Revised: A parameterized supernet search method is employed to explore the relaxed continuous SPD search space following a bi-level optimization problem with probability simplex constraint for effective

Original: SPD network design.
Revised: SPD network design.

Original: The solution to our proposed problem using back-propagation is carefully crafted, so that, the weight updates follow the geometry of the SPD manifold.
Revised: The solution to our proposed problem using back-propagation is carefully crafted, so that, the weight updates follow the geometry of the SPD manifold.

Original: Quantitative results on the benchmark dataset show a commendable performance gain over handcrafted SPD networks and
Revised: Quantitative results on the benchmark dataset show a commendable performance gain over handcrafted SPD networks and

Original: Euclidean NAS algorithms.
Revised: Euclidean NAS algorithms.

Original: Additionally, we demonstrate that the learned SPD architecture is much lighter than other NAS based architecture and, it is transferable to other datasets as well.
Revised: Additionally, we demonstrate that the learned SPD architecture is much lighter than other NAS based architecture and, it is transferable to other datasets as well.

Original: There can be many direction to improve our work, e.g., allowing SPDNetNAS to reduce the rigid constraints on the ﬁnal architecture.
Revised: Our work provides an architecture search methodology for the scenarios where the acquired input data are SPD’s, for example, diffusion tensor imaging for medical applications, drone recognition, etc. In addition, our method offers a paradigm to automate the neural architecture design for the scenarios that require the second-order representations/poolings for robust visual recognition (e.g., Wang et al. Engin et al.

Original: Another possible task is  to automate the design of SPD operation search space, presently, we manually deﬁne the set of possible candidate operations.
Revised: ; Wang et al. Accordingly, we encourage more future works to pursue these two directions. Also, it is fairly interesting to extend our proposed method to sequential manifold valued data (Zhen et al., 2019; Chakraborty et al., 2018).

Original: A A DDITIONAL E XPERIMENTAL A NALYSIS A. E FFECT OF MODIFYING PREPROCESSING LAYERS FOR MULTIPLE DIMENSIONALITY REDUCTION
Revised: A A DDITIONAL E XPERIMENTAL A NALYSIS A. E FFECT OF MODIFYING PREPROCESSING LAYERS FOR MULTIPLE DIMENSIONALITY REDUCTION

Original: Unlike Huang & Van Gool (2017) work on the SPD network
Revised: Unlike Huang & Van Gool (2017) work on the SPD network

Original: [Table]
Revised: [Table]

Original: where multiple transformation matrices are applied at multiple layers to reduce the dimension of the input data, our reduction cell presented in the main paper is one step.
Revised: where multiple transformation matrices are applied at multiple layers to reduce the dimension of the input data, our reduction cell presented in the main paper is one step.

Original: For example: For HDM05 dataset (M¨uller et al., 2007), the author’s of
Revised: For example: For HDM05 dataset (M¨uller et al., 2007), the author’s of

Original: SDPNet (Huang & Van Gool, 2017) apply 93 × 70 , 70 × 50 , 50 × 30 , transformation matrices to reduce the dimension of the input matrix, on the contrary, we reduce the dimension in one step from 93 to 30 which is inline with Brooks et al.
Revised: SPDNet (Huang & Van Gool, 2017) apply 93 × 70 , 70 × 50 , 50 × 30 , transformation matrices to reduce the dimension of the input matrix, on the contrary, we reduce the dimension in one step from 93 to 30 which is inline with Brooks et al.

Original: (2019) work.
Revised: (2019) work.

Original: To study the behaviour of our method under multiple dimesionality reduction pipeline on HDM
Revised: To study the behaviour of our method under multiple dimesionality reduction pipeline on HDM

Original: [Table]
Revised: [Table]

Original: we use the preprocessing layers to perform dimensionality reduction.
Revised: we use the preprocessing layers to perform dimensionality reduction.

Original: To be precise
Revised: To be precise

Original: [Table]
Revised: [Table]

Original: we consider a preprocessing step to reduce the dimension from 93 to 70 to 50 and then, a reduction cell that reduced the dimension from 50 to 24.
Revised: we consider a preprocessing step to reduce the dimension from 93 to 70 to 50 and then, a reduction cell that reduced the dimension from 50 to 24.

Original: This modiﬁcation has the advantage that it reduces the search time from 3 CPU days to 2.5 CPU days, and in addition, provides a performance gain (see Table (5))
Revised: This modiﬁcation has the advantage that it reduces the search time from 3 CPU days to 2.5 CPU days, and in addition, provides a performance gain (see Table (6))

Original: [Table]
Revised: [Table]

Original: The normal and the reduction cells for the multiple dimension reduction are shown in Figure (3)
Revised: The normal and the reduction cells for the multiple dimension reduction are shown in Figure (3)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: A. E FFECT OF ADDING NODES TO THE CELL
Revised: A. E FFECT OF ADDING NODES TO THE CELL

Original: Experiments presented in the main paper consists of N = 5 nodes per cell which includes two input nodes, one output node, and two intermediate nodes.
Revised: Experiments presented in the main paper consists of N = 5 nodes per cell which includes two input nodes, one output node, and two intermediate nodes.

Original: To do further analysis of our design choice, we added nodes to the cell.
Revised: To do further analysis of our design choice, we added nodes to the cell.

Original: Such analysis can help us study the critical behaviour of our cell design i.e, whether adding an intermediate nodes can improve the performance or not?, and how it affects the computational complexity of our algorithm?
Revised: Such analysis can help us study the critical behaviour of our cell design i.e, whether adding an intermediate nodes can improve the performance or not?, and how it affects the computational complexity of our algorithm?

Original: To perform this experimental analysis, we used
Revised: To perform this experimental analysis, we used

Original: HDM05 dataset (M¨uller et al., 2007).
Revised: HDM05 dataset (M¨uller et al., 2007).

Original: We added one extra intermediate node ( N = 6 ) to the cell design.
Revised: We added one extra intermediate node ( N = 6 ) to the cell design.

Original: We observe that we converge towards an architecture design that is very much similar in terms of operations (see Figure 4).
Revised: We observe that we converge towards an architecture design that is very much similar in terms of operations (see Figure 4).

Original: The evaluation results shown in Table (6) help us to deduce that adding more intermediate nodes increases the number of channels for output node, subsequently leading to increased complexity and almost double the computation time.
Revised: The evaluation results shown in Table (7) help us to deduce that adding more intermediate nodes increases the number of channels for output node, subsequently leading to increased complexity and almost double the computation time.

Original: [Table]
Revised: [Table]

Original: A. E FFECT OF ADDING MULTIPLE CELLS
Revised: A. E FFECT OF ADDING MULTIPLE CELLS

Original: In our paper we stack 1 normal cell over 1 reduction cell for all the experiments.
Revised: In our paper we stack 1 normal cell over 1 reduction cell for all the experiments.

Original: For more extensive analysis of the proposed method, we conducted training experiments by stacking multiple cells which
Revised: For more extensive analysis of the proposed method, we conducted training experiments by stacking multiple cells which

Original: [Figure]
Revised: [Figure]

Original: is in-line with the experiments conducted by Liu et al.
Revised: is in-line with the experiments conducted by Liu et al.

Original: (2018b).
Revised: (2018b).

Original: We then transfer the optimized architectures from the singe cell search directly to the multi-cell architectures for training.
Revised: We then transfer the optimized architectures from the singe cell search directly to the multi-cell architectures for training.

Original: Hence, the search time for all our experiments is same as for a single cell search i.e. 3 CPU days.
Revised: Hence, the search time for all our experiments is same as for a single cell search i.e. 3 CPU days.

Original: Results for this experiment are provided in Table 7.
Revised: Results for this experiment are provided in Table 8.

Original: The ﬁrst row in the table shows the performance for single cell model, while the second and third rows show the performance with multi-cell stacking.
Revised: The ﬁrst row in the table shows the performance for single cell model, while the second and third rows show the performance with multi-cell stacking.

Original: Remarkably, by stacking multiple cells our proposed SPDNetNAS outperforms SPDNetBN Brooks et al.
Revised: Remarkably, by stacking multiple cells our proposed SPDNetNAS outperforms SPDNetBN Brooks et al.

Original: by a large margin (about 8% , i.e., about 12% for the relative improvement) . 
Revised: by a large margin (about 8% , i.e., about 12% for the relative improvement) . [Figure]

Original: [Table]
Revised: [Table]

Original: A. AFEW P ERFORMANCE C OMPARISON ON RAW SPD FEATURES
Revised: A. AFEW P ERFORMANCE C OMPARISON ON RAW SPD FEATURES

Original: In addition to the evaluation on CNN features in the major paper, we also use the raw SPD features (extracted from gray video frames) from Huang & Van Gool (2017); Brooks et al.
Revised: In addition to the evaluation on CNN features in the major paper, we also use the raw SPD features (extracted from gray video frames) from Huang & Van Gool (2017); Brooks et al.

Original: (2019) to compare the competing methods.
Revised: (2019) to compare the competing methods.

Original: To be speciﬁc, each frame is normalized to 20 × 20 and then represent each video using a 400 × 400 covariance matrix (Wang et al., 2012; Huang & Van Gool, 2017).
Revised: To be speciﬁc, each frame is normalized to 20 × 20 and then represent each video using a 400 × 400 covariance matrix (Wang et al., 2012; Huang & Van Gool, 2017).

Original: Tablesummarizes the results.
Revised: Tablesummarizes the results.

Original: As we can see, the transferred architecture can handle the new dataset quite convincingly.
Revised: As we can see, the transferred architecture can handle the new dataset quite convincingly.

Original: The test accuracy is comparable to the best SPD network method for RADAR model transfer.
Revised: The test accuracy is comparable to the best SPD network method for RADAR model transfer.

Original: For HDM05 model transfer, the test accuracy is much better than the existing SPD networks.
Revised: For HDM05 model transfer, the test accuracy is much better than the existing SPD networks.

Original: [Table]
Revised: [Table]

Original: A. D ERIVED CELL ARCHITECTURE USING SIGMOID ON F R ´ ECHET MIXTURE OF SPD OPERATION
Revised: A. D ERIVED CELL ARCHITECTURE USING SIGMOID ON F R ´ ECHET MIXTURE OF SPD OPERATION

Original: Figure 5(a) and Figure 5(b) show the cell architecture obtained using the softmax and sigmoid respectively on the Fr´echet mixture of SPD operation.
Revised: Figure 5(a) and Figure 5(b) show the cell architecture obtained using the softmax and sigmoid respectively on the Fr´echet mixture of SPD operation.

Original: It can be observed that it has relatively more skip and pooling operation than sparsemax ((see Figure 2(b))).
Revised: It can be observed that it has relatively more skip and pooling operation than sparsemax ((see Figure 2(b))).

Original: In contrast to softmax and sigmoid, the SPD cell obtained using sparsemax is composed of more convolution type operation in the architecture, which in fact is important for better representation of the data.
Revised: In contrast to softmax and sigmoid, the SPD cell obtained using sparsemax is composed of more convolution type operation in the architecture, which in fact is important for better representation of the data.

Original: 1 c_{k-2} c_{k-1}
Revised: 1 c_{k-2} c_{k-1}

Original: B i M a p_2_no r m a l
Revised: B i M a p_2_no r m a l

Original: S k i p_no
Revised: S k i p_no

Original: r m a l
Revised: r m a l

Original: S k i p_no
Revised: S k i p_no

Original: r m a l
Revised: r m a l

Original: S k i p_no
Revised: S k i p_no

Original: r m a l c_{k} c_{k-2} c_{k-1} c_{k}

Revised: r m a l c_{k} c_{k-2} c_{k-1} c_{k}

Original: MaxPooling
Revised: MaxPooling

Original: [Table]
Revised: [Table]

Original: MaxPooling
Revised: MaxPooling

Original: [Table]
Revised: [Table]

Original: reduced
Revised: reduced

Original: M a x P oo li
Revised: M a x P oo li

Original: [Table]
Revised: [Table]

Original: _ r e du ce d
Revised: _ r e du ce d

Original: M
Revised: M

Original: [Table] [Table]
Revised: [Table] [Table]

Original: c_{k
Revised: c_{k

Original: [Table]
Revised: [Table]

Original: 2} c_{k-1}
Revised: 2} c_{k-1}

Original: [Table]
Revised: [Table]

Original: i M
Revised: i M

Original: [Table]
Revised: [Table]

Original: p
Revised: p

Original: [Table]
Revised: [Table]

Original: no r m
Revised: no r m

Original: [Table]
Revised: [Table]

Original: c_
Revised: c_

Original: [Table]
Revised: [Table]

Original: k
Revised: k

Original: [Table] [Table]
Revised: [Table] [Table]

Original: p
Revised: p

Original: [Table]
Revised: [Table]

Original: c_{k
Revised: c_{k

Original: [Table]
Revised: [Table]

Original: k
Revised: k

Original: [Table]
Revised: [Table]

Original: } c_{k
Revised: } c_{k

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B
Revised: B

Original: i M a p_1_ r e d u ce d B i M a p_1_ r e du ce d
Revised: i M a p_1_ r e d u ce d B i M a p_1_ r e du ce d

Original: [Table]
Revised: [Table]

Original: k
Revised: k

Original: [Table] [Table]
Revised: [Table] [Table]

Original: k i p
Revised: k i p

Original: [Table]
Revised: [Table]

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: S k i p
Revised: S k i p

Original: [Table]
Revised: [Table]

Original: n o r m a l c_{k} c_{k-2} c_{k-1} c_{k}
Revised: n o r m a l c_{k} c_{k-2} c_{k-1} c_{k}

Original: M a x P o o li ng_ r e du c e d
Revised: M a x P o o li ng_ r e du c e d

Original: B i M a p_2_ r e du ce d (a) Normal Cell learned on HDM (b) Reduced Cell learned on HDM (c) Normal Cell learned on RADAR (d) Reduced Cell learned on RADAR
Revised: B i M a p_2_ r e du ce d (a) Normal Cell learned on HDM (b) Reduced Cell learned on HDM (c) Normal Cell learned on RADAR (d) Reduced Cell learned on RADAR

Original: M a x P o o li ng_ r e du ce d M a x P oo li ng_ r e du ce d
Revised: M a x P o o li ng_ r e du ce d M a x P oo li ng_ r e du ce d

Original: S k i p_no
Revised: S k i p_no

Original: r m a l
Revised: r m a l

Original: S k i p_no
Revised: S k i p_no

Original: r m a l S k i p_no
Revised: r m a l S k i p_no

Original: r m a l
Revised: r m a l

Original: S k i p_no
Revised: S k i p_no

Original: r m a l
Revised: r m a l

Original: [Figure]
Revised: [Figure]

Original: A. C OMPARISON BETWEEN K ARCHER FLOW AND RECURSIVE APPORACH FOR WEIGHTED F RECHET MEAN
Revised: A. C OMPARISON BETWEEN K ARCHER FLOW AND RECURSIVE APPORACH FOR WEIGHTED F RECHET MEAN

Original: The proposed NAS algorithm is based on Fr ´ echet Mean computations.
Revised: The proposed NAS algorithm is based on Fr ´ echet Mean computations.

Original: From the weighted mixture of operations between nodes to the derivation of intermediate nodes, both compute the Fr ´ echet mean of a set of points on the SPD manifold.
Revised: From the weighted mixture of operations between nodes to the derivation of intermediate nodes, both compute the Fr ´ echet mean of a set of points on the SPD manifold.

Original: It is well known that there is no closed form solution when the number of input samples is bigger than 2 (Brooks et al., 2019).
Revised: It is well known that there is no closed form solution when the number of input samples is bigger than 2 (Brooks et al., 2019).

Original: We can only compute an approximation using the famous Karcher ﬂow algorithm (Brooks et al., 2019) or recursive geodesic mean (Chakraborty et al., 2020).
Revised: We can only compute an approximation using the famous Karcher ﬂow algorithm (Brooks et al., 2019) or recursive geodesic mean (Chakraborty et al., 2020).

Original: For comparison, we replace our used Karcher ﬂow algorithm with the recursive approach under our SPDNetNAS framework.
Revised: For comparison, we replace our used Karcher ﬂow algorithm with the recursive approach under our SPDNetNAS framework.

Original: Table 9 sumarizes the comparison between these two algorithms.
Revised: Table 10 sumarizes the comparison between these two algorithms.

Original: We observe considerable decrease in accuracy for both the training and test set when using the recursive methods, showing that the Karcher ﬂow algorithm favors our proposed algorithm more.
Revised: We observe considerable decrease in accuracy for both the training and test set when using the recursive methods, showing that the Karcher ﬂow algorithm favors our proposed algorithm more.

Original: [Table]
Revised: [Table]

Original: A. C ONVERGENCE CURVE ANALYSIS
Revised: A. C ONVERGENCE CURVE ANALYSIS

Original: Figure 6(a) shows the validation curve which almost saturates at 200 epoch demonstrating the stability of our training process.
Revised: Figure 6(a) shows the validation curve which almost saturates at 200 epoch demonstrating the stability of our training process.

Original: First column bar of Figure (6(b)) show the test accuracy comparison when only 10% of the data is used for training our architecture which demonstrate the effectiveness of our algorithm.
Revised: First column bar of Figure (6(b)) show the test accuracy comparison when only 10% of the data is used for training our architecture which demonstrate the effectiveness of our algorithm.

Original: Further, we study this for our SPDNetNAS architecture by taking 10%, 33%, 80% of the data for training.
Revised: Further, we study this for our SPDNetNAS architecture by taking 10%, 33%, 80% of the data for training.

Original: Figure 6(b)) clealy show our superiority of SPDNetNAS algorithm than handcrafted
Revised: Figure 6(b)) clealy show our superiority of SPDNetNAS algorithm than handcrafted

Original: SPD networks.
Revised: SPD networks.

Original: Figure (7(a)) and Figure (7(b)) show the convergence curve of our loss function on the RADAR and HDM05 datasets respectively.
Revised: Figure (7(a)) and Figure (7(b)) show the convergence curve of our loss function on the RADAR and HDM05 datasets respectively.

Original: For the RADAR dataset the validation and training losses follow a similar trend and converges at 200 epochs.
Revised: For the RADAR dataset the validation and training losses follow a similar trend and converges at 200 epochs.

Original: For the HDM05 dataset, we observe the training curve plateaus after 60 epochs, where as the validation curve takes 100 epochs to provide a stable performance.
Revised: For the HDM05 dataset, we observe the training curve plateaus after 60 epochs, where as the validation curve takes 100 epochs to provide a stable performance.

Original: Additionally, we noticed a reasonable gap between the training loss and validation loss for the HDM05 dataset (M¨uller et al., 2007).
Revised: Additionally, we noticed a reasonable gap between the training loss and validation loss for the HDM05 dataset (M¨uller et al., 2007).

Original: A similar pattern of convergence gap between validation loss and training loss has been observed by Huang & Van Gool (2017) work.
Revised: A similar pattern of convergence gap between validation loss and training loss has been observed by Huang & Van Gool (2017) work.

Original: A. W HY WE PREFERRED TO SIMULATE OUR EXPERIMENTS ON CPU RATHER THAN GPU?
Revised: A. W HY WE PREFERRED TO SIMULATE OUR EXPERIMENTS ON CPU RATHER THAN GPU?

Original: When dealing with SPD matrices, we need to carry out complex computations.
Revised: When dealing with SPD matrices, we need to carry out complex computations.

Original: These computations are performed to make sure that our transformed representation and corresponding operations respect the underlying manifold structure.
Revised: These computations are performed to make sure that our transformed representation and corresponding operations respect the underlying manifold structure.

Original: In our study, we analyzed SPD matrices with the Afﬁne Invariant
Revised: In our study, we analyzed SPD matrices with the Afﬁne Invariant

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Riemannian Metric (AIRM), this induces operations heavily dependent on singular value decomposition (SVD) or eigendecomposition (EIG).
Revised: Riemannian Metric (AIRM), this induces operations heavily dependent on singular value decomposition (SVD) or eigendecomposition (EIG).

Original: Both decompositions suffer from weak support on
Revised: Both decompositions suffer from weak support on

Original: GPU platforms.
Revised: GPU platforms.

Original: Hence, our training did not beneﬁt from GPU acceleration and we decided to train on CPU.
Revised: Hence, our training did not beneﬁt from GPU acceleration and we decided to train on CPU.

Original: As a future work, we aim to speedup our implementation on GPU by optimizing the SVD
Revised: As a future work, we aim to speedup our implementation on GPU by optimizing the SVD

Original: Householder bi-diagonalization process as studied in some existing works like Dong et al.
Revised: Householder bi-diagonalization process as studied in some existing works like Dong et al.

Revised: (2017a);

Original: Gates et al.
Revised: Gates et al.

Original: B D ETAILED DESCRIPTION OF OUR PROPOSED OPERATIONS
Revised: B D ETAILED DESCRIPTION OF OUR PROPOSED OPERATIONS

Original: In this section, we describe some of the major operations deﬁned in the main paper from an intuitive point of view.
Revised: In this section, we describe some of the major operations deﬁned in the main paper from an intuitive point of view.

Original: We particularly focus on some of the new operations that are deﬁned for the input
Revised: We particularly focus on some of the new operations that are deﬁned for the input

Original: SPDs, i
Revised: SPDs, i

Original: [Figure]
Revised: [Figure]

Original: e., the Weighted Riemannian Pooling, the Average/Max Pooling, the Skip Reduced operation and the Mixture of Operations.
Revised: e., the Weighted Riemannian Pooling, the Average/Max Pooling, the Skip Reduced operation and the Mixture of Operations.

Original: B. W EIGHTED R IEMANNIAN P OOLING
Revised: B. W EIGHTED R IEMANNIAN P OOLING

Original: Figure 8 provides an intuition behind the Weighted Riemannian Pooling operation.
Revised: Figure 8 provides an intuition behind the Weighted Riemannian Pooling operation.

Original: Here, w 11, w 21, etc.,
Revised: Here, w 11, w 21, etc.

Revised: The channel —shown in orange, is then computed as weighted Fr´echet mean over these two input channels.

Revised: This procedure is repeated to achieve the desired number of output channels (here two) and ﬁnally all the output channels are concatenated.

Revised: The weights are learnt as a part of the optimization procedure ensuring the explicit convex constraint is imposed

Original: [Figure]
Revised: [Figure]

Original: B. A VERAGE AND M AX P OOLING
Revised: B. A VERAGE AND M AX P OOLING

Original: In Figure 9 we show our average and max pooling operations.
Revised: In Figure 9 we show our average and max pooling operations.

Original: We ﬁrst perform a LogEig map on the
Revised: We ﬁrst perform a LogEig map on the

Original: SPD matrices to project them to the Euclidean space.
Revised: SPD matrices to project them to the Euclidean space.

Original: Next
Revised: Next

Original: [Figure]
Revised: [Figure]

Original: we perform average and max pooling on these Euclidean matrices similar to classical convolutional neural networks.
Revised: we perform average and max pooling on these Euclidean matrices similar to classical convolutional neural networks.

Original: We further perform an
Revised: We further perform an

Original: ExpEig map to project the Euclidean matrices back on the SPD manifold.
Revised: ExpEig map to project the Euclidean matrices back on the SPD manifold.

Original: The diagram shown in
Revised: The diagram shown in

Original: Figure 9 is inspired by Huang & Van Gool (2017) work.
Revised: Figure 9 is inspired by Huang & Van Gool (2017) work.

Original: The kernel size of AveragePooling reduced and MaxPooling reduced is set to 2 or 4 for all experiments according to the speciﬁc dimensionality reduction factors.
Revised: The kernel size of AveragePooling reduced and MaxPooling reduced is set to 2 or 4 for all experiments according to the speciﬁc dimensionality reduction factors.

Original: [Figure]
Revised: [Figure]

Original: B. S KIP R EDUCED
Revised: B. S KIP R EDUCED

Original: Following Liu et al.
Revised: Following Liu et al.

Original: (2018b)
Revised: (2018b)

Original: [Figure]
Revised: [Figure]

Original: we deﬁned an analogous of Skip operation on a single channel for the reduced cell (Figure 10).
Revised: we deﬁned an analogous of Skip operation on a single channel for the reduced cell (Figure 10).

Original: We start by using a BiMap layer —equivalent to Conv in Liu et al.
Revised: We start by using a BiMap layer —equivalent to Conv in Liu et al.

Original: (2018b)
Revised: (2018b)

Original: , to map the input channel to an SPD whose space dimension is half of the input dimension.
Revised: , to map the input channel to an SPD whose space dimension is half of the input dimension.

Original: We further perform an SVD decomposition on the two SPDs followed by concatenating the Us, Vs and Ds obtained from SVD to block diagonal matrices.
Revised: We further perform an SVD decomposition on the two SPDs followed by concatenating the Us, Vs and Ds obtained from SVD to block diagonal matrices.

Original: Finally
Revised: Finally

Original: [Figure]
Revised: [Figure]

Original: we compute the output by multiplying the block diagonal U, V and D computed before.
Revised: we compute the output by multiplying the block diagonal U, V and D computed before.

Original: [Figure]
Revised: [Figure]

Original: B. M IXED O PERATION ON SPD S
Revised: B. M IXED O PERATION ON SPD S

Original: In Figure 11 we provide an intuition of the mixed operation we have proposed in the main paper.
Revised: In Figure 11 we provide an intuition of the mixed operation we have proposed in the main paper.

Original: We consider a very simple base case of three nodes, two input nodes (1 and 2) and one output node (node 3).
Revised: We consider a very simple base case of three nodes, two input nodes (1 and 2) and one output node (node 3).

Original: The goal is to compute the output node 3 from input nodes 1 and 2.
Revised: The goal is to compute the output node 3 from input nodes 1 and 2.

Original: We perform a candidate set of operations on the input node, which correspond to edges between the nodes (here two for simplicity).
Revised: We perform a candidate set of operations on the input node, which correspond to edges between the nodes (here two for simplicity).

Original: Each operation has a weight
Revised: Each operation has a weight

Original: α
Revised: α

Original: i j where i corresponds to the node index and j is the candidate operation identiﬁer.
Revised: i j where i corresponds to the node index and j is the candidate operation identiﬁer.

Original: In Figure 11 below i and j ∈ { 1 , 2 } and α 1 = { α 1 1 , α 1 2 } , α 2 = { α 2 1 , α 2 2 } .
Revised: In Figure 11 below i and j ∈ { 1 , 2 } and α 1 = { α 1 1 , α 1 2 } , α 2 = { α 2 1 , α 2 2 } .

Original: α ’s are optimized as a part of the bi-level optimization procedure proposed in the main paper.
Revised: α ’s are optimized as a part of the bi-level optimization procedure proposed in the main paper.

Original: Using these alpha’s, we perform a channel-wise weighted Fr´echet mean (wFM) as depicted in the ﬁgure below.
Revised: Using these alpha’s, we perform a channel-wise weighted Fr´echet mean (wFM) as depicted in the ﬁgure below.

Original: This effectively corresponds to a mixture of the candidate operations.
Revised: This effectively corresponds to a mixture of the candidate operations.

Original: Note that the alpha’s corresponding to all channels of a single operation are assumed to be the same.
Revised: Note that the alpha’s corresponding to all channels of a single operation are assumed to be the same.

Original: Once the weighted Fr´echet means have been computed for nodes 1 and 2, we perform a channel-wise concatenation on the outputs of the two nodes, effectively doubling the number of channels in node 3.
Revised: Once the weighted Fr´echet means have been computed for nodes 1 and 2, we perform a channel-wise concatenation on the outputs of the two nodes, effectively doubling the number of channels in node 3.

Original: [Figure]
Revised: [Figure]

Original: C D IFFERENTIABLE CONVEX LAYER FOR SPARSEMAX OPTIMIZATION
Revised: C D IFFERENTIABLE CONVEX LAYER FOR SPARSEMAX OPTIMIZATION

Original: 1 import cvxpy as cp 2 from cvxpylayers.torch import CvxpyLayer 3 4 def sparsemax_convex_layer(x, n): w
Revised: 1 import cvxpy as cp 2 from cvxpylayers.torch import CvxpyLayer 3 4 def sparsemax_convex_layer(x, n): w_ = cp.

Original:  = cp.Variable(n)
Revised: Variable(n) x_ = cp.

Original: = the objective constraint  cp
Revised: Parameter( constraint = cp

Original: ) constraint cpw problem cp return w
Revised: Minimize cp cp multiply(w )) constraint = cp sum w

Original: Listing 1: Function to solve the sparsemax constraint optimization D F URTHER STUDY It has been reported that convolutional architectures outperform the recurrent networks for sequential data (Bai et al., 2018). To study a similar behaviour for sequential manifold valued data Zhen et al.
Revised: < cp

Original: ; Chakraborty et al. deﬁned dilated CNN to increase the receptive ﬁeld, and used residual connections for a deeper and stable network design For neural architecture search algorithm to design such an architecture on manifold-valued sequential data, we need to deﬁne the search space that is composed of both residual connections of the architectures and dilated convolution based operations on the manifold-valued data. In addition, another potential impact of our work to SPD pooling is that it might enable the optimization on architecture designs of various SPD pooling layers studied by exiting works like Wang et al.
Revised: ) layer

Original: Engin et al.
Revised: CvxpyLayer problem x_]

Original: ; Wang et al. We believe such a study can be very useful for future SPD based NAS algorithm’s to build deeper, better and more stable SPD manifold networks.
Revised: ]) w, = layer(x) return w