Original: coVariance Neural Networks
Revised: coVariance Neural Networks

Original: Abstract
Revised: Abstract

Original: Graph neural networks (GNN) are an effective framework that exploit interrelationships within graph-structured data for learning.
Revised: Graph neural networks (GNN) are an effective framework that exploit interrelationships within graph-structured data for learning.

Original: Principal component analysis (PCA) involves the projection of data on the eigenspace of the covariance matrix and draws similarities with the graph convolutional filters in GNNs.
Revised: Principal component analysis (PCA) involves the projection of data on the eigenspace of the covariance matrix and draws similarities with the graph convolutional filters in GNNs.

Original: Motivated by this observation, we study a GNN architecture, called coVariance neural network (VNN), that operates on sample covariance matrices as graphs.
Revised: Motivated by this observation, we study a GNN architecture, called coVariance neural network (VNN), that operates on sample covariance matrices as graphs.

Original: We theoretically establish the stability of VNNs to perturbations in the covariance matrix, thus, implying an advantage over standard PCA-based data analysis approaches that are prone to instability due to principal components associated with close eigenvalues.
Revised: We theoretically establish the stability of VNNs to perturbations in the covariance matrix, thus, implying an advantage over standard PCA-based data analysis approaches that are prone to instability due to principal components associated with close eigenvalues.

Original: Our experiments on real-world datasets validate our theoretical results and show that VNN performance is indeed more stable than PCA-based statistical approaches.
Revised: Our experiments on real-world datasets validate our theoretical results and show that VNN performance is indeed more stable than PCA-based statistical approaches.

Original: Moreover, our experiments on multi-resolution datasets also demonstrate that VNNs are amenable to transferability of performance over covariance matrices of different dimensions; a feature that is infeasible for PCA-based approaches.
Revised: Moreover, our experiments on multi-resolution datasets also demonstrate that VNNs are amenable to transferability of performance over covariance matrices of different dimensions; a feature that is infeasible for PCA-based approaches.

Original:  Introduction
Revised:  Introduction

Original: Convolutional neural networks (CNN) are among the most popular deep learning (DL) architectures due to their demonstrated ability to learn latent features from Euclidean data adaptively and automatically [1].
Revised: Convolutional neural networks (CNN) are among the most popular deep learning (DL) architectures due to their demonstrated ability to learn latent features from Euclidean data adaptively and automatically [1].

Original: Motivated by the fact that graph models can naturally represent data in many practical applications [2], the generalizations of CNNs to adapt to graph data using GNN architectures have rapidly emerged in recent years [3].
Revised: Motivated by the fact that graph models can naturally represent data in many practical applications [2], the generalizations of CNNs to adapt to graph data using GNN architectures have rapidly emerged in recent years [3].

Original: A practical challenge to DL architectures is scalability to high-dimensional data [4].
Revised: A practical challenge to DL architectures is scalability to high-dimensional data [4].

Original: Specifically, DL models can typically have millions of parameters that require long training times and high storage capacity [5].
Revised: Specifically, DL models can typically have millions of parameters that require long training times and high storage capacity [5].

Original: GNN architectures have two attractive features towards learning from high-dimensional data: 1.
Revised: GNN architectures have two attractive features towards learning from high-dimensional data: 1.

Original: the number of parameters associated with graph convolution operations in GNNs is independent of the graph size; and 2.
Revised: the number of parameters associated with graph convolution operations in GNNs is independent of the graph size; and 2.

Original: under certain assumptions, GNN performance can be transferred across graphs of different dimensions [6].
Revised: under certain assumptions, GNN performance can be transferred across graphs of different dimensions [6].

Original: Principal component analysis (PCA) is a traditional, non-parametric approach for linear dimensionality reduction and increase of interpretability of high dimensional datasets [7].
Revised: Principal component analysis (PCA) is a traditional, non-parametric approach for linear dimensionality reduction and increase of interpretability of high dimensional datasets [7].

Original: Accordingly, it is common to embed the PCA transform within a DL pipeline [4, 8, 9].
Revised: Accordingly, it is common to embed the PCA transform within a DL pipeline [4, 8, 9].

Original: Such approaches usually use
Revised: Such approaches usually use

Original: PCA to optimize the DL model parameters by removing redundancy in the features in intermediate layers.
Revised: PCA to optimize the DL model parameters by removing redundancy in the features in intermediate layers.

Original: The scope of our work is different.
Revised: The scope of our work is different.

Original: In this paper, we leverage the connections between PCA and the mathematical concepts used in GNNs to propose a learning framework that can recover PCA and also, inherit additional desirable features associated with GNNs.
Revised: In this paper, we leverage the connections between PCA and the mathematical concepts used in GNNs to propose a learning framework that can recover PCA and also, inherit additional desirable features associated with GNNs.

Original: 1.1 PCA and Graph Fourier Transform
Revised: 1.1 PCA and Graph Fourier Transform

Original: PCA is an orthonormal transform which identifies the modes of maximum variance in the dataset.
Revised: PCA is an orthonormal transform which identifies the modes of maximum variance in the dataset.

Original: The transformed dimensions of the data (known as principal components) are uncorrelated.
Revised: The transformed dimensions of the data (known as principal components) are uncorrelated.

Original: Computing the PCA transform reduces to finding the eigenbasis of the covariance matrix or equivalently, the singular value decomposition of the data matrix [10].
Revised: Computing the PCA transform reduces to finding the eigenbasis of the covariance matrix or equivalently, the singular value decomposition of the data matrix [10].

Original: We note that PCA computation using eigendecomposition of a covariance matrix has similarities with the graph Fourier transform (GFT) in the graph signal processing literature [11].
Revised: We note that PCA computation using eigendecomposition of a covariance matrix has similarities with the graph Fourier transform (GFT) in the graph signal processing literature [11].

Original: This observation is discussed more formally next.
Revised: This observation is discussed more formally next.

Original: Consider an m − dimensional random vector X ∈ R m × 1 with the true or ensemble covariance matrix of X defined as C △ =
Revised: Consider an m − dimensional random vector X ∈ R m × 1 with the true or ensemble covariance matrix of X defined as C △ =

Original: E [( X − E
Revised: E [( X − E

Original: [ X ])( X − E [ X ]) T ] .
Revised: [ X ])( X − E [ X ]) T ] .

Original: In practice, we may not observe the data model for X and its statistics, such as the covariance matrix C directly.
Revised: In practice, we may not observe the data model for X and its statistics, such as the covariance matrix C directly.

Original: Alternatively, we have access to a dataset consisting of n random, independent and identically distributed (i.i.d) samples of X , given by x i ∈ R m × 1 , ∀ i ∈ { 1 , . . . , n } , where the data matrix is ˆ x n = [ x 1 , . . .
Revised: Alternatively, we have access to a dataset consisting of n random, independent and identically distributed (i.i.d) samples of X , given by x i ∈ R m × 1 , ∀ i ∈ { 1 , . . . , n } , where the data matrix is ˆ x n = [ x 1 , . . .

Original: , x n ] .
Revised: , x n ] .

Original: Using the samples x i , ∀ i ∈ { 1 , . . . , n } , we can form an estimate of the ensemble covariance matrix, conventionally referred to as the sample covariance matrix
Revised: Using the samples x i , ∀ i ∈ { 1 , . . . , n } , we can form an estimate of the ensemble covariance matrix, conventionally referred to as the sample covariance matrix

Original: [Equation]
Revised: [Equation]

Original: where ¯ x is the sample mean across n samples and · T refers to the transpose operator.
Revised: where ¯ x is the sample mean across n samples and · T refers to the transpose operator.

Original: Given the eigendecomposition of ˆ C n
Revised: Given the eigendecomposition of ˆ C n

Original: [Equation]
Revised: [Equation]

Original: where U =
Revised: where U =

Original: [ u 1 , . . .
Revised: [ u 1 , . . .

Original: , u m ] is the matrix with its columns as the eigenvectors and W = diag ( w 1 , . . .
Revised: , u m ] is the matrix with its columns as the eigenvectors and W = diag ( w 1 , . . .

Original: , w m ) is the diagonal matrix of eigenvalues, such that, w 1 ≥ w 2 · · · ≥ w m , the implementation of PCA is given by [10]
Revised: , w m ) is the diagonal matrix of eigenvalues, such that, w 1 ≥ w 2 · · · ≥ w m , the implementation of PCA is given by [10]

Original: [Equation]
Revised: [Equation]

Original: The eigenvectors of ˆ C n form the principal components and the transformation in (3) is equivalent to the projection of the sample
Revised: The eigenvectors of ˆ C n form the principal components and the transformation in (3) is equivalent to the projection of the sample

Original: x i in the eigenvector space of ˆ C n .
Revised: x i in the eigenvector space of ˆ C n .

Original: A detailed overview of PCA is provided in Appendix B.
Revised: A detailed overview of PCA is provided in Appendix B.

Original: We note that in the context of graph signal processing, (3) is equivalent to the graph Fourier transform (GFT) if the covariance matrix ˆ C n is considered the graph representation or a graph shift operator for the data ˆ x n [12].
Revised: We note that in the context of graph signal processing, (3) is equivalent to the graph Fourier transform (GFT) if the covariance matrix ˆ C n is considered the graph representation or a graph shift operator for the data ˆ x n [12].

Original: 1.2 Contributions
Revised: 1.2 Contributions

Original: Motivated by the observation in Section 1.1, we introduce the notion of coVariance Fourier transform and define coVariance filters similar to graph convolutional filters in GNNs.
Revised: Motivated by the observation in Section 1.1, we introduce the notion of coVariance Fourier transform and define coVariance filters similar to graph convolutional filters in GNNs.

Original: Our analysis shows that standard PCA can be recovered using coVariance filters.
Revised: Our analysis shows that standard PCA can be recovered using coVariance filters.

Original: Furthermore, we propose a DL architecture based on coVariance filters, referred to as coVariance neural networks (VNN) 1 : a permutationinvariant architecture derived from the GNN framework that uses convolutional filters defined on the covariance matrix.
Revised: Furthermore, we propose a DL architecture based on coVariance filters, referred to as coVariance neural networks (VNN) 1 : a permutationinvariant architecture derived from the GNN framework that uses convolutional filters defined on the covariance matrix.

Original: Understanding VNNs is relevant given the ubiquity of so-termed correlation networks [13, Chapter 7.3.1] and covariance matrices as graph models for neuroimaging [14], user preference [15], financial [16], and gene expression data[13, Chapter 7.3.4], to name a few timely domains.
Revised: Understanding VNNs is relevant given the ubiquity of so-termed correlation networks [13, Chapter 7.3.1] and covariance matrices as graph models for neuroimaging [14], user preference [15], financial [16], and gene expression data[13, Chapter 7.3.4], to name a few timely domains.

Original: Summary of our contributions is as follows:
Revised: Summary of our contributions is as follows:

Original: Stability of VNN: For finite n , the eigenvectors of the sample covariance matrix ˆ C n are perturbed from the eigenvectors of the ensemble covariance matrix C .
Revised: Stability of VNN: For finite n , the eigenvectors of the sample covariance matrix ˆ C n are perturbed from the eigenvectors of the ensemble covariance matrix C .

Original: Moreover, the principal components corresponding to eigenvalues that are close are unstable, i.e., small changes in the dataset may cause large changes in such principal components [17] and subsequently, irreproducible statistical results [18].
Revised: Moreover, the principal components corresponding to eigenvalues that are close are unstable, i.e., small changes in the dataset may cause large changes in such principal components [17] and subsequently, irreproducible statistical results [18].

Original: Statistical learning approaches such as regression, classification, clustering that use PCA as the first step for dimension reduction inherit this instability.
Revised: Statistical learning approaches such as regression, classification, clustering that use PCA as the first step for dimension reduction inherit this instability.

Original: Our main theoretical contribution is to establish the stability of VNNs to perturbations in the sample covariance matrix in terms of sample size n .
Revised: Our main theoretical contribution is to establish the stability of VNNs to perturbations in the sample covariance matrix in terms of sample size n .

Original: For this purpose, we leverage the perturbation theory for sample covariance matrices and show that for a given ˆ C n , the perturbation in the VNN output Φ( ˆ C n ) with respect to the VNN output for C , Φ( C ) satisfies ∥ Φ( ˆ C n ) − Φ( C ) ∥
Revised: For this purpose, we leverage the perturbation theory for sample covariance matrices and show that for a given ˆ C n , the perturbation in the VNN output Φ( ˆ C n ) with respect to the VNN output for C , Φ( C ) satisfies ∥ Φ( ˆ C n ) − Φ( C ) ∥

Original: =
Revised: =

Original: O (1 /n 12 − ε ) for some ε < 1 / 2 with high probability.
Revised: O (1 /n 12 − ε ) for some ε < 1 / 2 with high probability.

Original: Therefore, our analysis ties the stability of VNNs to the quality of sample covariance matrices, which is distinct from existing studies that study stability of GNNs under abstract, data-independent discrete or geometric perturbations to graphs [19, 20].
Revised: Therefore, our analysis ties the stability of VNNs to the quality of sample covariance matrices, which is distinct from existing studies that study stability of GNNs under abstract, data-independent discrete or geometric perturbations to graphs [19, 20].

Original: Empirical Validations: Our experiments on real-world and synthetic datasets show that VNNs are indeed stable to perturbations in the sample covariance matrix.
Revised: Empirical Validations: Our experiments on real-world and synthetic datasets show that VNNs are indeed stable to perturbations in the sample covariance matrix.

Original: Moreover, on a multi-resolution dataset, we also illustrate the transferability of VNN performance to datasets of different dimensions without re-training.
Revised: Moreover, on a multi-resolution dataset, we also illustrate the transferability of VNN performance to datasets of different dimensions without re-training.

Original: These observations provide convincing evidence for the advantages offered by
Revised: These observations provide convincing evidence for the advantages offered by

Original: VNNs over standard PCA-based approaches for data analysis.
Revised: VNNs over standard PCA-based approaches for data analysis.

Original: 1.3 Related Work
Revised: 1.3 Related Work

Original: GNNs : GNNs broadly encapsulate all DL models adapted to graph data [3, 21].
Revised: GNNs : GNNs broadly encapsulate all DL models adapted to graph data [3, 21].

Original: Recent survey articles in [3] and [21] categorize the variants of GNNs according to diverse criteria that include mathematical formulations, algorithms, and hardware/software implementations.
Revised: Recent survey articles in [3] and [21] categorize the variants of GNNs according to diverse criteria that include mathematical formulations, algorithms, and hardware/software implementations.

Original: Convolutional GNNs [22], graph autoencoders [23], recurrent GNNs, and gated GNNs [24] are among a few prominently studied and applied categories of GNNs.
Revised: Convolutional GNNs [22], graph autoencoders [23], recurrent GNNs, and gated GNNs [24] are among a few prominently studied and applied categories of GNNs.

Original: The taxonomy pertinent to this paper is that of graph convolutional network (GCN) that generalizes the concept of convolution to graphs.
Revised: The taxonomy pertinent to this paper is that of graph convolutional network (GCN) that generalizes the concept of convolution to graphs.

Original: Specifically, VNNs are based on GCNs that model the convolution operation via graph convolutional filters [25].
Revised: Specifically, VNNs are based on GCNs that model the convolution operation via graph convolutional filters [25].

Original: The graph convolutional filters are defined as polynomials over the graph representation where the order of the polynomial controls the aggregation of information over nodes in the graph [12].
Revised: The graph convolutional filters are defined as polynomials over the graph representation where the order of the polynomial controls the aggregation of information over nodes in the graph [12].

Original: We extend the definition of graph convolutional filters to filters over sample covariance matrices to define VNNs.
Revised: We extend the definition of graph convolutional filters to filters over sample covariance matrices to define VNNs.

Original: The ubiquity of PCA and prevalence of covariance matrices across disciplines for data analysis motivate us to study VNNs independently of aforementioned GCNs.
Revised: The ubiquity of PCA and prevalence of covariance matrices across disciplines for data analysis motivate us to study VNNs independently of aforementioned GCNs.

Original: Moreover, the unique structure inherent to covariance matrices (relative to arbitrary graphs) and their sample-based perturbations facilitates sharper stability analyses, which offer novel insights.
Revised: Moreover, the unique structure inherent to covariance matrices (relative to arbitrary graphs) and their sample-based perturbations facilitates sharper stability analyses, which offer novel insights.

Original: Robust PCA:
Revised: Robust PCA:

Original: Robust PCA is a variant of PCA that aims to recover the low rank representation of a dataset arbitrarily corrupted due to outliers [26, 27, 28].
Revised: Robust PCA is a variant of PCA that aims to recover the low rank representation of a dataset arbitrarily corrupted due to outliers [26, 27, 28].

Original: The outliers considered in robust
Revised: The outliers considered in robust

Original: PCA approaches could manifest in the dataset due to missing data, adversarial behavior, defects or contamination in data collection.
Revised: PCA approaches could manifest in the dataset due to missing data, adversarial behavior, defects or contamination in data collection.

Original: In contrast to such settings, here we consider the inherent statistical uncertainty in the sample covariance matrix, where ill-defined eigenvalues and eigenvectors of the sample covariance matrix can be the source of instability in statistical inference.
Revised: In contrast to such settings, here we consider the inherent statistical uncertainty in the sample covariance matrix, where ill-defined eigenvalues and eigenvectors of the sample covariance matrix can be the source of instability in statistical inference.

Original: Therefore, the notion of stability of VNNs in this paper is fundamentally distinct from the notion of stability or robustness in robust PCA approaches.
Revised: Therefore, the notion of stability of VNNs in this paper is fundamentally distinct from the notion of stability or robustness in robust PCA approaches.

Original:  coVariance Filters
Revised:  coVariance Filters

Original: If m dimensions of data ˆ x n can be represented as the nodes of an m -node, undirected graph, the sample covariance matrix ˆ C n is equivalent to its adjacency matrix.
Revised: If m dimensions of data ˆ x n can be represented as the nodes of an m -node, undirected graph, the sample covariance matrix ˆ C n is equivalent to its adjacency matrix.

Original: In GNNs, graph Fourier transform projects graph signals in the eigenspace of the graph and is leveraged to analyze graph convolutional filters [12].
Revised: In GNNs, graph Fourier transform projects graph signals in the eigenspace of the graph and is leveraged to analyze graph convolutional filters [12].

Original: Therefore, we start by formalizing the notion of coVariance Fourier transform (abbreviated as VFT).
Revised: Therefore, we start by formalizing the notion of coVariance Fourier transform (abbreviated as VFT).

Original: For this purpose, we leverage the eigendecomposition of ˆ C n in (2).
Revised: For this purpose, we leverage the eigendecomposition of ˆ C n in (2).

Original: Definition 1 (coVariance Fourier Transform) .
Revised: Definition 1 (coVariance Fourier Transform) .

Original: Consider a sample covariance matrix ˆ C n as defined in (1) .
Revised: Consider a sample covariance matrix ˆ C n as defined in (1) .

Original: The coVariance Fourier transform (VFT) of a random sample x is defined as its projection on the eigenspace of ˆ C n and is given by
Revised: The coVariance Fourier transform (VFT) of a random sample x is defined as its projection on the eigenspace of ˆ C n and is given by

Original: [Equation]
Revised: [Equation]

Original: The i -th entry of ˜ x , i.e., [ ˜ x ] i represents the i -th Fourier coefficient and is associated with the eigenvalue w i .
Revised: The i -th entry of ˜ x , i.e., [ ˜ x ] i represents the i -th Fourier coefficient and is associated with the eigenvalue w i .

Original: Note that the similarity between PCA and VFT implies that eigenvalue w i encodes the variability of dataset x n in the direction of the principal component u i .
Revised: Note that the similarity between PCA and VFT implies that eigenvalue w i encodes the variability of dataset x n in the direction of the principal component u i .

Original: In this context, the eigenvalues of the covariance matrix are mathematical equivalent of the notion of graph frequencies in graph signal processing [11].
Revised: In this context, the eigenvalues of the covariance matrix are mathematical equivalent of the notion of graph frequencies in graph signal processing [11].

Original: Next, we define the notion of coVariance filters (VF) that are polynomials in the covariance matrix.
Revised: Next, we define the notion of coVariance filters (VF) that are polynomials in the covariance matrix.

Original: Definition 2 (coVariance Filter) .
Revised: Definition 2 (coVariance Filter) .

Original: Given a set of real valued parameters { h k } mk =0 , the coVariance filter for the covariance matrix ˆ C n is defined as
Revised: Given a set of real valued parameters { h k } mk =0 , the coVariance filter for the covariance matrix ˆ C n is defined as

Original: [Equation]
Revised: [Equation]

Original: The output of the covariance filter H ( ˆ C n ) for an input x is given by
Revised: The output of the covariance filter H ( ˆ C n ) for an input x is given by

Original: [Equation]
Revised: [Equation]

Original: The coVariance filter H ( ˆ C n ) follows similar analytic concepts of combining information in different neighborhoods as in the well-studied graph convolutional filters [12].
Revised: The coVariance filter H ( ˆ C n ) follows similar analytic concepts of combining information in different neighborhoods as in the well-studied graph convolutional filters [12].

Original: Moreover, the filter is defined by the parameters { h k } mk =0 and therefore, for the ensemble covariance matrix C , the coVariance filter is given by H ( C ) .
Revised: Moreover, the filter is defined by the parameters { h k } mk =0 and therefore, for the ensemble covariance matrix C , the coVariance filter is given by H ( C ) .

Original: On taking the VFT of the output in (6) and leveraging (2), we have
Revised: On taking the VFT of the output in (6) and leveraging (2), we have

Original: [Equation]
Revised: [Equation]

Original: where (7) holds from the orthonormality of eigenvectors and definition of VFT in (4).
Revised: where (7) holds from the orthonormality of eigenvectors and definition of VFT in (4).

Original: Using (7), we can further define the frequency response of the coVariance filter over the covariance matrix ˆ C n in the domain of its principal components as
Revised: Using (7), we can further define the frequency response of the coVariance filter over the covariance matrix ˆ C n in the domain of its principal components as

Original: [Equation]
Revised: [Equation]

Original: such that, from (7) and (8), the i -th element of ˜ z i has the following relationship
Revised: such that, from (7) and (8), the i -th element of ˜ z i has the following relationship

Original: [Equation]
Revised: [Equation]

Original: Equation (9) reveals that performing the coVariance filtering operation boils down to processing (e.g., amplifying or attenuating)
Revised: Equation (9) reveals that performing the coVariance filtering operation boils down to processing (e.g., amplifying or attenuating)

Original: the principal components of the data.
Revised: the principal components of the data.

Original: This observation draws analogy with the linear-time invariant systems in signal processing where the different frequency modes (in this case, principal components) can be processed separately using coVariance filters, in a way determined by the frequency response values h ( w i ) .
Revised: This observation draws analogy with the linear-time invariant systems in signal processing where the different frequency modes (in this case, principal components) can be processed separately using coVariance filters, in a way determined by the frequency response values h ( w i ) .

Original: For instance, using a narrowband coVariance filter whose frequency response is h ( λ ) = 

Revised: For instance, using a narrowband coVariance filter whose frequency response is h ( λ ) =

Original: if λ = w i and h ( λ ) = 0 , otherwise, we recover the score corresponding to the projection of x on u i , i.e, the i -th principal component of ˆ C n .
Revised: if λ = w i and h ( λ ) = 0 , otherwise, we recover the score corresponding to the projection of x on u i , i.e, the i -th principal component of ˆ C n .

Original: Therefore, there exist filterbanks of narrowband coVariance filters that enable the recovery of the PCA transformation.
Revised: Therefore, there exist filterbanks of narrowband coVariance filters that enable the recovery of the PCA transformation.

Original: This observation is formalized in Theorem 1.
Revised: This observation is formalized in Theorem 1.

Original: Theorem 1 (coVariance Filter Implementation of PCA) .
Revised: Theorem 1 (coVariance Filter Implementation of PCA) .

Original: Given a covariance matrix ˆ C n with eigendecomposition in (2) , if the PCA transformation of input x is given by y
Revised: Given a covariance matrix ˆ C n with eigendecomposition in (2) , if the PCA transformation of input x is given by y

Original: = U T x , there exists a filterbank of coVariance filters { H i ( ˆ C n ) : i ∈ { 1 , . . .
Revised: = U T x , there exists a filterbank of coVariance filters { H i ( ˆ C n ) : i ∈ { 1 , . . .

Original: , m }} , such that, the score of the projection of input x on eigenvector u i can be recovered by the application of a coVariance filter H
Revised: , m }} , such that, the score of the projection of input x on eigenvector u i can be recovered by the application of a coVariance filter H

Original: i ( ˆ C n ) as:
Revised: i ( ˆ C n ) as:

Original: [Equation]
Revised: [Equation]

Original: where the frequency response h i ( λ ) of the filter H i ( ˆ C n ) is given by
Revised: where the frequency response h i ( λ ) of the filter H i ( ˆ C n ) is given by

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 shows that processing data samples with PCA and processing data samples with a specific polynomial on the covariance matrix are equivalent.
Revised: Theorem 1 establishes equivalence between processing data samples with PCA and processing data samples with a specific polynomial on the covariance matrix.

Original: As we shall see in subsequent sections, the processing on a polynomial of covariance matrix has advantages in terms of stability with respect to the perturbations in the sample covariance matrix.
Revised: As we shall see in subsequent sections, the processing on a polynomial of covariance matrix has advantages in terms of stability with respect to the perturbations in the sample covariance matrix.

Original: The design of frequency response of different coVariance filters sufficient to implement PCA transformation according to Theorem 1 is shown figuratively in Appendix D. If it is desired to have PCA-based data transformation be followed by dimensionality reduction or statistical learning tasks such as regression or classification, the coVariance filter-based PCA can be coupled with the post-hoc analysis model to have an end-toend framework that enables the optimization of the parameters
Revised: The design of frequency response of different coVariance filters sufficient to implement PCA transformation according to Theorem 1 is shown figuratively in Appendix D. If it is desired to have PCA-based data transformation be followed by dimensionality reduction or statistical learning tasks such as regression or classification, the coVariance filter-based PCA can be coupled with the post-hoc analysis model to have an end-toend framework that enables the optimization of the parameters

Original: η i in the frequency response of the coVariance filters.
Revised: η i in the frequency response of the coVariance filters.

Original:  coVariance Neural Networks (VNN)
Revised:  coVariance Neural Networks (VNN)

Original: In this section, we propose coVariance Neural Network (VNN), which provides an end-to-end, non-linear, parametric mapping from the input data x to any generic objective r and is defined as
Revised: In this section, we propose coVariance Neural Network (VNN), which provides an end-to-end, non-linear, parametric mapping from the input data x to any generic objective r and is defined as

Original: [Equation]
Revised: [Equation]

Original: for sample covariance matrix ˆ C n where H is the set of filter coefficients that characterize the representation space defined by the mapping Φ( · ) .
Revised: for sample covariance matrix ˆ C n where H is the set of filter coefficients that characterize the representation space defined by the mapping Φ( · ) .

Original: The VNN Φ( x ; ˆ C n , H ) may be formed by multiple layers, where each layer consists of two main components: i) a filter bank made of VFs similar to that in (6); and ii) a pointwise non-linear activation function (such as ReLU , tanh ).
Revised: The VNN Φ( x ; ˆ C n , H ) may be formed by multiple layers, where each layer consists of two main components: i) a filter bank made of VFs similar to that in (6); and ii) a pointwise non-linear activation function (such as ReLU , tanh ).

Original: Therefore, in principle, the architecture of VNNs is similar to that of graph neural networks with the covariance matrix ˆ C n as the graph shift operator [12].
Revised: Therefore, in principle, the architecture of VNNs is similar to that of graph neural networks with the covariance matrix ˆ C n as the graph shift operator [12].

Original: We next define the coVariance perceptron, which forms the building block of a VNN and is equivalent to a 1-layer VNN.
Revised: We next define the coVariance perceptron, which forms the building block of a VNN and is equivalent to a 1-layer VNN.

Original: Definition 3 (coVariance Perceptron) .
Revised: Definition 3 (coVariance Perceptron) .

Original: Consider a dataset with the sample covariance matrix ˆ C n .
Revised: Consider a dataset with the sample covariance matrix ˆ C n .

Original: For a given non-linear activation function σ ( · ) , input x , a coVariance filter
Revised: For a given non-linear activation function σ ( · ) , input x , a coVariance filter

Original: [Equation]
Revised: [Equation]

Original: and its corresponding coefficient set H , the coVariance perceptron is defined as
Revised: and its corresponding coefficient set H , the coVariance perceptron is defined as

Original: [Equation]
Revised: [Equation]

Original: The VNN can be constructed by cascading multiple layers of coVariance perceptrons (shown in
Revised: The VNN can be constructed by cascading multiple layers of coVariance perceptrons (shown in

Original: Appendix D in the Supplementary file).
Revised: Appendix D in the Supplementary file).

Original: Note that the non-linear activation functions across different layers allow for non-linear transformations, thus, increasing the expressiveness of VNNs beyond linear mappings such as coVariance filters.
Revised: Note that the non-linear activation functions across different layers allow for non-linear transformations, thus, increasing the expressiveness of VNNs beyond linear mappings such as coVariance filters.

Original: Furthermore, similar to GNNs, we can further increase the representation power of VNNs by incorporating multiple parallel features per layer to form filter banks at every layer [12, 29].
Revised: Furthermore, similar to GNNs, we can further increase the representation power of VNNs by incorporating multiple parallel inputs and outputs per layer enabled by filter banks at every layer [12, 29].

Original: In this context, we remark that the definition of a one layer perceptron in (13) can be expanded to the following.
Revised: In this context, we remark that the definition of a one layer perceptron in (13) can be expanded to the following.

Original: Remark 1 (coVariance Perceptron with Filter Banks) .
Revised: Remark 1 (coVariance Perceptron with Filter Banks) .

Original: Consider a coVariance perceptron with F in features at the input and F out features at the output.
Revised: Consider a coVariance perceptron with F in m -dimensional inputs and F out m -dimensional outputs.

Original: Denote the input at the perceptron by x in = (cid:104) x in [1] , . . .
Revised: Denote the input at the perceptron by x in = (cid:104) x in [1] , . . .

Original: , x in [ F in ] (cid:105) and the output at the perceptron by x out = (cid:104) x out [1] , . . .
Revised: , x in [ F in ] (cid:105) and the output at the perceptron by x out = (cid:104) x out [1] , . . .

Original: , x out [ F out ] (cid:105) .
Revised: , x out [ F out ] (cid:105) .

Original: Each inputfeature x in [ g ] , ∀ g ∈ { 1 , . .
Revised: Each input x in [ g ] , ∀ g ∈ { 1 , . .

Original: , F in } is processed by F out coVariance filters in parallel.
Revised: , F in } is processed by F out coVariance filters in parallel.

Original: For f ∈ { 1 , . . .
Revised: For f ∈ { 1 , . . .

Original: , F in } , the f -th output in x outcorresponding to x in is given by
Revised: , F out } , the f -th output in x out is given by

Original: [Equation]
Revised: [Equation]

Original: where H f is the set of all filter coefficients in coVariance filters [ H fg ( ˆ C n )] g in (14) .
Revised: where H f is the set of all filter coefficients in coVariance filters [ H fg ( ˆ C n )] g in (14) .

Original: Therefore, the VNN with filter bank implementation deploys F in × F out number of VFs in a layer defined by the covariance perceptron in Remark 1.
Revised: Therefore, the VNN with filter bank implementation deploys F in × F out number of VFs in a layer defined by the covariance perceptron in Remark 1.

Original: Next, we note that the definitions of coVariance filter in Section 2 and VNN are with respect to the sample covariance matrix ˆ C n .
Revised: Next, we note that the definitions of coVariance filter in Section 2 and VNN are with respect to the sample covariance matrix ˆ C n .

Original: However, due to finite sample size effects, the sample covariance matrix will be a perturbed version of the ensemble covariance matrix C .
Revised: However, due to finite sample size effects, the sample covariance matrix will be a perturbed version of the ensemble covariance matrix C .

Original: PCA-based approaches that rely on eigendecomposition of the sample covariance matrix can potentially be highly sensitive to such perturbations.
Revised: PCA-based approaches that rely on eigendecomposition of the sample covariance matrix can potentially be highly sensitive to such perturbations.

Original: Specifically, if small changes are made to the dataset x n , certain ill-defined eigenvalues and eigenvectors can induce instability in the outcomes of PCA-based statistical learning models [17].
Revised: Specifically, if small changes are made to the dataset x n , certain ill-defined eigenvalues and eigenvectors can induce instability in the outcomes of PCA-based statistical learning models [17].

Original: Therefore, it is desirable that the designs of coVariance filters and VNNs are not sensitive to random perturbations in the sample covariance matrix as compared to the ensemble covariance matrix C .
Revised: Therefore, it is desirable that the designs of coVariance filters and VNNs are not sensitive to random perturbations in the sample covariance matrix as compared to the ensemble covariance matrix C .

Original: In this context, we study the stability of coVariance filters and VNNs in the next section.
Revised: In this context, we study the stability of coVariance filters and VNNs in the next section.

Original:  Stability Analysis
Revised:  Stability Analysis

Original: We start with the stability analysis of coVariance filters, which will also be instrumental in establishing the stability of VNNs.
Revised: We start with the stability analysis of coVariance filters, which will also be instrumental in establishing the stability of VNNs.

Original: Our results will leverage the eigendecomposition of C , given by
Revised: Our results will leverage the eigendecomposition of C , given by

Original: [Equation]
Revised: [Equation]

Original:  where V ∈ R m × m is the matrix of eigenvectors such that, V =
Revised: 15) where V ∈ R m × m is the matrix of eigenvectors such that, V =

Original: [ v 1 , . .
Revised: [ v 1 , . .

Original: . , v m ] and Λ = diag ( λ 1 , . . .
Revised: . , v m ] and Λ = diag ( λ 1 , . . .

Original: , λ m ) is the diagonal matrix of eigenvalues, such that, λ 1 ≥ · · · ≥ λ m .
Revised: , λ m ) is the diagonal matrix of eigenvalues, such that, λ 1 ≥ · · · ≥ λ m .

Original: 4.1 Stability of coVariance Filters
Revised: 4.1 Stability of coVariance Filters

Original: To study the stability of coVariance filters, we analyze the effect of statistical uncertainty induced in the sample covariance matrix with respect to the ensemble covariance matrix on the coVariance filter output.
Revised: To study the stability of coVariance filters, we analyze the effect of statistical uncertainty induced in the sample covariance matrix with respect to the ensemble covariance matrix on the coVariance filter output.

Original: To this end, we compare the outputs H ( C ) x and H ( ˆ C n ) x for any random instance x of X .
Revised: To this end, we compare the outputs H ( C ) x and H ( ˆ C n ) x for any random instance x of X .

Original: Without loss of generality, we present our results over instances of X where ∥ X ∥ ≤ 1 .
Revised: Without loss of generality, we present our results over instances of X where ∥ X ∥ ≤ 1 .

Original: We also consider the following assumption on the frequency response.
Revised: We also consider the following assumption on the frequency response.

Original: Assumption : The frequency response of filter H ( C ) satisfies:
Revised: Assumption : The frequency response of filter H ( C ) satisfies:

Original: [Equation]
Revised: [Equation]

Original: where k
Revised: where k

Original: i △ = (cid:112) E [ ∥ XX T v i ∥ 2 ] − λ 2 i , for some constant M > 0 and for all non-zero eigenvalues
Revised: i △ = (cid:112) E [ ∥ XX T v i ∥ 2 ] − λ 2 i , for some constant M > 0 and for all non-zero eigenvalues

Original: λ i ̸
Revised: λ i ̸

Original: = λ j , i, j ∈ { 1 , . .
Revised: = λ j , i, j ∈ { 1 , . .

Original: , m } of C .
Revised: , m } of C .

Original: Here, k i is a measure of kurtosis of the distribution of X .
Revised: Here, k i is a measure of kurtosis of the distribution of X .

Original: Next, we provide the result that establishes the stability of coVariance filters in Theorem 2.
Revised: Next, we provide the result that establishes the stability of coVariance filters in Theorem 2.

Original: The impact of the assumption in (16) on the filter stability is discussed subsequently.
Revised: The impact of the assumption in (16) on the filter stability is discussed subsequently.

Original: To present the results in Theorem 2, we also use the following definitions:
Revised: To present the results in Theorem 2, we also use the following definitions:

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 (Stability of coVariance Filter) .
Revised: Theorem 2 (Stability of coVariance Filter) .

Original: Consider a random vector X ∈ R m × 1 , such that, its corresponding covariance matrix is given by C = E
Revised: Consider a random vector X ∈ R m × 1 , such that, its corresponding covariance matrix is given by C = E

Original: [( X − E [ X ])( X − E
Revised: [( X − E [ X ])( X − E

Original: [ X ]) T ] .
Revised: [ X ]) T ] .

Original: For a sample covariance matrix ˆ C n formed using n i.i.d instances of X and a random instance x of X , such that, ∥ x ∥ ≤ 1 and under assumption (16) , the following holds with probability at least 1 − n − 2 ε − 2 κm/n for any ε ∈ (0 , 1 / 2] :
Revised: For a sample covariance matrix ˆ C n formed using n i.i.d instances of X and a random instance x of X , such that, ∥ x ∥ ≤ 1 and under assumption (16) , the following holds with probability at least 1 − n − 2 ε − 2 κm/n for any ε ∈ (0 , 1 / 2] :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix C.
Revised: See Appendix C.

Original: The right-hand side term in (18) and the conditions in the assumption in (16) are obtained by the analysis of the finite sample size effect-driven perturbations in ˆ C n and its eigenvectors with respect to that in C .
Revised: The right-hand side term in (18) and the conditions in the assumption in (16) are obtained by the analysis of the finite sample size effect-driven perturbations in ˆ C n and its eigenvectors with respect to that in C .

Original: From Theorem 2, we note that ∥ H ( ˆ C n ) − H ( C ) ∥ decays with the number of samples n at least at the rate of 1 /n 12 − ε .
Revised: From Theorem 2, we note that ∥ H ( ˆ C n ) − H ( C ) ∥ decays with the number of samples n at least at the rate of 1 /n 12 − ε .

Original: Thus, we conclude that the stability of the coVariance filter improves as the number of samples n increases.
Revised: Thus, we conclude that the stability of the coVariance filter improves as the number of samples n increases.

Original: This observation is along the expected lines as the estimate ˆ C n becomes closer to the ensemble covariance matrix C by the virtue of the law of large numbers.
Revised: This observation is along the expected lines as the estimate ˆ C n becomes closer to the ensemble covariance matrix C by the virtue of the law of large numbers.

Original: Next, we briefly discuss two aspects of the assumption in (16).
Revised: Next, we briefly discuss two aspects of the assumption in (16).

Original: Note that the upper bound in (16) controls the variability of the frequency
Revised: Note that the upper bound in (16) controls the variability of the frequency

Original: response h ( λ ) with respect to λ .
Revised: response h ( λ ) with respect to λ .

Original: For any pair of eigenvalues λ i and λ j of C , this variability is tied to the eigengap | λ i − λ j | and the factor k i .
Revised: For any pair of eigenvalues λ i and λ j of C , this variability is tied to the eigengap | λ i − λ j | and the factor k i .

Original: We discuss this next.
Revised: We discuss this next.

Original: Discriminability between close eigenvalues : Firstly, for a given k i and eigenvalue λ i , the response of the filter for any eigenvalue λ j , j ̸
Revised: Discriminability between close eigenvalues : Firstly, for a given k i and eigenvalue λ i , the response of the filter for any eigenvalue λ j , j ̸

Original: = i becomes closer to h ( λ i ) if | λ i − λ
Revised: = i becomes closer to h ( λ i ) if | λ i − λ

Original: j | decreases.
Revised: j | decreases.

Original: From the perturbation theory of eigenvectors and eigenvalues of sample covariance matrices, we know that the sample-based estimates of the eigenspaces corresponding to eigenvalues λ
Revised: From the perturbation theory of eigenvectors and eigenvalues of sample covariance matrices, we know that the sample-based estimates of the eigenspaces corresponding to eigenvalues λ

Original: i and λ j become harder to distinguish as | λ i − λ
Revised: i and λ j become harder to distinguish as | λ i − λ

Original: j | decreases [30].
Revised: j | decreases [30].

Original: Therefore, the coVariance filter that satisfies (16) sacrifices discriminability between close eigenvalues to preserve its stability with respect to the statistical uncertainty inherent in the sample covariance matrix.
Revised: Therefore, the coVariance filter that satisfies (16) sacrifices discriminability between close eigenvalues to preserve its stability with respect to the statistical uncertainty inherent in the sample covariance matrix.

Original: Stability
Revised: Stability

Original: with respect to kurtosis and
Revised: with respect to kurtosis and

Original: estimation quality
Revised: estimation quality

Original: Since we have
Revised: Since we have

Original: E [ ∥ XX T v i ∥ 2 ] ≤ E [ ∥ X ∥ 4 ] , the factor k i is tied to the measure of kurtosis of the underlying distribution of X in the direction of v i .
Revised: E [ ∥ XX T v i ∥ 2 ] ≤ E [ ∥ X ∥ 4 ] , the factor k i is tied to the measure of kurtosis of the underlying distribution of X in the direction of v i .

Original: Distributions with high kurtosis tend to have heavier tails and more outliers.
Revised: Distributions with high kurtosis tend to have heavier tails and more outliers.

Original: Therefore, smaller k i indicates that the distribution of X has a fast decaying tail in the direction of v i which allows for a more accurate estimation of λ i and v i .
Revised: Therefore, smaller k i indicates that the distribution of X has a fast decaying tail in the direction of v i which allows for a more accurate estimation of λ i and v i .

Original: We refer the reader to [30,
Revised: We refer the reader to [30,

Original: Section 4.1.3] for additional details.
Revised: Section 4.1.3] for additional details.

Original: In the context of coVariance filters, we note that the upper bound in (16) is more liberal if λ i is associated with a smaller k i or equivalently, the distribution of X has a smaller kurtosis in the direction of v i .
Revised: In the context of coVariance filters, we note that the upper bound in (16) is more liberal if λ i is associated with a smaller k i or equivalently, the distribution of X has a smaller kurtosis in the direction of v i .

Original: This observation implies that if λ i and v i are ‘easier’ to estimate, the frequency response for λ j in the vicinity of λ i is less constrained.
Revised: This observation implies that if λ i and v i are ‘easier’ to estimate, the frequency response for λ j in the vicinity of λ i is less constrained.

Original: 4.2 Stability of coVariance Neural Networks
Revised: 4.2 Stability of coVariance Neural Networks

Original: The stability of VNNs is analyzed by comparing Φ( x ; ˆ C n , H ) and Φ( x ; C , H ) .
Revised: The stability of VNNs is analyzed by comparing Φ( x ; ˆ C n , H ) and Φ( x ; C , H ) .

Original: Note that stable graph convolutional filters imply the stability of GNNs for different perturbation models [19].
Revised: Note that stable graph convolutional filters imply the stability of GNNs for different perturbation models [19].

Original: In
Revised: In

Original: VNNs, the perturbations are derived from the finite sample effect in sample covariance matrix, which is distinct from the data-independent perturbation models considered in the existing literature on GNNs, and allow us to relate sample size n with VNN stability.
Revised: VNNs, the perturbations are derived from the finite sample effect in sample covariance matrix, which is distinct from the data-independent perturbation models considered in the existing literature on GNNs, and allow us to relate sample size n with VNN stability.

Original: We formalize the stability of VNNs under the assumption of stable coVariance filters in Theorem 3.
Revised: We formalize the stability of VNNs under the assumption of stable coVariance filters in Theorem 3.

Original: For this purpose, we consider a VNN Φ( · ; · , H ) with F features per layer  and L layers, with the filter bank given by 
Revised: For this purpose, we consider a VNN Φ( · ; · , H ) with number of m -dimensional inputs and outputs per layer as F and L layers, with the filter bank given by H = { H ℓfg } , ∀ f, g ∈ { 1 , . . . , F } , ℓ ∈ { 1 , . . .

Original: [Equation]
Revised: , L } .

Original: Theorem 3 (Stability of VNN) .
Revised: Theorem 3 (Stability of VNN) .

Original: Consider a sample covariance matrix ˆ C n and the ensemble covariance matrix C .
Revised: Consider a sample covariance matrix ˆ C n and the ensemble covariance matrix C .

Original: Given a bank of coVariance filters { H ℓfg } , such that | h ℓfg ( λ ) | ≤ 1 and a pointwise non-linearity σ ( · ) , such that, | σ ( a ) − σ ( b ) | ≤ | a − b | , if the covariance filters satisfy
Revised: Given a bank of coVariance filters { H ℓfg } , such that | h ℓfg ( λ ) | ≤ 1 and a pointwise non-linearity σ ( · ) , such that, | σ ( a ) − σ ( b ) | ≤ | a − b | , if the covariance filters satisfy

Original: [Equation]
Revised: [Equation]

Original: 19) for some α n > 0 , then, we have
Revised: 19) for some α n > 0 , then, we have

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 3 follows from [12, Appendix E] for any generic α n .
Revised: The proof of Theorem 3 follows from [12, Appendix E] for any generic α n .

Original: The parameter α n represents the finite sample effect on the perturbations in ˆ C n with respect to C .
Revised: The parameter α n represents the finite sample effect on the perturbations in ˆ C n with respect to C .

Original: From Theorem 2, we note that α n scales as 1 /n 12 − ε with respect to the number of samples n for the coVariance filters whose frequency response depends on the eigengap and kurtosis of the underlying distribution of the data in (16).
Revised: From Theorem 2, we note that α n scales as 1 /n 12 − ε with respect to the number of samples n for the coVariance filters whose frequency response depends on the eigengap and kurtosis of the underlying distribution of the data in (16).

Original: Furthermore, the stability of a VNN decreases with increase in number of features F and number of layers L , which is consistent with the stability properties of GNNs.
Revised: Furthermore, the stability of a VNN decreases with increase in number of m -dimensional inputs and outputs per layer F and number of layers L , which is consistent with the stability properties of GNNs.

Original: Therefore, our results present a more holistic perspective to the stability of VNNs than that possible for generic GNNs.
Revised: Therefore, our results present a more holistic perspective to the stability of VNNs than that possible for generic GNNs.

Original: Remark 2 (Computational Complexity of VNN) .
Revised: Remark 2 (Computational Complexity of VNN) .

Original: For a coVariance perceptron defined in (14), the computational cost is given by O ( m 2 TF in F out ) , where T ≤ m is the maximum number of filter taps in its associated filter bank.
Revised: For a coVariance perceptron defined in (14), the computational cost is given by O ( m 2 TF in F out ) , where T ≤ m is the maximum number of filter taps in its associated filter bank.

Original: From Remark 2, we note that the computational complexity of VNNs can be prohibitive for large m .
Revised: From Remark 2, we note that the computational complexity of VNNs can be prohibitive for large m .

Original: However, oftentimes sparsity is imposed as a regularization to estimate high-dimensional correlation matrices; see e.g. [31].
Revised: However, oftentimes sparsity is imposed as a regularization to estimate high-dimensional correlation matrices; see e.g. [31].

Original: As a result, the computational complexity becomes O ( | E | TF in F out ) , where | E | is the number of non-zero correlations (edges in the covariance graph) and can be markedly smaller than m 2 .
Revised: As a result, the computational complexity becomes O ( | E | TF in F out ) , where | E | is the number of non-zero correlations (edges in the covariance graph) and can be markedly smaller than m 2 .

Original: Since VNN architecture is analogous to that of GNN, the property of GNN transferability (see [6]) across different sized graphs can also establish scalability of VNNs to high-dimensional datasets for settings where multi-resolution datasets are available.
Revised: Since VNN architecture is analogous to that of GNN, the property of GNN transferability (see [6]) across different sized graphs can also establish scalability of VNNs to high-dimensional datasets for settings where multi-resolution datasets are available.

Original: In the next section, we empirically validate our theoretical results on the stability of VNNs.
Revised: In the next section, we empirically validate our theoretical results on the stability of VNNs.

Original: Moreover, on a set of multi-resolution datasets, we also empirically evaluate VNNs for transferability.
Revised: Moreover, on a set of multi-resolution datasets, we also empirically evaluate VNNs for transferability.

Original:  Experiments
Revised:  Experiments

Original: In this section, we discuss our experiments on different datasets.
Revised: In this section, we discuss our experiments on different datasets.

Original: Primarily, we evaluate VNNs on a regression problem on different neuroimaging datasets curated at University of Pennsylvania, where we regress human chronological age (time since birth) against cortical thickness data.
Revised: Primarily, we evaluate VNNs on a regression problem on different neuroimaging datasets curated at University of Pennsylvania, where we regress human chronological age (time since birth) against cortical thickness data.

Original: Cortical thickness is a measure of the gray matter width
Revised: Cortical thickness is a measure of the gray matter width

Original: and it is well established that cortical thickness varies with healthy aging [32].
Revised: and it is well established that cortical thickness varies with healthy aging [32].

Original: Additional details on the neurological utility of this experiment are included in Appendix E. Brief descriptions of these datasets are provided next.
Revised: Additional details on the neurological utility of this experiment are included in Appendix E. Brief descriptions of these datasets are provided next.

Original: ABC Dataset : ABC dataset consists of the cortical thickness data collected from a heterogeneous population of n = 341 subjects (mean age = 68 .
Revised: ABC Dataset : ABC dataset consists of the cortical thickness data collected from a heterogeneous population of n = 341 subjects (mean age = 68 .

Original: 1 years, standard deviation = 13 ) that consists of healthy adults, and subjects with mild cognitive impairment or Alzheimer’s disease.
Revised: 1 years, standard deviation = 13 ) that consists of healthy adults, and subjects with mild cognitive impairment or Alzheimer’s disease.

Original: For each individual, joint-label fusion [33] was used to quantify mean cortical thickness in m = 104 anatomical parcellations.
Revised: For each individual, joint-label fusion [33] was used to quantify mean cortical thickness in m = 104 anatomical parcellations.

Original: Therefore, for every subject, we have a 104 dimensional vector whose entries correspond to the cortical thickness in distinct brain regions.
Revised: Therefore, for every subject, we have a 104 dimensional vector whose entries correspond to the cortical thickness in distinct brain regions.

Original: Multi-resolution FTDC Datasets : FTDC Datasets consist of the cortical thickness data from n = 170 healthy subjects (mean age = 64 .
Revised: Multi-resolution FTDC Datasets : FTDC Datasets consist of the cortical thickness data from n = 170 healthy subjects (mean age = 64 .

Original: 26 years, standard deviation = 8 . 26 ).
Revised: 26 years, standard deviation = 8 . 26 ).

Original: For each subject, the cortical thickness data is extracted according to a multiresolution Schaefer parcellation [34], at 100 parcel, 300 parcel, and 500 parcel resolutions.
Revised: For each subject, the cortical thickness data is extracted according to a multiresolution Schaefer parcellation [34], at 100 parcel, 300 parcel, and 500 parcel resolutions.

Original: Therefore, for each subject, we have the cortical thickness data consisting of m = 100 features, m = 300 features or m = 500 features, with the higher number of features providing the cortical thickness data of a brain at a finer resolution.
Revised: Therefore, for each subject, we have the cortical thickness data consisting of m = 100 features, m = 300 features or m = 500 features, with the higher number of features providing the cortical thickness data of a brain at a finer resolution.

Original: We leverage the different resolutions of data available to form three datasets: FTDC100, FTDC300, and
Revised: We leverage the different resolutions of data available to form three datasets: FTDC100, FTDC300, and

Original: FTDC500, which form the cortical thickness datasets corresponding to 100, 300, and 500 features resolutions, respectively.
Revised: FTDC500, which form the cortical thickness datasets corresponding to 100, 300, and 500 features resolutions, respectively.

Original: Our primary objective is to illustrate the stability and transferability of VNNs that also imply advantages over traditional PCA-based approaches.
Revised: Our primary objective is to illustrate the stability and transferability of VNNs that also imply advantages over traditional PCA-based approaches.

Original: Hence, we use PCA-based regression as the primary baseline for comparison against VNNs.
Revised: Hence, we use PCA-based regression as the primary baseline for comparison against VNNs.

Original: We use the ABC dataset to demonstrate the higher stability of VNNs over PCA-based regression models in Section 5.1.
Revised: We use the ABC dataset to demonstrate the higher stability of VNNs over PCA-based regression models in Section 5.1.

Original: In Section 5.2, we use the FTDC datasets to demonstrate transferability of VNN performance across the multi-resolution datasets without re-training.
Revised: In Section 5.2, we use the FTDC datasets to demonstrate transferability of VNN performance across the multi-resolution datasets without re-training.

Original: The experiments in Section 5.2 clearly lay beyond the scope of PCA-based statistical learning models.
Revised: The experiments in Section 5.2 clearly lay beyond the scope of PCA-based statistical learning models.

Original: Details on data and code availability for the experiments are included in
Revised: Details on data and code availability for the experiments are included in

Original: Appendix A.
Revised: Appendix A.

Original: 5.1 Stability against Perturbations in Sample Covariance Matrix
Revised: 5.1 Stability against Perturbations in Sample Covariance Matrix

Original: In this section, we evaluate the stability or robustness of the trained VNN and PCA-regression models against perturbations in the sample covariance matrix used in training.
Revised: In this section, we evaluate the stability or robustness of the trained VNN and PCA-regression models against perturbations in the sample covariance matrix used in training.

Original: To this end, we first train nominal VNN and PCA-regression models.
Revised: To this end, we first train nominal VNN and PCA-regression models.

Original: The effects of perturbations in the sample covariance matrix on the performance of nominal models are evaluated subsequently.
Revised: The effects of perturbations in the sample covariance matrix on the performance of nominal models are evaluated subsequently.

Original: We first describe the experiment designs for nominal models based on VNN and PCA-regression.
Revised: We first describe the experiment designs for nominal models based on VNN and PCA-regression.

Original: VNN Experiments : We randomly split ABC dataset into a 90 / 10 train/test split.
Revised: VNN Experiments : We randomly split ABC dataset into a 90 / 10 train/test split.

Original: The sample covariance matrix is formed from 307 samples in the training set, i.e., we have ˆ C 307 of size 104 × 104 .
Revised: The sample covariance matrix is formed from 307 samples in the training set, i.e., we have ˆ C 307 of size 104 × 104 .

Original: The VNN consists of 2 layers with 2 filter taps each, a filter bank of 13 features per dimension in each layer for m = 104 dimensions of the input data, and a readout layer that calculates the unweighted mean of the outputs at the last layer to form an estimate for age.
Revised: The VNN consists of 2 layers with 2 filter taps each, a filter bank of 13 m -dimensional outputs per dimension in each layer for m = 104 dimensions of the input data, and a readout layer that calculates the unweighted mean of the outputs at the last layer to form an estimate for age.

Original: The hyperparameters for the VNN architecture and learning rate of the optimizer in this experiments and all subsequent
Revised: The hyperparameters for the VNN architecture and learning rate of the optimizer in this experiments and all subsequent

Original: VNN experiments in this section are chosen by a hyperparameter optimization framework called
Revised: VNN experiments in this section are chosen by a hyperparameter optimization framework called

Original: Optuna [35].
Revised: Optuna [35].

Original: The training set is randomly subdivided into subsets of 273 and 34 samples, such that, the VNN is trained with respect to the mean squared error loss between the predicted age and the true age in 273 samples.
Revised: The training set is randomly subdivided into subsets of 273 and 34 samples, such that, the VNN is trained with respect to the mean squared error loss between the predicted age and the true age in 273 samples.

Original: The loss is optimized using batch stochastic gradient descent with Adam optimizer available in PyTorch library [36] for up to 100 epochs.
Revised: The loss is optimized using batch stochastic gradient descent with Adam optimizer available in PyTorch library [36] for up to 100 epochs.

Original: The learning rate for the optimizer is set to 0 .
Revised: The learning rate for the optimizer is set to 0 .

Original: The VNN model with the best minimum mean squared error performance on the remaining 34 samples (which acts as a validation set) is included in the set of nominal models for this permutation of the training set.
Revised: The VNN model with the best minimum mean squared error performance on the remaining 34 samples (which acts as a validation set) is included in the set of nominal models for this permutation of the training set.

Original: PCA-regression : The PCA-regression pipeline consists of two steps: i) we first identify the principle components using the eigendecomposition of the sample covariance matrix ˆ C 307 ; and then, ii) to maintain consistency with VNN, transform the 273 samples from the training set used for VNN training to fit to the corresponding age data using a regression model.
Revised: PCA-regression : The PCA-regression pipeline consists of two steps: i) we first identify the principle components using the eigendecomposition of the sample covariance matrix ˆ C 307 ; and then, ii) to maintain consistency with VNN, transform the 273 samples from the training set used for VNN training to fit to the corresponding age data using a regression model.

Original: Regression is implemented using sklearn package in python [37] with ‘linear’ and radial basis function (‘rbf’) kernels.
Revised: Regression is implemented using sklearn package in python [37] with ‘linear’ and radial basis function (‘rbf’) kernels.

Original: PCA-regression with ‘rbf’ kernel enables us to accommodate non-linear relationships between cortical thickness and age.
Revised: PCA-regression with ‘rbf’ kernel enables us to accommodate non-linear relationships between cortical thickness and age.

Original: PCA-regression with linear kernel in the regression model is referred to as PCA-LR and that with ‘rbf’ kernel in the regression model is referred to as PCA-rbf .
Revised: PCA-regression with linear kernel in the regression model is referred to as PCA-LR and that with ‘rbf’ kernel in the regression model is referred to as PCA-rbf .

Original: The optimal number of principal components in the PCA-regression pipeline are selected through a 10-fold cross-validation procedure on the training set, repeated 5 times.
Revised: The optimal number of principal components in the PCA-regression pipeline are selected through a 10-fold cross-validation procedure on the training set, repeated 5 times.

Original: For 100 random permutations of the training set in VNN and PCA-regression experiments, we form a set of 100 nominal models and evaluate their stability.
Revised: For 100 random permutations of the training set in VNN and PCA-regression experiments, we form a set of 100 nominal models and evaluate their stability.

Original: To evaluate the stability of VNN, we replace the sample covariance matrix ˆ C 307 with ˆ C n ′ for n ′ ∈ [5 , 341] , n ′ ̸ = 307 .
Revised: To evaluate the stability of VNN, we replace the sample covariance matrix ˆ C 307 with ˆ C n ′ for n ′ ∈ [5 , 341] , n ′ ̸ = 307 .

Original: For PCA-regression models, we re-evaluate the principal components corresponding to ˆ C n ′ to transform the training data while keeping the regression model learnt for PCA transformation from ˆ C 307 fixed.
Revised: For PCA-regression models, we re-evaluate the principal components corresponding to ˆ C n ′ to transform the training data while keeping the regression model learnt for PCA transformation from ˆ C 307 fixed.

Original: Clearly, ˆ C n ′ will be perturbed with respect to ˆ C 307 due to finite sample size effect.
Revised: Clearly, ˆ C n ′ will be perturbed with respect to ˆ C 307 due to finite sample size effect.

Original: For each nominal model, we evaluate the model performances in terms of mean absolute error (MAE) and correlation between predicted age and true age for the training set and the test set.
Revised: For each nominal model, we evaluate the model performances in terms of mean absolute error (MAE) and correlation between predicted age and true age for the training set and the test set.

Original: Figures 1 a)-b) illustrate the variation in MAE performance with perturbations to ˆ C 307 on the training set.
Revised: Figures 1 a)-b) illustrate the variation in MAE performance with perturbations to ˆ C 307 on the training set.

Original: Figures 1 c)-f) correspond to similar results on the test set for MAE and correlation.
Revised: Figures 1 c)-f) correspond to similar results on the test set for MAE and correlation.

Original: The mean performance in terms of MAE over 100 nominal models is marked by a blue arrow in Fig.
Revised: The mean performance in terms of MAE over 100 nominal models is marked by a blue arrow in Fig.

Original: 1 a) for training set and in Fig.
Revised: 1 a) for training set and in Fig.

Original: 1 c) for the test set.
Revised: 1 c) for the test set.

Original: For PCA-regression models, we observe that both training and test performance in terms of MAE degrades significantly when the sample covariance matrix is perturbed from ˆ C 307 by removing or adding even a small number of samples (also seen in Fig.
Revised: For PCA-regression models, we observe that both training and test performance in terms of MAE degrades significantly when the sample covariance matrix is perturbed from ˆ C 307 by removing or adding even a small number of samples (also seen in Fig.

Original: 1 b) and d), which are the plots focused only on the range of 290 − 341 samples from Fig.
Revised: 1 b) and d), which are the plots focused only on the range of 290 − 341 samples from Fig.

Original: 1 a) and c), respectively and include error bars.).
Revised: 1 a) and c), respectively and include error bars.).

Original: In contrast, the VNN performances on both training and test sets are stable with respect to perturbations to ˆ C 307 , as suggested by our theoretical results.
Revised: In contrast, the VNN performances on both training and test sets are stable with respect to perturbations to ˆ C 307 , as suggested by our theoretical results.

Original: However, as the number of samples decrease to n ′ < 50 , we observe a significant decrease in VNN stability in Fig.
Revised: However, as the number of samples decrease to n ′ < 50 , we observe a significant decrease in VNN stability in Fig.

Original: 1 a) and c).
Revised: 1 a) and c).

Original: We also observe that the correlation between the predicted age and true age in the test set for VNN is consistently more stable than that for PCA-regression models over the entire range of samples evaluated (Fig. 1 e)).
Revised: We also observe that the correlation between the predicted age and true age in the test set for VNN is consistently more stable than that for PCA-regression models over the entire range of samples evaluated (Fig. 1 e)).

Original: Moreover, Fig. 1 e) also demonstrates that in the range of n ′ <
Revised: Moreover, Fig. 1 e) also demonstrates that in the range of n ′ <

Original: 50 samples, there is a sharper decline in the correlation for PCA-regression models as compared to VNN despite the MAE for PCA-regression models having smaller MAE than VNN in this range.
Revised: 50 samples, there is a sharper decline in the correlation for PCA-regression models as compared to VNN despite the MAE for PCA-regression models having smaller MAE than VNN in this range.

Original: Thus, our experiments demonstrate the stability of VNNs while also illustrating that PCA-regression models may be overfit on the principal components of the sample covariance matrix used in training.
Revised: Thus, our experiments demonstrate the stability of VNNs while also illustrating that PCA-regression models may be overfit on the principal components of the sample covariance matrix used in training.

Original: Additional experiments on synthetic data and FTDC datasets also illustrate the stability of VNNs (see Appendix E).
Revised: Additional experiments on synthetic data and FTDC datasets also illustrate the stability of VNNs (see Appendix E).

Original: 5.2 Transferability
Revised: 5.2 Transferability

Original: We evaluate the transferability for VNN models trained on FTDC datasets across different resolutions.
Revised: We evaluate the transferability for VNN models trained on FTDC datasets across different resolutions.

Original: For this purpose, the VNN is trained at a specific resolution and its sample covariance matrix is replaced by the sample covariance matrix at a different resolution for testing.
Revised: For this purpose, the VNN is trained at a specific resolution and its sample covariance matrix is replaced by the sample covariance matrix at a different resolution for testing.

Original: The training of VNNs
Revised: The training of VNNs

Original: - <latexit sha1_base64="PEH4hq9Byr31bEAwTWUtGG0J6es=">AAAB9HicbVBNSwMxEJ31s9avqkcvwSJ4Ktki6rFYEY8V+gXtUrJptg3NZtckWyhLf4cXD4p49cd489+YtnvQ1gcDj/dmmJnnx4Jrg/G3s7a+sbm1ndvJ7+7tHxwWjo6bOkoUZQ0aiUi1faKZ4JI1DDeCtWPFSOgL1vJH1ZnfGjOleSTrZhIzLyQDyQNOibGSl3Z1gO7rd1UX42mvUMQlPAdaJW5GipCh1it8dfsRTUImDRVE646LY+OlRBlOBZvmu4lmMaEjMmAdSyUJmfbS+dFTdG6VPgoiZUsaNFd/T6Qk1HoS+rYzJGaol72Z+J/XSUxw46Vcxolhki4WBYlAJkKzBFCfK0aNmFhCqOL2VkSHRBFqbE55G4K7/PIqaZZL7lXp8rFcrNxmceTgFM7gAly4hgo8QA0aQOEJnuEV3pyx8+K8Ox+L1jUnmzmBP3A+fwAXAJEC</latexit> FTDC <latexit sha1_base64="PEH4hq9Byr31bEAwTWUtGG0J6es=">AAAB9HicbVBNSwMxEJ31s9avqkcvwSJ4Ktki6rFYEY8V+gXtUrJptg3NZtckWyhLf4cXD4p49cd489+YtnvQ1gcDj/dmmJnnx4Jrg/G3s7a+sbm1ndvJ7+7tHxwWjo6bOkoUZQ0aiUi1faKZ4JI1DDeCtWPFSOgL1vJH1ZnfGjOleSTrZhIzLyQDyQNOibGSl3Z1gO7rd1UX42mvUMQlPAdaJW5GipCh1it8dfsRTUImDRVE646LY+OlRBlOBZvmu4lmMaEjMmAdSyUJmfbS+dFTdG6VPgoiZUsaNFd/T6Qk1HoS+rYzJGaol72Z+J/
Revised: - <latexit sha1_base64="PEH4hq9Byr31bEAwTWUtGG0J6es=">AAAB9HicbVBNSwMxEJ31s9avqkcvwSJ4Ktki6rFYEY8V+gXtUrJptg3NZtckWyhLf4cXD4p49cd489+YtnvQ1gcDj/dmmJnnx4Jrg/G3s7a+sbm1ndvJ7+7tHxwWjo6bOkoUZQ0aiUi1faKZ4JI1DDeCtWPFSOgL1vJH1ZnfGjOleSTrZhIzLyQDyQNOibGSl3Z1gO7rd1UX42mvUMQlPAdaJW5GipCh1it8dfsRTUImDRVE646LY+OlRBlOBZvmu4lmMaEjMmAdSyUJmfbS+dFTdG6VPgoiZUsaNFd/T6Qk1HoS+rYzJGaol72Z+J/XSUxw46Vcxolhki4WBYlAJkKzBFCfK0aNmFhCqOL2VkSHRBFqbE55G4K7/PIqaZZL7lXp8rFcrNxmceTgFM7gAly4hgo8QA0aQOEJnuEV3pyx8+K8Ox+L1jUnmzmBP3A+fwAXAJEC</latexit> FTDC <latexit sha1_base64="PEH4hq9Byr31bEAwTWUtGG0J6es=">AAAB9HicbVBNSwMxEJ31s9avqkcvwSJ4Ktki6rFYEY8V+gXtUrJptg3NZtckWyhLf4cXD4p49cd489+YtnvQ1gcDj/dmmJnnx4Jrg/G3s7a+sbm1ndvJ7+7tHxwWjo6bOkoUZQ0aiUi1faKZ4JI1DDeCtWPFSOgL1vJH1ZnfGjOleSTrZhIzLyQDyQNOibGSl3Z1gO7rd1UX42mvUMQlPAdaJW5GipCh1it8dfsRTUImDRVE646LY+OlRBlOBZvmu4lmMaEjMmAdSyUJmfbS+dFTdG6VPgoiZUsaNFd/T6Qk1HoS+rYzJGaol72Z+J/

Original: XSUxw46Vcxolhki4WBYlAJkKzBFCfK0aNmFhCqOL2VkSHRBFqbE55G4K
Revised: XSUxw46Vcxolhki4WBYlAJkKzBFCfK0aNmFhCqOL2VkSHRBFqbE55G4K

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: in this context follows similar procedure as described in Section 5.1 and results in 100 random VNN models for different permutations of the training set.
Revised: in this context follows similar procedure as described in Section 5.1 and results in 100 random VNN models for different permutations of the training set.

Original: For FTDC500, the VNN model consists of 1 -layer with a filter bank of 27 features per dimensions for m = 500 dimensions of the input data, and 2 filter taps in each layer.
Revised: For FTDC500, the VNN model consists of 1 -layer with a filter bank of 27 m -dimensional outputs per dimension for m = 500 dimensions of the input data, and 2 filter taps in each layer.

Original: The learning rate for the Adam optimizer is set to 0 .
Revised: The learning rate for the Adam optimizer is set to 0 .

Original: For FTDC300, the VNN model consists of 2 -layer architecture, with 44 features per dimension for m = 300 dimensions in both layers and 2 filter taps in each layer.
Revised: For FTDC300, the VNN model consists of 2 -layer architecture, with 44 m -dimensional outputs per dimension for m = 300 dimensions in both layers and 2 filter taps in each layer.

Original: The learning rate for the Adam optimizer is set to 0 .
Revised: The learning rate for the Adam optimizer is set to 0 .

Original: For FTDC100, the VNN model consists of 2 -layer architecture, with 93 features per dimensions for m = 100 dimensions in both layers and 2 filter taps in each layer.
Revised: For FTDC100, the VNN model consists of 2 -layer architecture, with 93 m -dimensional outputs per dimension for m = 100 dimensions in both layers and 2 filter taps in each layer.

Original: The learning rate for the optimizer is set to 0 . 0047 .
Revised: The learning rate for the optimizer is set to 0 . 0047 .

Original: The readout layer in each model evaluates the unweighted mean of the outputs of the final layer to form an estimate for age.
Revised: The readout layer in each model evaluates the unweighted mean of the outputs of the final layer to form an estimate for age.

Original: We tabulate the MAE in the matrix in Fig.
Revised: We tabulate the MAE in the matrix in Fig.

Original: 2 a) and correlation between true age and predicted age in the matrix in Fig.
Revised: 2 a) and correlation between true age and predicted age in the matrix in Fig.

Original: 2 b), where the row ID indicates the dataset for which VNN was trained and the column ID indicates the dataset on which the VNN was tested to evaluate transferability of VNN performance across datasets with different resolutions.
Revised: 2 b), where the row ID indicates the dataset for which VNN was trained and the column ID indicates the dataset on which the VNN was tested to evaluate transferability of VNN performance across datasets with different resolutions.

Original: For instance, the element at coordinates (1 , 2) in Fig.
Revised: For instance, the element at coordinates (1 , 2) in Fig.

Original: 2a) represents the MAE evaluated on complete FTDC300 dataset ( m = 300 ) for VNNs trained on FTDC100 dataset ( m = 100 ).
Revised: 2a) represents the MAE evaluated on complete FTDC300 dataset ( m = 300 ) for VNNs trained on FTDC100 dataset ( m = 100 ).

Original: The results in Fig.
Revised: The results in Fig.

Original: 2 show that the performance of VNNs in terms of MAE and correlation between predicted age and true age can be transferred across different resolutions in the FTDC datasets.
Revised: 2 show that the performance of VNNs in terms of MAE and correlation between predicted age and true age can be transferred across different resolutions in the FTDC datasets.

Original: Note that this experiment is not feasible for PCA-regression models, where the principal components and the regression model would need to be re-evaluated for data from different resolution.
Revised: Note that this experiment is not feasible for PCA-regression models, where the principal components and the regression model would need to be re-evaluated for data from different resolution.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we have introduced coVariance neural networks (VNN) that operate on covariance matrices and whose architecture is derived from graph neural networks.
Revised: In this paper, we have introduced coVariance neural networks (VNN) that operate on covariance matrices and whose architecture is derived from graph neural networks.

Original: We have studied the stability properties of VNNs for sample covariance matrices and shown that the stability of VNNs improves with the number of data samples n .
Revised: We have studied the stability properties of VNNs for sample covariance matrices and shown that the stability of VNNs improves with the number of data samples n .

Original: Our experiments on real datasets have demonstrated that VNNs are significantly more stable than PCA-based approaches with respect to perturbations in the sample covariance matrix.
Revised: Our experiments on real datasets have demonstrated that VNNs are significantly more stable than PCA-based approaches with respect to perturbations in the sample covariance matrix.

Original: Also, unlike PCA-based approaches, VNNs do not require the eigendecomposition of the sample covariance matrix.
Revised: Also, unlike PCA-based approaches, VNNs do not require the eigendecomposition of the sample covariance matrix.

Original: Furthermore, on a set of multiresolution datasets, we have observed that VNN performance is also transferable across cortical thickness data collected at multiple resolutions without re-training.
Revised: Furthermore, on a set of multiresolution datasets, we have observed that VNN performance is also transferable across cortical thickness data collected at multiple resolutions without re-training.

Original: Acknowledgements
Revised: Acknowledgements

Original: This research was supported by National Science Foundation under the grants CCF-1750428 and
Revised: This research was supported by National Science Foundation under the grants CCF-1750428 and

Original: CCF-1934962.
Revised: CCF-1934962.

Original: The ABC dataset was provided by the Penn Alzheimer’s Disease Research Center (ADRC; NIH AG072979) at University of Pennsylvania.The MRI data for FTDC datasets were provided by the Penn Frontotemporal Degeneration Center (NIH AG066597). Cortical thickness data were made available by Penn Image Computing and Science Lab at University of Pennsylvania. A Data and Code Availability
Revised: The ABC dataset was provided by the Penn Alzheimer’s Disease Research Center (ADRC; NIH AG072979) at University of Pennsylvania.

Original: Data for experiments on ABC and FTDC datasets may be requested through https://www.

Original: pennbindlab.com/data-sharing and upon review by the University of Pennsylvania Neurodegenerative Data Sharing Committee, access will be granted upon reasonable request.

Original: Data and code for synthetic experiments in Appendix D are available at https://github.com/pennbindlab/VNN .

Original: The implementations of the experiments on ABC and FTDC datasets are similar.

Original: B Overview of PCA

Original: The advantages of using PCA, its role in dimension reduction, and limitations are well documented [10].

Original: PCA is an orthogonal, linear transformation of the observed dataset ˆ x n via a change of basis, such that, the transformed data has minimal redundancy while the hidden structure or patterns of interest in the raw data are preserved for further statistical analyses.

Original: In practice, the implementation of PCA is given by

Original: [Equation]

Original: where the rows of P ∈ R m × m represent the new basis on which the data ˆ x n is projected and therefore, form the designed principal components in PCA.

Original: We next briefly discuss the construction of the transformation P and the intuition behind it.

Original: Note that in the diagonal elements of the sample covariance matrix ˆ C n represent the individual variances of the constituent m features and its off-diagonal elements characterize the redundancy in the data.

Original: A desired objective of PCA is to de-correlate the dataset ˆ x n by removing any second order dependencies.

Original: To achieve this, a linear transformation P is designed, such that, the covariance of the transformed data y n , given by ˆ C y is a diagonal matrix.

Original: It can readily be verified that the eigenvectors of the sample covariance matrix ˆ C n satisfy this property.

Original: The eigendecomposition of ˆ C n is formalized as

Original: [Equation]

Original: (22) where · T refers to the Hermitian operator, U is the matrix constituted by orthonormal eigenvectors of ˆ C n and W = diag ( w 1 , . . .

Original: , w m ) is the diagonal matrix of eigenvalues of ˆ C n , such that, w 1 ≥ w 2 · · · ≥ w m .

Original: Therefore, for transformation P = U T , such that, ˆ y n =

Original: U T ˆ x n we have

Original: [Equation]

Original: Hence, in practical implementation of PCA, the eigenvectors of the sample covariance matrix form the principal components that characterize the linear, orthogonal transformation of data ˆ x n .

Original: A common application of PCA is to extract the principal components that explain the most variance in ˆ x n (thus, resulting in dimension reduction while preserving the most relevant information under certain assumptions), followed by designing statistical models for inference tasks such as prediction, classification etc [7].

Original: C Stability of coVariance Filters

Original: We start by characterizing the perturbation of sample covariance matrix ˆ C n with respect to C in

Original: Lemma 1.

Original: For convenience, we use the notation ˆ C for ˆ C n .

Original: To this end, we define

Original: [Equation]

Original: and I m as an m × m identity matrix.

Original: Lemma 1.

Original: Consider an ensemble covariance matrix C with the eigendecomposition in (15) and a sample covariance matrix ˆ C with the eigendecomposition in (1) .

Original: For any eigenvalue λ i > 0 of C , the perturbation E satisfies

Original: [Equation]

Original: 25) where

Original: [Equation]

Original: i

Original: [Equation]

Original: Proof.

Original: Note that from the definition of eigenvectors and eigenvalues, we have

Original: [Equation]

Original: We can rewrite (27) in terms of perturbations with respect to the ensemble covariance matrix C and the outputs of its eigendecomposition as follows:

Original: [Equation]

Original: where we have used w i

Original: =

Original: λ

Original: i + δλ i and u i =

Original: v i

Original: + δ v i .

Original: Using the fact that Cv i

Original: =

Original: λ

Original: i v i and rearranging the terms in (28), we have

Original: [Equation]

Original: By setting E =

Original: ˆ C − C and β i

Original: =

Original: λ

Original: i

Original: I m − C , we can rewrite (29) as

Original: [Equation]

Original: Next, we leverage Lemm 1 to complete the proof of Theorem 2.

Original: Proof of Theorem

Original: Proof.

Original: To start with, we note that the coVariance filters with respect to ˆ C and C are given by

Original: [Equation]

Original: ( ˆ C )

Original: [Equation]

Original: We aim to study the stability of the coVariance filters by analyzing the difference between H ( ˆ C ) and

Original: H ( C ) .

Original: For this purpose, we next establish the first order approximation for ˆ C k in terms of C and E .

Original: Using ˆ C = C + E , the first order approximation of ˆ C k is given by

Original: [Equation]

Original: [Equation]

Original: where ˜ E satisfies ∥ ˜ E ∥ ≤ k (cid:80)

Original: [Equation]

Original: (cid:0) kr (cid:1)

Original: ∥

Original: E ∥ r ∥ C ∥

Original: k − r .

Original: Using (32), we have

Original: [Equation]

Original: ) where ˜ D satisfies ∥ ˜ D ∥ 2 = O ( ∥ E ∥ 2 ) [19].

Original: The focus of our subsequent analysis will be the first term in (34).

Original: For a random data sample x = [ x 1 , . . .

Original: , x m ] T , such that, ∥ x ∥ < R , for some R >and x ∈ R m × 1 , its VFT with respect to C is given by ˜ x

Original: = V T x , where ˜ x

Original: = [˜ x 1 , . .

Original: ˜ x m ] T .

Original: The relationship ˜ x and x can be expressed as

Original: [Equation]

Original: Multiplying both sides in (34) by x and by leveraging (35), we get [ H ( ˆ C

Original: [Equation]

Original: ( C

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: = m (cid:88) i =

Original: [Equation]

Original: ˜

Original: [Equation]

Original: m (cid:88) k

Original: [Equation]

Original: [Equation]

Original: k k

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: (37) = m (cid:88) i = ˜

Original: [Equation]

Original: (cid:88) k

Original: [Equation]

Original: [Equation]

Original: k

Original: [Equation]

Original: 1 (cid:88) r

Original: [Equation]

Original: C

Original: [Equation]

Original: k

Original: [Equation]

Original: −

Original: [Equation]

Original: where we have used Cv i = λ i v i in the transition from (37) to (38). We focus only on the first term in (38) and leverage the result from Lemma 1 that expands Ev i to get m (cid:88)
Revised: The MRI data for FTDC datasets were provided by the Penn Frontotemporal Degeneration Center (NIH AG066597).

Original: [Equation]

Original: Next, we analyze term 1, term 2, and term 3 in (39) separately.

Original: Analysis of Term 1 in (39):

Original: In the analysis of term 1, we start by noting that

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Using (42) and δ v i = u

Original: i − v i in term 1 in (39), we have m

Original: [Equation]

Original: Using C r = V Λ r V T in (43), term 1 in (39) is equivalent to

Original: [Equation]

Original: where L i is a diagonal matrix whose j -th element is given by

Original: [Equation]

Original: 49) where h ( λ i ) is the frequency response of the coVariance filter and is defined in (8).

Original: Therefore, we have L i = diag ([ h ( λ i )

Original: − h ( λ j )] j ) .

Original: Next, in (45), we note that

Original: [Equation]

Original: Using (50) and (49) in (45) and v T j v i = 0 , ∀ j ̸

Original: =

Original: i , we deduce that the term 1 in (39) is equivalent to m (cid:88)

Original: [Equation]

Original: where J i is a diagonal matrix whose j -th diagonal element is given by

Original: [Equation]

Original: For the stability analysis, we are interested in the norm of term 1.

Original: Therefore, by noting the equivalence between the term 1 in (39) and (51), we have

Original: [Equation]

Original: Note that v T j u

Original: i is the inner product between the eigenvector v j of ensemble covariance matrix C and the eigenvector u i of the sample covariance matrix ˆ C .

Original: The bounds on v T j u i in terms of the number of data samples n have been studied in the existing literature.

Original: Here, we leverage the result from [30, Theorem 4.1] to conclude that if sgn ( λ j − λ i )2 v j > sgn ( λ j − λ i )( λ j − λ i ) for λ i ̸

Original: = λ j , the condition

Original: [Equation]

Original: m

Original: [Equation]

Original: =

Original: [Equation]

Original: (55) is true with probability at least (cid:0) 1 − 1 n 2 ε (cid:1) for some ε ∈ (0 , 1 / 2] , where k i = (cid:16) E

Original: [ ∥ XX T v i ∥ 22 ] − λ 2 i (cid:17) 12 .

Original: [-Furthermore, we note that the condition sgn ( λ j − λ i )2 v j > sgn ( λ j − λ i )( λ j − λ i ) is satisfied with probability at least 1 − 2 k 
-]
Revised: [-Furthermore, we note that the condition sgn ( λ j − λ i )2 v j > sgn ( λ j − λ i )( λ j − λ i ) is satisfied with probability at least 1 − 2 k 
-]

Original: i | λ i − λ

Original: j | [30, Corollary 4.2], which via a union bound and first order approximation from Taylor series implies that (55) is true with probability at least 1 − 1 n 2 ε − 2 κmn for κ defined in (17).

Original: Therefore, for a coVariance filter with the property

Original: [Equation]

Original: for some real constant M > 0 , the condition in (55) is equivalent to

Original: [Equation]

Original: =

Original: [Equation]

Original: |

Original: [Equation]

Original: x

Original: [Equation]

Original: | , (57) which holds with probability at least 1 − 1 n 2 ε − 2 κmn .

Original: Furthermore, note that m (cid:80) i =1 | ˜ x i | ≤ √ m ∥ x ∥ .

Original: If the random sample x satisfies ∥ x ∥ ≤ R , then we have

Original: [Equation]

Original: for any ε ∈ (0 , 1 / 2] .

Original: Analysis of Term 2 in (39): Using Cv i

Original: =

Original: λ

Original: i v i , we note that term 2 in (39) is equivalent to m

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Next, using Weyl’s theorem [38, Theorem 8.1.6], we note that ∥ E ∥ ≤ α implies that | δλ i | ≤ α for any α > 0 .

Original: For a random instance x of random vector X whose probability distribution is supported within a ball of radius 1 w.l.o.g, such that, ∥ x ∥ ≤ 1 , we have

Original: [Equation]

Original: for some constant B > 0 and u > 0 .

Original: The result in (62) follows directly from [39, Theorem 5.6.1].

Original: Therefore, using (61), we have

Original: [Equation]

Original: Using (62), | h ′ ( λ i ) | ≤ M/k min (where k min = min i ∈{ 1 ,...,m } ,λ i > 0 k i ) from (56), and ∥ v i ∥ = 1 , we have

Original: [Equation]

Original: for some constant A > 0 and u > 0 .

Original: Analysis of Term 3 in (39): We remark that the term 3 in (39) consists of second order error terms that diminish faster with the number of samples n as compared to term 1 and term 2.

Original: To illustrate this, we note two facts.

Original: First, from (62) and Weyl’s theorem, we note that ∥ δλ i I m − E ∥ ≤ 2 ∥ E ∥ and

Original: ∥ E ∥ ≃ O (1 / √ n ) with high probability.

Original: Secondly, from the existing literature [40, 41], we note that if ˆ C follows the Wishart distribution, we have

Original: [Equation]

Original: and

Original: [Equation]

Original: [Equation]

Original: Using Chebyshev’s inequality, we note that

Original: [Equation]

Original: for any constant γ > 1 .

Original: Furthermore, (67) is equivalent to

Original: [Equation]

Original: which implies that ∥ δ v i ∥

Original: =

Original: O (1 / √ n ) with high probability.

Original: Therefore, the second order error term ( δλ i I m − E ) δ v i scales as O (1 /n ) , which diminishes faster with n as compared to terms 1 and 2, that individually scale as O (1 /n 1 / 2 − ε ) for ε ∈ (0 , 1 / 2] and O (1 / √ n ) , respectively.

Original: The proof of Theorem 2 is completed by noting that the condition on ∥ [ H ( ˆ C ) − H ( C )] x ∥ reduces to the condition on operator norm ∥

Original: [ H ( ˆ C ) − H ( C )] ∥ for any ∥ x ∥ ≤ 1 and that the terms scaling at 1 / √ n or slower in (58) and (64) dominate the scaling behavior of the upper bound on ∥ [ H ( ˆ C ) −

Original: H ( C )] ∥ . [Figure] E Additional Experiments and Discussions In this section, we provide additional discussions on the age prediction task using cortical thickness  and related experiments on the synthetic data.
Revised: Cortical thickness data were made available by Penn Image Computing and Science Lab at University of Pennsylvania.

Original: In Section E.1, we briefly discuss the motivation behind studying age prediction and PCA-based statistical analysis in this context.

Original: In Section E.2, we provide additional details on cortical thickness data acquisition.

Original: In Section E.3, we report the results for stability analysis of VNNs and PCA-regression models for FTDC100 ( m = 100 ) and

Original: FTDC300 ( m = 300 ) datasets.

Original: In Section E.4, we study the stability of VNNs on two simulated settings that include non-linear and linear data models.

Original: In Section E.5, we include additional figures that supplement the VNN transferability results in Fig.

Original: [Figure]

Original: E.1 Predicting Brain Age with Cortical Thickness Data

Original: Brain age estimation using various modalities of magnetic resonance imaging (MRI) is an active area of research [42, 43, 44].

Original: A promising application of brain age prediction is early detection of neurodegenerative diseases (such as Alzheimer’s, Huntingson’s disease) which may manifest themselves as error in age prediction in pathological contexts by machine learning models trained on healthy subjects.

Original: Cortex anatomical measures extracted from structural MRI scans have shown promising results in age prediction in existing studies [43, 44].

Original: Cortical measures across the brain usually have high collinearity and therefore, the age prediction pipeline consisting of dimension reduction on cortical features and regression models is commonly employed.

Original: The utility of dimension reduction is apparent in ABC and FTDC datasets used in this paper as well from Fig.

Original: 5a and Fig.

Original: 5b, where we observe that the eigenvalue distribution for ABC and FTDC300 datasets is skewed and therefore, PCA based dimensionality reduction is well-motivated.

Original: The studies in [43] and [44] report age prediction results on different datasets using various approaches, such as PCA-based regression (similar to the PCA-regression models evaluated in this paper), ridge regression, lasso based regression, and fully connected neural networks.

Original: All these approaches show comparable performances in predicting age using cortical anatomical measures [43].

Original: The objective of our experiments is to demonstrate the properties of VNNs, such as stability and transferability, against other covariance matrix driven statistical analyses.

Original: In this context, PCA-based regression models are the natural baselines for comparison against VNNs.

Original: E.2 Cortical Thickness Data Acquisition

Original: Cortical thickness measures in regions of interest (ROIs) were derived using 0 .

Original: 8 − 1 mm isotropic

Original: T1-weighted MRI.

Original: The complete pipeline for cortical thickness extraction for ABC data is similar to that in [45].

Original: E.3 Stability of VNNs on FTDC100 and FTDC300 datasets

Original: In this subsection, we study the stability of VNNs for FTDC100 and FTDC300 datasets in a similar fashion as for ABC dataset in Section 3.

Original: We split each dataset into a 90 / 10 train/test split, such that, we have 153 samples in the training set and 17 samples in the test set.

Original: The nominal VNN and PCA-regression models for FTDC100 and FTDC300 are trained on their respective sample covariance matrices ˆ C 153 derived from cortical thickness data.

Original: The architecture and hyperparamaters for VNN training for FTDC100 and FTDC300 are same as that reported in Section 5.2.

Original: Figure 6 a) and b) show the variance in MAE over training and test sets for nominal models based on VNN and

Original: PCA-regression with respect to ˆ C 153 for FTDC100 dataset.

Original: Figure 6 c) and d) show the variance in

Original: MAE over training and test sets for nominal models based on VNN and PCA-regression with respect to ˆ C 153 for FTDC300 dataset.

Original: For both datasets, we observe that VNNs are stable with respect to the perturbations in the sample covariance matrix.

Original: Also, note that our theoretical results in Theorem 5.

Original: and Theorem 3 suggest stability of coVariance filters and VNNs for n > m .

Original: Our observations from the experiments on FTDC300 dataset show evidence that VNNs retain stability even when this condition is violated.

Original: Significant randomness is induced into the performance of PCA − LR when the principal components are perturbed due to perturbations in ˆ C 153 for both datasets.

Original: However, in

Original: Fig.

Original: 6, we also observe that PCA-rbf retains stability in performance when ˆ C 153 is replaced by ˆ C n ′ for n ′ ∈ [65 , 152] .

Original: E.4 Stability of VNNs on Synthetic Data

Original: We consider two settings for synthetic data.

Original: Friedman regression problem : This setting is described as ‘Friedman1’ in [46] and is generated using the routine sklearn.datasets.make_friedman1 in python.

Original: In the simulated dataset, we have m independent features or predictors, each sampled uniformly from the range [0 , 1] .

Original: Out of m predictors, any 5 are used to generate the response variable and others are independent of the response.

Original: If the 5 relevant predictors are given by x 1 , x 2 , x 3 , x 4 , and x 5 , they are related to the response y as

Original: [Equation]

Original: 69) where ϑ represents noise.

Original: We generate a dataset of n = 1000 samples using m = 100 features and noise distributed according to N (0 , 1) .

Original: Figure 7 a) shows the distribution of eigenvalues in the covariance matrix for the dataset used.

Original: The dataset is split into a 90 / 10 train/test split and a sample covariance matrix ˆ C 900 is evaluated from the features in the training set.

Original: Next, we perform regression against the response using VNN, PCA-LR , and PCA-rbf models.

Original: For VNNs, we use the same architecture and hyperparameters as described in Section 5.1.

Original: Using different permutations of the training set, we train 100 nominal models for VNN, PCA-LR , and PCA-rbf .

Original: Figure 7 b) plots the variation in mean performance of the nominal models with respect to perturbations in the sample covariance matrix.

Original: The last data point in Fig.

Original: 7 b) corresponds to the performance of nominal models trained on ˆ C 900 .

Original: When ˆ C 900 is replaced with ˆ C n ′ for n ′ ∈ [5 , 899] , we observe that the VNN performance retains stability for sample covariance matrices generated with n ′ >samples.

Original: However, considerable randomness is introduced in the performance of PCA-regression models when the principal components are evaluated from ˆ C n ′ for n ′ ̸ = 900 .

Original: Similar phenomenon is observed for the performance on the test set in Fig.

Original: 7 c).

Original: Linear regression problem : In this setting, we generate random linear regression problems using the routine sklearn.datasets.make_regression in python, which allows us to specify the number of informative features; effective rank of the input dataset, i.e., the approximate number of singular vectors required to explain most of the input data by linear combinations; tail parameter which is the relative importance of the fat noisy tail of the singular values profile; and noise.

Original: In our experiments, we set the input dimension m = 100 , the dimension of the response to be 1 , number of samples n = 1000 , number of informative features to be 20 , effective rank of the input dataset to be 25 and noise to be distributed as N (0 , 3) .

Original: Since the stability of VNNs depends on the eigengap and kurtosis of underlying distribution of the data in direction of eigenvectors according to Theorem 3, we also aim to study the effects on VNN distribution with respect to variation in the strength of the tail of the eigenvalue distribution.

Original: To this end, we generate two datasets for linear regression with tail parameters set to 0 .

Original: 7 and 0 . 2 .

Original: The eigenvalues of the covariance matrix for tail set to 0 .

Original: 7 are more spread out as compared to that for tail set to 0 .

Original: 2 (see Fig.

Original: 8 a) and Fig.

Original: 9 a)).

Original: To evaluate the stability of VNNs, we split each dataset into a 90 / 10 train/test split and generate sample covariance matrices ˆ C 900 for both.

Original: Next, we perform regression using VNN and PCA-LR models.

Original: For VNNs, we use the same architecture and hyperparameters as described in Section 5.1.

Original: Using different permutations of the training set, we obtain 100 nominal models for VNN and

Original: PCA − lR .

Original: Figure 8 b) plots the variations in mean MAE performances of the nominal models with respect to perturbations in the sample covariance matrix, with the last data point corresponding to n = 900 , i.e., ˆ C 900 .

Original: When ˆ C 900 is replaces with ˆ C n ′ for any n ′ ∈ [5 , 899] , our experiments show that VNN performance is stable but significant randomness is induced into the performance of PCA-LR model.

Original: Similar phenomenon is observed for the performance on the test set in Fig.c).

Original: Same discussion also follows for the results in Fig.

Original: 9 b)-c), where we have set the tail parameter for eigenvalue distribution to be 0 .

Original: Comparison of MAE performances in Fig.

Original: 8 c) and Fig.

Original: 9 c) reveals that both VNN and PCA − LR perform significantly better for dataset with tail set to 0 .

Original: 2 than that for 0 .

Original: This observation is along the expected lines as the first 25 eigenvalues and eigenvectors that contain the most information are more separated (and hence, more accurately estimated by the sample covariance matrix) for the dataset with tail parameter 0 .

Original: 2 as compared to the dataset with tail parameter 0 .

Original: Moreover, VNNs retain the stability property for both datasets.

Original: Finally, we also note that the observations above also extend to data with higher dimensions.

Original: Figureillustrates the results on a setting for linear regression with m = 1000 , number of informative features to be 20 , effective rank of the input dataset to be 25 and noise to be distributed as N (0 , 3) .

Original: The tail is set to 0 .

Original: The number of samples is 5000 of which 4500 are used to form the covariance matrix ˆ C 4500 for training PCA-regression and VNN models.

Original: Figure 10 a) illustrates the eigenvalue distribution for this setting.

Original: Figure 10 b) and c) show the results of stability analysis by perturbing ˆ C 4500 for training set and test set, respectively.

Original: E.5 Transferability of VNNs

Original: In Fig.

Original: we show the predicted age vs true age plots corresponding to the first rows of the matrices in Fig.

Original: 2, i e., when the VNN is trained on FTDC100 dataset and its transferability is evaluated on

Original: FTDC300 and FTDC500 datasets.

Original: Similar plots are observed for transferability in other reported settings as well.