Original: S ELF - SUPERVISED LEARNING OF M AXIMUM M ANIFOLD C APACITY R EPRESENTATIONS
Revised: S ELF - SUPERVISED LEARNING OF M AXIMUM M ANIFOLD C APACITY R EPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-supervised Learning (SSL) provides a strategy for constructing useful representations of images without relying on hand-assigned labels.
Revised: Self-supervised Learning (SSL) provides a strategy for constructing useful representations of images without relying on hand-assigned labels.

Original: Many such methods aim to learn a function that maps distinct views of the same scene or object to nearby points in the representation space.
Revised: Many such methods aim to learn a function that maps distinct views of the same scene or object to nearby points in the representation space.

Original: These methods are often justified by showing that they optimize an objective that is an approximation of (or correlated with) the mutual information between representations of different views.
Revised: These methods are often justified by showing that they optimize an objective that is an approximation of (or correlated with) the mutual information between representations of different views.

Original: Here, we recast the problem from the perspective of manifold capacity , a measure that has been used to evaluate the classification capabilities of a representation.
Revised: Here, we recast the problem from the perspective of manifold capacity , a measure that has been used to evaluate the classification capabilities of a representation.

Original: Specifically, we develop a contrastive learning framework that aims to maximize the number of linearly separable object manifolds, yielding a Maximum Manifold
Revised: Specifically, we develop a contrastive learning framework that aims to maximize the number of linearly separable object manifolds, yielding a Maximum Manifold

Original: Capacity Representation (MMCR).
Revised: Capacity Representation (MMCR).

Original: We apply this method to unlabeled images, each augmented by a set of basic transformations, and find that it learns meaningful features using the standard linear evaluation protocol.
Revised: We apply this method to unlabeled images, each augmented by a set of basic transformations, and find that it learns meaningful features using the standard linear evaluation protocol.

Original: Specifically, we find that MMCRs support performance on object recognition comparable or better than recently developed SSL frameworks, while providing more robustness to adversarial attacks.
Revised: Specifically, we find that MMCRs support performance on object recognition comparable or better than recently developed SSL frameworks, while providing more robustness to adversarial attacks.

Original: Finally, empirical analysis reveals the means by which compression of object manifolds gives rise to class separability.
Revised: Finally, empirical analysis reveals the means by which compression of object manifolds gives rise to class separability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural images lie, at least locally, within manifolds whose intrinsic dimensionality is low relative to that of their embedding space (the set of pixel intensities).
Revised: Natural images lie, at least locally, within manifolds whose intrinsic dimensionality is low relative to that of their embedding space (the set of pixel intensities).

Original: Nevertheless, these manifolds are enormously complex, as evidenced by the variety of natural scenes.
Revised: Nevertheless, these manifolds are enormously complex, as evidenced by the variety of natural scenes.

Original: A fundamental goal of machine learning is to extract these structures from observations, and use them to perform inference tasks.
Revised: A fundamental goal of machine learning is to extract these structures from observations, and use them to perform inference tasks.

Original: In the context of recognition, consider the object submanifold , M j , which consists of all images of object j (for example, those taken from different camera locations, or under different lighting conditions).
Revised: In the context of recognition, consider the object submanifold , M j , which consists of all images of object j (for example, those taken from different camera locations, or under different lighting conditions).

Original: Object recognition networks act to map images within a submanifold to nearby representations, relative to images from other submanifolds, and this concept has been effictively exploited in recent self-supervised learning (SSL) methods (Zbontar et al., 2021;
Revised: Object recognition networks act to map images within a submanifold to nearby representations, relative to images from other submanifolds, and this concept has been effictively exploited in recent self-supervised learning (SSL) methods (Zbontar et al., 2021;

Original: Chen et al., 2020;
Revised: Chen et al., 2020;

Original: Caron et al., 2020; Bachman et al., 2019; Wang & Isola, 2020; Wang et al., 2022).
Revised: Caron et al., 2020; Bachman et al., 2019; Wang & Isola, 2020; Wang et al., 2022).

Original: Most of these operate by minimizing pairwise distances between images within submanifolds, while contrastively maximizing pairwise distances between images in different submanifolds.
Revised: Most of these operate by minimizing pairwise distances between images within submanifolds, while contrastively maximizing pairwise distances between images in different submanifolds.

Original: A parallel effort in computational neuroscience has aimed to characterize manifolds in neural representations, and their relationship to underlying neural circuits (Kriegeskorte & Kievit, 2013; Chung & Abbott, 2021).
Revised: A parallel effort in computational neuroscience has aimed to characterize manifolds in neural representations, and their relationship to underlying neural circuits (Kriegeskorte & Kievit, 2013; Chung & Abbott, 2021).

Original: Studies in various modalities have identified geometric structures in neural data that are associated with behavioral tasks (Bernardi et al., 2020; DiCarlo & Cox, 2007; H´enaff et al., 2021; Gallego et al., 2017; Nieh et al., 2021), and explored metrics for quantifying these representation geometries.
Revised: Studies in various modalities have identified geometric structures in neural data that are associated with behavioral tasks (Bernardi et al., 2020; DiCarlo & Cox, 2007; H´enaff et al., 2021; Gallego et al., 2017; Nieh et al., 2021), and explored metrics for quantifying these representation geometries.

Original: Here, we make use of a recently developed measure of manifold capacity , rooted in statistical physics (Chung et al., 2018), which has been used to evaluate how many manifolds can be linearly separated within the representation space of various models.
Revised: Here, we make use of a recently developed measure of manifold capacity , rooted in statistical physics (Chung et al., 2018), which has been used to evaluate how many manifolds can be linearly separated within the representation space of various models.

Original: We develop a simplified form of this meausre, and incorporate it into a novel contrastive objective, that maximizes the extent of the global image manifold while minimizing that of constituent object manifolds.
Revised: We develop a simplified form of this meausre, and incorporate it into a novel contrastive objective, that maximizes the extent of the global image manifold while minimizing that of constituent object manifolds.

Original: We apply this to an unlabeled set of images, each augmented to form a small set of samples from their corresponding manifold.
Revised: We apply this to an unlabeled set of images, each augmented to form a small set of samples from their corresponding manifold.

Original: We show that the learned representations
Revised: We show that the learned representations

Original: • support high-quality object recognition, when evaluated using the standard linear evaluation (Chen et al., 2020) paradigm (i.e., training a linear classifer to operate on the output of the unsupervised network).
Revised: • support high-quality object recognition, when evaluated using the standard linear evaluation (Chen et al., 2020) paradigm (i.e., training a linear classifer to operate on the output of the unsupervised network).

Original: In particular, performance is approximately matched to that of other recently proposed SSL methods.
Revised: In particular, performance is approximately matched to that of other recently proposed SSL methods.

Original: • extract semantically relevant features from the data, that can be revealed by examining the learning signal derived from the unsupervised task • have interpretable geometric properties • are more robust to adversarial attack than those of other recently proposed SSL methods.
Revised: • extract semantically relevant features from the data, that can be revealed by examining the learning signal derived from the unsupervised task • have interpretable geometric properties • are more robust to adversarial attack than those of other recently proposed SSL methods.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Our methodology is closely related to and inspired by recent advances in contrastive self-supervised representation learning (SSL), but has a distinctly different motivation.
Revised: Our methodology is closely related to and inspired by recent advances in contrastive self-supervised representation learning (SSL), but has a distinctly different motivation.

Original: Many recent frameworks craft objectives that are designed to maximize the mutual information between representations of different views of the same object (Oord et al., 2018;
Revised: Many recent frameworks craft objectives that are designed to maximize the mutual information between representations of different views of the same object (Oord et al., 2018;

Original: Chen et al., 2020; Oord et al., 2018; Tian et al., 2020; Bachman et al., 2019)).
Revised: Chen et al., 2020; Oord et al., 2018; Tian et al., 2020; Bachman et al., 2019)).

Original: However, estimating mutual information in high dimensional feature spaces (which is the regime of modern deep learning models models) has been difficult to compute historically (Belghazi et al., 2018), and furthermore it is not clear that more closely approximating mutual information in the objective produces improved representations (Wang &
Revised: However, estimating mutual information in high dimensional feature spaces (which is the regime of modern deep learning models models) has been difficult to compute historically (Belghazi et al., 2018), and furthermore it is not clear that more closely approximating mutual information in the objective produces improved representations (Wang &

Original: Isola, 2020).
Revised: Isola, 2020).

Original: 1 By contrast, capacity estimation theories operate in the regime of large ambient dimension as they are derived in the “large N (thermodynamic) limit” (Chung et al., 2018; Bahri et al., 2020).
Revised: 1 By contrast, capacity estimation theories operate in the regime of large ambient dimension as they are derived in the “large N (thermodynamic) limit” (Chung et al., 2018; Bahri et al., 2020).

Original: Therefore we test whether one such measure, which until now had been used to evaluate the quality of representations, might be useful as objective function in SSL.
Revised: Therefore we test whether one such measure, which until now had been used to evaluate the quality of representations, might be useful as objective function in SSL.

Original: Operationally, many existing methods are optimized to minimize some notion of distance between the representations of different augmented views of the same image, while maximizing the distance between representations of (augmented views of) distinct images (these are thought of as encouraging alignment and uniformity in the framework of Wang & Isola (2020)).
Revised: Operationally, many existing methods are optimized to minimize some notion of distance between the representations of different augmented views of the same image, while maximizing the distance between representations of (augmented views of) distinct images (these are thought of as encouraging alignment and uniformity in the framework of Wang & Isola (2020)).

Original: When taking the view that different views of an image form a continuous manifold that we aim to compress, the distance between two randomly sampled points from said manifold seems a strange choice for the size metric to optimize for.
Revised: When taking the view that different views of an image form a continuous manifold that we aim to compress, the distance between two randomly sampled points from said manifold seems a strange choice for the size metric to optimize for.

Original: Perhaps unsurprisingly it has been demonstrated on multiple occasions, notably by the success of the “multi-crop,” strategy implemented SwAV (Caron et al., 2020) and earlier in the contrastive multiview coding work by Tian et al.
Revised: Perhaps unsurprisingly it has been demonstrated on multiple occasions, notably by the success of the “multi-crop,” strategy implemented SwAV (Caron et al., 2020) and earlier in the contrastive multiview coding work by Tian et al.

Original: However most commonly the use of multiple views is such that the objective effectively becomes a Monte Carlo estimate with more than one sample of the same pairwise distance function.
Revised: However most commonly the use of multiple views is such that the objective effectively becomes a Monte Carlo estimate with more than one sample of the same pairwise distance function.

Original: Rather than using the mean distance or cosine similarity between pairs of points, we use a nuclear norm as a combined measure of size and dimensionality of groups of points, an idea that is strongly motivated by learning theory.
Revised: Rather than using the mean distance or cosine similarity between pairs of points, we use a nuclear norm as a combined measure of size and dimensionality of groups of points, an idea that is strongly motivated by learning theory.

Original: The nuclear norm has been previously used to induce or infer low rank structure in the representation of data, for example, in H ´ enaff et al. (2015);
Revised: The nuclear norm has been previously used to induce or infer low rank structure in the representation of data, for example, in H ´ enaff et al. (2015);

Original: Wang et al.
Revised: Wang et al.

Original: Lezama et al.
Revised: Lezama et al.

Original: In particular, Wang et al.
Revised: In particular, Wang et al.

Original: (2022) employ the nuclear norm as a regularizer to supplement an InfoNCE loss.
Revised: (2022) employ the nuclear norm as a regularizer to supplement an InfoNCE loss.

Original: Our approach represents a more radical departure from the traditional
Revised: Our approach represents a more radical departure from the traditional

Original: InfoNCE loss, as we will detail below.
Revised: InfoNCE loss, as we will detail below.

Original: Rather than pair a low-rank prior with a logistic regressionbased likelihood, we make the more symmetric choice of employing a high rank likelihood.
Revised: Rather than pair a low-rank prior with a logistic regressionbased likelihood, we make the more symmetric choice of employing a high rank likelihood.

Original: This allows the objective to explicitly discourage dimensional collapse, a well known issue in SSL (Jing et al., 2021).
Revised: This allows the objective to explicitly discourage dimensional collapse, a well known issue in SSL (Jing et al., 2021).

Original: Another consequence of encouraging maximal rank over the dataset is that the objective encourages the representation to form a simplex equiangular tight frame (sETF).
Revised: Another consequence of encouraging maximal rank over the dataset is that the objective encourages the representation to form a simplex equiangular tight frame (sETF).

Original: sETFs have been shown to be optimal in terms of cross-entropy loss when features lie on the unit hypersphere (Lu & Steinerberger, 2020), and such representations can be obtained in the supervised setting when optimizing either the traditional cross-entropy loss or a supervised contrastive loss (Papyan et al., 2020;
Revised: sETFs have been shown to be optimal in terms of cross-entropy loss when features lie on the unit hypersphere (Lu & Steinerberger, 2020), and such representations can be obtained in the supervised setting when optimizing either the traditional cross-entropy loss or a supervised contrastive loss (Papyan et al., 2020;

Original: Graf et al., 2021).
Revised: Graf et al., 2021).

Original: Recent work has shown that many popular objectives in SSL can be understood as different methods of approximating a loss function whose minima form sETFs (Dubois et al., 2022).
Revised: Recent work has shown that many popular objectives in SSL can be understood as different methods of approximating a loss function whose minima form sETFs (Dubois et al., 2022).

Original: Our approach is novel, in that it encourages sETF representations by directly optimizing the distribution of singular values, rather than minimizing a cross-entropy loss.
Revised: Our approach is novel, in that it encourages sETF representations by directly optimizing the distribution of singular values, rather than minimizing a cross-entropy loss.

Original: [Figure]
Revised: [Figure]

Original: 2. M ANIFOLD C APACITY T HEORY
Revised: 2. M ANIFOLD C APACITY T HEORY

Original: Consider a set of P manifolds embedded in a feature space of dimensionality D , each assigned a random binary class label.
Revised: Consider a set of P manifolds embedded in a feature space of dimensionality D , each assigned a random binary class label.

Original: Manifold capacity theory is concerned with the question: what is the largest value of PD such that there exists (with high probability) a hyperplane separating the two classes?
Revised: Manifold capacity theory is concerned with the question: what is the largest value of PD such that there exists (with high probability) a hyperplane separating the two classes?

Original: Recent theoretical work has demonstrated that there exists a critical value, dubbed the manifold capacity α C , such that when PD < α C the probability of finding a separating hyperplane is approximately 1 . 0 , and when PD > α C the probability is approximately 0 . 0 .
Revised: Recent theoretical work has demonstrated that there exists a critical value, dubbed the manifold capacity α C , such that when PD < α C the probability of finding a separating hyperplane is approximately 1 . 0 , and when PD > α C the probability is approximately 0 . 0 .

Original: Furthermore, α C can be accurately predicted from three key quantities: (1) the manifold radius R M , which measures the size of the manifold relative to its distance from the origin, (2) the manifold dimensionality D M which estimates the number of dimensions along which a manifold has significant extent, and (3) the centroid correlation (if the positions of manifolds are correlated with each other they will be more difficult to separate).
Revised: Furthermore, α C can be accurately predicted from three key quantities: (1) the manifold radius R M , which measures the size of the manifold relative to its distance from the origin, (2) the manifold dimensionality D M which estimates the number of dimensions along which a manifold has significant extent, and (3) the centroid correlation (if the positions of manifolds are correlated with each other they will be more difficult to separate).

Original: In particular, when the centroid correlation is low the manifold capacity can be approximated by ϕ ( R M √ D M ) where ϕ ( · ) is a monotonically decreasing function.
Revised: In particular, when the centroid correlation is low the manifold capacity can be approximated by ϕ ( R M √ D M ) where ϕ ( · ) is a monotonically decreasing function.

Original: For manifolds of arbitrary geometry calculating the manifold radii and dimensionalities involves an iterative process that alternates between determining the set of “anchor points” on each manifold that are relevant for the classification problem=, and computing the statistics of random projections of these anchor points (Cohen et al., 2020).
Revised: For manifolds of arbitrary geometry calculating the manifold radii and dimensionalities involves an iterative process that alternates between determining the set of “anchor points” on each manifold that are relevant for the classification problem, and computing the statistics of random projections of these anchor points (Cohen et al., 2020).

Original: This process is both computationally costly and nondifferentiable, and therefore not suitable for use as an objective function.
Revised: This process is both computationally costly and nondifferentiable, and therefore not suitable for use as an objective function.

Original: For more detail on the general theory see A.2.
Revised: For more detail on the general theory see A.2.

Original: However, if the submanifolds are assumed to be elliptical in shape there is an analytical expression for each of these,
Revised: However, if the submanifolds are assumed to be elliptical in shape there is an analytical expression for each of these,

Original: [Equation]
Revised: [Equation]

Original: where the λ 2 i are the eigenvalues of the covariance matrix of points on the manifold.
Revised: where the λ 2 i are the eigenvalues of the covariance matrix of points on the manifold.

Original: For reference, for a batch of 100 128-D manifolds with 100 points sampled from each, computing these ellipticalassuming measures is approximately 500 times faster in terms of wall-clock time.
Revised: For reference, for a batch of 100 128-D manifolds with 100 points sampled from each, computing these ellipticalassuming measures is approximately 500 times faster in terms of wall-clock time.

Original: Using these definitions for manifold radius and dimensionality we can write the capacity as α C = ϕ
Revised: Using these definitions for manifold radius and dimensionality we can write the capacity as α C = ϕ

Original: ( (cid:80) i σ i ) where σ i are the singular values of a matrix containing points on the manifold (which are the square roots of the eigenvalues of the covariance matrix).
Revised: ( (cid:80) i σ i ) where σ i are the singular values of a matrix containing points on the manifold (which are the square roots of the eigenvalues of the covariance matrix).

Original: In this form, the sum is the
Revised: In this form, the sum is the

Original: L 1 norm of the singular values, known as the Nuclear Norm of the matrix.
Revised: L 1 norm of the singular values, known as the Nuclear Norm of the matrix.

Original: When used as an objective function, this measure will prefer sparse solutions (i.e., a small number of non-zero singular values) corresponding to low dimensionality.
Revised: When used as an objective function, this measure will prefer sparse solutions (i.e., a small number of non-zero singular values) corresponding to low dimensionality.

Original: It is worth comparing this objective to another natural candidate for quantifying size: the determinant of the covariance matrix.
Revised: It is worth comparing this objective to another natural candidate for quantifying size: the determinant of the covariance matrix.

Original: The determinant is equal to the product of the eigenvalues (which captures the squared volume of the corresponding ellipse), but lacks the preference for lower dimensionality that comes with the Nuclear Norm.
Revised: The determinant is equal to the product of the eigenvalues (which captures the squared volume of the corresponding ellipse), but lacks the preference for lower dimensionality that comes with the Nuclear Norm.

Original: Specifically, since the determinant is zero whenever one (or more) eigenvalue is zero, it cannot distinguish zerovolume manifolds of different dimensionality.
Revised: Specifically, since the determinant is zero whenever one (or more) eigenvalue is zero, it cannot distinguish zerovolume manifolds of different dimensionality.

Original: In Yu et al.
Revised: In Yu et al.

Original: (2020), lossy coding rate (entropy) is used as a measure of compactness, which simplifies to the log determinant under a Gaussian model Ma et al.
Revised: (2020), lossy coding rate (entropy) is used as a measure of compactness, which simplifies to the log determinant under a Gaussian model Ma et al.

Original: In that work, the identity matrix is added to a multiple of the feature covariance matrix before evaluating the determinant, which solves the dimensionality issue described above.
Revised: In that work, the identity matrix is added to a multiple of the feature covariance matrix before evaluating the determinant, which solves the dimensionality issue described above.

Original: 2. O PTIMIZING M ANIFOLD C APACITY
Revised: 2. O PTIMIZING M ANIFOLD C APACITY

Original: Manifold Capacity has been previously used to evaluate and compare network representations (Chung et al., 2018; Cohen et al., 2020).
Revised: Manifold Capacity has been previously used to evaluate and compare network representations (Chung et al., 2018; Cohen et al., 2020).

Original: Here, we explore its use as an objective function for self-supervised learning.
Revised: Here, we explore its use as an objective function for self-supervised learning.

Original: For each input image (notated as a vector x b ∈ R D ) we generate K samples from the corresponding manifold by applying a set of random augmentations (each drawn from the same distribution), yielding a manifold sample matrix
Revised: For each input image (notated as a vector x b ∈ R D ) we generate K samples from the corresponding manifold by applying a set of random augmentations (each drawn from the same distribution), yielding a manifold sample matrix

Original: ˜ X b ∈ R D × K .
Revised: ˜ X b ∈ R D × K .

Original: Each augmented image is transformed by a Deep Neural Network, which computes nonlinear function f ( x b ; θ ) parameterized by θ , and the d -dimensional responses are projected onto the unit sphere yielding manifold response matrix Z b ∈ R d × K .
Revised: Each augmented image is transformed by a Deep Neural Network, which computes nonlinear function f ( x b ; θ ) parameterized by θ , and the d -dimensional responses are projected onto the unit sphere yielding manifold response matrix Z b ∈ R d × K .

Original: The centroid c b is approximated by averaging across the columns.
Revised: The centroid c b is approximated by averaging across the columns.

Original: For a set of images { x 1 , ..., x B } we compute normalized response matrices { Z 1 , ..., Z B } and assemble their corresponding centroids into matrix C ∈ R d × B .
Revised: For a set of images { x 1 , ..., x B } we compute normalized response matrices { Z 1 , ..., Z B } and assemble their corresponding centroids into matrix C ∈ R d × B .

Original: Given the responses and their centroids, the loss function is expressed as:
Revised: Given the responses and their centroids, the loss function is expressed as:

Original: [Equation]
Revised: [Equation]

Original: where || · || ∗ indicates the nuclear norm and λ is a tradeoff parameter.
Revised: where || · || ∗ indicates the nuclear norm and λ is a tradeoff parameter.

Original: The first term maximizes the extent of the “centroid manifold” to encourage separability while the second term encourages object manifold compression .
Revised: The first term maximizes the extent of the “centroid manifold” to encourage separability while the second term encourages object manifold compression .

Original: Compression by Maximizing Centroid Nuclear Norm Alone Interestingly, the first term also has a compressive effect.
Revised: Compression by Maximizing Centroid Nuclear Norm Alone Interestingly, the first term also has a compressive effect.

Original: This is because each centroid vector, as a mean of unit vectors, has norm that is linearly related to the average cosine similarity of vectors of said unit vectors.
Revised: This is because each centroid vector, as a mean of unit vectors, has norm that is linearly related to the average cosine similarity of vectors of said unit vectors.

Original: Specifically,
Revised: Specifically,

Original: [Equation]
Revised: [Equation]

Original: Here z b,i denotes the representation of the i th augmentation of x b .
Revised: Here z b,i denotes the representation of the i th augmentation of x b .

Original: Then because the nuclear norm is bounded below by the Frobenius norm (Recht et al., 2010), || C || ∗ ≥
Revised: Then because the nuclear norm is bounded below by the Frobenius norm (Recht et al., 2010), || C || ∗ ≥

Original: [Equation]
Revised: [Equation]

Original: = (cid:113)(cid:80)
Revised: = (cid:113)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: || 2 , maximizing the centroid nuclear norm optimizes an upper bound on the norms of centroid vectors, thus encouraging intra-object manifold similarity.
Revised: || 2 , maximizing the centroid nuclear norm optimizes an upper bound on the norms of centroid vectors, thus encouraging intra-object manifold similarity.

Original: We can gain further insight by considering how the distribution of singular vectors of a matrix depends on the norms and pairwise similarities of the constituent column vectors.
Revised: We can gain further insight by considering how the distribution of singular vectors of a matrix depends on the norms and pairwise similarities of the constituent column vectors.

Original: While no closed form solution exists for the singular values of an arbitrary matrix, the case where the matrix is composed of two column vectors can provide useful intuition.
Revised: While no closed form solution exists for the singular values of an arbitrary matrix, the case where the matrix is composed of two column vectors can provide useful intuition.

Original: If C = [ c 1 , c 2 ] , Z 1 = [ z 1 , 1 , z 1 , 2 ] , Z 2 =
Revised: If C = [ c 1 , c 2 ] , Z 1 = [ z 1 , 1 , z 1 , 2 ] , Z 2 =

Original: [ z 2 , 1 , z 2 , 2 ] , the singular values of C and Z i are:
Revised: [ z 2 , 1 , z 2 , 2 ] , the singular values of C and Z i are:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: So, || σ ( C ) ||
Revised: So, || σ ( C ) ||

Original: 1 = ||
Revised: 1 = ||

Original: C || ∗ is maximized when the centroid vectors have maximal norms (bounded above by 1, since they are the centroids of unit vectors), and are orthogonal to each other.
Revised: C || ∗ is maximized when the centroid vectors have maximal norms (bounded above by 1, since they are the centroids of unit vectors), and are orthogonal to each other.

Original: As we saw above the centroid norms is a linear function of within-manifold similarity.
Revised: As we saw above the centroid norms is a linear function of within-manifold similarity.

Original: Similarly, || σ ( Z i ) ||
Revised: Similarly, || σ ( Z i ) ||

Original: 1 = ||
Revised: 1 = ||

Original: Z
Revised: Z

Original: i || ∗ is minimized when the within-manifold similarity is maximal.
Revised: i || ∗ is minimized when the within-manifold similarity is maximal.

Original: So, both terms in the objective encourage object manifold compression (in the simple case described above the effect is nearly mathematically equivalent).
Revised: So, both terms in the objective encourage object manifold compression (in the simple case described above the effect is nearly mathematically equivalent).

Original: Surprisingly, this implies the first term alone encapsulates both of the key ingredients of a contrastive learning framework, and we do observe that simply maximizing ||
Revised: Surprisingly, this implies the first term alone encapsulates both of the key ingredients of a contrastive learning framework, and we do observe that simply maximizing ||

Original: C || ∗ is sufficient to learn a useful representation.
Revised: C || ∗ is sufficient to learn a useful representation.

Original: This is because the compressive role of “positives” in contrastive learning is carried out by forming the centroid vectors, so the objective is not positive-free even with λ = 0 .
Revised: This is because the compressive role of “positives” in contrastive learning is carried out by forming the centroid vectors, so the objective is not positive-free even with λ = 0 .

Original: For example, if only a single view is used the objective lacks a compressive component and fails to produce a useful representation.
Revised: For example, if only a single view is used the objective lacks a compressive component and fails to produce a useful representation.

Original: We will refer to this version of the objective (with λ = 0 ) as “implicit manifold compression.”
Revised: We will refer to this version of the objective (with λ = 0 ) as “implicit manifold compression.”

Original: In A.5 we demonstrate empirically that this implicit form effectively reduces || Z b || ∗ So, all three factors which determine the manifold capacity (radius, dimensionality, and centroid correlations) can be elegantly expressed in an objective function with a single term, −|| C || ∗ .
Revised: In A.5 we demonstrate empirically that this implicit form effectively reduces || Z b || ∗ So, all three factors which determine the manifold capacity (radius, dimensionality, and centroid correlations) can be elegantly expressed in an objective function with a single term, −|| C || ∗ .

Original: Computational Complexity : The implicit form of our method involves computing a singular value decomposition of C ∈ R d × B which has complexity O ( Bd × min ( B, d )) , where B is the batch size and d is the dimensionality of the output.
Revised: Computational Complexity : The implicit form of our method involves computing a singular value decomposition of C ∈ R d × B which has complexity O ( Bd × min ( B, d )) , where B is the batch size and d is the dimensionality of the output.

Original: By comparison, many popular methods in SSL involve computing the pairwise cosine similarity between members of the batch, which has complexity
Revised: By comparison, many popular methods in SSL involve computing the pairwise cosine similarity between members of the batch, which has complexity

Original: O ( B 2 d ) .
Revised: O ( B 2 d ) .

Original: Additionally, the complexity of our method (in implicit form) is constant with respect to the number of views used (though the feature extraction phase is linear in the number of views), while pairwise similarity metrics will have quadratic complexity with the number of views.
Revised: Additionally, the complexity of our method (in implicit form) is constant with respect to the number of views used (though the feature extraction phase is linear in the number of views), while pairwise similarity metrics will have quadratic complexity with the number of views.

Original:  M ETHODS 3. I MPLEMENTATION D ETAILS
Revised:  M ETHODS 3. I MPLEMENTATION D ETAILS

Original: Architecture.
Revised: Architecture.

Original: For all experiments we use ResNet-50 (He et al., 2016) as a backbone architecture (for variants trained on CIFAR we removed max pooling layers).
Revised: For all experiments we use ResNet-50 (He et al., 2016) as a backbone architecture (for variants trained on CIFAR we removed max pooling layers).

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (2020), we append a small perceptron with one hidden layer to the output of the average pooling layer of the
Revised: (2020), we append a small perceptron with one hidden layer to the output of the average pooling layer of the

Original: ResNet so that z
Revised: ResNet so that z

Original: i
Revised: i

Original: =
Revised: =

Original: g
Revised: g

Original: ( h ( x i )) , where h is the ResNet and g is the MLP.
Revised: ( h ( x i )) , where h is the ResNet and g is the MLP.

Original: Optimization We employ a standard set of augmenetations, taking them directly from (Zbontar et al., 2021).
Revised: Optimization We employ a standard set of augmenetations, taking them directly from (Zbontar et al., 2021).

Original: For CIFAR and STL-10 we used relatively small batch sizes and the Adam optimizer with a fixed learning rate, for ImageNet-100 we used batch size of 2048 for all three methods and the LARS optimizer with linear warmup and cosine decay.
Revised: For CIFAR and STL-10 we used relatively small batch sizes and the Adam optimizer with a fixed learning rate, for ImageNet-100 we used batch size of 2048 for all three methods and the LARS optimizer with linear warmup and cosine decay.

Original: We used a large number of views for small datasets with MMCR (40 for CIFAR and 20 for STL), and a much smaller number of views for ImageNet-100 (4).
Revised: We used a large number of views for small datasets with MMCR (40 for CIFAR and 20 for STL), and a much smaller number of views for ImageNet-100 (we report results for 4 views and 2 views).

Original: For more details on specific hyperparameter settings see A.3.
Revised: For more details on specific hyperparameter settings see A.3.

Original: 3. E VALUATION METHODS
Revised: 3. E VALUATION METHODS

Original: Classification Accuracy.
Revised: Classification Accuracy.

Original: We follow a standard linear evaluation protocol: the parameters of the encoder, h are frozen and a linear function is trained to classify images using the standard supervised cross entropy loss function, see A.6 for details.
Revised: We follow a standard linear evaluation protocol: the parameters of the encoder, h are frozen and a linear function is trained to classify images using the standard supervised cross entropy loss function, see A.6 for details.

Original: We pre-train and evaluate classification performance on CIFAR-10/100, STL-10, and ImageNet-100 (a 100 class subset of the full ImageNet Dataset) (Krizhevsky et al., 2009; Coates et al., 2011; Deng et al., 2009).
Revised: We pre-train and evaluate classification performance on CIFAR-10/100, STL-10, and ImageNet-100 (a 100 class subset of the full ImageNet Dataset) (Krizhevsky et al., 2009; Coates et al., 2011; Deng et al., 2009).

Original: Mean Field Theory Manifold Analysis.
Revised: Mean Field Theory Manifold Analysis.

Original: In manifold capacity theory Chung et al.
Revised: In manifold capacity theory Chung et al.

Original: (2018), the radius and dimensionality of manifolds with arbitrary geometry (i.e., not restricted to elliptical form) are determined by the statistics of particular anchor points within the convex hull of the manifold.
Revised: (2018), the radius and dimensionality of manifolds with arbitrary geometry (i.e., not restricted to elliptical form) are determined by the statistics of particular anchor points within the convex hull of the manifold.

Original: The anchor points of each manifold are those that uniquely specify the maximum margin separating hyperplanes between said manifold and other manifolds.
Revised: The anchor points of each manifold are those that uniquely specify the maximum margin separating hyperplanes between said manifold and other manifolds.

Original: This general measureof capacity is not well-suited fordirect use as an objective function, because computation of the anchor points requires a computationally expensive iterative procedure. But we use it to analyze the capacity of our selfsupervised representation (see Fig. 3).
Revised: Though this measure is not well suited for use as an objective, we use it to analyze the capacity of our self-supervised representation (see Fig. 3).

Original: Subspace Angle.
Revised: Subspace Angle.

Original: Besides measuring the size and dimensionality of individual object manifolds we also wish to characterize the degree of overlap between pairs of manifolds.
Revised: Besides measuring the size and dimensionality of individual object manifolds we also wish to characterize the degree of overlap between pairs of manifolds.

Original: For this, we measure the angle between their subspaces (Knyazev & Argentati, 2002).
Revised: For this, we measure the angle between their subspaces (Knyazev & Argentati, 2002).

Original: Shared Variance.
Revised: Shared Variance.

Original: Object manifolds will generally have a lower intrinsic dimensionality then the space they are embedded in.
Revised: Object manifolds will generally have a lower intrinsic dimensionality then the space they are embedded in.

Original: Therefore, the data will have low variance along several of the principal vectors used to calculate the set of subspace angles, and so many of the principal angles will have little meaning.
Revised: Therefore, the data will have low variance along several of the principal vectors used to calculate the set of subspace angles, and so many of the principal angles will have little meaning.

Original: To address this limitation we also compute the shared variance between the linear subspaces that contain object manifolds.
Revised: To address this limitation we also compute the shared variance between the linear subspaces that contain object manifolds.

Original: Adversarial Robustness.
Revised: Adversarial Robustness.

Original: Once a linear classifier has been trained on top of the representation, the end-to-end system can be subjected to standard adversarial attacks.
Revised: Once a linear classifier has been trained on top of the representation, the end-to-end system can be subjected to standard adversarial attacks.

Original: We 
Revised: We evaluate adversarial robustness under ℓ ∞

Original: [Table]
Revised: [Table]

Original: Baseline Models We chose SimCLR (Chen et al., 2020) and Barlow Twins (Zbontar et al., 2021) for primary comparisons with our model.
Revised: Baseline Models We chose SimCLR (Chen et al., 2020) and Barlow Twins (Zbontar et al. [Table] 2021) for primary comparisons with our model.

Original: These methods are recent, simple to implement, and achieve performance comparable to state of the art for self-supervised learning on the Imagenet-1k dataset.
Revised: These methods are recent, simple to implement, and achieve performance comparable to state of the art for self-supervised learning on the Imagenet-1k dataset.

Original: 3. P ERFORMANCE
Revised: 3. P ERFORMANCE

Original: Figure 2 details the evolution of the representation during the course of training.
Revised: Figure 2 details the evolution of the representation during the course of training.

Original: The centroid nuclear norm (Fig. 2b) increases steadily as the centroids become increasingly orthogonal to each other (Fig. 2c) and grow in norm (Fig. 2d).
Revised: The centroid nuclear norm (Fig. 2b) increases steadily as the centroids become increasingly orthogonal to each other (Fig. 2c) and grow in norm (Fig. 2d).

Original: The compression of individual augmentation manifolds is reflected in Fig.
Revised: The compression of individual augmentation manifolds is reflected in Fig.

Original: 2a.
Revised: 2a.

Original: The downstream classification accuracies are reported in Table 1.
Revised: The downstream classification accuracies are reported in Table 1.

Original: Subsequent analyses are carried out on the CIFAR-10 dataset using the implicit MMCR model as it achieved the highest performance.
Revised: Subsequent analyses are carried out on the CIFAR-10 dataset using the implicit MMCR model as it achieved the highest performance.

Original: 3. R EPRESENTATION GEOMETRIC ANALYSIS
Revised: 3. R EPRESENTATION GEOMETRIC ANALYSIS

Original: In figure 3 we show that our representation, which is optimized using an objective that assumes elliptical manifold geometry, nevertheless yields representations with high mean field manifold capacity (relative to baseline methods).
Revised: In figure 3 we show that our representation, which is optimized using an objective that assumes elliptical manifold geometry, nevertheless yields representations with high mean field manifold capacity (relative to baseline methods).

Original: For completeness we also analyzed the geometries of class manifolds, whose points are the representations of different examples from the same class.
Revised: For completeness we also analyzed the geometries of class manifolds, whose points are the representations of different examples from the same class.

Original: This analysis provided further evidence that learning to maximize augmentation manifold capacity compresses and separates class manifolds, leading to a useful representation.
Revised: This analysis provided further evidence that learning to maximize augmentation manifold capacity compresses and separates class manifolds, leading to a useful representation.

Original: Interestingly MMCRs seem to use a different strategy than the baseline methods to increase the capacity, namely MMCRs product class/augmentation manifolds with larger radii, but lower dimensionality (Fig. 3)
Revised: Interestingly MMCRs seem to use a different strategy than the baseline methods to increase the capacity, namely MMCRs product class/augmentation manifolds with larger radii, but lower dimensionality (Fig. 3)

Original: 3. E MERGENCE OF NEURAL MANIFOLDS VIA GRADIENT COHERENCE
Revised: 3. E MERGENCE OF NEURAL MANIFOLDS VIA GRADIENT COHERENCE

Original: We hypothesize the class separability in MMCRs arises because augmentation manifolds corresponding to examples from the same class are optimally compressed by more similar transformations than those stemming from distinct classes.
Revised: We hypothesize the class separability in MMCRs arises because augmentation manifolds corresponding to examples from the same class are optimally compressed by more similar transformations than those stemming from distinct classes.

Original: To investigate this empirically, we evaluate the gradient of the objective function for inputs belonging to the same class.
Revised: To investigate this empirically, we evaluate the gradient of the objective function for inputs belonging to the same class.

Original: We can then check whether gradients obtained from (distinct) batches of the same class are more similar to each other than those obtained from different classes, which would suggest that the strategy for compressing augmentation manifolds from the same class are relatively similar to each other.
Revised: We can then check whether gradients obtained from (distinct) batches of the same class are more similar to each other than those obtained from different classes, which would suggest that the strategy for compressing augmentation manifolds from the same class are relatively similar to each other.

Original: Figure 4 demonstrates that this is the case: within class gradient coherence, as measured by cosine similarity, is consistently higher than across class coherence across both training epochs and model hierarchy.
Revised: Figure 4 demonstrates that this is the case: within class gradient coherence, as measured by cosine similarity, is consistently higher than across class coherence across both training epochs and model hierarchy

Original: 3. M ANIFOLD SUBSPACE ALIGNMENT
Revised: 3. M ANIFOLD SUBSPACE ALIGNMENT

Original: Within-class gradient coherence constitutes a plausible mechanistic explanation for the emergence of class separability, but it does not explain why members of the same class share similar compression strategies.
Revised: Within-class gradient coherence constitutes a plausible mechanistic explanation for the emergence of class separability, but it does not explain why members of the same class share similar compression strategies.

Original: To begin answering this question we examine the geometric properties of augmentation manifolds in the pixel domain.
Revised: To begin answering this question we examine the geometric properties of augmentation manifolds in the pixel domain.

Original: Here we observe small but measurable differences between the distributions of within-class similarity and across-class similarity, as demonstrated in the top row of figure 5.
Revised: Here we observe small but measurable differences between the distributions of within-class similarity and across-class similarity, as demonstrated in the top row of figure 5.

Original: The subtle difference in the geometric properties of augmentation manifolds in the pixel domain in turn leads to the increased gradient coherence observed above, which over training leads to a representation that rearranges and reshapes augmentation manifolds from the same class in a similar fashion (bottom row of figure 5), thus allowing for the linear separation of classes.
Revised: The subtle difference in the geometric properties of augmentation manifolds in the pixel domain in turn leads to the increased gradient coherence observed above, which over training leads to a representation that rearranges and reshapes augmentation manifolds from the same class in a similar fashion (bottom row of figure 5), thus allowing for the linear separation of classes.

Original: Not only are centroids of same-class-manifolds in more similar regions of the representation space than those coming from distinct classes (Fig 5 third column bottom row) but additionally same-class-manifolds have more similar shapes to each other (Fig 5 bottom row columns 1 and 2).
Revised: Not only are centroids of same-class-manifolds in more similar regions of the representation space than those coming from distinct classes (Fig 5 third column bottom row) but additionally same-class-manifolds have more similar shapes to each other (Fig 5 bottom row columns 1 and 2).

Original: We next ask how the representation learned according to the MMCR objective differs from those optimized for other self supervised loss functions.
Revised: We next ask how the representation learned according to the MMCR objective differs from those optimized for other self supervised loss functions.

Original: While MMCR encourages centroids to be as close to orthogonal to each other, the InfoNCE loss employed in Chen et al.
Revised: While MMCR encourages centroids to be as close to orthogonal to each other, the InfoNCE loss employed in Chen et al.

Original: (2020) benefits when negative pairs are as dissimilar as possible, which is achieved when the two points lie in opposite regions of the same subspace rather than in distinct subspaces.
Revised: (2020) benefits when negative pairs are as dissimilar as possible, which is achieved when the two points lie in opposite regions of the same subspace rather than in distinct subspaces.

Original: The Barlow Twins (Zbontar et al., 2021) loss is not an explicit function of feature vector similarities, but instead encourages individual features to be correlated across the batch dimension and distinct features to be uncorrelated.
Revised: The Barlow Twins (Zbontar et al., 2021) loss is not an explicit function of feature vector similarities, but instead encourages individual features to be correlated across the batch dimension and distinct features to be uncorrelated.

Original: In 6 we demonstrate that these intuitions are borne out empirically: the MMCR representation produces augmentation manifold centroids that are significantly less similar to each other than the two baseline methods.
Revised: In 6 we demonstrate that these intuitions are borne out empirically: the MMCR representation produces augmentation manifold centroids that are significantly less similar to each other than the two baseline methods.

Original: 3. A DVERSARIAL R OBUSTNESS
Revised: 3. A DVERSARIAL R OBUSTNESS

Original: Previous works that used similar geometrically motivated loss functions such as the orthogonal low-rank embedding (Lezama et al., 2018) and maximal coding rate reduction Yu et al.
Revised: Previous works that used similar geometrically motivated loss functions such as the orthogonal low-rank embedding (Lezama et al., 2018) and maximal coding rate reduction Yu et al.

Original: (2020) have reported the resulting representations have increased inter-class margins and are more robust to label noise.
Revised: (2020) have reported the resulting representations have increased inter-class margins and are more robust to label noise.

Original: We therefore tested whether the increased tendency to orthogonalize in MMCR models leads to any benefit in terms of adversarial robustness.
Revised: We therefore tested whether the increased tendency to orthogonalize in MMCR models leads to any benefit in terms of adversarial robustness.

Original: In 7 we show that the MMCR model (and attached classifier) is indeed more robust than either Barlow Twins or SimCLR trained models against PGD attacks with a range of strengths (Madry et al., 2018).
Revised: In 7 we show that the MMCR model (and attached classifier) is indeed more robust than either Barlow Twins or SimCLR trained models against PGD attacks with a range of strengths (Madry et al., 2018).

Original: We found similar results using the stronger 
Revised: We found similar results using the stronger ditional details can be found in A.

Original: AutoAttack protocol (Croce & Hein, 2020), see A.7.
Revised: AutoAttack protocol (Croce & Hein, 2020), see A.7.

Original: Note that Barlow Twins models seem to beditional details can be found in A more robust than those arising from SimCLR.
Revised: Note that Barlow Twins models seem to be more robust than those arising from SimCLR.

Original: We speculate that this is a result of the decorrelation (rather than anti-correlation) encouraged by the Barlow Twins objective.
Revised: We speculate that this is a result of the decorrelation (rather than anti-correlation) encouraged by the Barlow Twins objective.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Previous work has proposed manifold capacity theory as a means of understanding the performance of deep neural networks trained using traditional supervised methods (Cohen et al., 2020).
Revised: Previous work has proposed manifold capacity theory as a means of understanding the performance of deep neural networks trained using traditional supervised methods (Cohen et al., 2020).

Original: Given that manifold capacity theory is particularly well suited for analyzing high dimensional feature spaces, while many SSL methods are motivated by information criterion, we wondered whether manifold capacity could serve as a useful objective function.
Revised: Given that manifold capacity theory is particularly well suited for analyzing high dimensional feature spaces, while many SSL methods are motivated by information criterion, we wondered whether manifold capacity could serve as a useful objective function.

Original: By approximating the manifold geometries as elliptical, which significantly reduces the computation required to calculate the geometric properties that dictate capacity, we were able to achieve exactly this.
Revised: By approximating the manifold geometries as elliptical, which significantly reduces the computation required to calculate the geometric properties that dictate capacity, we were able to achieve exactly this.

Original: Although representational manifold geometries are generally not elliptical, we have demonstrated that this approximation can nonetheless produce a useful learning signal, and leads to networks with high manifold capacity (Fig. 3).
Revised: Although representational manifold geometries are generally not elliptical, we have demonstrated that this approximation can nonetheless produce a useful learning signal, and leads to networks with high manifold capacity (Fig. 3).

Original: Nevertheless, it would be interesting to consider other reductions of the mean field manifold capacity theory that can capture non-elliptical structure of individual manifolds, perhaps by computing higher order statistics of constituent points.
Revised: Nevertheless, it would be interesting to consider other reductions of the mean field manifold capacity theory that can capture non-elliptical structure of individual manifolds, perhaps by computing higher order statistics of constituent points.

Original: We demonstrate the utility of our approach by applying the method to three small datasets of unlabeled images and demonstrating that the resulting representations can yield similar downstream task performance to two baselines.
Revised: We demonstrate the utility of our approach by applying the method to three small datasets of unlabeled images and demonstrating that the resulting representations can yield similar downstream task performance to two baselines.

Original: Furthermore we conducted a gradient based analysis to begin to understand why the self-supervised learning signal is capable of producing useful representations.
Revised: Furthermore we conducted a gradient based analysis to begin to understand why the self-supervised learning signal is capable of producing useful representations.

Original: Finally, motivated by an empirical exploration of the geometrical differences between the representations produced by the three considered methods, we demonstrate that MMCRs can offer improved robustness to adversarial attacks.
Revised: Finally, motivated by an empirical exploration of the geometrical differences between the representations produced by the three considered methods, we demonstrate that MMCRs can offer improved robustness to adversarial attacks.

Original: Additionally, we leveraged manifold capacity analysis in its full generality to gain insight into the geometry of the MMCR.
Revised: Additionally, we leveraged manifold capacity analysis in its full generality to gain insight into the geometry of the MMCR.

Original: Intriguingly, our method produces augmentation and class manifolds with lower dimensionality but larger radius than either Barlow Twins or SimCLR (Fig. 3).
Revised: Intriguingly, our method produces augmentation and class manifolds with lower dimensionality but larger radius than either Barlow Twins or SimCLR (Fig. 3).

Original: Future work will seek to understand why this is the case, but more generally this suggests that capacity analysis can be a fruitful way to understand the different encoding strategies encouraged by various
Revised: Future work will seek to understand why this is the case, but more generally this suggests that capacity analysis can be a fruitful way to understand the different encoding strategies encouraged by various

Original: SSL paradigms.
Revised: SSL paradigms.

Original: Another factor that distinguishs MMCRs from other models is a tendency to orthogonalize augmentation manifold centroids and thus form a representation that is globally high dimensional.
Revised: Another factor that distinguishs MMCRs from other models is a tendency to orthogonalize augmentation manifold centroids and thus form a representation that is globally high dimensional.

Original: Given that the recent observations that the representations in visual cortex and high performing models of visual cortex are surprisingly high dimensional (Stringer et al., 2019; Elmoznino & Bonner, 2022), it may be interesting to test how well MMCRs predict neural activity.
Revised: Given that the recent observations that the representations in visual cortex and high performing models of visual cortex are surprisingly high dimensional (Stringer et al., 2019; Elmoznino & Bonner, 2022), it may be interesting to test how well MMCRs predict neural activity.

Original: In this study, we introduced one specific model of learning using metrics based on a specific theory of representations: self-supervised learning via maximizing neural manifold capacity.
Revised: In this study, we introduced one specific model of learning using metrics based on a specific theory of representations: self-supervised learning via maximizing neural manifold capacity.

Original: More broadly, this work demonstrates the efficacy of a learning objective solely based on representations (rather than relying on classifiers), and motivates new directions of research where representation-level phenomena in neurobiology inform the design choice in neural networks.
Revised: More broadly, this work demonstrates the efficacy of a learning objective solely based on representations (rather than relying on classifiers), and motivates new directions of research where representation-level phenomena in neurobiology inform the design choice in neural networks.

Original: With recent trends in neuroscience focused on representation geometric observations in neural data, we hope that this work lays foundations for future studies in learning based on representation geometries informed by new discoveries in neuroscience.
Revised: With recent trends in neuroscience focused on representation geometric observations in neural data, we hope that this work lays foundations for future studies in learning based on representation geometries informed by new discoveries in neuroscience.

Original: A A PPENDIX A. P YTORCH S TYLE P SEUDOCODE FOR MMCR
Revised: A A PPENDIX A. P YTORCH S TYLE P SEUDOCODE FOR MMCR

Original: 1 # h: encoder 2 # g: projection head 3 # B: batch size 4 # K: number of augmentations 5 # D: projector output dimensionality 6 # 7 # lmbda: trade-off parameter 8 for x in loader: # K randomly augmented views x = multi_augment(x)
Revised: 1 # h: encoder 2 # g: projection head 3 # B: batch size 4 # K: number of augmentations 5 # D: projector output dimensionality 6 # 7 # lmbda: trade-off parameter 8 for x in loader: # K randomly augmented views x = multi_augment(x)

Original: # B x K x H x W 11# push through encoder and projector z = g(h(x))
Revised: # B x K x H x W 11# push through encoder and projector z = g(h(x))

Original: # B x K x D 14# project onto unitsphere z = normalize(z, dim=-1) 17# calculate centroids (mean over augmentation axis)
Revised: # B x K x D 14# project onto unitsphere z = normalize(z, dim=-1) 17# calculate centroids (mean over augmentation axis)

Original: c
Revised: c

Original: = z.mean(dim=1) # B x D 20# calculate singular values
Revised: = z.mean(dim=1) # B x D 20# calculate singular values

Original: U_z, S_z, V_z = svd(z) # batch svd
Revised: U_z, S_z, V_z = svd(z) # batch svd

Original: U_c, S_c, V_c = svd(c)
Revised: U_c, S_c, V_c = svd(c)

Original: 24# calculate loss loss =
Revised: 24# calculate loss loss =

Original: -1.0 * sum(S_c) + lmbda * sum(S_z) / B 27# backward pass and optimization step loss.backward() optim.step()
Revised: -1.0 * sum(S_c) + lmbda * sum(S_z) / B 27# backward pass and optimization step loss.backward() optim.step()

Original: A. 
Revised: A.

Original: M EAN F IELD
Revised: M EAN F IELD

Original: ACKGROUND I NFORMATION
Revised: ACKGROUND I NFORMATION

Original: Mean Field Theory Recall the problem setting for manifold capacity analysis: given a set of P manifolds embedded in a feature space of dimensionality D , each assigned a random binary class label Chung et al.
Revised: Mean Field Theory Recall the problem setting for manifold capacity analysis: given a set of P manifolds embedded in a feature space of dimensionality D , each assigned a random binary class label Chung et al.

Original: Manifold capacity theory is concerned with the question: what is the largest value of P D such that there exists (with high probability) a hyperplane separating the two classes?
Revised: Manifold capacity theory is concerned with the question: what is the largest value of P D such that there exists (with high probability) a hyperplane separating the two classes?

Original: In the thermodynamic limit, where P, D → ∞ but PD remains finite, the inverse capacity can be written exactly,
Revised: In the thermodynamic limit, where P, D → ∞ but PD remains finite, the inverse capacity can be written exactly,

Original: [Equation]
Revised: [Equation]

Original: where, F ( ⃗T ) = min ⃗V (cid:110) ∥
Revised: where, F ( ⃗T ) = min ⃗V (cid:110) ∥

Original: ⃗V − ⃗T ∥ 2 | g S ( ⃗V ) ≥ 0 (cid:111) , S is the set defining the manifold geometry (i.e. the set of vectors ⃗S that are points on an individual manifold), ⃗T are random vectors drawn from a white multivariate Gaussian distribution, and g S ( ⃗V )
Revised: ⃗V − ⃗T ∥ 2 | g S ( ⃗V ) ≥ 0 (cid:111) , S is the set defining the manifold geometry (i.e. the set of vectors ⃗S that are points on an individual manifold), ⃗T are random vectors drawn from a white multivariate Gaussian distribution, and g S ( ⃗V )

Original: = min ⃗S { ⃗V · ⃗S | ⃗S ∈ S} , is the concave support function.
Revised: = min ⃗S { ⃗V · ⃗S | ⃗S ∈ S} , is the concave support function.

Original: The KKT equations for this convex optimization problem are:
Revised: The KKT equations for this convex optimization problem are:

Original: [Equation]
Revised: [Equation]

Original: , where ˜ S ( ⃗T ) is a subgradient of the support function.
Revised: , where ˜ S ( ⃗T ) is a subgradient of the support function.

Original: When the support function is differentiable, the subgradient is unique and equal to the gradient,
Revised: When the support function is differentiable, the subgradient is unique and equal to the gradient,

Original: [Equation]
Revised: [Equation]

Original: ˜ S ( ⃗T ) is the unique point in the convex hull of S that satisfies the first KKT equation, and is called the “anchor point” for S induced by the random vector ⃗T .
Revised: ˜ S ( ⃗T ) is the unique point in the convex hull of S that satisfies the first KKT equation, and is called the “anchor point” for S induced by the random vector ⃗T .

Original: Equivalent Interpretation of Anchor Points For a given dichotomy (random binary class labelling)
Revised: Equivalent Interpretation of Anchor Points For a given dichotomy (random binary class labelling)

Original: the weight vector of the maximum margin separating hyperplane can be decomposed into a sum of at most P vectors, with each manifold contributing a single vector, which lies within the convex hull of the manifold.
Revised: the weight vector of the maximum margin separating hyperplane can be decomposed into a sum of at most P vectors, with each manifold contributing a single vector, which lies within the convex hull of the manifold.

Original: The position of said point point is a function of the manifolds position relative to all of the other manifolds in the space and depends on the particular set of random labels.
Revised: The position of said point point is a function of the manifolds position relative to all of the other manifolds in the space and depends on the particular set of random labels.

Original: Thus there exists a distribution of separating-hyperplane-determining-points for each individual manifold.
Revised: Thus there exists a distribution of separating-hyperplane-determining-points for each individual manifold.

Original: Using the “cavity” method it can be shown that these points are none other than the anchor points that are involved in solving the optimization problem described above Gerl & Krey (1994).
Revised: Using the “cavity” method it can be shown that these points are none other than the anchor points that are involved in solving the optimization problem described above Gerl & Krey (1994).

Original: Numerical Solution To solve the mean field equations numerically, one samples several random
Revised: Numerical Solution To solve the mean field equations numerically, one samples several random

Original: Gaussian vectors ⃗T , and then for each ⃗T , ⃗V and ⃗S are determined by solving the quadratic programming program given above.
Revised: Gaussian vectors ⃗T , and then for each ⃗T , ⃗V and ⃗S are determined by solving the quadratic programming program given above.

Original: The capacity is then estimated as the mean value of F or the samples ⃗T .
Revised: The capacity is then estimated as the mean value of F or the samples ⃗T .

Original: Manifold Geometries
Revised: Manifold Geometries

Original: The way the capacity varies in terms of the statistics of the anchor points can be simplified by introducing two key quantities, the manifold radius R M and manifold dimensionality R M :
Revised: The way the capacity varies in terms of the statistics of the anchor points can be simplified by introducing two key quantities, the manifold radius R M and manifold dimensionality R M :

Original: [Equation]
Revised: [Equation]

Original: where ˆ S ( ⃗T ) is a unit-vector in the direction of the anchor point ˜ S .
Revised: where ˆ S ( ⃗T ) is a unit-vector in the direction of the anchor point ˜ S .

Original: In particular as discussed in the main text, the manifold capacity can be approximated by ϕ ( R M √ D M ) where ϕ is a monotonically decreasing function.
Revised: In particular as discussed in the main text, the manifold capacity can be approximated by ϕ ( R M √ D M ) where ϕ is a monotonically decreasing function.

Original: Elliptical Geometries
Revised: Elliptical Geometries

Original: In the case where the manifolds exhibit elliptical symmetries, the manifold radius and dimensionality can be written in terms of the eigenvalues of the covariance matrix of the anchor points:
Revised: In the case where the manifolds exhibit elliptical symmetries, the manifold radius and dimensionality can be written in terms of the eigenvalues of the covariance matrix of the anchor points:

Original: [Equation]
Revised: [Equation]

Original: So, in this case R M is the total variability of the anchor points, and D M is a generalized participation ratio of the anchor point covariance, a well known soft measure of dimensionality.
Revised: So, in this case R M is the total variability of the anchor points, and D M is a generalized participation ratio of the anchor point covariance, a well known soft measure of dimensionality.

Original: A. A DDITIONAL P RE - TRAINING INFORMATION
Revised: A. A DDITIONAL P RE - TRAINING INFORMATION

Original: Settings for CIFAR/STL-10 We take the parameters of each augmentation directly from Zbontar et al.
Revised: Settings for CIFAR/STL-10 We take the parameters of each augmentation directly from Zbontar et al.

Original: (2021), but
Revised: (2021), but

Original: for these lower resolution images we omitted Gaussian blurring and solarization augmentations.
Revised: for these lower resolution images we omitted Gaussian blurring and solarization augmentations.

Original: All models were trained for 500 epochs using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 1 e − 3 and weight decay of 1 e − 6 .
Revised: All models were trained for 500 epochs using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 1 e − 3 and weight decay of 1 e − 6 .

Original: For all three methods we used a one hidden layer MLP with hidden dimension of 512 and output dimension of 128 for the projector head g .
Revised: For all three methods we used a one hidden layer MLP with hidden dimension of 512 and output dimension of 128 for the projector head g .

Original: We swept batch size for each method and chose the one that resulted in the highest downstream task performance.
Revised: We swept batch size for each method and chose the one that resulted in the highest downstream task performance.

Original: For both SimCLR and Barlow Twins we found that a batch size of 128 was optimal (among 32, 64, 128, 256, and 512) for all 3 datasets.
Revised: For both SimCLR and Barlow Twins we found that a batch size of 128 was optimal (among 32, 64, 128, 256, and 512) for all 3 datasets.

Original: For MMCR there is a trade-off between batch size and the number of augmentations used, and the optimal value of that trade-off is highly dataset dependent.
Revised: For MMCR there is a trade-off between batch size and the number of augmentations used, and the optimal value of that trade-off is highly dataset dependent.

Original: For CIFAR-10 and CIFAR-100 we used batch size of 32 and 40 views, and for
Revised: For CIFAR-10 and CIFAR-100 we used batch size of 32 and 40 views, and for

Original: STL-10 we used a batch of 64 with 20 views For Barlow Twins we used λ = 1128 which normalizes for the number of elements in the on-diagonal and off-diagonal terms in the loss.
Revised: STL-10 we used a batch of 64 with 20 views For Barlow Twins we used λ = 1128 which normalizes for the number of elements in the on-diagonal and off-diagonal terms in the loss.

Original: For SimCLR we used the recommended setting of τ = 0 .
Revised: For SimCLR we used the recommended setting of τ = 0 .

Original: The overall performance of both baseline methods (and likely MMCR as well) could be increased with a more thorough hyperparameter search and by employing methodology that more closely matches the original works.
Revised: The overall performance of both baseline methods (and likely MMCR as well) could be increased with a more thorough hyperparameter search and by employing methodology that more closely matches the original works.

Original: For example, both methods would likely benefit from the combination of larger batch size, the use of the LARS optimizer (which is designed for large batch optimization), a learning rate scheduler consisting of linear warmup followed by cosine annealing, longer training, and the use of more diverse augmentations (i.e. including solarization and gaussian blur).
Revised: For example, both methods would likely benefit from the combination of larger batch size, the use of the LARS optimizer (which is designed for large batch optimization), a learning rate scheduler consisting of linear warmup followed by cosine annealing, longer training, and the use of more diverse augmentations (i.e. including solarization and gaussian blur).

Original: Additionally Barlow Twins reports that the representation can benefit from using a much larger projector network than we use.
Revised: Additionally Barlow Twins reports that the representation can benefit from using a much larger projector network than we use.

Original: Because our goal was primarily to demonstrate that MMCR can produce representations that are comparable to these baselines rather than to produce state-of-the-art results on small scale datasets we opted for simplifications wherever possible (using off the shelf Adam for optimization with a fixed learning rate, and fixing architectural hyperparameters like the projector dimensionality).
Revised: Because our goal was primarily to demonstrate that MMCR can produce representations that are comparable to these baselines rather than to produce state-of-the-art results on small scale datasets we opted for simplifications wherever possible (using off the shelf Adam for optimization with a fixed learning rate, and fixing architectural hyperparameters like the projector dimensionality).

Original: Settings for ImageNet-

Revised: Settings for ImageNet-

Original: For ImageNet we more closely match the pre-training procedures of previous works.
Revised: For ImageNet we more closely match the pre-training procedures of previous works.

Original: We use a batch size of 2048 and a smaller number of views for MMCR (4), and also use the full suite of augmentations from Zbontar et al.
Revised: We use a batch size of 2048 and a smaller number of views for MMCR (4), and also use the full suite of augmentations from Zbontar et al.

Original: For the sake of efficiency we train for a reduced number of epochs (200).
Revised: For the sake of efficiency we train for a reduced number of epochs (200).

Original: For MMCR and SimCLR we modified the projector hidden dimensionality to be 4096 for the projector head, following the original work Chen et al.
Revised: For MMCR and SimCLR we modified the projector hidden dimensionality to be 4096 for the projector head, following the original work Chen et al.

Original: For
Revised: For

Original: Barlow Twins we used the recommended 2-layer MLP with hidden and output dimensions of 8192, and set λ = 5 e − 3 , however these hyperparameters were optimal for the full ImageNet dataset, and not neccesarrily for ImageNet-100.
Revised: Barlow Twins we used the recommended 2-layer MLP with hidden and output dimensions of 8192, and set λ = 5 e − 3 , however these hyperparameters were optimal for the full ImageNet dataset, and not neccesarrily for ImageNet-100.

Original: We were unable to achieve better downstream performance using a ResNet-50 backbone than what has previously been reported in the literature for this dataset with a ResNet-18 backbone, therefore we report the ResNet-18 performance reported in (da Costa et al., 2022).
Revised: We were unable to achieve better downstream performance using a ResNet-50 backbone than what has previously been reported in the literature for this dataset with a ResNet-18 backbone, therefore we report the ResNet-18 performance reported in (da Costa et al., 2022).

Original: For SimCLR we use τ = 0 .
Revised: For SimCLR we use τ = 0 .

Original: 1 which is the recommended setting for larger batch sizes.
Revised: 1 which is the recommended setting for larger batch sizes.

Original: A. D ETAILS OF R EPRESENTATIONAL A NALYSES
Revised: A. D ETAILS OF R EPRESENTATIONAL A NALYSES

Original: Manifold Capacity Analysis For each pre-trained model, we extract layer activations across the
Revised: Manifold Capacity Analysis For each pre-trained model, we extract layer activations across the

Original: ResNet hierarchy after a forward pass of a set of images.
Revised: ResNet hierarchy after a forward pass of a set of images.

Original: For class manifold analysis, the set of images contain 10 classes, where each class has 100 examples.
Revised: For class manifold analysis, the set of images contain 10 classes, where each class has 100 examples.

Original: Augmentation manifolds instead have 100 exemplars with 100 examples each.
Revised: Augmentation manifolds instead have 100 exemplars with 100 examples each.

Original: Following Cohen et al.
Revised: Following Cohen et al.

Original: (2020), we take activations from all convolutional layers in ResNet-50 after a ReLU non-linearity.
Revised: (2020), we take activations from all convolutional layers in ResNet-50 after a ReLU non-linearity.

Original: The specific extracted layers highlighted in bold fonts are given by Table 2.
Revised: The specific extracted layers highlighted in bold fonts are given by Table 2.

Original: The final analysis results are averaged over five data samplings with different random seeds and random projections of intermediate features to lowerdimension spaces (default 5000 dimensions).
Revised: The final analysis results are averaged over five data samplings with different random seeds and random projections of intermediate features to lowerdimension spaces (default 5000 dimensions).

Original: [Table]
Revised: [Table]

Original: Gradient Coherence Analysis In Fig 4, for each of the classes of CIFAR-10, we generatebatches of 32 augmentation manifolds of samples from a specific class (with 40 augmentations each).
Revised: Gradient Coherence Analysis In Fig 4, for each of the classes of CIFAR-10, we generatebatches of 32 augmentation manifolds of samples from a specific class (with 40 augmentations each).

Original: We then measure the gradient of the loss function for each batch during different stages of training
Revised: We then measure the gradient of the loss function for each batch during different stages of training

Original: [Table]
Revised: [Table]

Original: and compute the cosine similarity between every pair of gradients.
Revised: and compute the cosine similarity between every pair of gradients.

Original: Across all stages of training the mean cosine similarity between gradients generated from batches of the same class is larger than those from distinct classes (left column).
Revised: Across all stages of training the mean cosine similarity between gradients generated from batches of the same class is larger than those from distinct classes (left column).

Original: This observation remains true when isolating the gradients of parameters from different stages of in the resnet-50 hierarchy (center and right columns, respectively).
Revised: This observation remains true when isolating the gradients of parameters from different stages of in the resnet-50 hierarchy (center and right columns, respectively).

Original: Manifold Subspace Alignment For Fig.
Revised: Manifold Subspace Alignment For Fig.

Original: 5 we generated 100 samples from the augmentation manfiolds of 500 images in the CIFAR-10 dataset.
Revised: 5 we generated 100 samples from the augmentation manfiolds of 500 images in the CIFAR-10 dataset.

Original: We then measure the mean subspsace angle (left column), fraction of shared variance (middle column) and centroid cosine similarity between each pair of manifolds.
Revised: We then measure the mean subspsace angle (left column), fraction of shared variance (middle column) and centroid cosine similarity between each pair of manifolds.

Original: The same procedure was used for generating the data for 6.
Revised: The same procedure was used for generating the data for 6.

Original: A. I MPLICIT MMCR E FFECTIVELY R EDUCES A UGMENTATION M ANIFOLD N UCLEAR N ORM
Revised: A. I MPLICIT MMCR E FFECTIVELY R EDUCES A UGMENTATION M ANIFOLD N UCLEAR N ORM

Original: To test whether or not implicit manifold compression actually reduces the mean augmentation manifold nuclear norm, we can vary the value of λ .
Revised: To test whether or not implicit manifold compression actually reduces the mean augmentation manifold nuclear norm, we can vary the value of λ .

Original: Below we see the evolution of bother terms of the loss for several different values of lambda during training on CIFAR-10.
Revised: Below we see the evolution of bother terms of the loss for several different values of lambda during training on CIFAR-10.

Original: For these experiments the batch size was 64 and the number of augmentations per image was 4.0.
Revised: For these experiments the batch size was 64 and the number of augmentations per image was 4.0.

Original: As shown in 8, the level of compression of individual manifolds is nearly the same across all values of the tradeoff parameter tested.
Revised: As shown in 8, the level of compression of individual manifolds is nearly the same across all values of the tradeoff parameter tested.

Original: A. C LASSIFICATION E VALUATION P ROCEDURE
Revised: A. C LASSIFICATION E VALUATION P ROCEDURE

Original: During pre-training all models were monitored with a k-nearest neighbor classifier (k=200) and checkpointed every 5 epochs.
Revised: During pre-training all models were monitored with a k-nearest neighbor classifier (k=200) and checkpointed every 5 epochs.

Original: After pre-training, we trained linear classifiers on all checkpoints whose monitor accuracy was within 1% of the highest observed accuracy, and select the model that achieves the highest linear classification accuracy.
Revised: After pre-training, we trained linear classifiers on all checkpoints whose monitor accuracy was within 1% of the highest observed accuracy, and select the model that achieves the highest linear classification accuracy.

Original: Linear classifiers were trained using the Adam optimizer with batch size of 1024 and an initial learning rate of 0.1, which decayed according to a cosine scheduler over the course of 50 epochs.
Revised: Linear classifiers were trained using the Adam optimizer with batch size of 1024 and an initial learning rate of 0.1, which decayed according to a cosine scheduler over the course of 50 epochs.

Original: For the linear classifier training, at train time we use the same set of augmentations as during unsupervised pretraining, at test time we only use center cropping and random horizontal flipping.
Revised: For the linear classifier training, at train time we use the same set of augmentations as during unsupervised pretraining, at test time we only use center cropping and random horizontal flipping.

Original: A. A DDITIONAL D ETAILS FOR A DVERSARIAL R OBUSTNESS A NALYSES
Revised: A. A DDITIONAL D ETAILS FOR A DVERSARIAL R OBUSTNESS A NALYSES

Original: In Figure 7, we choose 50 iterations for the PGD ℓ ∞ -norm since it guarantees a robust accuracy value not far away from asymptotically larger PGD attack iterations (Madry et al., 2018; Croce &
Revised: In Figure 7, we choose 50 iterations for the PGD ℓ ∞ -norm since it guarantees a robust accuracy value not far away from asymptotically larger PGD attack iterations (Madry et al., 2018; Croce &

Original: Hein, 2020).
Revised: Hein, 2020).

Original: In our experiment, we have shown that the PGD attack indeed converges in a similar fashion (See Figure 9).
Revised: In our experiment, we have shown that the PGD attack indeed converges in a similar fashion (See Figure 9).

Original: However, the robust accuracy for MMCR tends to converge at larger PGD attack iterations.
Revised: However, the robust accuracy for MMCR tends to converge at larger PGD attack iterations.

Original: We therefore also analyzed the robust accuracies for the three SSL methods with varying iterations across all epsilon attack strength.
Revised: We therefore also analyzed the robust accuracies for the three SSL methods with varying iterations across all epsilon attack strength.

Original: Figure 10 shows MMCR exhibits a significantly higher robust accuracy compared to Barlow-Twins and SimCLR in the low iterations regime.
Revised: Figure 10 shows MMCR exhibits a significantly higher robust accuracy compared to Barlow-Twins and SimCLR in the low iterations regime.

Original: Aside from the standard PGD adversarial attack, we also tested three SSL methods under the AutoAttack protocol.
Revised: Aside from the standard PGD adversarial attack, we also tested three SSL methods under the AutoAttack protocol.

Original: The ℓ ∞ -norm AutoAttack accuracy is given by Table 3.
Revised: The ℓ ∞ -norm AutoAttack accuracy is given by Table 3.

Original: [Table]
Revised: [Table]