Original: A C RITIQUE OF S ELF -E XPRESSIVE D EEP S UBSPACE
Revised: A C RITIQUE OF S ELF -E XPRESSIVE D EEP S UBSPACE

Original: C LUSTERING
Revised: C LUSTERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Subspace clustering is an unsupervised clustering technique designed to cluster data that is supported on a union of linear subspaces, with each subspace deﬁning a cluster with dimension lower than the ambient space.
Revised: Subspace clustering is an unsupervised clustering technique designed to cluster data that is supported on a union of linear subspaces, with each subspace deﬁning a cluster with dimension lower than the ambient space.

Original: Many existing formulations for this problem are based on exploiting the self-expressive property of linear subspaces, where any point within a subspace can be represented as linear combination of other points within the subspace.
Revised: Many existing formulations for this problem are based on exploiting the self-expressive property of linear subspaces, where any point within a subspace can be represented as linear combination of other points within the subspace.

Original: To extend this approach to data supported on a union of non-linear manifolds, numerous studies have proposed learning an appropriate kernel embedding of the original data using a neural network, which is regularized by a self-expressive loss function on the data in the embedded space to encourage a union of linear subspaces prior on the data in the embedded space.
Revised: To extend this approach to data supported on a union of non-linear manifolds, numerous studies have proposed learning an appropriate kernel embedding of the original data using a neural network, which is regularized by a self-expressive loss function on the data in the embedded space to encourage a union of linear subspaces prior on the data in the embedded space.

Original: Here we show that there are a number of potential ﬂaws with this approach which have not been adequately addressed in prior work.
Revised: Here we show that there are a number of potential ﬂaws with this approach which have not been adequately addressed in prior work.

Original: In particular, we show the optimization problem is often ill-posed in multiple ways, which can lead to a degenerate embedding of the data, which need not correspond to a union of subspaces at all.
Revised: In particular, we show the model formulation is often ill-posed in multiple ways, which can lead to a degenerate embedding of the data, which need not correspond to a union of subspaces at all.

Original: We validate our theoretical results experimentally and additionally repeat prior experiments reported in the literature, where we conclude that a signiﬁcant portion of the previously claimed performance beneﬁts can be attributed to an ad-hoc post processing step rather than the clustering model.
Revised: We validate our theoretical results experimentally and additionally repeat prior experiments reported in the literature, where we conclude that a signiﬁcant portion of the previously claimed performance beneﬁts can be attributed to an ad-hoc post processing step rather than the clustering model.

Original:  I NTRODUCTION AND B ACKGROUND
Revised:  I NTRODUCTION AND B ACKGROUND

Original: Subspace clustering is a classical unsupervised learning problem, where one wishes to segment a given dataset into a prescribed number of clusters, and each cluster is deﬁned as a linear (or afﬁne) subspace with dimension lower than the ambient space.
Revised: Subspace clustering is a classical unsupervised learning problem, where one wishes to segment a given dataset into a prescribed number of clusters, and each cluster is deﬁned as a linear (or afﬁne) subspace with dimension lower than the ambient space.

Original: There have been a wide variety of approaches proposed in the literature to solve this problem Vidal et al.
Revised: There have been a wide variety of approaches proposed in the literature to solve this problem Vidal et al.

Original: (2016), but a large family of state-of-the-art approaches are based on exploiting the self-expressive property of linear subspaces.
Revised: (2016), but a large family of state-of-the-art approaches are based on exploiting the self-expressive property of linear subspaces.

Original: That is, if a point lies in a linear subspace, then it can be represented as a linear combination of other points within the subspace.
Revised: That is, if a point lies in a linear subspace, then it can be represented as a linear combination of other points within the subspace.

Original: Based on this fact, a wide variety of methods have been proposed which, given a dataset Z ∈ R d × N of N d -dimensional points, ﬁnd a matrix of coefﬁcients C ∈ R N × N by solving the problem:
Revised: Based on this fact, a wide variety of methods have been proposed which, given a dataset Z ∈ R d × N of N d -dimensional points, ﬁnd a matrix of coefﬁcients C ∈ R N × N by solving the problem:

Original: [Equation]
Revised: [Equation]

Original: Here, the ﬁrst term (cid:107) ZC − C (cid:107) 2 F captures the self-expressive property by requiring every datapoint to represent itself as an approximate linear combination of other points, i.e., Z i ≈ ZC i , where Z i and C i are the i th columns of Z and C , respectively.
Revised: Here, the ﬁrst term (cid:107) ZC − C (cid:107) 2 F captures the self-expressive property by requiring every datapoint to represent itself as an approximate linear combination of other points, i.e., Z i ≈ ZC i , where Z i and C i are the i th columns of Z and C , respectively.

Original: The second term, θ ( C ) , is some regularization function designed to encourage each data point to only select other points within the correct subspace in its representation and to avoid trivial solutions (such as C = I ).
Revised: The second term, θ ( C ) , is some regularization function designed to encourage each data point to only select other points within the correct subspace in its representation and to avoid trivial solutions (such as C = I ).

Original: Once the C matrix has been solved for, one can then deﬁne a graph afﬁnity between data points, typically based on the magnitudes of the entries of C , and use an appropriate graph-based clustering method (e.g., spectral clustering von Luxburg (2007)) to produce the ﬁnal clustering of the data points.
Revised: Once the C matrix has been solved for, one can then deﬁne a graph afﬁnity between data points, typically based on the magnitudes of the entries of C , and use an appropriate graph-based clustering method (e.g., spectral clustering von Luxburg (2007)) to produce the ﬁnal clustering of the data points.

Original: One of the ﬁrst methods to utilize this approach was Sparse Subspace Clustering (SSC) Elhamifar & Vidal (2009; 2013), where θ takes the form θ SSC ( C )
Revised: One of the ﬁrst methods to utilize this approach was Sparse Subspace Clustering (SSC) Elhamifar & Vidal (2009; 2013), where θ takes the form θ SSC ( C )

Original: = (cid:107) C (cid:107) 

Revised: = (cid:107) C (cid:107)

Original: + δ (diag( C ) = 0 ) , with (cid:107) · (cid:107)denoting the (cid:96) 1 norm and δ an indicator function which takes value ∞ if an element of the diagonal of C is non-zero and 0 otherwise.
Revised: + δ (diag( C ) = 0 ) , with (cid:107) · (cid:107)denoting the (cid:96) 1 norm and δ an indicator function which takes value ∞ if an element of the diagonal of C is non-zero and 0 otherwise.

Original: By regularizing C to be sparse, a point must represent itself using the smallest number of other points within the dataset, which in turn ideally requires a point to only select other points within its own subspace in the representation.
Revised: By regularizing C to be sparse, a point must represent itself using the smallest number of other points within the dataset, which in turn ideally requires a point to only select other points within its own subspace in the representation.

Original: Likewise other variants, with LowRank Representation (LRR)
Revised: Likewise other variants, with LowRank Representation (LRR)

Original: Liu et al.
Revised: Liu et al.

Original: and Elastic-net Subspace Clustering (EnSC)
Revised: and Elastic-net Subspace Clustering (EnSC)

Original: You et al.
Revised: You et al.

Original: (2016) being well-known examples, take the same form as (1) with different choices of regularization, θ LRR ( C )
Revised: (2016) being well-known examples, take the same form as (1) with different choices of regularization, θ LRR ( C )

Original: = (cid:107) C (cid:107) ∗ and θ EnSC ( C )
Revised: = (cid:107) C (cid:107) ∗ and θ EnSC ( C )

Original: = (cid:107) C (cid:107) 

Revised: = (cid:107) C (cid:107)

Original: + τ (cid:107) C (cid:107) 2 F + δ (diag( C ) = 0 ) , respectively, where (cid:107) · (cid:107) ∗ denotes the nuclear norm (sum of the singular values).
Revised: + τ (cid:107) C (cid:107) 2 F + δ (diag( C ) = 0 ) , respectively, where (cid:107) · (cid:107) ∗ denotes the nuclear norm (sum of the singular values).

Original: A signiﬁcant advantage of the majority of these methods is that it can be proven (typically subject to some technical assumptions regarding the angles between the underlying subspaces and the distribution of the sampled data points within the subspaces) that the optimal C matrix in (1) will be “correct” in the sense that if C i,j is non-zero then the i th and j th columns of Z lie in the same linear subspace Soltanolkotabi & Cand`es (2012);
Revised: A signiﬁcant advantage of the majority of these methods is that it can be proven (typically subject to some technical assumptions regarding the angles between the underlying subspaces and the distribution of the sampled data points within the subspaces) that the optimal C matrix in (1) will be “correct” in the sense that if C i,j is non-zero then the i th and j th columns of Z lie in the same linear subspace Soltanolkotabi & Cand`es (2012);

Original: Lu et al.
Revised: Lu et al.

Original: (2012); Elhamifar & Vidal (2013)
Revised: (2012); Elhamifar & Vidal (2013)

Original: ; Soltanolkotabi et al.
Revised: ; Soltanolkotabi et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ; Wang &
Revised: ; Wang &

Original: Xu (2016); You & Vidal (2015a;b); Tsakiris & Vidal (2018); Li et al.
Revised: Xu (2016); You & Vidal (2015a;b); Tsakiris & Vidal (2018); Li et al.

Original: ; Robinson et al.
Revised: ; Robinson et al.

Original: (2019), which has led to these methods achieving state-of-the-art performance in many applications.
Revised: (2019), which has led to these methods achieving state-of-the-art performance in many applications.

Original: 1. S ELF -E XPRESSIVE D EEP S UBSPACE C LUSTERING
Revised: 1. S ELF -E XPRESSIVE D EEP S UBSPACE C LUSTERING

Original: Although subspace clustering techniques based on self-expression display strong empirical performance and provide theoretical guarantees, a signiﬁcant limitation of these techniques is the requirement that the underlying dataset needs to be approximately supported on a union of linear subspaces.
Revised: Although subspace clustering techniques based on self-expression display strong empirical performance and provide theoretical guarantees, a signiﬁcant limitation of these techniques is the requirement that the underlying dataset needs to be approximately supported on a union of linear subspaces.

Original: This has led to a strong motivation to extend these techniques to more general datasets, such as data supported on a union of non-linear low-dimensional manifolds.
Revised: This has led to a strong motivation to extend these techniques to more general datasets, such as data supported on a union of non-linear low-dimensional manifolds.

Original: From inspection of the right side of (1), one can observe that the only dependence on the data Z comes in the form of the Gram matrix
Revised: From inspection of the right side of (1), one can observe that the only dependence on the data Z comes in the form of the Gram matrix

Original: Z (cid:62) Z .
Revised: Z (cid:62) Z .

Original: As a result, self-expressive subspace clustering techniques are amendable to the “kerneltrick”, where instead of taking an inner product kernel between data points, one can instead use a general kernel κ ( · , · ) Patel & Vidal (2014);
Revised: As a result, self-expressive subspace clustering techniques are amendable to the “kerneltrick”, where instead of taking an inner product kernel between data points, one can instead use a general kernel κ ( · , · ) Patel & Vidal (2014);

Original: Patel et al.
Revised: Patel et al.

Original: Of course, such an approach comes with the traditional challenge of how to select an appropriate kernel so that the embedding of the data in the Hilbert space associated with the choice of kernel results in a union of linear subspaces in the embedded space.
Revised: Of course, such an approach comes with the traditional challenge of how to select an appropriate kernel so that the embedding of the data in the Hilbert space associated with the choice of kernel results in a union of linear subspaces in the embedded space.

Original: The ﬁrst approach to propose learning an appropriate embedding of an initial dataset X ∈ R d x × N (which does not necessarily have a union of subspaces structure) was given by Patel et al.
Revised: The ﬁrst approach to propose learning an appropriate embedding of an initial dataset X ∈ R d x × N (which does not necessarily have a union of subspaces structure) was given by Patel et al.

Original: ; Patel & Vidal (2014);
Revised: ; Patel & Vidal (2014);

Original: Patel et al.
Revised: Patel et al.

Original: (2015) who proposed ﬁrst projecting the data into a lower dimensional space via a learned linear projector, Z = P l X , where P l ∈ R d × d x ( d < d x ) is also optimized over in addition to C in (1).
Revised: (2015) who proposed ﬁrst projecting the data into a lower dimensional space via a learned linear projector, Z = P l X , where P l ∈ R d × d x ( d < d x ) is also optimized over in addition to C in (1).

Original: To ensure that sufﬁcient information about the original data X is preserved in the low-dimensional embedding Z , the authors further required that the linear projector satisfy the constraint that P l P (cid:62) l
Revised: To ensure that sufﬁcient information about the original data X is preserved in the low-dimensional embedding Z , the authors further required that the linear projector satisfy the constraint that P l P (cid:62) l

Original: = I and added an additional term to the objective with form (cid:107) X − P (cid:62) l P l X (cid:107) 2 F .
Revised: = I and added an additional term to the objective with form (cid:107) X − P (cid:62) l P l X (cid:107) 2 F .

Original: More recently, given the recent success of deep neural networks, a large number of studies Peng et al.
Revised: More recently, given the recent success of deep neural networks, a large number of studies Peng et al.

Original: ; Ji et al.
Revised: ; Ji et al.

Original: ; Zeng et al.
Revised: ; Zeng et al.

Original: (2019b;a);
Revised: (2019b;a);

Original: Xie et al.
Revised: Xie et al.

Original: (2020); Sun et al.
Revised: (2020); Sun et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: ; Tang et al.
Revised: ; Tang et al.

Original: ; Kheirandishfard et al.
Revised: ; Kheirandishfard et al.

Original: (2020b);
Revised: (2020b);

Original: Zhou et al.
Revised: Zhou et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2019b)
Revised: (2019b)

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2018); Abavisani & Patel (2018)
Revised: (2018); Abavisani & Patel (2018)

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018; 2019c;a); Kheirandishfard et al.
Revised: (2018; 2019c;a); Kheirandishfard et al.

Original: (2020a) have attempted to learn an appropriate embedding of the data (which ideally would have a union of linear subspaces structure) via a neural network, Φ E ( X , W e ) , where W e denotes the parameters of a network mapping deﬁned by Φ E , which takes a dataset X ∈ R d x × N as input.
Revised: (2020a) have attempted to learn an appropriate embedding of the data (which ideally would have a union of linear subspaces structure) via a neural network, Φ E ( X , W e ) , where W e denotes the parameters of a network mapping deﬁned by Φ E , which takes a dataset X ∈ R d x × N as input.

Original: In an attempt to encourage the embedding of the data, Φ E ( X , W e ) , to have this union of subspaces structure, these approaches minimize a selfexpressive loss term, with form given in (1), on the embedded data, and a large majority of these proposed techniques can be described by the general form:
Revised: In an attempt to encourage the embedding of the data, Φ E ( X , W e ) , to have this union of subspaces structure, these approaches minimize a selfexpressive loss term, with form given in (1), on the embedded data, and a large majority of these proposed techniques can be described by the general form:

Original: [Equation]
Revised: [Equation]

Original: where g is some function designed to discourage trivial solutions (for example Φ E ( X , W e ) = 0 ) and γ > 0 is some hyper-parameter to balance the terms.
Revised: where g is some function designed to discourage trivial solutions (for example Φ E ( X , W e ) = 0 ) and γ > 0 is some hyper-parameter to balance the terms.

Original: Several different choices of g have been proposed in the literature.
Revised: Several different choices of g have been proposed in the literature.

Original: The ﬁrst is to place some form of normalization directly on Z .
Revised: The ﬁrst is to place some form of normalization directly on Z .

Original: For example, the authors of Peng et al.
Revised: For example, the authors of Peng et al.

Original: (2017) propose an Instance
Revised: (2017) propose an Instance

Original: Normalization regularization, which attempts to constrain the norm of the embedded data points to be 1, which takes the form g ( Z , X , C ) = (cid:80) Ni
Revised: Normalization regularization, which attempts to constrain the norm of the embedded data points to be 1, which takes the form g ( Z , X , C ) = (cid:80) Ni

Original: =

Revised: =

Original: ( Z (cid:62)
Revised: ( Z (cid:62)

Original: i Z i − 1) 2 .
Revised: i Z i − 1) 2 .

Original: Likewise, one could also consider
Revised: Likewise, one could also consider

Original: Dataset Normalization schemes, which bound the norm of the entire embedded representation (cid:107) Z (cid:107) 2 F ≥ τ or Channel/Batch Normalization schemes, which bound the norm of a channel of the embedded representation (i.e., a row of Z ), (cid:107)
Revised: Dataset Normalization schemes, which bound the norm of the entire embedded representation (cid:107) Z (cid:107) 2 F ≥ τ or Channel/Batch Normalization schemes, which bound the norm of a channel of the embedded representation (i.e., a row of Z ), (cid:107)

Original: Z i (cid:107) 2 F ≥ τ, ∀ i .
Revised: Z i (cid:107) 2 F ≥ τ, ∀ i .

Original: We note that this is quite similar to the common Batch Norm operator Ioffe & Szegedy (2015) used in neural network training which attempts to constrain each row of Z to have zero mean and constant norm.
Revised: We note that this is quite similar to the common Batch Norm operator Ioffe & Szegedy (2015) used in neural network training which attempts to constrain each row of Z to have zero mean and constant norm.

Original: Another popular form of g is to also train a decoding network Φ D ( · , W d ) with parameters W d to map the self-expressive representation, Φ E ( X , W e ) C , back to the original data to ensure that sufﬁcient information is preserved in the self-expressive representation to recover the original data.
Revised: Another popular form of g is to also train a decoding network Φ D ( · , W d ) with parameters W d to map the self-expressive representation, Φ E ( X , W e ) C , back to the original data to ensure that sufﬁcient information is preserved in the self-expressive representation to recover the original data.

Original: This idea is essentially a generalization of the previously discussed work, which considered constrained linear encoder/decoder mappings Patel et al.
Revised: This idea is essentially a generalization of the previously discussed work, which considered constrained linear encoder/decoder mappings Patel et al.

Original: ; Patel & Vidal (2014);
Revised: ; Patel & Vidal (2014);

Original: Patel et al.
Revised: Patel et al.

Original: (2015), to non-linear autoencodering neural networks
Revised: (2015), to non-linear autoencodering neural networks

Original: and was ﬁrst proposed by the authors of Ji et al.
Revised: and was ﬁrst proposed by the authors of Ji et al.

Original: We will refer to this as Autoencoder Regularization , and the problem takes the general form:
Revised: We will refer to this as Autoencoder Regularization , and the problem takes the general form:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term is the standard self-expressive subspace clustering loss applied to the embedded representation, and the second term is a standard auto-encoder loss, with (cid:96) typically chosen to be the squared loss.
Revised: where the ﬁrst term is the standard self-expressive subspace clustering loss applied to the embedded representation, and the second term is a standard auto-encoder loss, with (cid:96) typically chosen to be the squared loss.

Original: Note that here both the kernel (in the form of the encoding network) and the optimal self-expression encoding, C , are trained jointly, and once problem (3) is solved one can use the recovered C matrix directly for clustering.
Revised: Note that here both the kernel (in the form of the encoding network) and the optimal self-expression encoding, C , are trained jointly, and once problem (3) is solved one can use the recovered C matrix directly for clustering.

Original: 1. P APER C ONTRIBUTIONS
Revised: 1. P APER C ONTRIBUTIONS

Original: Using the general formulation in (2) and the popular speciﬁc case in (3), Self-Expressive Deep Subspace Clustering (SEDSC) has been applied to a variety of applications, but there is relatively little that it known about it from a theoretical standpoint.
Revised: Using the general formulation in (2) and the popular speciﬁc case in (3), Self-Expressive Deep Subspace Clustering (SEDSC) has been applied to a variety of applications, but there is relatively little that it known about it from a theoretical standpoint.

Original: Initial formulations for SEDSC were guided by the intuition that if the dataset is drawn from a union of linear subspaces, then solving problem (1) is known to induce desirable properties in C for clustering.
Revised: Initial formulations for SEDSC were guided by the intuition that if the dataset is drawn from a union of linear subspaces, then solving problem (1) is known to induce desirable properties in C for clustering.

Original: By extension one might assume that if one also optimizes over the geometry of the learned embedding ( Z ) this objective might induce a desirable geometry in the embedded space (e.g., a union of linear subspaces).
Revised: By extension one might assume that if one also optimizes over the geometry of the learned embedding ( Z ) this objective might induce a desirable geometry in the embedded space (e.g., a union of linear subspaces).

Original: However, a vast majority of the prior theoretical analysis for problems of the form in (1) only considers the case where the data is held ﬁxed and analyze the properties of the optimal C matrix.
Revised: However, a vast majority of the prior theoretical analysis for problems of the form in (1) only considers the case where the data is held ﬁxed and analyze the properties of the optimal C matrix.

Original: Due to the well-known fact that neural networks are capable of producing highly-expressive mapping functions (and hence a network could produce many potential values for Z ), the use of a model such as (2)/(3) is essentially using (1) as a regularization function on Z to encourage a union of subspaces geometry.
Revised: Due to the well-known fact that neural networks are capable of producing highly-expressive mapping functions (and hence a network could produce many potential values for Z ), the use of a model such as (2)/(3) is essentially using (1) as a regularization function on Z to encourage a union of subspaces geometry.

Original: To date, however, models such as (2)/(3) have been guided largely by intuition and signiﬁcant questions remain regarding what type of data geometry is encouraged by F ( Z , C ) when one optimizes over both the encoding matrix, C , and the network producing the embedded data representation, Z .
Revised: To date, however, models such as (2)/(3) have been guided largely by intuition and signiﬁcant questions remain regarding what type of data geometry is encouraged by F ( Z , C ) when one optimizes over both the encoding matrix, C , and the network producing the embedded data representation, Z .

Original: Here we explore these questions via theoretical analysis where we show that the use of F ( Z , C ) as a regularization function when learning a kernel from the data in an attempt to promote beneﬁcial data geometries in the embedded space, as is done in (2), is largely insufﬁcient in the sense that the optimal data geometries are trivial and not conducive to successful clustering in most cases.
Revised: Here we explore these questions via theoretical analysis where we show that the use of F ( Z , C ) as a regularization function when learning a kernel from the data in an attempt to promote beneﬁcial data geometries in the embedded space, as is done in (2), is largely insufﬁcient in the sense that the optimal data geometries are trivial and not conducive to successful clustering in most cases.

Original: Speciﬁcally, we ﬁrst note a basic fact that the Autoencoder Regularization formulation in (3) is typically ill-posed for most commonly used networks if constraints are not placed on the magnitude of the embedded data, Φ E ( X , W e ) , either through regularization/constraints on the autoencoder weights/architecture or an explicit normalization of Φ E ( X , W e ) , such as in the Instance/Batch/Dataset Normalization schemes.
Revised: Speciﬁcally, we ﬁrst note a basic fact that the Autoencoder Regularization formulation in (3) is typically ill-posed for most commonly used networks if constraints are not placed on the magnitude of the embedded data, Φ E ( X , W e ) , either through regularization/constraints on the autoencoder weights/architecture or an explicit normalization of Φ E ( X , W e ) , such as in the Instance/Batch/Dataset Normalization schemes.

Original: Then, even assuming that the embedded representation has been suitably normalized, we show that the optimal embedded data geometry encouraged by F ( Z , C ) is trivial in various ways, which will depend on how the data is normalized.
Revised: Then, even assuming that the embedded representation has been suitably normalized, we show that the optimal embedded data geometry encouraged by F ( Z , C ) is trivial in various ways, which will depend on how the data is normalized.

Original: We illustrate our theoretical predictions with experiments on both real and synthetic data.
Revised: We illustrate our theoretical predictions with experiments on both real and synthetic data.

Original: Further, we show experimentally that much of the claimed performance beneﬁt of the SEDSC model which has been reported in previous work can be attributed to an ad-hoc post-processing of the C matrix ﬁrst proposed in Ji et al.
Revised: Further, we show experimentally that much of the claimed performance beneﬁt of the SEDSC model which has been reported in previous work can be attributed to an ad-hoc post-processing of the C matrix ﬁrst proposed in Ji et al.

Original: Notation.
Revised: Notation.

Original: We will denote matrices with capital boldfaced letters, Z , vectors which are not rows/columns of a larger matrix with lower-case boldfaced letters, z , and sets with calligraphic letters, Z .
Revised: We will denote matrices with capital boldfaced letters, Z , vectors which are not rows/columns of a larger matrix with lower-case boldfaced letters, z , and sets with calligraphic letters, Z .

Original: The i th row of a matrix will be denoted with a superscript, Z i ; the i th column of a matrix will be denoted with a subscript, Z i ; and the ( i, j ) th entry of a matrix will be denoted as Z i,j .
Revised: The i th row of a matrix will be denoted with a superscript, Z i ; the i th column of a matrix will be denoted with a subscript, Z i ; and the ( i, j ) th entry of a matrix will be denoted as Z i,j .

Original: We will denote the minimum singular value of a matrix Z as σ min ( Z ) , and we will denote the nuclear,spectral, Frobenius,and (cid:96) p  norms for a matrix/vector as (cid:107) Z (cid:107) ∗ , (cid:107) Z (cid:107) ◦∗ , (cid:107) Z (cid:107) F , and (cid:107) Z (cid:107)  p, respectively.
Revised: We will denote the minimum singular value of a matrix Z as σ min ( Z ) , and we will denote the nuclear, Frobenius, (cid:96) p , and Schatten- p norms for a matrix/vector as (cid:107) Z (cid:107) ∗ , (cid:107) Z (cid:107) F , (cid:107) Z (cid:107) p , and (cid:107) Z (cid:107) S p respectively.

Original: δ ( cnd ) denotes an indicator function which takes value [-
-] if condition cnd is true and ∞ otherwise.
Revised: δ ( cnd ) denotes an indicator function with value [-
-]0 if condition cnd is true and ∞ otherwise.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: We begin our analysis of the SEDSC model by considering the most popular formulation, which is to employ Autoencoder Regularization as in (3).
Revised: We begin our analysis of the SEDSC model by considering the most popular formulation, which is to employ Autoencoder Regularization as in (3).

Original: Speciﬁcally, we note that without any regularization on the autoencoder network parameters ( W e , W d ) or any normalization placed on the embedded representation, Φ E ( X , W e ) , the formulation in (3) is often ill-posed in the sense that the value of
Revised: Speciﬁcally, we note that without any regularization on the autoencoder network parameters ( W e , W d ) or any normalization placed on the embedded representation, Φ E ( X , W e ) , the formulation in (3) is often ill-posed in the sense that the value of

Original: F (Φ E ( X , W e ) , C ) can be made arbitrarily small without changing the value of the autoencoder loss by simply scaling-down the weights in the encoding network and scaling-up the weights in the decoding network in a way which doesn’t change the output of the autoencoder but reduces the magnitude of the embedded representation.
Revised: F (Φ E ( X , W e ) , C ) can be made arbitrarily small without changing the value of the autoencoder loss by simply scaling-down the weights in the encoding network and scaling-up the weights in the decoding network in a way which doesn’t change the output of the autoencoder but reduces the magnitude of the embedded representation.

Original: As a basic example, consider fully-connected single-hidden-layer networks for the encoder/decoder, Φ E ( X , W e )
Revised: As a basic example, consider fully-connected single-hidden-layer networks for the encoder/decoder, Φ E ( X , W e )

Original: =
Revised: =

Original: W 2 e ( W 1 e X ) + and Φ D ( Z , W d ) =
Revised: W 2 e ( W 1 e X ) + and Φ D ( Z , W d ) =

Original: W 2 d ( W 1 d Z ) + where ( x ) + = max { x, 0 } denotes the Rectiﬁed Linear Unit (ReLU) non-linearity applied entrywise.
Revised: W 2 d ( W 1 d Z ) + where ( x ) + = max { x, 0 } denotes the Rectiﬁed Linear Unit (ReLU) non-linearity applied entrywise.

Original: Then, note that for any α ≥one has Φ E ( X , α W e ) =
Revised: Then, note that for any α ≥one has Φ E ( X , α W e ) =

Original: ( α W 2 e )( α W 1 e X ) + = α 2 W 2 e
Revised: ( α W 2 e )( α W 1 e X ) + = α 2 W 2 e

Original: ( W 1 e X )
Revised: ( W 1 e X )

Original: + = α 2 Φ E ( X , W e ) and similarly for Φ D .
Revised: + = α 2 Φ E ( X , W e ) and similarly for Φ D .

Original: As a result one can scale W e by any α > 0 and W d by α − 1 without changing the output of the autoencoder, Φ D (Φ E ( X , α W e ) , α − 1 W d ) =
Revised: As a result one can scale W e by any α > 0 and W d by α − 1 without changing the output of the autoencoder, Φ D (Φ E ( X , α W e ) , α − 1 W d ) =

Original: Φ D (Φ E ( X , W e ) , W d ) , but (cid:107) Φ E
Revised: Φ D (Φ E ( X , W e ) , W d ) , but (cid:107) Φ E

Original: X , α W e ) (cid:107) F = α 2 (cid:107) Φ E
Revised: X , α W e ) (cid:107) F = α 2 (cid:107) Φ E

Original: ( X , W e ) (cid:107) F .
Revised: ( X , W e ) (cid:107) F .

Original: From this, taking α to be arbitrarily small the magnitude of the embedding can also be made arbitrarily small, so if θ ( C ) is something like a norm (as is typically the case) the value of F ( Z , C ) in (3) can also be made arbitrarily small without changing the reconstruction loss of the autoencoder (cid:96) .
Revised: From this, taking α to be arbitrarily small the magnitude of the embedding can also be made arbitrarily small, so if θ ( C ) is something like a norm (as is typically the case) the value of F ( Z , C ) in (3) can also be made arbitrarily small without changing the reconstruction loss of the autoencoder (cid:96) .

Original: This implies one is simply training an autoencoder without any contribution from the F ( Z , C ) term (other than trying to reduce the magnitude of the embedded representation).
Revised: This implies one is simply training an autoencoder without any contribution from the F ( Z , C ) term (other than trying to reduce the magnitude of the embedded representation).

Original: This basic idea can be easily generalized and formalized in the following statement:
Revised: This basic idea can be easily generalized and formalized in the following statement:

Original: Proposition 1.
Revised: Proposition 1.

Original: Consider the objective in (3) and suppose θ ( C ) is a function which achieves its minimum at C = 0 and satisﬁes θ ( µ C ) <
Revised: Consider the objective in (3) and suppose θ ( C ) is a function which achieves its minimum at C = 0 and satisﬁes θ ( µ C ) <

Original: θ ( C ) , ∀ µ ∈ (0 , 1) .
Revised: θ ( C ) , ∀ µ ∈ (0 , 1) .

Original: If for any choice of ( W d , W e ) and τ 1 , τ 2 ∈ (0 , 1) in there exists ( ˆ W d , ˆ W e ) such that Φ E ( X , ˆ W e )
Revised: If for any choice of ( W d , W e ) and τ 1 , τ 2 ∈ (0 , 1) in there exists ( ˆ W d , ˆ W e ) such that Φ E ( X , ˆ W e )

Original: = τ 1 Φ E ( X , W e ) and Φ D ( τ 2 Z , ˆ W d ) =
Revised: = τ 1 Φ E ( X , W e ) and Φ D ( τ 2 Z , ˆ W d ) =

Original: Φ D ( Z , W d ) then the F term in (3) can be made arbitrarily small without changing the value of the loss function (cid:96) .
Revised: Φ D ( Z , W d ) then the F term in (3) can be made arbitrarily small without changing the value of the loss function (cid:96) .

Original: Further, if the ﬁnal layers of the encoding and decoding networks are positively-homogeneous with degree p (cid:54) = 0 , such a ( ˆ W d , ˆ W e ) will always exist simply by scaling the network weights of the linear (or afﬁne) operator parameters.
Revised: Further, if the ﬁnal layer of the encoding network is positively-homogeneous with degree p (cid:54) = 0 , such a ( ˆ W d , ˆ W e ) will always exist simply by scaling the network weights of the linear (or afﬁne) operator parameters.

Original: Proof.
Revised: Proof.

Original: All proofs are provided in the Supplementary Material.
Revised: All proofs are provided in the Supplementary Material.

Original: Recall, a positively-homogeneous function of degree p is any function which satisﬁes f ( αx ) =
Revised: Recall, a positively-homogeneous function of degree p is any function which satisﬁes f ( αx ) =

Original: α p f ( x ) , ∀ α ≥ 0 , and that most commonly used non-linearities in neural networks are positively homogenous, with Rectiﬁed Linear Units (ReLUs), leaky ReLUs, and max-pooling being common examples (note that the above proposition also applies to networks employing bias terms).
Revised: α p f ( x ) , ∀ α ≥ 0 , and that most commonly used non-linearities in neural networks are positively homogenous, with Rectiﬁed Linear Units (ReLUs), leaky ReLUs, and max-pooling being common examples (note that the above proposition also applies to networks employing bias terms).

Original: As a result, most autoencoder architectures employed for SEDSC require some form or regularization or normalization for the F term to have any impact on the geometry of the embedded representation (other than trivially becoming smaller in magnitude).
Revised: As a result, most autoencoder architectures employed for SEDSC require some form or regularization or normalization for the F term to have any impact on the geometry of the embedded representation (other than trivially becoming smaller in magnitude).

Original: In practice, most of the previously cited studies employing the SEDSC model use networks which satisfy the conditions of Proposition 1 without regularization, and we note that such models can never be solved to global optimality (only approach it asymptotically) and are inherently ill-posed.
Revised: In practice, most of the previously cited studies employing the SEDSC model use networks which satisfy the conditions of Proposition 1 without regularization, and we note that such models can never be solved to global optimality (only approach it asymptotically) and are inherently ill-posed.

Original: From this, one can conclude that Autoencoder Regularization by itself is often insufﬁcient to prevent trival solutions to (2).
Revised: From this, one can conclude that Autoencoder Regularization by itself is often insufﬁcient to prevent trival solutions to (2).

Original: This speciﬁc issue can be easily ﬁxed (although prior work often does not) if one ensures that the magnitude of the embedded representation is constrained to be larger than some minimum value - either through regularization/constraints placed on the network weights, such as using weight decay or coupling weights between the encoder and decoder, or explicit normalization of the embedded representation.
Revised: This speciﬁc issue can be easily ﬁxed (although prior work often does not) if one ensures that the magnitude of the embedded representation is constrained to be larger than some minimum value - either through regularization/constraints placed on the network weights, such as using weight decay or coupling weights between the encoder and decoder, or explicit normalization of the embedded representation.

Original: However, the question remains what geometries are promoted by (1) even if the basic issue described above is corrected, which we explore next.
Revised: However, the question remains what geometries are promoted by (1) even if the basic issue described above is corrected, which we explore next.

Original: 2. T HE E FFECT OF THE A UTOENCODER L OSS
Revised: 2. T HE E FFECT OF THE A UTOENCODER L OSS

Original: Before presenting the remainder of our formal results, we ﬁrst pause to discuss the effect of the autoencoder loss ( (cid:96) ) in (3).
Revised: Before presenting the remainder of our formal results, we ﬁrst pause to discuss the effect of the autoencoder loss ( (cid:96) ) in (3).

Original: The use of the autoencoder loss is designed to ensure that the embedded representation Z retains sufﬁcient information about the original data X so that the data can be reconstructed by the decoder, but we note that this does not necessarily impose signiﬁcant constraints on the geometric arrangement of the embedded points in Z .
Revised: The use of the autoencoder loss is designed to ensure that the embedded representation Z retains sufﬁcient information about the original data X so that the data can be reconstructed by the decoder, but we note that this does not necessarily impose signiﬁcant constraints on the geometric arrangement of the embedded points in Z .

Original: While there is potentially some constraint on the possible geometries of Z which can be expressed by the choice of encoder/decoder architectures, for reasonably expressive encoders and decoders data points can be mapped essentially arbitrarily into the embedded space and still decoded accurately, provided the embedding of any two points is still distinct to within some (cid:15) perturbation.
Revised: While there is potentially some constraint on the possible geometries of Z which can be expressed by the choice of encoder/decoder architectures, for reasonably expressive encoders and decoders data points can be mapped essentially arbitrarily into the embedded space and still decoded accurately, provided the embedding of any two points is still distinct to within some (cid:15) perturbation.

Original: Further, because we only evaluate the mapping on a ﬁnite set of points X , this is a much easier (and can be achieved with much simpler networks) than the well-known universal approximation regime of neural networks (which requires good approximation over the entire continuous data domain), and it is well documented that typical network architectures can optimally ﬁt fairly arbitrary ﬁnite training data Zhang et al.
Revised: Further, because we only evaluate the mapping on a ﬁnite set of points X , this is a much easier (and can be achieved with much simpler networks) than the well-known universal approximation regime of neural networks (which requires good approximation over the entire continuous data domain), and it is well documented that typical network architectures can optimally ﬁt fairly arbitrary ﬁnite training data Zhang et al.

Original: Given the above discussion, in our analysis we will ﬁrst focus on the setting where the autoencoder is highly expressive.
Revised: Given the above discussion, in our analysis we will ﬁrst focus on the setting where the autoencoder is highly expressive.

Original: In this regime, the autoencoder can select weight parameters to produce an essentially arbitrary choice of Z embedding, so the global optimum of the model in (3) (and likewise (2)) will approach a choice of parameters which minimizes F ( Z , C ) .
Revised: In this regime, the autoencoder can select weight parameters to produce an essentially arbitrary choice of Z embedding, so the global optimum of the model in (3) (and likewise (2)) will approach a choice of parameters which minimizes F ( Z , C ) .

Original: As we already know from
Revised: As we already know from

Original: Prop 1, this is ill-posed without some additional form of regularization, so in the following subsections we explore optimal solutions to F ( Z , C ) when one optimizes over both C and the embedded representation Z = Φ E ( X , W e ) subject to three different constraints on Z : (1) Dataset Normalization (cid:107) Z (cid:107) 2 F ≥ τ , (2) Channel/Batch Normalization (cid:107)
Revised: Prop 1, this is ill-posed without some additional form of regularization, so in the following subsections we explore optimal solutions to F ( Z , C ) when one optimizes over both C and the embedded representation Z = Φ E ( X , W e ) subject to three different constraints on Z : (1) Dataset Normalization (cid:107) Z (cid:107) 2 F ≥ τ , (2) Channel/Batch Normalization (cid:107)

Original: Z i (cid:107) 2 = 1 ∀ i , and (3) Instance Normalization (cid:107)
Revised: Z i (cid:107) 2 = 1 ∀ i , and (3) Instance Normalization (cid:107)

Original: Z i (cid:107) 2 = 1 ∀ i .
Revised: Z i (cid:107) 2 = 1 ∀ i .

Original: Finally, after characterizing solutions to F ( Z , C ) we then give a speciﬁc example of a very simple (i.e., not highly expressive) family of architectures which can achieve these solutions.
Revised: Finally, after characterizing solutions to F ( Z , C ) we then give a speciﬁc example of a very simple (i.e., not highly expressive) family of architectures which can achieve these solutions.

Original: 2. D ATASET AND B ATCH /C HANNEL N ORMALIZATION
Revised: 2. D ATASET AND B ATCH /C HANNEL N ORMALIZATION

Original: We will ﬁrst consider the Dataset and Batch/Channel Normalization schemes, which will both result in very similar optimal solutions for the embedded data geometry.
Revised: We will ﬁrst consider the Dataset and Batch/Channel Normalization schemes, which will both result in very similar optimal solutions for the embedded data geometry.

Original: Recall, that this considers when the entire dataset is constrained to have a norm greater than some minimum value in the embedded space or when the norms of the rows of the dataset are constrained to have a norm greater than some minimum value.
Revised: Recall, that this considers when the entire dataset is constrained to have a norm greater than some minimum value in the embedded space or when the norms of the rows of the dataset are constrained to have a norm greater than some minimum value.

Original: We note that the latter case is very closely related to batch normalization Ioffe & Szegedy (2015), which requires that each channel/feature (in this case a row of the embedded representation) has zero mean and constant norm in expectation over the draw of a mini-batch.
Revised: We note that the latter case is very closely related to batch normalization Ioffe & Szegedy (2015), which requires that each channel/feature (in this case a row of the embedded representation) has zero mean and constant norm in expectation over the draw of a mini-batch.

Original: Additionally, while we do not explicitly enforce a zero-mean constraint, we will see that optimal solutions will exist which have zero mean.
Revised: Additionally, while we do not explicitly enforce a zero-mean constraint, we will see that optimal solutions will exist which have zero mean.

Original: Now, if one optimizes F ( Z , C ) jointly over ( Z , C ) subject to the above constraint(s) on Z , then the following holds:
Revised: Now, if one optimizes F ( Z , C ) jointly over ( Z , C ) subject to the above constraint(s) on Z , then the following holds:

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider the following optimization problems which jointly optimize over Z and C :
Revised: Consider the following optimization problems which jointly optimize over Z and C :

Original: [Equation]
Revised: [Equation]

Original: Then optimal values for C for both (P1) and (P2) are given by
Revised: Then optimal values for C for both (P1) and (P2) are given by

Original: [Equation]
Revised: [Equation]

Original: Moreover, for any optimal C ∗ , let r be the multiplicity of the smallest singular value of C ∗ − I and let Q ∈ R N × r be an orthonormal basis for the subspace spanned by the left singular vectors associated with the smallest singular value of C ∗ − I .
Revised: Moreover, for any optimal C ∗ , let r be the multiplicity of the smallest singular value of C ∗ − I and let Q ∈ R N × r be an orthonormal basis for the subspace spanned by the left singular vectors associated with the smallest singular value of C ∗ − I .

Original: Then we have that optimal values for Z are given by:
Revised: Then we have that optimal values for Z are given by:

Original: [Equation]
Revised: [Equation]

Original: From the above result, one notes from (5) that optimizing F ( Z , C ) jointly over both Z and C is equivalent to ﬁnding a C which minimizes a trade-off between the minimum singular value of C − I and the regularization θ ( C ) .
Revised: From the above result, one notes from (5) that optimizing F ( Z , C ) jointly over both Z and C is equivalent to ﬁnding a C which minimizes a trade-off between the minimum singular value of C − I and the regularization θ ( C ) .

Original: Further, we note that if such an optimal C results in the minimum singular value of C − I having a multiplicity of 1, then this implies that every data point in Z will simply be a scaled version of the same point.
Revised: Further, we note that if such an optimal C results in the minimum singular value of C − I having a multiplicity of 1, then this implies that every data point in Z will simply be a scaled version of the same point.

Original: Obviously, such an embedding is not useful for subspace clustering.
Revised: Obviously, such an embedding is not useful for subspace clustering.

Original: Characterizing optimal solutions to (5) is somewhat complicated in the general case due to the fact that the smallest singular value is a concave function of a matrix and θ is typically chosen to be a convex regularization function, resulting in the minimization of a convex+concave function.
Revised: Characterizing optimal solutions to (5) is somewhat complicated in the general case due to the fact that the smallest singular value is a concave function of a matrix and θ is typically chosen to be a convex regularization function, resulting in the minimization of a convex+concave function.

Original: Instead, we will  focus on the most commonly used choice of regularization, θ SSC ( C )
Revised: Instead, we will ﬁrst focus on the most commonly used choice of regularization, θ SSC ( C )

Original: = (cid:107) C (cid:107) 

Revised: = (cid:107) C (cid:107)

Original: + δ (diag( C ) = 0 ) , where we derive the optimal solution in the case where σ min ( C − I ) = 0 .
Revised: + δ (diag( C ) = 0 ) , where we derive the optimal solution in the case where σ min ( C − I ) = 0 .

Original: We note that this corresponds to the case where Z ∗ =
Revised: We note that this corresponds to the case where Z ∗ =

Original: Z ∗ C ∗ which one typically obtains as λ in (1) is taken to be small.
Revised: Z ∗ C ∗ which one typically obtains as λ in (1) is taken to be small.

Original: We also provide results for other popular choices of θ ( C )

Original: = (cid:107) C (cid:107) ∗ and θ ( C )

Original: = (cid:107) C (cid:107) F , which have qualitatively similar optimal geometries, in the Supplementary Material.

Original: Theorem 2.
Revised: Theorem 2.

Original: Optimal solutions to the problems
Revised: Optimal solutions to the problems

Original: [Equation]
Revised: [Equation]

Original: are characterized by the set
Revised: are characterized by the set

Original: [Equation]
Revised: [Equation]

Original:  [Equation] where P ∈ R N × N is an arbitrary signed-permutation matrix and z ∈ R d is an arbitrary vector which satisﬁes (cid:107) z (cid:107) 2 F ≥ τ/ 2 for (P1) and z 2 i ≥ τ/ (2 d ) , ∀ i ∈ [1 , d ] for (P2) .
Revised: where P ∈ R N × N is an arbitrary signed-permutation matrix and z ∈ R d is an arbitrary vector which satisﬁes (cid:107) z (cid:107) 2 F ≥ τ/ 2 for (P1) and z 2 i ≥ τ/ (2 d ) , ∀ i ∈ [1 , d ] for (P2) .

Original: From the above result, we have shown that if Z is normalized to have a lower-bounded norm on either the entire embedded representation or for each row, then the effect of the F ( Z , C ) loss will be largely similar to the situation described by Proposition 1 in the sense that the loss will still attempt to push all of the points to 0 with the exception of two points, which will be copies of each other (potentially to within a sign-ﬂip).
Revised: From the above result, we have shown that if Z is normalized to have a lower-bounded norm on either the entire embedded representation or for each row, then the effect of the F ( Z , C ) loss will be largely similar to the situation described by Proposition 1 in the sense that the loss will still attempt to push all of the points to 0 with the exception of two points, which will be copies of each other (potentially to within a sign-ﬂip).

Original: Again, the optimal embedded representation is clearly ill-posed for clustering since all but two of the points are trivially driven towards 0 in the embedded space.
Revised: Again, the optimal embedded representation is clearly ill-posed for clustering since all but two of the points are trivially driven towards 0 in the embedded space.

Original: 2. I NSTANCE N ORMALIZATION
Revised: In addition, we also present a result similar to Theorem 2 when C is regularized by any Schatten- p norm 1 , which includes two other popular choices of regularization that have appeared in the literature – the Frobenius norm θ ( C ) = (cid:107) C (cid:107) F (for p = 2 ) and the nuclear norm θ ( C ) = (cid:107) C (cid:107) ∗ (for p = 1 ) – as special cases. Theorem 3.

Revised: Optimal solutions to the problems

Revised: [Equation]

Revised: [Equation]

Revised: where (cid:107) C (cid:107) S p is any Schatten- p norm on C , are characterized by the set

Revised: [Equation]

Revised: where z ∈ R d is an arbitrary vector which satisﬁes (cid:107) z (cid:107) 2 F ≥ τ for (P1) and z 2 i ≥ τd , ∀ i for (P2) .

Revised: Again, note that this is obviously not a good geometry for successful spectral clustering as all the points in the dataset are simply arranged on a single line and the optimal C is a rank-one matrix.

Revised: 2. I NSTANCE N ORMALIZATION

Original: To explicitly prevent the case where most of the points in the embedded space are trivially driven to 0 as in the prior two normalization schemes, another potential normalization strategy which has been proposed for (2) is to use Instance Normalization Peng et al.
Revised: To explicitly prevent the case where most of the points in the embedded space are trivially driven to 0 as in the prior two normalization schemes, another potential normalization strategy which has been proposed for (2) is to use Instance Normalization Peng et al.

Original: (2017), where the (cid:96) 2 norm of each embedded data point is constrained to be equal to some constant.
Revised: (2017), where the (cid:96) 2 norm of each embedded data point is constrained to be equal to some constant.

Original: Here again we will see that this results in somewhat trivial data geometries.
Revised: Here again we will see that this results in somewhat trivial data geometries.

Original: Speciﬁcally, we will again focus on the choice of the θ SSC ( C ) regularization function when we have exact equality, Z = ZC , for simplicity of presentation.
Revised: Speciﬁcally, we will again focus on the choice of the θ SSC ( C ) regularization function when we have exact equality, Z = ZC , for simplicity of presentation.

Original: From this we have the following result:
Revised: From this we have the following result:

Original: Theorem 3.
Revised: Theorem 4.

Original: Optimal solutions to the problem
Revised: Optimal solutions to the problem

Original: [Equation]
Revised: [Equation]

Original: must have the property that for any column in Z ∗ , Z ∗ i , there exists another column, Z ∗ j
Revised: must have the property that for any column in Z ∗ , Z ∗ i , there exists another column, Z ∗ j

Original: ( i (cid:54) =
Revised: ( i (cid:54) =

Original: j ) , such that Z ∗ i
Revised: j ) , such that Z ∗ i

Original: = s i,j Z ∗ j where s i,j ∈ {− 1 , 1 } .
Revised: = s i,j Z ∗ j where s i,j ∈ {− 1 , 1 } .

Original: Further, (cid:107) C ∗ i (cid:107) 1 = 1 ∀ i and C i,j (cid:54) = 

Revised: Further, (cid:107) C ∗ i (cid:107) 1 = 1 ∀ i and C i,j (cid:54) =

Original: = ⇒
Revised: = ⇒

Original: Z ∗ i = ±
Revised: Z ∗ i = ±

Original: Z ∗ j .
Revised: Z ∗ j .

Original:  The above result is quite intuitive in the sense that because a given point cannot use itself in its representation (due to the diag( C ) = 0 constraint), the next best thing is to have an exact copy of itself in another column.
Revised: [Figure] The above result is quite intuitive in the sense that because a given point cannot use itself in its representation (due to the diag( C ) = 0 constraint), the next best thing is to have an exact copy of itself in another column.

Original: While this result is more conducive to successful clustering in the sense that points which are close in the embedded space are encouraged to ‘merge’ into a single point, there are still numerous pathological geometries that can result.
Revised: While this result is more conducive to successful clustering in the sense that points which are close in the embedded space are encouraged to ‘merge’ into a single point, there are still numerous pathological geometries that can result.

Original: Speciﬁcally, there is no constraint on the number of ‘distinct’ points in the representation (i.e., the number of vectors which are not copies of each other to within a sign ﬂip), other than it must be less than N/ 2 .
Revised: Speciﬁcally, there is no constraint on the number of ‘distinct’ points in the representation (i.e., the number of vectors which are not copies of each other to within a sign ﬂip), other than it must be less than N/ 2 .

Original: As a result, the optimal C ∗ matrix can also contain an arbitrary number (in the range [1 , N/ 2] ) of connected components in the afﬁnity graph, resulting in somewhat arbitrary spectral clustering.
Revised: As a result, the optimal C ∗ matrix can also contain an arbitrary number (in the range [1 , N/ 2] ) of connected components in the afﬁnity graph, resulting in somewhat arbitrary spectral clustering.

Original: Example of Degenerate Geometry with Simple Networks.
Revised: Example of Degenerate Geometry with Simple Networks.

Original: In section 2.1 we discussed how if the encoder/decoder are highly expressive then the optimal embedding will approach the solutions we give in our theoretical analysis.
Revised: In section 2.1 we discussed how if the encoder/decoder are highly expressive then the optimal embedding will approach the solutions we give in our theoretical analysis.

Original: Here we show that trivial embeddings can also occur with relatively simple encoders/decoders.
Revised: Here we show that trivial embeddings can also occur with relatively simple encoders/decoders.

Original: Speciﬁcally, consider basic encoder/decoder architectures which consist of two afﬁne mappings with a ReLU non-linearity (denoted as ( · ) + ) on the hidden layer:
Revised: Speciﬁcally, consider basic encoder/decoder architectures which consist of two afﬁne mappings with a ReLU non-linearity (denoted as ( · ) + ) on the hidden layer:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Figure] where the linear operators ( W matrices) can optionally be constrained (for example for convolution operations W could be required to be a Toeplitz matrix).
Revised: where the linear operators ( W matrices) can optionally be constrained (for example for convolution operations W could be required to be a Toeplitz matrix).

Original: Now if we have that the embedded dimension d is equal to the data dimension d x we will say that linear operators can express identity if there exists parameters ( W 2 , W 1 ) such that W 2 W 

Revised: Now if we have that the embedded dimension d is equal to the data dimension d x we will say that linear operators can express identity if there exists parameters ( W 2 , W 1 ) such that W 2 W

Original: =
Revised: =

Original: I .
Revised: I .

Original: Note that if the architectures in (11) are fully-connected this implies that for a general X the number of hidden units is greater than or equal to d = d x , while if the linear operators are convolutions, then we only need one convolutional channel (with the kernel being the delta function).
Revised: Note that if the architectures in (14) are fully-connected this implies that for a general X the number of hidden units is greater than or equal to d = d x , while if the linear operators are convolutions, then we only need one convolutional channel (with the kernel being the delta function).

Original: Within this setting we have the following result:
Revised: Within this setting we have the following result:

Original: Proposition 2.
Revised: Proposition 2.

Original: Consider encoder and decoder networks with the form given in (11) where the linear operators in both the encoder/decoder can express identity.
Revised: Consider encoder and decoder networks with the form given in (14) where the linear operators in both the encoder/decoder can express identity.

Original: Then, given any dataset X ∈ R d x × N  there exist network parameters ( W e , W d ) which satisfy the following:
Revised: Then, given any dataset X ∈ R d x × N and τ > 0 there exist network parameters ( W e , W d ) which satisfy the following:

Original: Embedded points are arbitrarily close: (cid:107) Φ E
Revised: Embedded points are arbitrarily close: (cid:107) Φ E

Original: ( X i , W e ) − Φ E ( X j , W e ) (cid:107) ≤ (cid:15) ∀ ( i, j ) and ∀ (cid:15) > 0 .
Revised: ( X i , W e ) − Φ E ( X j , W e ) (cid:107) ≤ (cid:15) ∀ ( i, j ) and ∀ (cid:15) > 0 .

Original: Embedded points have norm arbitrarily close to 1: |(cid:107) Φ E ( X i , W e ) (cid:107) F − 1 | ≤ (cid:15) ∀ i and ∀ (cid:15) > 0 .
Revised: Embedded points have norm arbitrarily close to τ : |(cid:107) Φ E ( X i , W e ) (cid:107) F − τ | ≤ (cid:15) ∀ i and ∀ (cid:15) > 0 .

Original: Embedded points can be decoded exactly: Φ D (Φ E ( X i , W e ) , W d ) =
Revised: Embedded points can be decoded exactly: Φ D (Φ E ( X i , W e ) , W d ) =

Original: X i , ∀ i .
Revised: X i , ∀ i .

Original: From the above simple example, we can see that even with very simple network architectures (i.e., not necessarily highly expressive) it is still possible to have solutions which are arbitrarily close to the global optimum described in Theorem 3 while still having a perfect reconstruction of X .
Revised: From the above simple example, we can see that even with very simple network architectures (i.e., not necessarily highly expressive) it is still possible to have solutions which are arbitrarily close to the global optimum described in Theorem 4 while still having a perfect reconstruction of X .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Here, we present experiments on both real and synthetic data that verify our theoretical predictions experimentally.
Revised: Here, we present experiments on both real and synthetic data that verify our theoretical predictions experimentally.

Original: We ﬁrst evaluate the Autoencoder Regularization form given in (3) by repeating all of the experiments from Ji et al.
Revised: We ﬁrst evaluate the Autoencoder Regularization form given in (3) by repeating all of the experiments from Ji et al.

Original: (2017), showing that the optimization problem never reaches a stationary point due to the pathology described by Prop 1 (see Supplementary Material) and that the improvement in performance is largely attributable to an ad-hoc post-processing step.
Revised: (2017), showing that the optimization problem never reaches a stationary point due to the pathology described by Prop 1 (see Supplementary Material) and that the improvement in performance is largely attributable to an ad-hoc post-processing step.

Original: Then, we present experiments on a simple synthetic dataset to illustrate our theoretical results.
Revised: Then, we present experiments on a simple synthetic dataset to illustrate our theoretical results.

Original: 3. R EPEATING THE E XPERIMENTS OF J I ET AL . (2017)

Original: First we use the code provided by the authors of Ji et al.

Original: (2017) to repeat all of their original clustering experiments on the Extended Yale-B (38 faces), ORL, and COIL100 datasets.

Original: As baseline methods, we also train the autoencoder network without the F ( Z , C ) term (i.e., γ = 0 and C ﬁxed at I in (3)), then using the autoencoder as a ﬁxed feature extractor perform subspace clustering on embedded features.

Original: Likewise, we also perform subspace clustering on the raw data (see Supplementary Material for further details).

Original: [Table]

Original: Original Data Reconstructed Data
Revised: Original Data Reconstructed Data

Original: AutoEncoderAutoEncoder + Self-Express
Revised: AutoEncoderAutoEncoder + Self-Express

Original: Original Data Domain
Revised: Original Data Domain

Original: Embedded Data Domain |C| |C| - log color scale
Revised: Embedded Data Domain |C| |C| - log color scale

Original: D a t a s e t N o r m 0.
Revised: D a t a s e t N o r m 0.

Revised: 3. R EPEATING THE E XPERIMENTS OF J I ET AL . (2017)

Revised: First we use the code provided by the authors of Ji et al.

Revised: (2017) to repeat all of their original clustering experiments on the Extended Yale-B (38 faces), ORL, and COIL100 datasets.

Revised: As baseline methods, we also train the autoencoder network without the F ( Z , C ) term (i.e., γ = 0 and C ﬁxed at I in (3)), then using the autoencoder as a ﬁxed feature extractor perform subspace clustering on embedded features.

Revised: Likewise, we also perform subspace clustering on the raw data (see Supplementary Material for further details).

Original: In addition to proposing the model in (3), the authors of Ji et al.
Revised: In addition to proposing the model in (3), the authors of Ji et al.

Original: (2017) also implement a somewhat arbitrary post-processing of the C matrix recovered from the SEDSC model before the ﬁnal spectral clustering, which involves 1) entrywise hard thresholding, 2) applying the shape interaction matrix method Costeira & Kanade (1995) to C , and 3) raising C to a power, entrywise.
Revised: (2017) also implement a somewhat arbitrary post-processing of the C matrix recovered from the SEDSC model before the ﬁnal spectral clustering, which involves 1) entrywise hard thresholding, 2) applying the shape interaction matrix method Costeira & Kanade (1995) to C , and 3) raising C to a power, entrywise.

Original: Likewise, many subsequent works on SEDSC follow Ji et al.
Revised: Likewise, many subsequent works on SEDSC follow Ji et al.

Original: and employ very similar post-processing steps on C .
Revised: and employ very similar post-processing steps on C .

Original: As shown in Figure 1 and Table 1 there is little observed beneﬁt for using SEDSC, as it achieves comparable (or worse) performance than baseline methods in almost all settings when the post-processing of C is applied consistently (or not) across all methods.
Revised: As shown in Figure 1 and Table 1 there is little observed beneﬁt for using SEDSC, as it achieves comparable (or worse) performance than baseline methods in almost all settings when the post-processing of C is applied consistently (or not) across all methods.

Original: 3. S YNTHETIC D ATA E XPERIMENTS
Revised: 3. S YNTHETIC D ATA E XPERIMENTS

Original: Finally, to illustrate our theoretical predictions we construct a simple synthetic dataset which consists of 100 points drawn from the union of two parabolas in R 2 , where the space of the embedding is also R 2 .
Revised: Finally, to illustrate our theoretical predictions we construct a simple synthetic dataset which consists of 100 points drawn from the union of two parabolas in R 2 , where the space of the embedding is also R 2 .

Original: We then train the model given in (3) with θ ( C )
Revised: We then train the model given in (3) with θ ( C )

Original: =
Revised: =

Original: θ SSC ( C )
Revised: θ SSC ( C )

Original: = (cid:107) C (cid:107) 

Revised: = (cid:107) C (cid:107)

Original: + δ (diag( C ) = 0 ) , the encoder/decoder networks being simple single-hidden-layer fully-connected networks withhidden units, and ReLU activations on the hidden units.
Revised: + δ (diag( C ) = 0 ) , the encoder/decoder networks being simple single-hidden-layer fully-connected networks withhidden units, and ReLU activations on the hidden units.

Original: Fig 2 shows the resulting solution that results when we directly add a normalization operator to the encoder network which normalizes the output of the encoder to have unit Frobenius norm over the entire dataset (Dataset Normalization).
Revised: Fig 2 shows the resulting solution that results when we directly add a normalization operator to the encoder network which normalizes the output of the encoder to have unit Frobenius norm over the entire dataset (Dataset Normalization).

Original: Additional experiments for other normalization schemes and a description of the details of our synthetic data experiments can be found in the Supplementary Material.
Revised: Additional experiments for other normalization schemes and a description of the details of our synthetic data experiments can be found in the Supplementary Material.

Original: From Figure 2 one can see that our theoretical predictions are largely conﬁrmed experimentally.
Revised: From Figure 2 one can see that our theoretical predictions are largely conﬁrmed experimentally.

Original: Namely, one sees that when the full SEDSC model is trained the embedded representation is very close to as predicted by Theorem 2, with almost all of the embedded points (Left Center - Red points) being near the origin with the exception of two points, which are co-linear with each other. 
Revised: Namely, one sees that when the full SEDSC model is trained the embedded representation is very close to as predicted by Theorem 2, with almost all of the embedded points (Left Center - Red points) being near the origin with the exception of two points, which are co-linear with each other. [Table]

Original: Likewise, the C matrix is dominated by two non-zero entries with the remaining non-zero entries only appearing on the log-scale color scale.
Revised: Likewise, the C matrix is dominated by two non-zero entries with the remaining non-zero entries only appearing on the log-scale color scale.

Original: We note that as this is a very simple dataset (i.e., two parabolas without any added noise) one would expect most reasonable manifold clustering/learning algorithms to succeed; however, due to the deﬁciencies of the SEDSC model we have shown in our analysis a trivial solution results.
Revised: We note that as this is a very simple dataset (i e., two parabolas without any added noise) one would expect most reasonable manifold clustering/learning algorithms to succeed; however, due to the deﬁciencies of the SEDSC model we have shown in our analysis a trivial solution results.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have presented a theoretical and experimental analysis of the Self-Expressive Deep Subspace
Revised: We have presented a theoretical and experimental analysis of the Self-Expressive Deep Subspace

Original: Clustering (SEDSC) model.
Revised: Clustering (SEDSC) model.

Original: We have shown that in many cases the SEDSC model is ill-posed and results in trivial data geometries in the embedded space.
Revised: We have shown that in many cases the SEDSC model is ill-posed and results in trivial data geometries in the embedded space.

Original: Further, our attempts to replicate previously reported experiments lead us to conclude that much of the claimed beneﬁt of SEDSC is attributable to other factors such as post-processing of the recovered encoding matrix, C, and not the SEDSC model itself.
Revised: Further, our attempts to replicate previously reported experiments lead us to conclude that much of the claimed beneﬁt of SEDSC is attributable to other factors such as post-processing of the recovered encoding matrix, C and not the SEDSC model itself.

Original: Overall, we conclude that considerably more attention needs to be given to the issues we have raised in this paper in terms of both how models for this problem are designed and how they are evaluated to ensure that one arrives at meaningful solutions and can clearly demonstrate the performance of the resulting model without other confounding factors.
Revised: Overall, we conclude that considerably more attention needs to be given to the issues we have raised in this paper in terms of both how models for this problem are designed and how they are evaluated to ensure that one arrives at meaningful solutions and can clearly demonstrate the performance of the resulting model without other confounding factors.