Original: C O - MANIFOLD LEARNING WITH MISSING DATA
Revised: C O - MANIFOLD LEARNING WITH MISSING DATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Representation learning is typically applied to only one mode of a data matrix, either its rows or columns.
Revised: Representation learning is typically applied to only one mode of a data matrix, either its rows or columns.

Original: Yet in many applications, there is an underlying geometry to both the rows and the columns.
Revised: Yet in many applications, there is an underlying geometry to both the rows and the columns.

Original: We propose utilizing this coupled structure to perform co-manifold learning: uncovering the underlying geometry of both the rows and the columns of a given matrix, where we focus on a missing data setting.
Revised: We propose utilizing this coupled structure to perform co-manifold learning: uncovering the underlying geometry of both the rows and the columns of a given matrix, where we focus on a missing data setting.

Original: Our unsupervised approach consists of three components.
Revised: Our unsupervised approach consists of three components.

Original: We ﬁrst solve a family of optimization problems to estimate a complete matrix at multiple scales of smoothness.
Revised: We ﬁrst solve a family of optimization problems to estimate a complete matrix at multiple scales of smoothness.

Original: We then use this collection of smooth matrix estimates to compute pairwise distances on the rows and columns based on a new multi-scale metric that implicitly introduces a coupling between the rows and the columns.
Revised: We then use this collection of smooth matrix estimates to compute pairwise distances on the rows and columns based on a new multi-scale metric that implicitly introduces a coupling between the rows and the columns.

Original: Finally, we construct row and column representations from these multi-scale metrics.
Revised: Finally, we construct row and column representations from these multi-scale metrics.

Original: We demonstrate that our approach outperforms competing methods in both data visualization and clustering.
Revised: We demonstrate that our approach outperforms competing methods in both data visualization and clustering.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dimension reduction plays a key role in exploratory data analysis, data visualization, clustering and classiﬁcation.
Revised: Dimension reduction plays a key role in exploratory data analysis, data visualization, clustering and classiﬁcation.

Original: Techniques range from the classical PCA and nonlinear manifold learning to deep autoencoders (Tenenbaum et al., 2000; Roweis & Saul, 2000; Belkin & Niyogi, 2003; Coifman &
Revised: Techniques range from the classical PCA and nonlinear manifold learning to deep autoencoders (Tenenbaum et al., 2000; Roweis & Saul, 2000; Belkin & Niyogi, 2003; Coifman &

Original: Lafon, 2006; Vincent et al., 2008; Rifai et al., 2011; Kingma & Welling, 2014).
Revised: Lafon, 2006; Vincent et al., 2008; Rifai et al., 2011; Kingma & Welling, 2014).

Original: These techniques focus on only one mode of the data, often the observations (columns) which areare measurements in a high-dimensional feature space (rows), and exploit correlations among the features to reduce the dimension of the feature vectors and obtain the underlying low-dimensional geometry of the observations.
Revised: These techniques focus on only one mode of the data, often the observations (columns) which are measurements in a high-dimensional feature space (rows), and exploit correlations among the features to reduce the dimension of the feature vectors and obtain the underlying low-dimensional geometry of the observations.

Original: Yet for many data matrices, for example in gene expression studies, recommendation systems,  and word-document analysis, correlations exist among both observations and features.
Revised: Yet for many data matrices, for example in gene expression studies, recommendation systems, sensor networks, and word-document analysis, correlations exist among both observations and features.

Original: In these cases, we seek a method that can exploit the correlations among both the rows and columns of a data matrix to better learn lower-dimensional representations of both.
Revised: In these cases, we seek a method that can exploit the correlations among both the rows and columns of a data matrix to better learn lower-dimensional representations of both.

Original: Biclustering methods, which extract distinct biclusters along both rows and columns, give a partial solution to performing simultaneous dimension reduction on the rows and columns of a data matrix.
Revised: Biclustering methods, which extract distinct biclusters along both rows and columns, give a partial solution to performing simultaneous dimension reduction on the rows and columns of a data matrix.

Original: Such methods, however, can break up a  smooth geometry into artiﬁcial clusters.
Revised: In certain settings, however, assuming a bi-clustering model is too restrictive and results in breaking up smooth geometries into artiﬁcial disjoint clusters that do not match the actual structure of the data. This can occur when the true geometry is one of overlapping rather than disjoint clusters, for example in word-document analysis (Ahn et al., 2010), or when the underlying structure is not one of clusters at all but rather a smooth manifold (Gavish & Coifman, 2012).

Original: A more general viewpoint to consider is that data matrices possess geometric relationships between their rows (features) and columns (observations) such that both modes lie on low-dimensional manifolds .
Revised: Thus, we consider a more general viewpoint: data matrices possess geometric relationships between their rows (features) and columns (observations) such that both modes lie on low-dimensional manifolds .

Original: Furthermore, the relationships between the rows may be informed by the relationships between the columns, and vice versa.
Revised: Furthermore, the relationships between the rows may be informed by the relationships between the columns, and vice versa.

Original: Several recent papers (Gavish & Coifman, 2012; Ankenman, 2014; Mishne et al., 2016; Shahid et al., 2016; Mishne et al., 2017; Yair et al., 2017) exploit this coupled relationship to co-organize matrices and infer underlying row and column embeddings.
Revised: Several recent papers (Gavish & Coifman, 2012; Ankenman, 2014; Mishne et al., 2016; Shahid et al., 2016; Mishne et al., 2017; Yair et al., 2017) exploit this coupled relationship to coorganize matrices and infer underlying row and column embeddings.

Original: Further complicating the story is that such matrices may suffer from missing values, due to measurement corruptions and limitations.
Revised: Further complicating the story is that such matrices may suffer from missing values, due to measurement corruptions and limitations.

Original: Missing values can sabotage efforts to learn the low dimensional manifold underlying the data.
Revised: Missing values can sabotage efforts to learn the low dimensional manifold underlying the data.

Original: Speciﬁcally, kernel-based methods rely on calculating a similarity matrix between observations, whose eigendecomposition yields a new embedding of the data.
Revised: Speciﬁcally, kernel-based methods rely on calculating a similarity matrix between observations, whose eigendecomposition yields a new embedding of the data.

Original: As the number of missing entries grows, the distances between points are increasingly distorted, resulting in poor representation of the data in the low-dimensional space (Gilbert & Sonthalia, 2018).
Revised: As the number of missing entries grows, the distances between points are increasingly distorted, resulting in poor representation of the data in the low-dimensional space (Gilbert & Sonthalia, 2018).

Original: Matrix completion algorithms assume the data is low-rank and ﬁll in the missing values by ﬁtting a global linear subspace to the data.
Revised: Matrix completion algorithms assume the data is low-rank and ﬁll in the missing values by ﬁtting a global linear subspace to the data.

Original: Yet, this may fail when the data lies on a nonlinear manifold. 
Revised: Yet, this may fail when the data lies on a nonlinear manifold. [Figure]

Original: Manifold learning in the missing data scenario has been addressed by a few recent papers.
Revised: Manifold learning in the missing data scenario has been addressed by a few recent papers.

Original: Nonlinear Principle Component Analysis (NLPCA) (Scholz et al., 2005) uses an autoencoder neural network, where the middle layer serves to learn a low-dimensional embedding of the data, and the[Figure] trained autoencoder is used to ﬁll in missing values.
Revised: Nonlinear Principle Component Analysis (NLPCA) (Scholz et al., 2005) uses an autoencoder neural network, where the middle layer serves to learn a low-dimensional embedding of the data, and the trained autoencoder is used to ﬁll in missing values.

Original: Missing Data Recovery through Unsupervised
Revised: Missing Data Recovery through Unsupervised

Original: Regression (Carreira-Perpin & Lu, 2011)
Revised: Regression (Carreira-Perpin & Lu, 2011)

Original: ﬁrst ﬁlls in the missing data with linear matrix completion methods, then calculates a non-linear embedding of the data and incorporates this embedding in an optimization problem to ﬁll in the missing values.
Revised: ﬁrst ﬁlls in the missing data with linear matrix completion methods, then calculates a non-linear embedding of the data and incorporates this embedding in an optimization problem to ﬁll in the missing values.

Original: Recently Gilbert & Sonthalia (2018) proposed
Revised: Recently Gilbert & Sonthalia (2018) proposed

Original: MR-MISSING which ﬁrst calculates an initial distance matrix using only non-missing entries and then uses the increase-only-metric-repair (IOMR) method to ﬁx the distance matrix so that it is a metric from which they calculate an embedding.
Revised: MR-MISSING which ﬁrst calculates an initial distance matrix using only non-missing entries and then uses the increase-only-metric-repair (IOMR) method to ﬁx the distance matrix so that it is a metric from which they calculate an embedding.

Original: None of these methods consider the co-manifold setting, where the coupled structure of the rows and the columns can be used to ﬁll in the data, and to calculate an embedding.
Revised: None of these methods consider the co-manifold setting, where the coupled structure of the rows and the columns can be used to ﬁll in the data, and to calculate an embedding.

Original: In this paper, we introduce a new method for performing joint dimension reduction on the rows and columns of a data matrix, which we term co-manifold learning, in the missing data setting.
Revised: In this paper, we introduce a new method for performing joint dimension reduction on the rows and columns of a data matrix, which we term co-manifold learning, in the missing data setting.

Original: We build on two recent lines of work on co-organizing the rows and columns of a data matrix (Gavish & Coifman, 2012; Mishne et al., 2016; 2017) and convex optimization methods for performing coclustering (Chi et al., 2017; 2018).
Revised: We build on two recent lines of work on co-organizing the rows and columns of a data matrix (Gavish & Coifman, 2012; Mishne et al., 2016; 2017) and convex optimization methods for performing coclustering (Chi et al., 2017; 2018).

Original: The former provide a ﬂexible framework for jointly organizing rows and columns but lacks algorithmic convergence guarantees.
Revised: The former provide a ﬂexible framework for jointly organizing rows and columns but lacks algorithmic convergence guarantees.

Original: The latter provides convergence guarantees but does not take full advantage of the multiple scales of the data revealed in the solution.
Revised: The latter provides convergence guarantees but does not take full advantage of the multiple scales of the data revealed in the solution.

Original: Instead of  inferring biclusters at a single scale, we use a multi-scale optimization framework to ﬁll in the data, imposing smoothness on both the rows and the columns at ﬁne to coarse scales.
Revised: In the ﬁrst stage of our approach, rather than inferring biclusters at a single scale, we use a multiscale optimization framework to ﬁll in the data at ﬁne to coarse scales while imposing smoothness on both the rows and the columns.

Original: The scale of the solution is encoded in a pair of joint cost parameters along the rows and columns.
Revised: The scales of the solutions are encoded in a pair of joint cost parameters along the rows and columns.

Original: The solutions to the optimization for each such pair yields a smooth estimate of the data along both the rows and columns, whose values are used to ﬁll in the missing values of the given matrix.

Original: We deﬁne a new multi-scale metric based on the ﬁlled-in matrix across all scales, which we  use to calculate nonlinear embeddings of the rows and columns.
Revised: Next, we deﬁne a new multi-scale metric based on the ﬁlled-in matrix across all scales, which we then use to calculate nonlinear embeddings of the rows and columns.

Original: Thus our approach yields three results: a collection of smoothed estimates of the matrix, pairwise distances on the rows and columns that better estimate the geometry of the complete data matrix, and corresponding nonlinear embeddings (see Figure 1).
Revised: Thus our approach yields three results: a collection of smoothed estimates of the matrix, pairwise distances on the rows and columns that better estimate the geometry of the complete data matrix, and corresponding nonlinear embeddings (see Figure 1).

Original: We will demonstrate in experimental results that our method reveals meaningful representations in coupled datasets with missing entries, whereas other methods are capable of revealing a meaningful representation only along one of the modes.
Revised: We will demonstrate in experimental results that our method reveals meaningful representations in coupled datasets with missing entries, whereas other methods are capable of revealing a meaningful representation only along one of the modes.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: We present the optimization framework in Section 2, the new multi-scale metric for co-manifold learning in Section 3 and experimental results in Section 4.
Revised: We present the optimization framework in Section 2, the new multi-scale metric for co-manifold learning in Section 3 and experimental results in Section 4.

Original:  C O - CLUSTERING AN I NCOMPLETE D ATA M ATRIX
Revised:  C O - CLUSTERING AN I NCOMPLETE D ATA M ATRIX

Original: We seek a collection of complete matrix approximations of a partially observed data matrix X ∈ R m × n that have been smoothed along their row and columns to varying degrees.
Revised: We seek a collection of complete matrix approximations of a partially observed data matrix X ∈ R m × n that have been smoothed along their row and columns to varying degrees.

Original: This collection will serve in computing row and column multi-scale distances to better estimate therow and column pairwise distance of the complete data matrix.
Revised: This collection will serve in computing row and column multi-scale distances to better estimate the pairwise distances of the complete data matrix.

Original: Let [ m ] denote the set of indices { 1 , . . .
Revised: Let [ m ] denote the set of indices { 1 , . . .

Original: , m } , and let Θ ⊆
Revised: , m } , and let Θ ⊆

Original: [ m ] × [ n ] be a subset of the indices that correspond to observed entries of X , and let P Θ denote the projection operator of m × n matrices onto an index set Θ , i.e. [ P Θ ( X )] ij is x ij if ( i, j ) ∈ Θ and is 0 otherwise.
Revised: [ m ] × [ n ] be a subset of the indices that correspond to observed entries of X , and let P Θ denote the projection operator of m × n matrices onto an index set Θ , i.e. [ P Θ ( X )] ij is x ij if ( i, j ) ∈ Θ and is 0 otherwise.

Original: We seek a minimizer U ( γ r , γ c ) of the following function.
Revised: We seek a minimizer U ( γ r , γ c ) of the following function.

Original: [Equation]
Revised: [Equation]

Original: The quadratic term quantiﬁes how well U approximates X on the observed entries, while the two roughness penalties, J r ( U ) and J c ( U ) , incentivize smoothness across the rows and columns of U .
Revised: The quadratic term quantiﬁes how well U approximates X on the observed entries, while the two roughness penalties, J r ( U ) and J c ( U ) , incentivize smoothness across the rows and columns of U .

Original: The nonnegative parameters γ r and γ c tune the tradeoff between how well U agrees with X over Θ and how smooth U is with respect to its rows and columns.
Revised: The nonnegative parameters γ r and γ c tune the tradeoff between how well U agrees with X over Θ and how smooth U is with respect to its rows and columns.

Original: By tuning γ r and γ c , we obtain estimates of X at varying scales of row and column smoothness.
Revised: By tuning γ r and γ c , we obtain estimates of X at varying scales of row and column smoothness.

Original: In this paper, we use roughness penalties of the following forms
Revised: In this paper, we use roughness penalties of the following forms

Original: [Equation]
Revised: [Equation]

Original: where U i · ( U · i ) denotes the i th row (column) of the matrix U .
Revised: where U i · ( U · i ) denotes the i th row (column) of the matrix U .

Original: The index sets E r and E c denote the edge sets of row and column graphs that encode a preliminary data-driven assessment of the similarities between rows and columns of the data matrix.
Revised: The index sets E r and E c denote the edge sets of row and column graphs that encode a preliminary data-driven assessment of the similarities between rows and columns of the data matrix.

Original: The function Ω , which maps [0 , ∞ ) into [0 , ∞ ) , will be explained shortly.
Revised: The function Ω , which maps [0 , ∞ ) into [0 , ∞ ) , will be explained shortly.

Revised: Variations on the optimization problem of minimizing (1) have been previously proposed in the literature.

Revised: When there is no data missing, i.e. Θ = [ m ] × [ n ] and Ω is a linear mapping, minimizing the objective in (1) produces a convex biclustering problem (Chi et al., 2017).

Revised: Additionally, if either γ r or γ c is zero, then we obtain convex clustering (Pelckmans et al., 2005; Hocking et al., 2011; Lindsten et al., 2011; Chi & Lange, 2015).

Revised: If we take Ω to be a nonlinear concave function, problem (1) reduces to an instance of concave penalized regressionbased clustering (Pan et al., 2013;

Revised: Marchetti & Zhou, 2014; Wu et al., 2016).

Original: The convergence properties of our co-clustering procedure will rely on the following two assumptions.
Revised: The convergence properties of our co-clustering procedure will rely on the following two assumptions.

Original: Assumption 2.1 The row and column graphs E r and E c are connected, i.e. the row graph is connected if for any pair of rows, indexed by i and j with i (cid:54) =
Revised: Assumption 2.1 The row and column graphs E r and E c are connected, i.e. the row graph is connected if for any pair of rows, indexed by i and j with i (cid:54) =

Original: j , there exists a sequence of indices i
Revised: j , there exists a sequence of indices i

Original: → k → · · · → l → j such that ( i, k ) , . . .
Revised: → k → · · · → l → j such that ( i, k ) , . . .

Original: , ( l, j ) ∈
Revised: , ( l, j ) ∈

Original: E r .
Revised: E r .

Original: A column graph is connected under analogous conditions.
Revised: A column graph is connected under analogous conditions.

Original: Assumption 2.2 The function Ω : [0 , ∞ ) (cid:55)→ [0 , ∞ ) is (i) concave and continuously differentiable on (0 , ∞ ) , (ii) vanishes at the origin, i.e. Ω(0) = 0 , (iii) is increasing on [0 , ∞ ) , and (iv) has ﬁnite directional derivative at the origin. 
Revised: Assumption 2.2 The function Ω : [0 , ∞ ) (cid:55)→ [0 , ∞ ) is (i) concave and continuously differentiable on (0 , ∞ ) , (ii) vanishes at the origin, i.e. Ω(0) = 0 , (iii) is increasing on [0 , ∞ ) , and (iv) has ﬁnite directional derivative at the origin. Figure 4 in Appendix A plots an example of a function Ω that satisﬁes Assumption 2.2.

Original: Variations on the optimization problem of minimizing (1) have been previously proposed in the literature.
Revised: For concreteness, in the rest of this paper, we use the following function Ω Ω( [Equation]

Original: When there is no data missing, i.e. Θ = [ m ] × [ n ] and Ω  is a linear mapping, minimizing the objective in (1) produces a convex biclustering problem (Chi et al., 2017).
Revised: where (cid:15) is a small positive number, e.g. 10 − 12 . The key feature of Ω in (3), which satisﬁes Assumption 2.2, is that when it is used in the roughness penalties (2) small differences between rows and columns are penalized signiﬁcantly more than larger differences resulting in more aggressive smoothing of small noisy variations and leaving intact more signiﬁcant systematic variations. Speciﬁcally, functions that satisfy Assumption 2.2 are not differentiable at the origin and therefore incentivize sparsity in the differences in the rows and columns.

Original: Additionally, if either γ r or γ c is zero, then we obtain convex clustering (Pelckmans et al., 2005; Hocking et al., 2011; Lindsten et al., 2011; Chi & Lange, 2015).
Revised: Consequently, small noisy variations between pairs of rows and columns are eliminated completely for sufﬁciently large γ r and γ c .

Original: If we take Ω to be a nonlinear concave function, problem (1) reduces to an instance of concave penalized regression-based clustering (Pan et al., 2013; Marchetti & Zhou, 2014; Wu et al., 2016).
Revised: This is in contrast with commonly used quadratic penalties.

Original: Replacing J r ( U ) and J c ( U ) by quadratic row and columnLaplacian penalties
Revised: For example, replacing J r ( U ) and J c ( U ) by quadratic row and column penalties

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: ( i,j ) ∈E c (cid:107) U

Original: [Equation] U [Equation] (cid:107) 22 , gives a version of matrix completion on graphs (Kalofolias et al., 2014;
Revised: gives a version of matrix completion on graphs (Kalofolias et al., 2014;

Original: Rao et al., 2015).
Revised: Rao et al., 2015), where w ij and ˜ w ij are ﬁxed row and column weights inferred from the data.

Original: Shahid et al.
Revised: Shahid et al.

Original: (2016) also use  row and columnLaplacian penalties to perform joint linear dimension reduction on the rows and columns of the data matrix. 
Revised: (2016) also use quadratic row and column penalties to perform joint linear dimension reduction on the rows and columns of a data matrix. Penalties like the ones given in (4), unlike the ones considered in

Original: Our work generalizes both Shahid et al.
Revised: Assumption 2.2, smooth out more signiﬁcant systematic variations more aggressively and shrink, but do not completely eliminate, small noisy variations. A simple concrete example illuminates the differences between roughness penalties considered in this paper and commonly used existing penalties. Let U 1 · = (1 0) T and U 2 · = (1 + δ 0) T and take Ω to be as deﬁned in (3). Then,

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Revised: ) ≈ √ δ, and (cid:107) U 1 · − U 2 · (cid:107)= δ 2 .

Revised: Suppose there is a small difference between the ﬁrst and second rows of U , e.g. δ = 10 − 4 .

Revised: Then (cid:107) U 1 · − U 2 · (cid:107)= 10 − 4 , Ω( (cid:107) U 1 · − U 2 · (cid:107) 2 )

Revised: ≈ 10 − 2 , and (cid:107) U 1 · − U 2 · (cid:107)= 10 − 8 .

Revised: Thus, small differences are penalized the most using the concave Ω and the least by the convex quadratic penalty.

Revised: Suppose there is a large difference between the ﬁrst and second rows of U , e.g. δ = 10 4 .

Revised: Then (cid:107) U 1 · − U 2 · (cid:107)= 10 4 , Ω( (cid:107) U 1 · − U 2 · (cid:107) 2 )

Revised: ≈ 10 2 , and (cid:107)

Revised: [Equation]

Revised: Thus, large differences are penalized the least using the concave Ω and the most by the convex quadratic penalty.

Revised: The practical consequence of the differences highlighted by the example above is that the convex penalties, either when Ω is linear or quadratic, do not introduce enough smoothing for small differences and too much smoothing for large differences.

Revised: Indeed, (Pan et al., 2013; Marchetti & Zhou, 2014; Wu et al., 2016) showed that superior clustering results could be had, when only J r or J c is used, by using concave Ω .

Revised: Chi et al.

Revised: (2017) also showed that empirically that the solution to the convex biclustering problem tended to identify too many biclusters and consequently also introduced a one-step reweighted convex biclustering reﬁnement, which recovered the true biclusters more accurately in simulation experiments.

Revised: The reweighting reﬁnement can be seen as taking a single step in an iterative algorithm for minimizing (1) inexactly when Ω is concave (Chi et al., 2018;

Revised: Chi &

Revised: Steinerberger, 2018).

Revised: As our method combines biclustering of incomplete data at different scales, or values of γ r and γ c , consequently we extend the reweighting reﬁnement introduced by Chi et al.

Revised: (2017) in Section 2.1.

Revised: Our problem formulation (1) is distinct from related problem formulations in the following ways:

Original: (2016) and 
Revised: Rows and columns of U are simultaneously shrunk towards each other as the parameters γ r and γ c increase. Note that this shrinkage procedure is fundamentally different from methods like the clustered dendrogram, which independently cluster the rows and columns as well as alternating partition tree construction procedures (Gavish

Revised: Coifman

Original: Chi et al.
Revised: Mishne et al.

Revised: Our ultimate goal is not to perform matrix completion (though this is a by-product of our approach) but rather to perform joint row and column dimension reduction.

Revised: Our work generalizes both Shahid et al.

Original:  (2017) in that we seek the ﬂexibility of performing non-linear dimension reduction on the rows and columns of the data matrix and seek more general manifold organization than  co-clustered structure.
Revised: (2016) and Chi et al. (2017) in that we seek the ﬂexibility of performing non-linear dimension reduction on the rows and columns of the data matrix, i e. a more general manifold organization than a co-clustered structure.

Revised: Instead of determining an optimal single scale of the solution as in Shahid et al.

Revised: ; Chi et al.

Revised: , we recognize that the multiple scales of the different solutions can be aggregated to better estimate the underlying geometry, similar to the tree-based Earth mover’s distance proposed in Ankenman (2014);

Revised: Mishne et al.

Original: 2. C O -C LUSTERING A LGORITHM
Revised: 2. C O -C LUSTERING A LGORITHM

Original: We now introduce a majorization-minimization (MM) algorithm (Sun et al., 2017) for solving the minimization in (1).
Revised: We now introduce a majorization-minimization (MM) algorithm (Sun et al., 2017) for solving the minimization in (1).

Original: The basic strategy behind an MM algorithm is to convert a hard optimization problem into a sequence of simpler ones.
Revised: The basic strategy behind an MM algorithm is to convert a hard optimization problem into a sequence of simpler ones.

Original: The MM principle requires majorizing the objective function f ( U ) by a surrogate function g ( U | ˜U ) anchored at ˜U .Majorization is a combination of ˜X
Revised: The MM principle requires majorizing the objective function f ( U ) by a surrogate function g ( U | ˜U ) anchored at ˜U .

Original: P { U (cid:16) ˜X γ r , γ c r c 5: ˜ r
Revised: Majorization is a combination of the tangency condition g ( U | ˜U ) = f ( ˜U ) and the domination condition g ( U | ˜U ) ≥ f

Original: ← Ω (cid:48) ( (cid:107) U · − U the tangency condition g ( U | ˜U ) = f ( ˜U ) and the domination condition g ( U | ˜U ) ≥ f ( U ) for all
Revised: ( U ) for all

Original: U ∈ R m × n .
Revised: U ∈ R m × n .

Original: The associated MM algorithm is deﬁned by the iterates U t +1 =
Revised: The associated MM algorithm is deﬁned by the iterates U t +1 =

Original: arg min U g
Revised: arg min U g

Original: ( U | U t ) .
Revised: ( U | U t ) .

Original: It is straightforward to verify that the MM iterates generate a descent algorithm driving the objective function downhill, i.e. that f ( U t +1 ) ≤ f ( U t ) for all t .
Revised: It is straightforward to verify that the MM iterates generate a descent algorithm driving the objective function downhill, i.e. that f ( U t +1 ) ≤ f ( U t ) for all t .

Original: The following function
Revised: The following function

Original: [Equation]
Revised: [Equation]

Original:  κ majorizes our objective function (1) at ˜U , where κ is a constant that does not depend on U and ˜ w r,ij and ˜ w c,ij are weights that depend on ˜U , i.e. 
Revised: ,j ) ∈E c ˜ w c,ij (cid:107) U · i − U · j (cid:107) 2 + κ majorizes our objective function (1) at ˜U , where κ is a constant that does not depend on U and ˜ w r,ij and ˜ w c,ij are weights that depend on ˜U , i.e. ˜ w r,ij = Ω

Original: [Equation]
Revised: [Equation]

Original: r,ij
Revised: ( (cid:107) ˜U i · − ˜U j · (cid:107) 2 ) and ˜ w c,ij = Ω (cid:48) ( (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: (5) where Ω (cid:48) denotes the ﬁrst derivative of Ω .

Original: We give a detailed derivation of this majorization in Appendix A.
Revised: We give a detailed derivation of this majorization in Appendix A.

Original: Minimizing g ( U | ˜U ) is equivalent to minimizing the objective function of the convex biclustering problem for which efﬁcient algorithms have been introduced (Chi et al., 2017).
Revised: Minimizing g ( U | ˜U ) is equivalent to minimizing the objective function of the convex biclustering problem for which efﬁcient algorithms have been introduced (Chi et al., 2017).

Original: Thus, in the t + 1 th iteration, our MM algorithm solves a convex biclustering problem where the missing values in X have been replaced with the values of ˜U = U t and the weights ˜ w r,ij and ˜ w c,ij have been computed based on ˜U = U t according to (2).
Revised: Thus, in the t + 1 th iteration, our MM algorithm solves a convex biclustering problem where the missing values in X have been replaced with the values of ˜U = U t and the weights ˜ w r,ij and ˜ w c,ij have been computed based on ˜U = U t according to (5). Note that the weights are continuously updated throughout the optimization as opposed to the ﬁxed weights in (Chi et al., 2017).

Original:  Algorithm 1 summarizes our MM algorithm, CO - CLUSTER - MISSING , which returns a smooth output matrix U ( γ r , γ c ) , a ﬁlled-in matrix ˜X = P Θ ( X ) + P Θ c ( U ( γ r , γ c )) as well as n r and n c , which are respectively the number of distinct rows and distinct columns in U ( γ r , γ c ) .
Revised: This introduces a notion of the scale of the solution into the weights. Algorithm 1 summarizes our MM algorithm, CO - CLUSTER - MISSING , which returns a smooth output matrix U ( γ r , γ c ) , a ﬁlled-in matrix ˜X = P Θ ( X ) + P Θ c ( U ( γ r , γ c )) as well as n r and n c , which are respectively the number of distinct rows and distinct columns in U ( γ r , γ c ) .

Original: The CO - CLUSTERMISSING algorithm has the following convergence guarantee whose proof is in Appendix B.
Revised: The CO - CLUSTERMISSING algorithm has the following convergence guarantee.

Original: Proposition 1 Under Assumption 2.1 and Assumption 2.2 , the sequence U t generated by Algorithm 1 has at least one limit point, and all limit points are stationary points of (1).
Revised: Proposition 1 Under Assumption 2.1 and Assumption 2.2 , the sequence U t generated by Algorithm 1 has at least one limit point, and all limit points are stationary points of (1).

Original: In the rest of this paper, we use the following function Ω which satisﬁes Assumption 2. [Equation] where (cid:15) is a small positive number, e.g. 10 − 12 . We brieﬂy explain the rationale in our choice.
Revised: The proof of Proposition 1 is presented in Appendix B.

Original: By the monotone convergence theorem, as (cid:15) tends to zero, Ω( z ) converges to the mapping z (cid:55)→ √ z .

Original: Thus, Ω( (cid:107) U i · − U j · (cid:107) 2 ) approximates a snowﬂake metric d ( U i · , U j · ) =

Original: (cid:112) (cid:107) U i · − U j · (cid:107) 2 .

Original: When the approximate snowﬂake metric is employed in the penalty term, small differences between rows and columns are penalized signiﬁcantly more than larger differences resulting in more aggressive smoothing of small noisy variations and less smoothing of more signiﬁcant systematic variations.

Original: Note that the weights are continuously updated throughout the optimization as opposed to the ﬁxed weights in (Chi et al., 2017).

Original: This introduces a notion of the scale of the solution into the weights.

Original: 2. C O - CLUSTERING AT MULTIPLE SCALES
Revised: 2. C O - CLUSTERING AT MULTIPLE SCALES

Original: Initializing Algorithm 1 is very important as the objective function in (1) is not convex.
Revised: Initializing Algorithm 1 is very important as the objective function in (1) is not convex.

Original: The matrix
Revised: The matrix

Original: U (0) is initialized to be the mean of all non-missing values.
Revised: U (0) is initialized to be the mean of all non-missing values.

Original: The connectivity graphs E r and E c

Original: CO - CLUSTER - MISSING (cid:0) P Θ ( X ) , γ r = 2 l , γ c = 2 k (cid:1) 7: distances: d ( X i · , X j · ) += d (cid:16) ˜X ( l,k ) i · , ˜X ( l,k ) j · (cid:17) distances d ( X · i

Original: X · j ) += d (cid:16) ˜X ( l,k ) · i , ˜X ( l,k ) · j (cid:17) 14: Calculate are initialized at the beginning using k -nearest-neighbor graphs, and remain ﬁxed throughout all considered scales.
Revised: The connectivity graphs E r and E c are initialized at the beginning using k -nearest-neighbor graphs, and remain ﬁxed throughout all considered scales.

Original: If we observed the complete matrix, employing a sparse Gaussian kernel is a natural way to quantify the local similarity between pairs of rows and pairs of columns.
Revised: If we observed the complete matrix, employing a sparse Gaussian kernel is a natural way to quantify the local similarity between pairs of rows and pairs of columns.

Original: The challenge is that we do not have the complete data matrix X
Revised: The challenge is that we do not have the complete data matrix X

Original: but only the partially observed one
Revised: but only the partially observed one

Original: P Θ ( X ) .
Revised: P Θ ( X ) .

Original: Therefore, we rely only on the observed values to calculate the k-nearest-neighbor graph, using the distance used by Ram et al.
Revised: Therefore, we rely only on the observed values to calculate the k -nearest-neighbor graph, based on the distance used by Ram et al.

Original: in an image inpainting problem.
Revised: in an image inpainting problem.

Revised: CO - CLUSTER - MISSING (cid:0) P Θ ( X ) , γ r = 2 l , γ c = 2 k (cid:1) distances:

Revised: d ( X i · , X j · ) += d (cid:16)

Revised: ˜X

Revised: ( l,k ) i · , ˜X ( l,k ) j · (cid:17)

Revised: distances d

Original:  To obtain a collection of estimates at multiple scales, weneed to solve the optimization problem for pairs of  γ r , γ c  .
Revised: · i · j ) += d (cid:16) ˜X ( l,k ) · i , ˜X ( l,k ) · j (cid:17) Solving the co-clustering problem in Algorithm 1 at a single scale yields the smooth estimate U , the ﬁlled-in data matrix ˜X , and n r and n c which are the number of distinct row and column clusters, respectively, identiﬁed at that scale through columns and rows merging in U . To obtain a collection of estimates at multiple scales, we solve the optimization problem for pairs of values for γ r , γ c set at logarithmic scale (Chi & Steinerberger, 2018) until we have converged to single global bi-cluster, i.e. n r = n c = 1 .

Original: We start with small values of γ r = 2 l 0 and γ c = 2 k 0 , where l 0 , k 0 < 0 .
Revised: We start with small values of γ r = 2 l 0 and γ c = 2 k 0 , where l 0 , k 0 < 0 .

Original: We calculate the co-clustering (Algorithm 1) and obtain the smooth estimate U ( l 0 ,k 0 ) =
Revised: We apply the co-clustering (Algorithm 1) and obtain the smooth estimate U ( l 0 ,k 0 ) =

Original: U (2 l 0 , 2 k 0 ) , theﬁlled-in data matrix ˜X
Revised: U (2 l 0 , 2 k 0 ) used to ﬁll in the data matrix ˜X

Original: ( l 0 ,k 0 ) , and n r and n c which are the number of distinct row and column clusters, respectively, identiﬁed at that scale.
Revised: ( l 0 ,k 0 ) .

Original: Keeping γ r ﬁxed, we keep increasing γ c by power of 2 and biclustering the data until the algorithm converges to one cluster along the columns.
Revised: Keeping γ r ﬁxed, we continue increasing γ c by power of 2 and applying the biclustering until the algorithm converges to one cluster along the columns ( n c = 1 ).

Original: We then increase γ r by power of 2 and reset γ c = 2 k 0 .
Revised: We then increase γ r by power of 2 and reset γ c = 2 k 0 .

Original: We repeat this procedure at increasing scales of γ r = 2 l , γ c = 2 k , untiln r = n c = 1 , indicating we have converged to a single global bicluster.
Revised: We repeat this procedure at increasing scales of γ r = 2 l , γ c = 2 k , until we have converged to a single global bicluster.

Original: Thus we traverse a solution surface at logarithmic scale (Chi & Steinerberger, 2018).

Original: This  yields a collection of ﬁlled-in matrices at all scales (cid:110) ˜X
Revised: This multiscale procedure yields a collection of ﬁlled-in matrices at all scales (cid:110) ˜X

Original: ( l,k ) (cid:111) l,k .
Revised: ( l,k ) (cid:111) l,k .

Revised: Note that the l and k denote the power of 2 taken for speciﬁc row and column cost parameters ( γ c , γ r ) in the solution.

Revised: This is intended as a compact notation that corresponds a pair of parameters ( γ r , γ c ) to their solution U ( l,k ) and ﬁlled in estimate ˜X ( l,k ) .

Original:  C O - MANIFOLD LEARNING
Revised:  C O - MANIFOLD LEARNING

Original: Kernel-based manifold learning relies on constructing a “good” similarity measure between points, and a dimension reduction method based on this similarity.
Revised: Kernel-based manifold learning relies on constructing a good similarity measure between points, and a dimension reduction method based on this similarity.

Original: The eigenvectors of these kernels is typically used as the new low-dimensional coordinates for the data.
Revised: The eigenvectors of these kernels are typically used as the new low-dimensional coordinates for the data.

Original: Here we leverage having calculated an estimate of the ﬁlled-in matrix at multiple scales (cid:110) ˜X
Revised: Here we leverage having calculated an estimate of the ﬁlled-in matrix at multiple scales (cid:110) ˜X

Original: ( l,k ) (cid:111) l,k , to deﬁne a new metric between rows and columns.
Revised: ( l,k ) (cid:111) l,k , to deﬁne a new metric between rows and columns.

Original: This metric will encompass all bi-scales as deﬁned by joint pairs of optimization cost parameters γ r , γ c .
Revised: This metric will encompass all bi-scales as deﬁned by joint pairs of optimization cost parameters γ r , γ c .

Original: Given a new metric we employ diffusion maps  to obtain a new embedding of the rows and columns.
Revised: Given a new metric we employ Diffusion maps (Coifman & Lafon, 2006) to obtain a new embedding of the rows and columns.

Original: Note that other methods can be used for embedding based on our new metric.

Original: The full algorithm is given in Algorithm 2.
Revised: The full algorithm is given in Algorithm 2.

Original: 3. M ULTI - SCALE METRIC
Revised: 3. M ULTI - SCALE METRIC

Original: We deﬁne a new metric to estimate the geometryboth locally and globally of the complete data matrix.
Revised: We deﬁne a new metric to estimate the geometry of the complete data matrix both locally and globally.

Original: For a given pair γ r , γ c , we calculate the Euclidean distance between rows for the ﬁlled-in matrix at that joint scale, weighted by the cost parameters:
Revised: For a given pair γ r , γ c , we calculate the Euclidean distance between rows for the ﬁlled-in matrix at that joint scale, weighted by the cost parameters:

Original: [Equation]
Revised: [Equation]

Original: where ˜X ( [Equation] ,k ) 
Revised: where ˜X ( l,k ) =

Original: = P Θ ( X ) 
Revised: P Θ ( X ) + P Θ c

Original: + P Θ c ( U ( l,k ) ) .
Revised: ( U ( l,k ) ) , and we set α = − 1 / 2 to favor local over global structure in our simulations. Our goal is to aggregate distances between a pair of rows (columns) across multiple scales of the solution, to calculate a metric that better recovers the local and global geometry of the data despite the missing values, thus “ﬁxing” the missing data metric.

Original: Having solved for multiple paris from the solution surface, we sum over all the distances to obtain a multi-scale distance on the data rows:
Revised: Having solved for multiple pairs from the solution surface, we sum over all the distances to obtain a multi-scale distance on the data rows:

Original: [Equation]
Revised: [Equation]

Original: An analogous multi-scale distance is computed for pairs of columns.
Revised: An analogous multi-scale distance is computed for pairs of columns.

Original: This metric takes advantage of solving the optimization for multiple pairs of cost parameters and ﬁlling in the missing values with increasingly smooth estimates. Note that if there are no missing values, this metric is just the Euclidean pairwise distance scaled by a scalar, so that we recover the  embedding of the complete matrix. 
Revised: Note that if there are no missing values, this metric is just the Euclidean pairwise distance scaled by a scalar, so that we recover the original embedding of the complete matrix. This metric takes advantage of solving the optimization for multiple pairs of cost parameters and ﬁlling in the missing values with increasingly smooth estimates (as γ r and γ c increase).

Original: In our simulations, we set α = − 1 / 2 to favor local over global structure.
Revised: It also alleviates the need to identify the ideal scale at which to ﬁll in the points; it is not clear that a single “optimal” scale actually exists, but rather different points in the matrix may have different optimal scales.

Original: As opposed to the partition-tree based metric of Mishne et al.
Revised: As opposed to the partition-tree based metric of Mishne et al.

Original: (2017), this metric takes into account all joint scales of the data as the matrix U is smoothed across rows and columns simultaneously, thus fully taking advantage of the coupling between both modes.
Revised: (2017), this metric takes into account all joint scales of the data as the matrix U is smoothed across rows and columns simultaneously, thus fully taking advantage of the coupling between both modes.

Original: 3. D IFFUSION MAPS
Revised: 3. D IFFUSION MAPS

Original: Having calculated a multi-scale metric on the rows and columns throughout the joint optimization procedure, we can now construct a pair of low-dimensional embeddings based on these distances.
Revised: Having calculated a multi-scale metric on the rows and columns throughout the joint optimization procedure, we can now construct a pair of low-dimensional embeddings based on these distances.

Original: Speciﬁcally we use diffusion maps (Coifman & Lafon, 2006), but any dimension reduction technique relying on the construction of a distance kernel could be used instead.
Revised: Speciﬁcally we use Diffusion maps (Coifman & Lafon, 2006), but any dimension reduction technique relying on the construction of a distance kernel could be used instead.

Original: We brieﬂy review the construction of the diffusion maps for the columns (observations) of a matrix but the same can be applied to the rows (features).
Revised: We brieﬂy review the construction of the diffusion maps for the rows (features) of a matrix but the same can be applied to the columns (observations).

Original: Given a distance between two columns of the matrix d ( X· i  , X· j  ) , we construct an afﬁnity kernel on the columns.
Revised: Given a distance between two rows of the matrix d ( X i · , X j · ) , we construct an afﬁnity kernel on the rows.

Original: We choose an exponential function, but other kernels can be considered depending on the application:
Revised: We choose an exponential function, but other kernels can be considered depending on the application:

Original: [Equation]
Revised: [Equation]

Original: where σ is a scale parameter.
Revised: where σ is a scale parameter.

Original: The exponential function enhances locality, as pairs of samples whose distance exceed σ have negligible afﬁnity.
Revised: The exponential function enhances locality, as pairs of samples whose distance exceed σ have negligible afﬁnity.

Original: One possible choice for σ is to be the median of distances within the data.
Revised: One possible choice for σ is to be the median of distances within the data.

Original: We derive a row-stochastic matrix on the columns by normalizing the rows of  A : 
Revised: We derive a row-stochastic matrix P by normalizing the rows of matrix A : P

Original: [Equation]

Original:  where D is a diagonal matrix whose elements are given by D [ i, i ] 
Revised: = D − 1 A , where D is a diagonal matrix whose elements are given by D [ i, i ] =

Original: = (cid:80)
Revised: (cid:80)

Original: j A [ i, j ] .
Revised: j A [ i, j ] .

Original: The eigendecomposition of P yields a sequence of positive decreasing eigenvalues: 1 = λ 0 ≥ λ 1 ≥ ... , and right eigenvectors { ψ (cid:96) } (cid:96) .
Revised: The eigendecomposition of P yields a sequence of positive decreasing eigenvalues: 1 = λ 0 ≥ λ 1 ≥ . . . , and right eigenvectors { ψ (cid:96) } (cid:96) .

Original: Retaining only the ﬁrst d eigenvalues and eigenvectors, the mapping Ψ embeds the columns into the Euclidean space R d :
Revised: Retaining only the ﬁrst d eigenvalues and eigenvectors, the mapping Ψ embeds the rows into the Euclidean space R d :

Original: [Equation]
Revised: [Equation]

Original: The embedding integrates the local connections found in the data into a global representation, which enables visualization of the data, organizes the data into meaningful clusters, and identiﬁes outliers and singular samples.
Revised: The embedding integrates the local connections found in the data into a global representation, which enables visualization of the data, organizes the data into meaningful clusters, and identiﬁes outliers and singular samples.

Original: This embedding is also equipped with a noise-robust distance, the diffusion distance.
Revised: This embedding is also equipped with a noise-robust distance, the diffusion distance.

Original: For more details on diffusion maps, see (Coifman & Lafon, 2006).
Revised: For more details on diffusion maps, see Coifman & Lafon (2006).

Original:  N UMERICAL E XPERIMENTS
Revised:  N UMERICAL E XPERIMENTS

Original: We run experiments on  three datasets, and evaluate results both qualitatively and quantitatively:
Revised: The model we consider in the paper is such that the data is not represented by a biclustering model but rather at least one of the modes (rows/columns) lies on a low-dimensional manifold. In our experiments we consider three such examples. In the ﬁrst a manifold structure exists along both rows and columns, and for the second and third the columns belong to disjoint clusters while the rows lie on a manifold:

Original: • linkage A synthetic dataset with a one-dimensional manifold along the rows and a twodimensional manifold along the columns.
Revised: • linkage A synthetic dataset with a one-dimensional manifold along the rows and a twodimensional manifold along the columns.

Original: Let { z i } N 

Revised: Let { z i } N

Original: i
Revised: i

Original: =1 ∈ R 3 be points along a helix and let { y j } N 2 j =1 ∈ R 3 be a two dimensional surface.
Revised: =1 ∈ R 3 be points along a helix and [Figure] NLPCA [Figure]

Revised: Co

Revised: [Figure]

Original:  We analyze the matrix of Euclidean distances between the two spatially distant sets of points to reveal the underlying geometry of both rows and columns,
Revised: manifold [Figure] let { y j } N 2 j =1 ∈ R 3 be a two dimensional surface, where we set N 1 = 190 , N 2 = 300 . We analyze the matrix of Euclidean distances between the two spatially distant sets of points to reveal the underlying geometry of both rows and columns,

Original: [Equation]
Revised: [Equation]

Original: Other functions of the distance can also be used such as the elastic or Coulomb potential operator(Coifman & Gavish, 2011).
Revised: Other functions of the distance can also be used such as the elastic or Coulomb potential operator

Revised: (Coifman & Gavish, 2011)

Original: Missing values correspond to having access to only some of the distances between pairs of points across the two sets.
Revised: Missing values correspond to having access to only some of the distances between pairs of points across the two sets.

Original: Note that thisis unlike
Revised: Note that this

Original:  MDS as we do not have pairwise distances between all datapoints, but rather distances between two sets of points with different geometries.[Figure]
Revised: is unlike MDS as we do not have pairwise distances between all datapoints, but rather distances between two sets of points with different geometries.

Original: • linkage

Revised: • linkage

Original: A synthetic dataset with a clustered structure along the rows and a twodimensional manifold along the columns.
Revised: A synthetic dataset with a clustered structure along the rows and a twodimensional manifold along the columns.

Original: Let { x i } N 

Revised: Let { x i } N

Original: i
Revised: i

Original: =1 ∈ R 3 be composed of points in 3 Gaussian clouds in 3D and let { y j } N 2 j =1 ∈ R 3 be a two dimensional surface as before.
Revised: =1 ∈ R 3 be composed of points in 3 Gaussian clouds in 3D and let { y j } N 2 j =1 ∈ R 3 be a two dimensional surface as before, where we set N 1 = 200 , N 2 = 300 .

Original: • lung500 A real-world dataset composed of 56 lung cancer patients and their gene expression (Lee et al., 2010).
Revised: • lung500 A real-world dataset composed of 56 lung cancer patients and their gene expression (Lee et al., 2010).

Original: We selected the 500 genes with the greatest variance from the original collection of 12,625 genes.
Revised: We selected the 500 genes with the greatest variance from the original collection of 12,625 genes.

Original: Subjects belong to one of four subgroups; they are either normal subjects (Normal) or have been diagnosed with one of three types of cancers: pulmonary carcinoid tumors (Carcinoid), colon metastases (Colon), and small cell carcinoma (Small Cell).
Revised: Subjects belong to one of four subgroups; they are either normal subjects (Normal) or have been diagnosed with one of three types of cancers: pulmonary carcinoid tumors (Carcinoid), colon metastases (Colon) , and small cell carcinoma (

Original: The rows and columns of the data matrix are randomly permuted so their natural order does not play a role in inferring the geometry.
Revised: Small Cell). For all datasets, the rows and columns of the data matrix are randomly permuted so their natural order does not play a role in inferring the geometry.

Revised: We evaluate results both qualitatively and quantitatively.

Original: In Figure 2, we compare our embeddings to both NLPCA with missing data completion (Scholz et al., 2005) and Diffusion maps (DM) (Coifman & Lafon, 2006) on the missing data, where both methods  are applied to each mode separately, while our method takes into account the coupled geometry.
Revised: [Figure] We compare our embeddings to three approaches: NLPCA with missing data completion (Scholz et al., 2005), Fast Robust PCA on Graphs (FRPCAG) (Shahid et al., 2016) and Diffusion maps (DM) (Coifman & Lafon, 2006) on the missing data. FRPCAG provides a linear embedding of a low-rank estimate of a data matrix using quadratic row and column penalties, while the other methods output nonlinear embeddings. NLPCA and DM are applied to each mode separately, while our method and FRPCAG take into account the coupled geometry.

Original: Comparing to Diffusion maps demonstrates how missing values corrupt the embedding.
Revised: Comparing to Diffusion maps demonstrates how missing values corrupt the embedding.

Revised: Note that this is also equivalent to applying our approach for only a single scale of the cost parameters ( γ r , γ c → ∞ ), as for this choice of parameters the solution U converges to the grand mean of the data.

Original: In all examples 50% of the entries are missing. For each of the three methods we display the embedding for both the rows (top) and the columns (bottom),  Both
Revised: In Figure 2, we display the embeddings of the different methods for each of the three datasets for both their rows (top) and their columns (bottom), where 50% of the entries have been removed. Both

Original: NLPCA and DM reveal the underlying 2D surface structure on the rows in only one of the linkage datasets, and err greatly on the other.
Revised: NLPCA and DM reveal the underlying 2D surface structure on the rows in only one of the linkage datasets, and err greatly on the other.

Original: DM correctly infers a 1D path for the linkage dataset but it is increasingly noisy.
Revised: DM correctly infers a 1D path for the linkage dataset but it is increasingly noisy.

Original: For NLPCA the 1D
Revised: For NLPCA the 1D

Original: embedding is not as smooth and clean  the embedding 
Revised: embedding is not as smooth and clean as the embedding inferred by the co-manifold approach.

Original: [Figure]
Revised: Our method reveals the 2D surface in both cases. FRPCAG is unsuccessful in uncovering the non-linear manifolds underlying either rows or columns in the linkage dataset. For linkage2 , the rows do not separate cleanly into disjoint clusters, and for the columns only one of the parameters of the 2D plane is uncovered. For the lung500 data, NLPCA and DM embed the cancer samples such that the normal subjects (yellow) are close to the Colon type (cyan), whereas both our method and FRPCAG separates the normal subjects from the cancer types.

Revised: This is due to taking into account the coupled structure of the genes and the samples.

Revised: As opposed to the clustered structure along the samples, the three nonlinear methods reveal a smooth manifold structure to the genes, which is different than the assumed clustered structure a biclustering method would infer.

Revised: The representation yielded by FRPCAG is the least structured but also does not reveal disjoint gene clusters.

Revised: For plots presenting the datasets and ﬁlled-in values at multiple scales see Appendix C.

Original: Manifold learning is not used only for data visualization but also for calculating new data representations that can then be used for signal processing and machine learning tasks.
Revised: Manifold learning is not used only for data visualization but also for calculating new data representations that can then be used for signal processing and machine learning tasks.

Original: The left panel ofFigure 3 compares clustering the embedding of the cancer patients in lung500 by each method for increasing percentage of missing values in the data, where we averaged over 30 realizations of missing entries.
Revised: Here we calculate clustering accuracy of clustering the low-dimensional representation of the data. We apply k -means to the column embeddings of each method, with k set to the correct number of clusters in the data, as we want to evaluate the ability of the methods to properly represent the data without being sensitive to empirical estimation of the number of clusters in the data.

Original: We use the Adjusted Rand Index (ARI) (Hubert & Arabie, 1985), to measure the similarity between the k-means clustering of the embedding and the ground-truth labels. 
Revised: We use the Adjusted Rand Index (ARI) (Hubert & Arabie, 1985), to measure the similarity between the k -means clustering of the embedding and the ground-truth labels. We note that Shahid et al.

Revised: (2016) do not provide guidelines by which to select the appropriate cost parameters γ r and γ c in the FRPCAG solution.

Revised: Therefore we calculated the solution using a wide range of possible values and plot the results for worst and best performance to demonstrate the sensitivity to setting these parameters.

Original: Our embedding (blue plot) gives the best clustering result and its performance is only slightly degraded by increasing the percentage of missing values, as opposed to Diffusion maps (red plot).
Revised: In contrast, our approach alleviates the need to perform this parameter selection. The left panel of Figure 3 compares clustering the embedding of the cancer patients in lung500 by each method for increasing percentage of missing values in the data, where we averaged overrealizations of missing entries. For low values of missing data, FRPCAG performs the best but its performance degrades as the percentage of missing values increases. For higher values of missing data (50% and above), our embedding (blue plot) gives the best clustering result and its performance is only slightly degraded by increasing the percentage of missing values, as opposed to Diffusion maps (red plot).

Original: This demonstrates that the metric we calculate is a good estimate of the metric of the complete data matrix.
Revised: This demonstrates that the metric we calculate is a good estimate of the metric of the complete data matrix.

Original: NLPCA (yellow plot) performs worst.
Revised: NLPCA (yellow plot) performs worst.

Original: The right panel of Figure 3 compares clustering the embedding of the three Gaussian clusters in linkage2 for increasing percentage of missing values in the data, where we averaged over 30 realizations of  missing entries.
Revised: The right panel of Figure 3 compares clustering the embedding of the three Gaussian clusters in linkage2 for increasing percentage of missing values in the data, where we averaged over 30 realizations of the data itself and the missing entries.

Original: Our embedding (blue plot) gives the best clustering result  and its performance is unaffected by increasing the percentage of missing values, as opposed to Diffusion maps (red plot) which is greatly degraded by the missing values.
Revised: Our embedding (blue plot) gives the best clustering result (for 20% missing values and above) and its performance is unaffected by increasing the percentage of missing values up to 80% , as opposed to Diffusion maps (red plot) which is greatly degraded by the missing values.

Original: NLPCA (yellow plot) does not perform as well as our approach, with performance decreasing as the percentage of missing values increases. 
Revised: NLPCA (yellow plot) does not perform as well as our approach, with performance decreasing as the percentage of missing values increases. For a good parameter selection

Revised: FRPCAG (purple plot) performs well for low percentage of missing values, and then performance is drastically impaired above 50% .

Revised: For a poor parameter selection FRPCAG (green plot) has the worst performance for practically all percentages of missing values.

Revised: Note that the overall poor performance of FRPCAG in this setting is due to the underlying manifold being non-linear and the data being high-rank, demonstrating the need for nonlinear embedding approaches.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper we presented a new method for learning nonlinear manifold representations of both the rows and columns of a matrix with missing data.
Revised: In this paper we presented a new method for learning nonlinear manifold representations of both the rows and columns of a matrix with missing data.

Original: We proposed a new optimization problem to obtain a smooth estimate of the missing data matrix, and solved this problem for different values of the cost parameters, which encode the smoothness scale of the estimate along the rows and columns.
Revised: We proposed a new optimization problem to obtain a smooth estimate of the missing data matrix, and solved this problem for different values of the cost parameters, which encode the smoothness scale of the estimate along the rows and columns.

Original: We leverage calculating these multi-scale estimates into a new metric that aims to capture the geometry of the complete data matrix.
Revised: We leverage calculating these multi-scale estimates into a new metric that aims to capture the geometry of the complete data matrix.

Original: This metric is then used in a kernel-based manifold learning technique to obtain new representations of both the rows and the columns.
Revised: This metric is then used in a kernel-based manifold learning technique to obtain new representations of both the rows and the columns.

Original: In future work we will investigate additional metrics in a general co-manifold setting and relate them to optimal transport problem and
Revised: In future work we will investigate additional metrics in a general co-manifold setting and relate them to optimal transport problem and

Original: Earth Mover’s Distance (Coifman & Leeb, 2013).
Revised: Earth Mover’s Distance (Coifman & Leeb, 2013).

Revised: We also intend to develop efﬁcient solutions to accelerate the optimization in order to address large-scale datasets, in addition to the small-scale regime we demonstrate here.

Revised: We note that the analysis of data in the small-scale regime is nontrivial, and is indeed a limitation for supervised methods such as deep learning due to limited training data.

Original: A D ERIVATION OF M AJORIZATION
Revised: A D ERIVATION OF M AJORIZATION

Original: We ﬁrst construct a majorization of the data-ﬁdelity term.
Revised: We ﬁrst construct a majorization of the data-ﬁdelity term.

Original: It is easy to verify that the following function of U
Revised: It is easy to verify that the following function of U

Original: [Equation]
Revised: [Equation]

Original: where ˜X = P Θ ( X ) + P Θ c ( ˜U ) , majorizes the data-ﬁdelity term 12 (cid:107)P Θ ( X ) − P Θ ( U ) (cid:107) 2 F at ˜U .
Revised: where ˜X = P Θ ( X ) + P Θ c ( ˜U ) , majorizes the data-ﬁdelity term 12 (cid:107)P [Equation] (cid:107) [Equation]

Original:  We next construct a majorization of the penalty term.
Revised: ˜U [Equation] We next construct a majorization of the penalty term.

Original: Recall that the ﬁrst-order Taylor approximation of a differentiable concave function provides a tight bound on the function.
Revised: Recall that the ﬁrst-order Taylor approximation of a differentiable concave function provides a global upper bound on the function.

Original: Therefore, under
Revised: Therefore, under

Original: Assumption 2.2, we have the following inequality
Revised: Assumption 2.2, we have the following inequality

Original: [Equation]
Revised: [Equation]

Revised: for all z, ˜ z ∈ [0 , ∞ ) .

Revised: Figure 4 shows the relationship between Ω given in (3) with (cid:15) = 10 − 12 and its ﬁrst-order Taylor approximation at ˜ z

Original:  Thus, we can majorize the penalty term γ r J r ( U ) + γ c J c ( U ) with the function
Revised: = 1 . Thus, we can majorize the penalty term γ r J r ( U ) + γ c J c ( U ) with the function

Original: [Equation]
Revised: [Equation]

Original: where κ is a constant that does not depend on U and ˜ w r,ij and ˜ w c,ij (2) are weights that depend on ˜U .The sum of functions (4) and (5) [Equation] j
Revised: where κ is a constant that does not depend on U and ˜ w r,ij and ˜ w c,ij (5) are weights that depend on ˜U .

Original: [Equation] ∈E c [Equation] w c
Revised: The sum of functions (7) and (8)

Original: [Equation]
Revised: [Equation]

Original: U

Original: [Equation]

Original: − U · j

Original: [Equation]

Original: κ majorizes our objective function (1) at ˜U .
Revised: majorizes our objective function (1) at ˜U .

Original: B C ONVERGENCE
Revised: B P ROOF OF P ROPOSITION

Original: The MM algorithm generates a sequence of iterates that has at least one limit point, and the limit points are stationary points of the objective function
Revised: The MM algorithm generates a sequence of iterates that has at least one limit point, and the limit points are stationary points of the objective function

Original: [Equation]
Revised: [Equation]

Original: To reduce notational clutter, we suppress the dependency of f on γ r and γ c since they are ﬁxed during Algorithm 1.
Revised: To reduce notational clutter, we suppress the dependency of f on γ r and γ c since they are ﬁxed during Algorithm 1.

Original: We prove Proposition 1 in three stages.
Revised: We prove Proposition 1 in three stages.

Original: First, we show that all limit points of the MM algorithm are ﬁxed points of the MM algorithm map.
Revised: First, we show that all limit points of the MM algorithm are ﬁxed points of the MM algorithm map.

Original: Second, we show that ﬁxed points of the MM algorithm are stationary points of f in (7).
Revised: Second, we show that ﬁxed points of the MM algorithm are stationary points of f in (10).

Original: Finally, we show that the MM algorithm has at least one limit point.
Revised: Finally, we show that the MM algorithm has at least one limit point.

Original: B. L IMIT POINTS ARE FIXED POINTS
Revised: B. L IMIT POINTS ARE FIXED POINTS

Original: The convergence theory of MM algorithms relies on the properties of the algorithm map ψ ( U ) that returns the next iterate given the last iterate.
Revised: The convergence theory of MM algorithms relies on the properties of the algorithm map ψ ( U ) that returns the next iterate given the last iterate.

Original: For easy reference, we state a simple version of Meyer’s monotone convergence theorem Meyer (1976), which is instrumental in proving convergence in our setting.
Revised: For easy reference, we state a simple version of Meyer’s monotone convergence theorem Meyer (1976), which is instrumental in proving convergence in our setting.

Original: Theorem 1 Let f ( U ) be a continuous function on a domain S and ψ ( U ) be a continuous algorithm map from S into S satisfying f ( ψ ( U ))
Revised: Theorem 1 Let f ( U ) be a continuous function on a domain S and ψ ( U ) be a continuous algorithm map from S into S satisfying f ( ψ ( U ))

Original: <
Revised: <

Original: f ( U ) for all U ∈ S with ψ ( U ) (cid:54) =
Revised: f ( U ) for all U ∈ S with ψ ( U ) (cid:54) =

Original: U .
Revised: U .

Original: Then all limit points of the iterate sequence U k
Revised: Then all limit points of the iterate sequence U k

Original: = ψ ( U k − 1 ) are ﬁxed points of ψ ( U ) .
Revised: = ψ ( U k − 1 ) are ﬁxed points of ψ ( U ) .

Original: In order to apply Theorem 1, we need to identify elements in the assumption with speciﬁc functions and sets corresponding to the problem of minimizing (7).
Revised: In order to apply Theorem 1, we need to identify elements in the assumption with speciﬁc functions and sets corresponding to the problem of minimizing (10).

Original: Throughout the following proof, it will sometimes be convenient to work with the column major vectorization of a matrix.
Revised: Throughout the following proof, it will sometimes be convenient to work with the column major vectorization of a matrix.

Original: The vector b = vec( B ) is obtained by stacking the columns of B on top of each other.
Revised: The vector b = vec( B ) is obtained by stacking the columns of B on top of each other.

Original: The function f : Take S = R m × n and f : S (cid:55)→ R to be the objective function in (7) and majorize f with g ( U | ˜U ) given in (6).
Revised: The function f : Take S = R m × n and f : S (cid:55)→ R to be the objective function in (10) and majorize f with g ( U | ˜U ) given in (9).

Original: The function f is continuous.
Revised: The function f is continuous.

Original: Let ψ ( ˜U ) =
Revised: Let ψ ( ˜U ) =

Original: arg min U g
Revised: arg min U g

Original: ( U | ˜U ) denote the algorithm map for the MM algorithm.
Revised: ( U | ˜U ) denote the algorithm map for the MM algorithm.

Original: Since g ( U | ˜U ) is strongly convex in U , it has a unique global minimizer.
Revised: Since g ( U | ˜U ) is strongly convex in U , it has a unique global minimizer.

Original: Consequently, f ( ψ ( U )) <
Revised: Consequently, f ( ψ ( U )) <

Original: f ( U ) for all ψ ( U ) (cid:54) =
Revised: f ( U ) for all ψ ( U ) (cid:54) =

Original: U .
Revised: U .

Original: Continuity of the algorithm map ψ : Continuity of ψ follows from the fact that the solution to the convex biclustering problem is jointly continuous in the weights and data matrix (Chi et al., 2017)[Proposition 2].
Revised: Continuity of the algorithm map ψ : Continuity of ψ follows from the fact that the solution to the convex biclustering problem is jointly continuous in the weights and data matrix (Chi et al., 2017)[Proposition 2].

Original: The weight ˜ w r,ij ( ˜U ) = Ω (cid:48)
Revised: The weight ˜ w r,ij ( ˜U ) = Ω (cid:48)

Original: ( (cid:107) U i · − U j · (cid:107) 2 ) is a continuous function of ˜U , since Ω (cid:48) is continuous according to Assumption 2.2.
Revised: ( (cid:107) U i · − U j · (cid:107) 2 ) is a continuous function of ˜U , since Ω (cid:48) is continuous according to Assumption 2.2.

Original: The weight ˜ w c,ij ( ˜U ) is likewise continuous in ˜U .
Revised: The weight ˜ w c,ij ( ˜U ) is likewise continuous in ˜U .

Original: The data matrix passed into the convex biclustering algorithm is ˜X = P Θ ( X ) + P Θ c ( ˜U ) , which is a continuous function of ˜U since the projection mapping P Θ c is continuous.
Revised: The data matrix passed into the convex biclustering algorithm is ˜X = P Θ ( X ) + P Θ c ( ˜U ) , which is a continuous function of ˜U since the projection mapping P Θ c is continuous.

Original: B. F IXED POINTS ARE STATIONARY POINTS
Revised: B. F IXED POINTS ARE STATIONARY POINTS

Original: Let L ij =
Revised: Let L ij =

Original: ( e i − e j ) T ⊗ I and ˜L ij = I ⊗ ( e i − e j ) T , where ⊗ denotes the Kronecker product. 
Revised: ( e i − e j ) T ⊗ I and ˜L ij = I ⊗ ( e i − e j ) T , where ⊗ denotes the Kronecker product. Then vec [Equation] i

Original: Then vec( U i · − U j · ) =
Revised: [Equation] U [Equation] [Equation]

Original:  L ij u and vec( U · i − U · j ) =
Revised: = L ij u and vec( U · i − U · j ) =

Original: ˜L ij u .
Revised: ˜L ij u .

Original: The directional derivative of f in the direction v at a point u is given by
Revised: The directional derivative of f in the direction v at a point u is given by

Original: [Equation]
Revised: [Equation]

Original: otherwise.
Revised: otherwise.

Original: A point u is a stationary point of f , if 
Revised: A point u is a stationary point of f , if for all direction vectors v

Original: [Equation]
Revised: [Equation]

Original: where P Θ ( u− x ) =
Revised: j [Equation] u

Original:  vec( P Θ ( U ) − P Θ ( X )) .
Revised: [Equation] , where P Θ ( u − x ) = vec( P Θ ( U ) − P Θ ( X )) .

Original: A point u is a ﬁxed point of ψ , if 0 is in the subdifferential of g ( u | u ) , i.e. 
Revised: A point u is a ﬁxed point of ψ , if 0 is in the subdifferential of g ( u | u ) , i.e. 0 ∈

Original: [Equation]
Revised: [Equation]

Original:  where the set on the right is the subdifferential ∂g ( u | u ) .
Revised: 11) where the set on the right is the subdifferential ∂g ( u | u ) .

Original: If L ij u (cid:54) = 0 , then ∂ (cid:107) L ij u (cid:107) 

Revised: If L ij u (cid:54) = 0 , then ∂ (cid:107) L ij u (cid:107)

Original: = (cid:110)
Revised: = (cid:110)

Original: L T ij
Revised: L T ij

Original: L ij u (cid:107) L ij u (cid:107) 2 (cid:111) .
Revised: L ij u (cid:107) L ij u (cid:107) 2 (cid:111) .

Original: On the other hand, if L ij u = 0 , then ∂ (cid:107) L ij u (cid:107) 2 = ∂ (cid:107) 0 (cid:107) 2 = { d : (cid:107) d (cid:107) 2 ≤ 1 } .
Revised: On the other hand, if L ij u = 0 , then ∂ (cid:107) L ij u (cid:107) 2 = ∂ (cid:107) 0 (cid:107) 2 = { d : (cid:107) d (cid:107) 2 ≤ 1 } .

Original: Fix an arbitrary direction vector v .
Revised: Fix an arbitrary direction vector v .

Original: The inner product of v with an element in the set on right hand side of (8) is given by[Equation] (cid:88) ( i
Revised: The inner product of v with an element in the set on right hand side of (11) is given by

Original: [Equation]

Original: r Ω (cid:48) (

Original: [Equation]
Revised: [Equation]

Original: L

Original: [Equation]
Revised: [Equation]

Original: (cid:107) [Equation] (cid:104) d ij 
Revised: where d ij ∈ ∂ (cid:107) L ij u (cid:107) 2 and ˜d ij ∈ ∂ (cid:107) ˜L ij u (cid:107) 2 .

Original: [Equation]

Original: (cid:105)

Original: +

Original: [Equation]

Original: c (cid:88) ( i,j ) ∈E c Ω (cid:48) ( (cid:107) ˜L ij u (cid:107) 2 ) (cid:104) d ij , v (cid:105) , (9) where d ij ∈ ∂ (cid:107) L ij u (cid:107) 2 and ˜d ij ∈ ∂ (cid:107) ˜L ij u (cid:107) 2 . Then the supremum of the right hand side of (9) over all d ij ∈ ∂ (cid:107) L ij u (cid:107) 2 and ˜d ij ∈ ∂ (cid:107) ˜L ij u (cid:107) 2 is nonnegative, because 0 ∈ ∂g ( u | u ) .
Revised: Then the supremum of the right hand side of (12) over all d ij ∈ ∂ (cid:107) L ij u (cid:107) 2 and ˜d ij ∈ ∂ (cid:107) ˜L ij u (cid:107) 2 is nonnegative, because 0 ∈ ∂g ( u | u ) .

Original: Consequently, all ﬁxed points of ψ are also stationary points of f .
Revised: Consequently, all ﬁxed points of ψ are also stationary points of f .

Original: B. T HE MM ITERATE SEQUENCE HAS A LIMIT POINT
Revised: B. T HE MM ITERATE SEQUENCE HAS A LIMIT POINT

Original: To ensure the existence of a limit point, we show that the function f is coercive, i.e. f ( U t ) → ∞ for any sequence (cid:107) U t (cid:107) F → ∞ .
Revised: To ensure the existence of a limit point, we show that the function f is coercive, i.e. f ( U t ) → ∞ for any sequence (cid:107) U t (cid:107) F → ∞ .

Original: Recall that according to Assumption 2.1 we assume that the row and column edge sets E r and E c form connected graphs.
Revised: Recall that according to Assumption 2.1 we assume that the row and column edge sets E r and E c form connected graphs.

Original: Therefore, J r ( U ) =
Revised: Therefore, J r ( U ) =

Original: J c ( U ) =
Revised: J c ( U ) =

Original: if and only if U = a 11 T (Chi et al., 2017, Proposition 3).
Revised: if and only if U = a 11 T (Chi et al., 2017, Proposition 3).

Original: The edge-incidence matrix of the column graph Φ c ∈ R |E c |× n encodes its connectivity and is deﬁned asφ
Revised: The edge-incidence matrix of the column graph Φ c ∈ R |E c |× n encodes its connectivity and is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: =   If node i is the head of edge l , − 1 If node i is the tail of edge l , otherwise.
Revised: otherwise.

Original: The row edge-incidence matrix Φ r ∈ R |E r |× m is deﬁned similarly.
Revised: The row edge-incidence matrix Φ r ∈ R |E r |× m is deﬁned similarly.

Original: Assume that Θ non-empty, i.e. at least one entry of the matrix has been observed.
Revised: Assume that Θ non-empty, i.e. at least one entry of the matrix has been observed.

Original: Finally, assume that Ω is also coercive.
Revised: Finally, assume that Ω is also coercive.

Original: Note that any sequence U t = a t 11 T + B t where (cid:104) B t , 11 T (cid:105) = 0 .
Revised: Note that any sequence U t = a t 11 T + B t where (cid:104) B t , 11 T (cid:105) = 0 .

Original: Note that J r ( U t ) =
Revised: Note that J r ( U t ) =

Original: J r ( B t ) and J c ( U t ) =
Revised: J r ( B t ) and J c ( U t ) =

Original: J c ( B t ) .
Revised: J c ( B t ) .

Original: Let U t be a diverging sequence, i.e. (cid:107) U t (cid:107) F → ∞ .
Revised: Let U t be a diverging sequence, i.e. (cid:107) U t (cid:107) F → ∞ .

Original: There are two cases to consider.
Revised: There are two cases to consider.

Original: Case I: Suppose that (cid:107) B t (cid:107) F → ∞ .
Revised: Case I: Suppose that (cid:107) B t (cid:107) F → ∞ .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: and let σ min denote the smallest singular value of L .
Revised: and let σ min denote the smallest singular value of L .

Original: Note that the null space of L is the span of 1 .
Revised: Note that the null space of L is the span of 1 .

Original: Therefore, since (cid:104) 1 , b t (cid:105) =(cid:107)
Revised: Therefore, since (cid:104) 1 , b t (cid:105) =(cid:107)

Original: Lb
Revised: Lb

Original: [Equation]
Revised: [Equation]

Revised: ≥

Original: [Equation]
Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: (cid:107) F .

Original: Also note that
Revised: Also note that

Original: [Equation]
Revised: [Equation]

Original: Since the mapping x = (cid:0) x Tx
Revised: Since the mapping x = (cid:0) x Tx

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  max {(cid:107) x 1 (cid:107) 2 , (cid:107) x 2 (cid:107) 2 } is a norm, and all ﬁnite dimensional norms are equivalent, there exists some η > 0 such that
Revised: T (cid:55)→ max {(cid:107) x 1 (cid:107) 2 , (cid:107) x 2 (cid:107) 2 } is a norm, and all ﬁnite dimensional norms are equivalent, there exists some η > 0 such that

Original: [Equation]
Revised: [Equation]

Original: By the triangle inequality
Revised: By the triangle inequality

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Revised: ( i

Revised: [Equation]

Revised: [Equation]

Revised: t (cid:107)

Original: Let M = max {|E r |
Revised: [Equation] ( i [Equation] j )

Original: [Equation]
Revised: [Equation]

Original: |E c |} then
Revised: c (cid:107) ˜L ij b t (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Putting inequalities (10), (11)
Revised:   . Let M = max {|E r | , |E c |} then max   (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original: and (13) together gives us
Revised: j

Original: [Equation]
Revised: [Equation]

Revised: ∈E r

Original: [Equation]
Revised: [Equation]

Original: Since Ω is increasing according to Assumption

Original: [Equation]
Revised: [Equation]

Revised: Putting inequalities (13), (14), (15), and (16) together gives us

Original: [Equation]
Revised: [Equation]

Original:  it follows that
Revised: Since Ω is increasing according to Assumption 2.2, it follows that

Original: [Equation]
Revised: [Equation]

Original: Inequality (15) implies that
Revised: Inequality (18) implies that

Original: [Equation]
Revised: [Equation]

Original: Consequently, since Ω is increasing and (cid:107) B t (cid:107) F → ∞ implies that f ( U t ) → ∞ .
Revised: Consequently, since Ω is increasing and (cid:107) B t (cid:107) F → ∞ implies that f ( U t ) → ∞ .

Original: [Figure]

Original: Case II: Suppose (cid:107) B t (cid:107) F ≤ B for some B .
Revised: Case II: Suppose (cid:107) B t (cid:107) F ≤ B for some B .

Original: Then | a t | → ∞ .
Revised: Then | a t | → ∞ .

Original: Note that we have the following inequality
Revised: Note that we have the following inequality

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: C = | Θ | − 1 sup
Revised: C = | Θ | − 1 sup

Original: [Equation]
Revised: [Equation]

Original: The function ( a t − C ) 2 diverges since | a t | → ∞ .
Revised: The function ( a t − C ) 2 diverges since | a t | → ∞ .

Original: Therefore, the function f is coercive.
Revised: Therefore, the function f is coercive.

Original: C F ILLING IN MISSING DATA
Revised: C F ILLING IN MISSING DATA

Original: We present the original underlying structure of 3D points used to generate the Euclidean distance matrix X for the datasets linkage and linkage2 in Figure 4 and Figure 7.
Revised: We present the original underlying structure of 3D points used to generate the Euclidean distance matrix X for the datasets linkage and linkage2 in Figure 5 and Figure 8.

Original: In Figure 5 and Figure 8, on the left we plot the original complete matrix where the rows and columns have been ordered according to the geometry of the 3D points.
Revised: In Figure 6 and Figure 9, on the left we plot the original complete matrix where the rows and columns have been ordered according to the geometry of the 3D points.

Original: On the right we plot the matrix we analyze whose rows and columns have been permuted and 50% of the entries have been removed.
Revised: On the right we plot the matrix we analyze whose rows and columns have been permuted and 50% of the entries have been removed.

Original: In Figure 6 and
Revised: In Figure 7 and

Original: Figure 9 we display the matrix ˜X ( l,k ) for three pairs of values l, k to demonstrate the smoothing that is occurring across the different scales of the rows and columns.
Revised: Figure 10 we display the matrix ˜X ( l,k ) for three pairs of values l, k to demonstrate the smoothing that is occurring across the different scales of the rows and columns.

Original: [Figure] 
Revised: [Figure] [Figure]