Original: I NVERTIBLE M ANIFOLD L EARNING FOR D IMENSION R EDUCTION
Revised: I NVERTIBLE M ANIFOLD L EARNING FOR D IMENSION R EDUCTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: It is widely believed that anonlinear dimension reduction (NLDR) process drops information inevitably in most practical scenarios, and even with the manifold assumption, mostexisting methods are unable to preserve structure of data after 
Revised: It is widely believed that a dimension reduction (DR) process drops information inevitably in most practical scenarios. Thus, most methods try to preserve some essential information of data after DR, as well as manifold based DR methods.

Original: DR due to the loss of information, especially in high-dimensional cases.
Revised: However, they usually fail to yield satisfying results, especially in high-dimensional cases.

Original: In the context of manifold learning, we think  a good low-dimensional representation should preserve  topological and geometric properties of data manifold.
Revised: In the context of manifold learning, we think that a good low-dimensional representation should preserve the topological and geometric properties of data manifolds, which involve exactly the entire information of the data manifolds.

Original: To achieve this, the inveribility of a NLDR transformation is required such that the learned representation is reconstructible via its inverse transformation. In this paper, we  propose a novel  method, called invertible manifold learning ( inv-ML ), to tackle this problem.
Revised: In this paper, we deﬁne the problem of information-lossless NLDR with the manifold assumption and propose a novel two-stage NLDR method, called invertible manifold learning ( inv-ML ), to tackle this problem.

Original: A locally isometric smoothness ( LIS ) constraint for preserving local geometry is applied to a two-stage inv-ML algorithm.
Revised: A local isometry constraint of preserving local geometry is applied under this assumption in inv-ML .

Original: Firstly, a homeomorphic sparse coordinate transformation is learned to ﬁnd the low-dimensional representation without loss of topological information.
Revised: Firstly, a homeomorphic sparse coordinate transformation is learned to ﬁnd the lowdimensional representation without losing topological information.

Original: Secondly, a linear compression is performed on the learned sparse coding, with the trade-off between the target dimension and the incurred information loss.
Revised: Secondly, a linear compression is performed on the learned sparse coding, with the trade-off between the target dimension and the incurred information loss.

Original: Experiments are conducted on seven datasets, whose results demonstrate that the proposed inv-ML not only achievesbetter invertible NLDR in comparison with typical existing methods but also reveals the characteristics of the learned manifolds through linear interpolation in latent space.
Revised: Experiments are conducted on seven datasets with a neural network implementation of inv-ML , called i-ML-Enc , which demonstrate that the proposed inv-ML not only achieves invertible NLDR in comparison with typical existing methods but also reveals the characteristics of the learned manifolds through linear interpolation in latent space.

Original: Moreover, we ﬁnd that the reliability of tangent space approximated by its local neighborhood on real-world datasets isa key to the success of manifold based DR algorithms.
Revised: Moreover, we ﬁnd that the reliability of tangent space approximated by the local neighborhood on real-world datasets is key to the success of manifold based DR algorithms.

Original: The code will be made available soon.
Revised: The code will be made available soon.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In real-world scenario, it is widely believed that the loss of data information is inevitable after dimension reduction (DR), though the goal of DR is to preserve as much information as possible in the low-dimensional space.
Revised: In real-world scenarios, it is widely believed that the loss of data information is inevitable after dimension reduction (DR), though the goal of DR is to preserve as much information as possible in the low-dimensional space.

Original: In the case of linear DR, compressed sensing (Donoho, 2006) breaks this common sense with practical sparse conditions of the given data.
Revised: In the case of linear DR, compressed sensing (Donoho, 2006) breaks this common sense with practical sparse conditions of the given data.

Original: In the case of nonlinear dimension reduction (NLDR), however, it has not been clearly discussed, e.g. what is the structure within data and how to maintain these structure after DR?
Revised: In the case of nonlinear dimension reduction (NLDR), however, it has not been clearly discussed, e.g. what is the structure within data and how to maintain these structures after NLDR?

Original: From the perspective of manifold learning, the manifold Assumption is widely adopted, but classical manifold based DR algorithms are usually performed not well in the complex practical case.
Revised: From the perspective of manifold learning, the manifold assumption is widely adopted, but classical manifold based DR methods usually fail to yield good results in the many practical case.

Original: Therefore, what is the gap between theoretical and real-world applications of manifold learning?
Revised: Therefore, what is the gap between theoretical and real-world applications of manifold based DR?

Original: Here, we give the ﬁrst detailed discussion of this two problem in the context of manifold learning.
Revised: Here, we give the ﬁrst detailed discussion of these two problems in the context of manifold learning.

Original: We think that a good low-dimensional representation should preserve the topology and geometry of input data, and thus introduce a homeomorphism based invertible NLDR process,  combining sparse coordinate transformation and local isometry smoothness (LIS) constraints which preserve the property of topology and geometry, to explain the information-lossless NLDR in manifold learning theoretically.
Revised: We think that a good low-dimensional representation should preserve the topology and geometry of input data, which require the NLDR transformation to be homeomorphic. Thus, we propose an invertible NLDR process, called inv-ML , combining sparse coordinate transformation and local isometry constraint which preserve the property of topology and geometry, to explain the information-lossless NLDR in manifold learning theoretically.

Original: We instantiatethe proposed inv-ML as a neural network called i-ML-Enc via a cascade of equi-dimensional layers and a linear compression layer, and conduct sufﬁcient experiments  to validatethe invertible DR ability of i-ML-Enc and analyze  inherent difﬁculties of classical manifold learning.
Revised: We instantiate inv-ML as a neural network called i-ML-Enc via a cascade of equidimensional layers and a linear transform layer. Sufﬁcient experiments are conduct to validate invertible NLDR abilities of i-ML-Enc and analyze learned representations to reveal inherent difﬁculties of classical manifold learning.

Original: Topology preserving dimension reduction.
Revised: Topology preserving dimension reduction.

Original: To start, we ﬁrst make out the  deﬁnition of  DR on a manifold.
Revised: To start, we ﬁrst make out the theoretical deﬁnition of information-lossless DR on a manifold.

Revised: The topological property is what is invariant under a homeomorphism, and thus what we want to achieve is to construct a homeomorphism for dimension reduction, removing the redundant dimensions while preserving invariant topology.

Original:  f : M d 0 → R m is a smooth mapping of a differential manifold into another, and if f is a homeomorphism of M d 0 into M d 

Revised: To be more speciﬁc, f : M d 0 → R m is a smooth mapping of a differential manifold into another, and if f is a homeomorphism of M d 0 into M d

Original: = f
Revised: = f

Original: ( M d 0 )
Revised: ( M d 0 )

Original: ⊂ R m , we call f is an embedding of M d 0 into R m .
Revised: ⊂ R m , we call f is an embedding of M d 0 into R m .

Original: Assume that the data set X
Revised: Assume that the data set X

Original: = { x j | 1 ≤ j ≤ n } sampled from the compact manifold M d 1 ⊂ R m which we call the data manifold and is homeomorphic to M d 0 .
Revised: = { x j | 1 ≤ j ≤ n } sampled from the compact manifold M d 1 ⊂ R m which we call the data manifold and is homeomorphic to M d 0 .

Original: For the sample points we get are represented in the coordinate after inclusion mapping i 1 , we can only regard them as points from
Revised: For the sample points we get are represented in the coordinate after inclusion mapping i 1 , we can only regard them as points from

Original: Euclidean space R m without any prior knowledge.
Revised: Euclidean space R m without any prior knowledge, and learn to approximate the data manifold in the latent space Z .

Original: According to the Whitney Embedding Theorem (Seshadri & Verma, 2016), M d 0 is can be embedded smoothly into R 2 d by a homeomorphism g .
Revised: According to the Whitney Embedding Theorem (Seshadri & Verma, 2016), M d 0 is can be embedded smoothly into R 2 d by a homeomorphism g .

Original: Rather than to ﬁnd the f − 1 : M d 1 → M d 0 , our goal is to seek a smooth map h : M d 1 → R s ⊂ R 2 d , where h
Revised: Rather than to ﬁnd the f − 1 : M d 1 → M d 0 , our goal is to seek a smooth map h : M d 1 → R s ⊂ R 2 d , where h

Original: = g ◦ f − 1 is a homeomorphism of M d 1 into M d 

Revised: = g ◦ f − 1 is a homeomorphism of M d 1 into M d

Original: = h ( M d 1 ) and d ≤ s ≤ 2 d (cid:28) m , and thus the dim ( h ( X ))
Revised: = h ( M d 1 ) and d ≤ s ≤ 2 d (cid:28) m , and thus the dim ( h ( X ))

Original: =
Revised: =

Original: s , which achieves the NLDR while preserving the topology.
Revised: s , which achieves the DR while preserving the topology.

Original: Owing to the homeomorphism h we seek as a NLDR mapping, the data manifold M d 1 is reconstructible via M d 1 = h − 1 ◦ h ( M d 1 ) , by which we mean h a topology preserving NLDR as well as informationlossless NLDR.
Revised: Owing to the homeomorphism h we seek as a DR mapping, the data manifold M d 1 is reconstructible via M d 1 = h − 1 ◦ h ( M d 1 ) , by which we mean h a topology preserving DR as well as information-lossless DR.

Original: Geometry preserving dimension reduction.
Revised: Geometry preserving dimension reduction.

Original: While the topology of the data manifold M d 1 can be preserved by the homeomorphism h discussed above, it may distort the geometry.
Revised: While the topology of the data manifold M d 1 can be preserved by the homeomorphism h discussed above, it may distort the geometry.

Original: To preserve the local geometry of the data manifold, the map should be isometric on the tangent space T p M dfor every p ∈ M d 1 , indicating that d M d 1 ( u, v ) =
Revised: To preserve the local geometry of the data manifold, the map should be isometric on the tangent space T p M dfor every p ∈ M d 1 , indicating that d M d 1 ( u, v ) =

Original: d M d 2 ( h ( u ) , h ( v )) , ∀ u, v ∈ T p M d 1 .
Revised: d M d 2 ( h ( u ) , h ( v )) , ∀ u, v ∈ T p M d 1 .

Original: By Nash’s
Revised: By Nash’s

Original: Embedding Theorem (Nash, 1956), any smooth manifold of class C k with k ≥ 3 and dimension d can be embedded isometrically in the Euclidean space R s with s polynomial in d . 
Revised: Embedding Theorem (Nash, 1956), any smooth manifold of class C k with k ≥ 3 and dimension d can be embedded isometrically in the Euclidean space R s with s polynomial in d . Noise perturbation.

Revised: In the real-world scenarios, sample points are not lied on the ideal manifold strictly due to the limitation of sampling, e.g. non-uniform sampling noises.

Revised: When the DR method is very robust to the noise, it is reasonable to ignore the effects of the noise and learn the representation

Revised: Z from the given data.

Revised: Therefore, the intrinsic dimension of X is approximate to d , resulting in the lowest isometric embedding dimension is larger than s .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Manifold learning.
Revised: Manifold learning.

Original: Most classical linear or nonlinear DR methods aim to preserve  geometric properties of the manifold.
Revised: Most classical linear or nonlinear DR methods aim to preserve the geometric properties of manifolds.

Original: The Isomap (Tenenbaum et al., 2000) based methods aim to preserve the global metric between every pairs of sample points.
Revised: The Isomap (Tenenbaum et al., 2000) based methods aim to preserve the global metric between every pair of sample points.

Original: For example, McQueen et al.
Revised: For example, McQueen et al.

Original: (2016) can be regarded as such methods based on the push-forward Riemannian metric.
Revised: (2016) can be regarded as such methods based on the push-forward Riemannian metric.

Original: For the other aspect, LLE (Roweis & Saul, 2000) based methods try to preserve local geometry after DR, whose derivatives like LTSA (Zhang & Zha, 2004), MLLE (Zhang & Wang, 2007), etc. have been widely used but usually fail in the high-dimensional case.
Revised: For the other aspect, LLE (Roweis & Saul, 2000) based methods try to preserve local geometry after DR, whose derivatives like LTSA (Zhang & Zha, 2004), MLLE (Zhang & Wang, 2007), etc. have been widely used but usually fail in the high-dimensional case.

Original: Recently, based onthe local properties of mainifold, MLDL (Li et al., 2020) has been proposed as a robust locally geometry preserving method implemented by  neural network,  abandoning the retention of topology.
Revised: Recently, based on local properties of manifolds, MLDL (Li et al., 2020) was proposed as a robust NLDR method implemented by a neural network, preserving the local geometry but abandoning the retention of topology.

Original: In contrast, our method takes  preservation of  geometry as well as topology into consideration, trying to maintain these properties of manifolds even in  excessive dimension reduction, where the target dimension s (cid:48) is smaller than s .
Revised: In contrast, our method takes the preservation of both geometry and topology into consideration, trying to maintain these properties of manifolds even in cases of excessive dimension reduction when the target dimension s (cid:48) is smaller than s .

Original: Invertible model.
Revised: Invertible model.

Original: From AutoEncoder (AE) (Hinton & Salakhutdinov, 2006), the fundamental neural network based model, having achieved DR and cut information loss by minimizing the reconstruction loss, some AE based generative models like VAE (Kingma & Welling, 2014) and manifold-based NLDR models like TopoAE (Moor et al., 2020) has emerged.
Revised: From AutoEncoder (AE) (Hinton & Salakhutdinov, 2006), the fundamental neural network based model, having achieved DR and cut information loss by minimizing the reconstruction loss, some AE based generative models like VAE (Kingma & Welling, 2014) and manifold-based NLDR models like TopoAE (Moor et al., 2020) has emerged.

Original: These methods cannot avoid information loss after NLDR, and thus, some invertible models consist of a series of equidimensional layers have been proposed, some of which aim to generate samples by density estimation through layers (Dinh et al., 2015)
Revised: These methods cannot avoid information loss after NLDR, and thus, some invertible models consist of a series of equidimensional layers have been proposed, some of which aim to generate samples by density estimation through layers (Dinh et al., 2015)

Original: (Dinh et al., 2017) (Behrmann et al., 2019), and the other of which are established for other targets, e.g. validating the mutual information bottleneck (Jacobsen et al., 2018).
Revised: (Dinh et al., 2017) (Behrmann et al., 2019), and the other of which are established for other targets, e.g. validating the mutual information bottleneck (Jacobsen et al., 2018).

Original: Different from methods mentioned above, our proposed i-ML-Enc is a neural network based encoder, with DR as well as maintaining structure of raw data points based on manifold assumption via a series of equi-dimensional layers.
Revised: Different from methods mentioned above, our proposed i-ML-Enc is a neural network based encoder, with NLDR as well as maintaining structures of raw data points based on manifold assumption via a series of equidimensional layers.

Original: Compressed sensing.
Revised: Compressed sensing.

Original: The JohnsonLindenstrauss Theorem (Johnson & Lindenstrauss, 1984) provides the lower bound of target dimension for linear DR with the pairwise distance loss.
Revised: The JohnsonLindenstrauss Theorem (Johnson & Lindenstrauss, 1984) provides the lower bound of target dimension for linear DR with the pairwise distance loss.

Original: Given a small constant (cid:15) ∈ (0 , 1) and n samples { x i } ni =1 in R m , a linear projection W : R m → R s , s > O ( logm(cid:15) 2 ) can be found, which embeds samples into a s -dimensional space with (1+ (cid:15) ) distortion of any sample pairs ( x i , x j ) .
Revised: Given a small constant (cid:15) ∈ (0 , 1) and n samples { x i } ni =1 in R m , a linear projection W : R m → R s , s > O ( logm(cid:15) 2 ) can be found, which embeds samples into a s -dimensional space with (1+ (cid:15) ) distortion of any sample pairs ( x i , x j ) .

Original: It adopts a prior assumption that the given samples in high-dimensional space have a relevant low-dimensional structure constraint which can be maintained by keeping the pairwise distance.
Revised: It adopts a prior assumption that the given samples in high-dimensional space have a relevant low-dimensional structure constraint which can be maintained by keeping the pairwise distance.

Original: Further, compressed sensing (CS) provides strict sparse conditions of linear DR with great probability to recover the compressed signal, which usually cooperates with sparse dictionary learning (Hawe et al., 2013).
Revised: Further, compressed sensing (CS) provides strict sparse conditions of linear DR with great probability to recover the compressed signal, which usually cooperates with sparse dictionary learning (Hawe et al., 2013).

Original: The core of CS is Restricted Isometry Property (RIP) condition, which reads
Revised: The core of CS is Restricted Isometry Property (RIP) condition, which reads

Original: [Equation]
Revised: [Equation]

Original:  where (cid:15) ∈ (0 , 1) is a rather small constant and W is a linear measurement of signal x 1 and x 2 .
Revised: (1) where (cid:15) ∈ (0 , 1) is a rather small constant and W is a linear measurement of signal x 1 and x 2 .

Original: Given a signal x ∈ R m with s -sparse representation α =
Revised: Given a signal x ∈ R m with s -sparse representation α =

Original: Φ x on an m -dimensional orthogonal basis Φ , α can be recovered from the linear measurement y =
Revised: Φ x on an m -dimensional orthogonal basis Φ , α can be recovered from the linear measurement y =

Original: Wα with great probability by the sparse optimization if W m × s satisﬁes the RIP condition: arg min ˜ a || ˜ α || 0 , s.t.
Revised: Wα with great probability by the sparse optimization if W m × s satisﬁes the RIP condition: arg min ˜ a || ˜ α || 0 , s.t.

Original: y
Revised: y

Original: =
Revised: =

Original: W ˜ α .
Revised: W ˜ α .

Original: The linear measurement is rewritten as y = ΨΦ
Revised: The linear measurement is rewritten as y = ΨΦ

Original: α
Revised: α

Original: = Ψ x where Ψ is a low-dimensional orthogonal basis and Φ can be found by the nonlinear dictionary learning.
Revised: = Ψ x where Ψ is a low-dimensional orthogonal basis and Φ can be found by the nonlinear dictionary learning.

Original: There are some reconstructible CS-based NLDR methods (Wei et al., 2015)
Revised: Some reconstructible CS-based NLDR methods (Wei et al., 2015)

Original: (Wei et al., 2019), but their performance is not impressive.
Revised: (Wei et al., 2019) are proposed, which are achieved by preserving local geometry on AE-based networks, but usually with unsatisfying embedding qualities.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: We will speciﬁcally discuss the proposed two stage NLDR  as the ﬁrst stage in Sec 3.1, in which a s -dimensional representation is learned by a homeomorphism transformation while keeping all topological and geometric structure of the data manifold; then give applicable conditions in real-world scenarios as the second stage in Sec 3.2, in which the dimension is further compressed to s (cid:48) .
Revised: We will speciﬁcally discuss the proposed two-stage invertible NLDR process inv-ML as the ﬁrst stage in Sec 3.1, in which a s -dimensional representation is learned by a homeomorphism transformation while keeping all topological and geometric structure of the data manifold; then give applicable conditions in real-world scenarios as the second stage in Sec 3.2, in which the dimension is further compressed to s (cid:48) .

Original: We instantiate the proposed DR process as a neural network based on ML-Enc (Li et al., 2020) in Sec 3.3.
Revised: We instantiate the proposed inv-ML as a neural network i-ML-Enc in Sec 3.3.

Original: (# $ % & ) ( ⊂ ℝ + , (-) ⊂ ℝ . / ......
Revised: (# $ % & ) ( ⊂ ℝ + , (-) ⊂ ℝ . / ......

Original: LL !
Revised: LL !

Original: (# 0 1 ($) ) !
Revised: (# 0 1 ($) ) !

Original: (a) 3 ($) #1 ($) , (-4$) , (0) , ($) # - 1 (-4$) 3 (0) #1 (0) 3 (-4$) #1 (-4$) (b) (c) (a) (d) 5 $ 55 -4$ (a) (d) 0’ 1’ 2’ (L-1)’ ......
Revised: (a) 3 ($) #1 ($) , (-4$) , (0) , ($) # - 1 (-4$) 3 (0) #1 (0) 3 (-4$) #1 (-4$) (b) (c) (a) (d) 5 $ 55 -4$ (a) (d) 0’ 1’ 2’ (L-1)’ ......

Original: W 4$
Revised: W $4$

Original: Decoder # 04$ ! 4$ (1 (0 / ) )
Revised: Decoder # 04$ ! 4$ (1 (0 / ) )

Original: # 24$ ! 4$ (1 (27$) / ) 8(⊂ ℝ +
Revised: # 24$ ! 4$ (1 (27$) / ) 8(

Original: Encoder W 4$ W 4$ # $ 4$ ! 4$ (1 ($ / ) )
Revised: [Figure]

Original: 3. T OPOLOGY AND G EOMETRY P RESERVATION
Revised: 3. T OPOLOGY AND G EOMETRY P RESERVATION

Original: Canonical embedding for homeomorphism.
Revised: Canonical embedding for homeomorphism.

Original: To seek the smooth homeomorphism h , we turn to the theorem of local canonical form of immersion (Mei, 2013).
Revised: To seek the smooth homeomorphism h , we turn to the theorem of local canonical form of immersion (Mei, 2013).

Original: Let f : M → N an immersion, and for any p ∈ M , there exist local coordinate systems ( U, φ ) around p and ( V, ψ ) around f ( p ) such that ψ ◦ f ◦ φ − 1 : φ ( U ) → ψ ( V ) is a canonical embedding, which reads
Revised: Let f : M → N an immersion, and for any p ∈ M , there exist local coordinate systems ( U, φ ) around p and ( V, ψ ) around f ( p ) such that ψ ◦ f ◦ φ − 1 : φ ( U ) → ψ ( V ) is a canonical embedding, which reads

Original: [Equation]
Revised: [Equation]

Original: , x 2 , · · · , x d ) = ( x 1 , x 2 , · · ·, x d , 0 , 0 , · · · , 0) .
Revised: In our case, let M = M d 2 , and N = M d 1 , any point z = ( z 1 , z 2 , · · ·

Original: In our case, let M = M d 2 , and N = M d 1 , any point z 
Revised: [Equation] z s [Equation] M

Original: = ( z 1 , z 2 , · · · , z s ) ∈ M d 1 ⊂ R s can be mapped to a point in R m by the canonical embedding
Revised: [Equation] 1 ⊂ R s can be mapped to a point in R m by the canonical embedding

Original: [Equation]
Revised: [Equation]

Original: For the point z is regarded as a point in R s , φ
Revised: For the point z is regarded as a point in R s , φ

Original: = I is an identity mapping, and for h = g ◦ f − 1 is a homeomorphism, h − 1 is continuous.
Revised: = I is an identity mapping, and for h = g ◦ f − 1 is a homeomorphism, h − 1 is continuous.

Original: The Eq.2 can be written as ( [Equation] , z [Equation]
Revised: The Eq.

Original: , · · · , z s ) [Equation] h ◦ ψ
Revised: (3) can be written as

Original: [Equation]
Revised: [Equation]

Original: z 1 , z 2 , · · · , z s , 0 , 0 , · · · , 0) = h ( x 1 , x 2 , · · · , x m ) Therefore, to reduce dim ( X ) = m to s , we can decompose h into ψ and h ◦ ψ − 1 , by ﬁrstly ﬁnding a homeomorphic coordinate transformation ψ to map x = ( x 1 , x 2 , · · · , x m ) into ψ ( x ) =
Revised: Therefore, to reduce dim ( X ) = m to s , we can decompose h into ψ and h ◦ ψ − 1 , by ﬁrstly ﬁnding a homeomorphic coordinate transformation ψ to map x = ( x 1 , x 2 , · · · , x m ) into ψ ( x ) =

Original: ( z 1 , z 2 , · · · , z s , 0 , 0 , · · · , 0) , which is called a sparse coordinate transformation , and h ◦ ψ − 1 can be easily obtained by Eq.2.
Revised: ( z 1 , z 2 , · · · , z s , 0 , 0 , · · · , 0) , which is called a sparse coordinate transformation , and h ◦ ψ − 1 can be easily obtained by Eq.

Original: We denote h ◦ ψ − 1 by h 0 and call it a sparse compression .
Revised: We denote h ◦ ψ − 1 by h 0 and call it a sparse compression .

Original: The theorem holds for any manifold, while in our case, we aims to ﬁnd the mapping of X ⊂ R m into R s , so the local coordinate systems can be extended to the whole space of R m .
Revised: The theorem holds for any manifold, while in our case, we aims to ﬁnd the mapping of X ⊂ R m into R s , so the local coordinate systems can be extended to the whole space of R m .

Original: Local isometry constraint.
Revised: Local isometry constraint.

Original: The  local isometry proposed by MLDL (Li et al., 2020) is employed, imposing the prior LIS constraint, which aims to preserve distances (or some other metrics) locally so that d M d 1 ( u, v ) =
Revised: The prior local isometry constraint is applied under the manifold assumption, which aims to preserve distances (or some other metrics) locally so that d M d 1 ( u, v ) =

Original: d M d 2 ( h ( u ) , h ( v )) , ∀ u, v ∈ T p M d 1 .
Revised: d M d 2 ( h ( u ) , h ( v )) , ∀ u, v ∈ T p M d 1 .

Original: 3. L INEAR C OMPRESSION
Revised: 3. L INEAR C OMPRESSION

Original: With the former discussed method, manifold-based NLDR can be achieved with topology and geometry preserved, i.e. s -sparse representation in R m .
Revised: With the former discussed method, manifold-based NLDR can be achieved with topology and geometry preserved, i.e. s -sparse representation in R m .

Original: However, the target dimension s (cid:48) may be even less than s , further compression can be performed through the linear compression h (cid:48) 

Revised: However, the target dimension s (cid:48) may be even less than s , further compression can be performed through the linear compression h (cid:48)

Original: : R m → R s (cid:48) instead of sparse compression , where h (cid:48) 

Revised: : R m → R s (cid:48) instead of sparse compression , where h (cid:48)

Original: ( z ) =
Revised: ( z ) =

Original: W m × s (cid:48)
Revised: W m × s (cid:48)

Original: z , with minor information loss.
Revised: z , with minor information loss.

Original: In general, the sparse compression is a particular case of linear compression with h 0 ( z )
Revised: In general, the sparse compression is a particular case of linear compression with h 0 ( z )

Original: =
Revised: =

Original: h (cid:48) 

Revised: h (cid:48)

Original: ( z ) =
Revised: ( z ) =

Original: Λ z , where Λ =
Revised: Λ z , where Λ =

Original: ( δ i,j ) m × s and δ i,j is the Kronecker delta.
Revised: ( δ i,j ) m × s and δ i,j is the Kronecker delta.

Original: We discusses the information loss caused by a linear compression under different target dimensions s (cid:48) as following cases.
Revised: We discusses the information loss caused by a linear compression under different target dimensions s (cid:48) as following cases.

Original: Theoretical case.
Revised: Ideal case.

Original: In the case of d ≤ s ≤ s (cid:48) , based on compressed sensing, we can reconstruct the raw input data after NLDR process without loss of any information by solving the sparse optimization problem mensioned in Sec. 2 when the transformation matrix W m × s (cid:48) has full rank of  column.
Revised: In the case of d ≤ s ≤ s (cid:48) , based on compressed sensing, we can reconstruct the raw input data after NLDR process without loss of any information by solving the sparse optimization problem mentioned in Sec. 2 when the transformation matrix W m × s (cid:48) has full rank of the column.

Original: In the case of d ≤ s (cid:48) < s , it is inevitable to drop the topological properties because the two spaces before and after NLDR are not homeomorphic, and it is reduced to local geometry preservation by
Revised: In the case of d ≤ s (cid:48) < s , it is inevitable to drop the topological properties because the two spaces before and after NLDR are not homeomorphic, and it is reduced to local geometry preservation by

Original: LIS constraint.
Revised: LIS constraint.

Original: However, in the case of s (cid:48) ≤ d < s , both topological and geometric information is lost to varying degrees.
Revised: However, in the case of s (cid:48) ≤ d < s , both topological and geometric information is lost to varying degrees.

Original: Therefore, we can only try to retain as much geometric structure as possible.
Revised: Therefore, we can only try to retain as much geometric structure as possible.

Original: Practical case.
Revised: Practical case.

Original: In real-world scenarios, the target dimension s (cid:48) is usually lower than s , even lower than d .
Revised: In real-world scenarios, the target dimension s (cid:48) is usually lower than s , even lower than d .

Original: Meanwhile, the data sampling rate is quite low, and  clustering effect is extremely signiﬁcant, indicating  it is possible to approximate M 1 by low-dimensional hyper-plane in the Euclidean space.
Revised: Meanwhile, the data sampling rate is quite low, and the clustering effect is extremely signiﬁcant, indicating that it is possible to approximate M 1 by low-dimensional hyperplane in the Euclidean space.

Original: In the case of s (cid:48) <
Revised: In the case of s (cid:48) <

Original: s , we can retain as the prior Euclidean topological structure as additional topological information ofthe raw data points.
Revised: s , we can retain as the prior Euclidean topological structure as additional topological information of raw data points.

Revised: It is reduced to replace the global topology with some relative structures between each cluster.

Original: 3. N ETWORK FOR I MPLEMENTATION
Revised: 3. N ETWORK FOR I MPLEMENTATION

Original: Based on Sec 3.1 and Sec 3.2, we propose a neural network i-ML-Enc which achieves two stages
Revised: Based on Sec 3.1 and Sec 3.2, we propose a neural network i-ML-Enc which achieves two-stage

Original: NLDR preserving both topology and geometry, as shown in Fig.
Revised: NLDR preserving both topology and geometry, as shown in Fig.

Original: In this section, we will introduce the function of network structures and loss functions respectively, including the orthogonal loss, padding loss and extra heads for the ﬁrst stage, and the LIS loss, push-away loss for the second stage.
Revised: In this section, we will introduce the function of network structures and loss functions respectively, including the orthogonal loss, padding loss and extra heads for the ﬁrst stage, and the LIS loss, push-away loss for the second stage.

Original: Cascade of homeomorphisms.
Revised: Cascade of homeomorphisms.

Original: Since the sparse coordinate transformation ψ (and its inverse) can be highly nonlinear and complex, we decompose it into a cascade of L − 1 isometric homeomorphisms ψ = ψ
Revised: Since the sparse coordinate transformation ψ (and its inverse) can be highly nonlinear and complex, we decompose it into a cascade of L − 1 isometric homeomorphisms ψ = ψ

Original: ( L − 1) ◦· · ·◦ ψ (2) ◦ ψ (1) , which can be achieved by L − 1 equi-dimensional network layers.
Revised: ( L − 1) ◦· · ·◦ ψ (2) ◦ ψ (1) , which can be achieved by L − 1 equidimensional network layers.

Original: For each ψ ( l ) , it is a sparse coordinate transformation , where ψ l ( z 1 , ( l ) , z 2 , ( l ) , · · · , z s l , ( l ) , 0 , · · · , 0) =
Revised: For each ψ ( l ) , it is a sparse coordinate transformation , where ψ l ( z 1 , ( l ) , z 2 , ( l ) , · · · , z s l , ( l ) , 0 , · · · , 0) =

Original: ( z 1 , ( l +1) , z 2 , ( l +1) , · · · , z s l +1 , ( l +1) , 0 , · · · , 0) with s l +1 <
Revised: ( z 1 , ( l +1) , z 2 , ( l +1) , · · · , z s l +1 , ( l +1) , 0 , · · · , 0) with s l +1 <

Original: s l and s L − 1 =
Revised: s l and s L − 1 =

Original: s .
Revised: s .

Original: The layer-wise transformation Z ( l +1) =
Revised: The layer-wise transformation Z ( l +1) =

Original: ψ ( l ) ( Z ( l ) ) and its inverse is written as
Revised: ψ ( l ) ( Z ( l ) ) and its inverse can be written as

Original: [Equation]
Revised: [Equation]

Original:  in which W l is the l -th weight matrix of the neural network to be learned, and σ ( . )
Revised: ) in which W l is the l -th weight matrix of the neural network to be learned, and σ ( . )

Original: is a nonlinear activation.
Revised: is a nonlinear activation.

Original: The bias term is removed here to facilitate its simple inverse structure.
Revised: The bias term is removed here to facilitate its simple inverse structure.

Original: Orthogonal loss.
Revised: Orthogonal loss.

Original: Each layer-wise transformation is thought to be a homeomorphism between
Revised: Each layer-wise transformation is thought to be a homeomorphism between

Original: Z ( l ) and Z ( l +1) , and we want it to be a nearly isometric.
Revised: Z ( l ) and Z ( l +1) , and we want it to be a nearly isometric.

Original: We force each W l to be an orthogonal matrix, which allows simple calculation of the inverse of W l .
Revised: We force each W l to be an orthogonal matrix, which allows simple calculation of the inverse of W l .

Original: Based on RIP condition, the orthogonal constraint of the weight matrix in the ﬁrst L − 1 layers can be obtained as
Revised: Based on RIP condition, the orthogonal constraint of the weight matrix in the ﬁrst L − 1 layers can be obtained as

Original: [Equation]
Revised: [Equation]

Original: where { α ( l ) } are the loss weights.
Revised: where { α ( l ) } are the loss weights.

Original: Notice that ρ ( W ) = sup z ∈ R m , z (cid:54) = 

Revised: Notice that ρ ( W ) = sup z ∈ R m , z (cid:54) =

Original: | W z | | z | is the spectral norm of
Revised: | W z | | z | is the spectral norm of

Original: W , and the loss term can be written as ρ ( W Tl W l − I )
Revised: W , and the loss term can be written as ρ ( W Tl W l − I )

Original: =
Revised: =

Original: sup z ∈ R m , z (cid:54) = 

Revised: sup z ∈ R m , z (cid:54) =

Original: | | W z | | z | | which is equivalent to
Revised: | | W z | | z | | which is equivalent to

Original: RIP condition.
Revised: RIP condition in Eq.

Original: Padding loss.
Revised: Padding loss.

Original: To force sparsity from the second to  L − 1 -th layers, we add a zero padding loss to each of these layers.
Revised: To force sparsity from the second to ( L − 1) -th layers, we add a zero padding loss to each of these layers.

Original: For the l -th layer whose target dimension is s( l) , pad the last m − s l elements of z ( l +1) with zeros and panish these elements with L 1 norm loss:
Revised: For the l -th layer whose target dimension is s l , pad the last m − s l elements of z ( l +1) with zeros and panish these elements with L 1 norm loss:

Original: [Equation]
Revised: [Equation]

Original: where { β ( l ) } are loss weights.
Revised: where { β ( l ) } are loss weights.

Original: The target dimension s( l) can be set heuristically.
Revised: The target dimension s l can be set heuristically.

Original: Linear transformation head.
Revised: Linear transformation head.

Original: We use the linear transformation head to achieve the linear compression step in our NLDR process, which is a transformation between the orthogonal basis of high dimension and lower dimension.
Revised: We use the linear transformation head to achieve the linear compression step in our NLDR process, which is a transformation between the orthogonal basis of high dimension and lower dimension.

Original: Thus, we apply the row orthogonal constraint to W L .
Revised: Thus, we apply the row orthogonal constraint to W L .

Original: LIS loss.
Revised: LIS loss.

Original: Since the linear DR is applied at the end of the NLDR process, we apply LIS constraint  to preserve the local geometric properties.
Revised: Since the linear DR is applied at the end of the NLDR process, we apply locally isometric smoothness (LIS) constraint (Li et al., 2020) to preserve the local geometric properties.

Original: Take the LIS loss in the l -th layer as an example:
Revised: Take the LIS loss in the l -th layer as an example:

Original: [Equation]
Revised: [Equation]

Original: where N ki is the set of x i ’s k -nearest neighborhood.
Revised: where N ki is a set of x i ’s k -nearest neighborhood in the input space, and d X and d Z are the distance of the input and the latent space, which can be approximated by Euclidean distance in local open sets.

Original: Push-away loss.
Revised: Push-away loss.

Original: In the real case discussed in Sec 3.2, the latent space of the  L − 1 -th layer can approximately to be a hyper-plane in high-dimensional, so that we introduce push-away loss to repel the non-adjacent sample points of each x i in its B -radius neighbourhood in the latent space.
Revised: In the real case discussed in Sec 3.2, the latent space of the ( L − 1) -th layer can approximately to be a hyperplane in Euclidean space, so that we introduce push-away loss to repel the non-adjacent sample points of each x i in its B -radius neighbourhood in the latent space.

Original: It deﬂates the manifold locally when acting together with L LIS .
Revised: It deﬂates the manifold locally when acting together with L LIS in the linear DR.

Original: Similarly, L push is applied after the linear transformation in the L -th layer:
Revised: Similarly, L push is applied after the linear transformation in the l -th layer:

Original: [Equation]
Revised: [Equation]

Revised: where 1 ( . ) ∈ { 0 , 1 } is the indicator function for the bound of B .

Original: Extra heads.
Revised: Extra heads.

Original: In order to force the ﬁrst L − 1 layers of the network to achieve NLDR gradually, we introduce auxiliary DR branchs, called extra head , at layers from the second to the  L − 1 -th.
Revised: In order to force the ﬁrst L − 1 layers of the network to achieve NLDR gradually, we introduce auxiliary DR branchs, called extra head , at layers from the second to the ( L − 1) -th.

Original: The structure of each extra head is same as the linear transformation head and will be discarded after training.
Revised: The structure of each extra head is same as the linear transformation head and will be discarded after training.

Original: L extra is written as
Revised: L extra is written as

Original: [Equation]
Revised: [Equation]

Original: where { γ ( l ) } and { µ ( l ) } are loss weights which can be set based on { s( l) } .
Revised: where { γ ( l ) } and { µ ( l ) } are loss weights which can be set based on { s l } .

Original: Inverse process.
Revised: Inverse process.

Original: The inverse process is the decoder directly obtained by the ﬁrst L − 1 layers of the encoder given by Eq. 3, which does not involved in the training process.
Revised: The inverse process is the decoder directly obtained by the ﬁrst L − 1 layers of the encoder given by Eq. (5), which does not involved in the training process.

Original: When the target dimension s (cid:48) is equal to s , the inverse of the layer- L can be solved by  compressed sensing rather than pseudo-inverse.
Revised: When the target dimension s (cid:48) is equal to s , the inverse of the layer- L can be solved by some existing methods such as compressed sensing or eigenvalue decomposition.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: To evaluate the invertibility of i-ML-Enc and analyze the property of data manifolds, we carry out experiments on seven datasets : (i) Swiss roll (Pedregosa et al., 2011), (ii)
Revised: In this section, we ﬁrst evaluate the proposed invertible NLDR achieved by i-ML-Enc in Sec 4.1, then investigate the property of data manifolds with i-ML-Enc in Sec 4.2. The properties of i-ML-Enc are further studied in Sec 4.3.

Original:  Spheres (Moor et al., 2020) and Half Spheres, (iii) USPS (Hull, 1994), (iv) MNIST (LeCun et al., 1998), (v) KMNIST (Clanuwat et al., 2018), (vi) FMNIST (Xiao et al., 2017), (vii)[-COIL-
-]
Revised: We carry out experiments on seven datasets : (i) Swiss roll (Pedregosa et al., 2011), (ii) Spheres (Moor et al., 2020) and Half Spheres, (iii) USPS (Hull, 1994), (iv) MNIST (LeCun et al., 1998), (v) KMNIST (Clanuwat et al., 2018), (vi) FMNIST (Xiao et al., 2017), (vii)[-COIL-
-]

Original:  (Nene et al., 1996b).
Revised: COIL-20 (Nene et al., 1996b).

Original: The implementation is based on the PyTorch 1.3.0 library running on NVIDIA [-v
-]
Revised: The implementation is based on the PyTorch 1.3.0 library running on NVIDIA [-v
-]v100 GPU.

Original: GPU.
Revised: The following settings of i-ML-Enc are used for all datasets: LeakyReLU with α = 0 . 1 ; Adam optimizer (Kingma & Ba, 2015) with learning rate lr = 0 . 001 for 8000 epochs; the local neighborhood is determined by kNN with k = 15 ; L layers neural network as shown in Fig.

Original: 4. M ETHODS C OMPARISON
Revised: 4. M ETHODS C OMPARISON

Original: We compare the proposed i-ML-Enc  with several typical methods in NLDR and inverse scenarios on Swiss roll, Spheres and Half Spheres, USPS, MNIST, FMNIST and COIL-20 datasets.
Revised: To verify the invertible NLDR ability of i-ML-Enc and analyze different cases of NLDR, we compare it with several typical methods in NLDR and inverse scenarios on both synthetic (Swiss roll, Spheres and Half Spheres) and real-world datasets (USPS, MNIST, FMNIST and COIL-20).

Original: Six methods for manifold learning : MLLE (Zhang & Wang, 2007), t-SNE (Maaten & Hinton, 2008) and ML-Enc (Li et al., 2020) are compared for NLDR; three AE-based methods VAE (Kingma & Welling, 2014),
Revised: Six methods for manifold learning : MLLE (Zhang & Wang, 2007), t-SNE (Maaten & Hinton, 2008) and ML-Enc (Li et al., 2020) are compared for NLDR; three AE-based methods VAE (Kingma & Welling, 2014),

Original: TopoAE (Moor et al., 2020) and ML-AE (Li et al., 2020) are compared for reconstructible manifold learning.
Revised: TopoAE (Moor et al., 2020) and ML-AE (Li et al., 2020) are compared for reconstructible manifold learning.

Original: Three methods for inverse models : INN (Nguyen et al., 2019), i-RevNet (Jacobsen et al., 2018), and i-ResNet (Behrmann et al., 2019) are compared for bijective property.
Revised: Three methods for inverse models : INN (Nguyen et al., 2019), i-RevNet (Jacobsen et al., 2018), and i-ResNet (Behrmann et al., 2019) are compared for bijective property.

Original: Among them, i-RevNet and i-ResNet are supervised algorithms while the rest are unsupervised.
Revised: Among them, i-RevNet and i-ResNet are supervised algorithms while the rest are unsupervised.

Revised: For a fair comparison in this experiment, we adopt 8 layers neural network for all the network-based methods except i-RevNet and i-ResNet.

Original: Hyperparameter values of i-ML-Enc and conﬁgurations of these datasets such as the input and target dimension are provided in Appendix A.2 .
Revised: Hyperparameter values of i-ML-Enc and conﬁgurations of these datasets such as the input and target dimension are provided in Appendix A.2 .

Original: [Table]
Revised: [Table]

Original: () - *+, - - () - *+, , ) .
Revised: [Figure]

Original: reconstruction - - () - *+, , )- - () - 5+, , ) .(b) Spere (PCA) - - () - *+, , )(c)
Revised: All the high-dimensional results are visualized by PCA and the target dimension s (cid:48) = 2 .

Original: HalfSpere (PCA) - - () - *+, , )- - () - *+, , ) < - - () - *+, , )  <
Revised: ) shows the NLDR and its inverse process of i-ML-Enc on the test set of Swiss roll in the case of d = s = s (cid:48) . We show the cases of s (cid:48) <

Original: (d) MNIST (PCA)
Revised: d ≤ s and s (cid:48)

Original: A - BC* () - *+, - - () - *+, , ) <
Revised: = d ≤ s by comparing (b)(c): (b) shows the failure case of reducing spheres S 100 sampled in R 101 into 10 -D, while (c) shows results of reducing half-spheres S 10 sampled in R 101 into 10 -D. The L 7 layers of i-ML-Enc show the same topology as the input data in both cases, but ML-Enc shows bad topological structures.

Original:  (e) COIL - 20 (PCA)
Revised: (d) and (e) show results of two sparse cases on MNIST and COIL-20:

Original: A - BC* () - *+, - - () - *+, , )() - *+, () - *+, - - () - 5+, , ) 46 - - () - *+, , ) <
Revised: The clustering effects of ML-Enc and t-SNE show the local geometric structure but dropping the relationship between sub-manifolds. With both of the geometric and topological structures, i-ML-Enc provides more reliable representations of the data manifold.

Original: Evalution metrics.
Revised: Evalution metrics.

Original: We evaluate the proposed invertible NLDR algorithm from three aspects: (1) Invertible property.
Revised: We evaluate an invertible NLDR algorithm from three aspects: (1) Invertible property.

Original: Reconstruction MSE ( RMSE ) and maximum norm error ( MNE ) measure the difference between the input  and reconstruction results by norm-based errors.
Revised: Reconstruction MSE ( RMSE ) and maximum norm error ( MNE ) measure the difference between the input data and reconstruction results by norm-based errors.

Original: (2) NLDR quality.
Revised: (2) NLDR quality.

Original: Trustworthiness ( Trust ) and Continuity ( Cont ) (Moor et al., 2020), latent MSE ( l-MSE ), Minimum ( K min ) and Maximum ( K max ) local Lipschitz constant (Li et al., 2020) are used to evaluate the quality of the low-dimensional representation.
Revised: Trustworthiness ( Trust ) and Continuity ( Cont ) (Kaski & Venna, 2006), latent MSE ( l-MSE ), Minimum ( K min ) and Maximum ( K max ) local Lipschitz constant (Li et al., 2020) are used to evaluate the quality of the low-dimensional representation.

Original: (3) Generalization ability of the representation.
Revised: (3) Generalization ability of the representation.

Original: Mean accuracy ( Acc ) of linear classiﬁcation on the representation measures models’ generalization ability to downstream tasks.
Revised: Mean accuracy ( Acc ) of linear classiﬁcation on the representation measures models’ generalization ability to downstream tasks.

Original: Their exact deﬁnitions and purpose are given in Appendix A.1 .
Revised: Their exact deﬁnitions and purpose are given in Appendix A.1 .

Original: Conclusion.
Revised: Conclusion.

Original: Table 1 compares the i-ML-Enc with the related methods on MNIST, more  detailed analysisand results on Swiss roll, Half Spheres, USPS, FMNIST and COIL-20 are given in Appendix 
Revised: Table 1 compares the i-ML-Enc with the related methods on MNIST, more results and detailed analysis on other datasets are given in Appendix A.2 .

Original: A.2 .
Revised: The process of invertible NLDR of i-ML-Enc and comparing results of typical methods are visualized in Fig.

Original: Compared with NLDR algorithms, the L -th layer of i-ML-Enc achieves second best NLDR results while preserving the all information of the dataset manifold in the L − 1 -th layer. Compared with inverse models, i-ML-Enc nearly outperforms the other methods in both invertible and NLDR metrics, which indicates that good low-dimensional representation ofmanifolds can be learned by the equi-dimensional layers.
Revised: We can conclude: (1) i-ML-Enc achieves invertible NLDR in the ﬁrst stage with great NLDR and generalization qualities. The representation in the L − 1 -th layer of i-ML-Enc mostly outperforms all comparing methods for both invertible and NLDR metrics without losing information of the data manifold, while other methods drop geometric and topological information to some extent. (2) i-ML-Enc tries to keep more geometric and topological structure in the second stage in the case of s (cid:48) <

Original: The NLDR and its inverse process of i-ML-Enc are visualized in Fig.
Revised: d ≤ s . Though the representation of the L -th layer of i-ML-Enc achieves the second best in NLDR metrics, it shows high consistency with the L − 1 -th layer in visualization results.

Original: 4. L ATENT S PACE I NTERPOLATION
Revised: 4. L ATENT S PACE I NTERPOLATION

Original: Since the ﬁrst L − 1 layers of i-ML-Enc are nearly homeomorphism, we carry out linear interpolation experiments on the discrete data points in both the input space and the  L − 1 -th layer latent space to approximate the intrinsic continuous manifold, and verify the latent results by its inverse process.
Revised: Since the ﬁrst stage of i-ML-Enc is nearly homeomorphism, we carry out linear interpolation experiments on the discrete data points in both the input space and the ( L − 1) -th layer latent space to analyze the intrinsic continuous manifold, and verify the latent results by its inverse process.

Original: A good low-dimensional representation of the manifold shall not only preserves the local properties, but also is ﬂatter and denser  with lower curvature.
Revised: A good low-dimensional representation of the manifold should not only preserve the local properties, but also be ﬂatter and denser than the high-dimensional input with lower curvature.

Original: Thus, we expect that the local linear interpolation results in the latent space should be more reliable than in the input space.
Revised: Thus, we expect that the local linear interpolation results in the latent space should be more reliable than in the input space.

Original: The complexity of data manifolds increases from USPS(256), MNIST(256), MNIST(784), KMNIST(784) to FMNIST(784), which is analyzed in Appendix A.3.1 .[Figure]
Revised: The complexity of data manifolds increases from USPS(256), MNIST(256), MNIST(784), KMNIST(784) to FMNIST(784), which is analyzed in Appendix A.3.1 .

Original: K-nearest neighbor interpolation.
Revised: K-nearest neighbor interpolation.

Original: We ﬁrst verify the reliability of the low-dimensional representation in a small local system by kNN interpolation.
Revised: We ﬁrst verify the reliability of the low-dimensional repre- [Figure] sentation in a small local system by kNN interpolation.

Original: Given a sample x i , randomly select x j in x i ’s k-nearest neighborhood in the latent space to form a sample pair ( x i , x j ) .
Revised: Given a sample x i , randomly select x j in x i ’s k-nearest neighborhood in the latent space to form a sample pair ( x i , x j ) .

Original: Perform linear interpolation of the latent representation of the pair and get reconstruction results for evaluation as: ˆ x ti,j = ψ − 1 ( tψ ( x i ) + (1 − t ) ψ ( x j )) , t ∈ [0 , 1] .
Revised: Perform linear interpolation of the latent representation of the pair and get reconstruction results for evaluation as: ˆ x ti,j = ψ − 1 ( tψ ( x i ) + (1 − t ) ψ ( x j )) , t ∈ [0 , 1] .

Original: The experiment is performed on i-ML-Enc with
Revised: The experiment is performed on i-ML-Enc with

Original: L = 6 and K = 15 , training with 8000 samples for USPS and MNIST(256), 20000 sapmles for
Revised: L = 6 and K = 15 , training with 8000 samples for USPS and MNIST(256), 20000 sapmles for

Original: MNIST(784), KMNIST, FMNIST.
Revised: MNIST(784), KMNIST, FMNIST.

Original: Evaluation.
Revised: Evaluation.

Original: (1) Calculate the MSE loss between reconstruction results of the latent interpolation ˆ x ti,j and the input space result x ti,j which is the corresponding interpolation results in the local neighborhood of the input space with x ti,j = t x i + (1 − t ) x j . Fig.
Revised: (1) Calculate the MSE loss between reconstruction results of the latent interpolation ˆ x ti,j and the input space result x ti,j which is the corresponding interpolation results in the local neighborhood of the input space with x ti,j = t x i + (1 − t ) x j . Fig.

Original: 5 shows the results of k = 1 , 2 , ..., 10 .
Revised: 5 shows the results of k = 1 , 2 , ..., 10 .

Original: (2) Visualize the typical results of the input space and the latent space for comparison, as shown in Fig.
Revised: (2) Visualize the typical results of the input space and the latent space for comparison, as shown in Fig.

Original: More results and detailed analysis are given in Appendix A.3.2 . 
Revised: More results and detailed analysis are given in Appendix A.3.2 . [Figure]

Original: Geodesic interpolation.
Revised: Geodesic interpolation.

Original: Based on 4.2.1, we further employ a more reasonable method to generate the sampling points between two distant samples pairs in the latent space.
Revised: Based on 4.2.1, we further employ a more reasonable method to generate the sampling points between two distant samples pairs in the latent space.

Original: Given a sample pair ( x i , x j )[Figure] with k ≥ 45 from different clusters, we select the three intermediate sample pairs ( x i , x i 1 ) , ( x i 1 , x i 2 ) , ( x i 2 , x j ) with k ≤ 20 along the geodesic path in latent space for piece-wise linear interpolation in both space.
Revised: Given a sample pair ( x i , x j ) with k ≥ 45 from different clusters, we select the three intermediate sample pairs ( x i , x i 1 ) , ( x i 1 , x i 2 ) , ( x i 2 , x j ) with k ≤ 20 along the geodesic path in latent space for piece-wise linear interpolation in both space.

Original: Visualization results are given in Appendix A.3.2 .
Revised: Visualization results are given in Appendix A.3.2 .

Original: Conclusion.
Revised: Conclusion.

Original: Compared with results of the kNN and geodesic interpolation, we can conclude: (1)
Revised: Compared with results of the kNN and geodesic interpolation, we can conclude: (1)

Original: Because of the sparsity of the high-dimensional latent space, noises are inevitable on the latent results indicating the limitation of linear approximation.
Revised: Because of the sparsity of the high-dimensional latent space, noises are inevitable on the latent results indicating the limitation of linear approximation.

Original: Empirically, the reliability of the latent interpolation decreases with the expansion of the local neighborhood on the same dataset.
Revised: Empirically, the reliability of the latent interpolation decreases with the expansion of the local neighborhood on the same dataset.

Original: (2) We will get worse latent results in the following cases: on the similar manifolds, the sampling rate is lower or the input dimension is higher indicated by USPS(256), MNIST(256) and MNIST(784); with the same sampling rate and input dimension, the manifold is more complex indicated by MNIST(784),
Revised: (2) We will get worse latent results in the following cases: on the similar manifolds, the sampling rate is lower or the input dimension is higher indicated by USPS(256), MNIST(256) and MNIST(784); with the same sampling rate and input dimension, the manifold is more complex indicated by MNIST(784),

Original: KMNIST to FMNIST.
Revised: KMNIST to FMNIST.

Original: They indicate that the conﬁdence of the tangent space estimated by local neighborhood decreases on more complex manifolds with sparse sampling.
Revised: They indicate that the conﬁdence of the tangent space estimated by local neighborhood decreases on more complex manifolds with sparse sampling.

Original: The interpolation between two samples in latent space is more smooth than that  the input space, validating the ﬂatness and density of the lower-dimensional representation learned by i-ML-Enc .
Revised: The interpolation between two samples in latent space is smoother than that in the input space, validating the ﬂatness and density of the lower-dimensional representation learned by i-ML-Enc .

Original: Overall, we infer that the unreliable approximation of the local tangent space by the local neighborhood is the basic reason of the manifold learning fails in the real-world case, because the geometry should be preserved in the ﬁrst place.
Revised: Overall, we infer that the unreliable approximation of the local tangent space by the local neighborhood is the basic reason for the manifold learning fails in the real-world case, because the geometry should be preserved in the ﬁrst place.

Original: To come up with this common situation, it is necessary to import other prior assumption or knowledge when the sampling rate of the data manifold is quite low, e.g. the Euclidean space assumption, semantic information of down-steam tasks.
Revised: To come up with this common situation, it is necessary to import other prior assumption or knowledge when the sampling rate of the data manifold is quite low, e.g. the Euclidean space assumption, semantic information of down-steam tasks.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original:  We perform  ablation study on MNIST, USPS, KMNIST, FMNIST and COIL-20 to evaluate  effects of the  network structure andthe loss terms in i-ML-Enc formanifold learning and invertible property.
Revised: Analysis on loss terms. We perform an ablation study on MNIST, USPS, KMNIST, FMNIST and COIL-20 to evaluate the effects of the proposed network structure and loss terms in i-ML-Enc for invertible manifold learning.

Original: Based on ML-Enc, three proposed parts are added: the extra head ( Ex ), the orthogonal loss L orth ( Orth ), the zero padding loss L pad ( Pad ).
Revised: Based on ML-Enc, three proposed parts are added: the extra head ( Ex ), the orthogonal loss L orth ( Orth ), the zero padding loss L pad ( Pad ).

Original: Besides the previous 8 indicators, we introduce the rank of the output matrix of the layer L − 1 as r ( Z L − 1 ) , to measure the sparsity of the high-dimensional representation.
Revised: Besides the previousindicators, we introduce the rank of the output matrix of the layer L − 1 as r ( Z L − 1 ) , to measure the sparsity of the high-dimensional representation.

Original: We conclude that the  Ex+Orth+Pad is the bestcombination to achieve invertible NLDR with s -sparse.
Revised: We conclude that the combination Ex+Orth+Pad is the best to achieve invertible NLDR of s -sparse by a series of equidimensional layers.

Original: The detailed analysis of experiment resultsand further discussion of the s -sparse are given in Appendix A.4 . 
Revised: The detailed analysis of experiment results are given in Appendix A.4.1 . Orthogonality and sparsity.

Original: C ONCLUSION
Revised: We further discuss the orthogonality of weight matrices and learned s -sparse representations in the ﬁrst stage of i-ML-Enc . We ﬁnd that the ﬁrst L − 1 layers of i-ML-Enc are nearly strict orthogonal mappings and the output from the L − 1 -th layer can be converted to s -dimensional representation without information loss. The detailed analysis are provided in Appendix A.4.2 .

Revised: Thus, we conclude that an invertible NLDR of data manifolds can be learned by i-ML-Enc in the sparse coordinate transformation .

Revised:  C ONCLUSION

Original: A novel invertible DR process inv-ML and a neural network implementation inv-ML-Enc are proposed to tackle two problems of manifold-based DR in practical scenarios, i.e., the condition for informationlossless NLDR and the key issue of manifold learning.
Revised: A novel invertible NLDR process inv-ML and a neural network implementation inv-ML-Enc are proposed to tackle two problems of manifold-based DR in practical scenarios, i e., the condition for information-lossless NLDR and the key issue of manifold learning.

Original: Firstly, the sparse coordinate transformation is learned to ﬁnd a ﬂatter and denser low-dimensional representation with preservation of geometry and topology of data manifolds.
Revised: Firstly, the sparse coordinate transformation is learned to ﬁnd a ﬂatter and denser low-dimensional representation with preservation of geometry and topology of data manifolds.

Original: Secondly, we discuss the information loss with different target dimensions in linear compression .
Revised: Secondly, we discuss the information loss with different target dimensions in linear compression .

Original: Experiment results of i-ML-Enc on seven datasets validate its invertibility.
Revised: Experiment results of i-ML-Enc on seven datasets validate its invertibility.

Original: Further, the interpolation experiments reveal that ﬁnding a reliable tangent space by the local neighborhood on real-world datasets is the inherent defect of manifold based DR methods.
Revised: Further, the interpolation experiments reveal that ﬁnding a reliable tangent space by the local neighborhood on real-world datasets is the inherent defect of manifold based DR methods.

Original: A A PPENDIX A. D EFINITIONS OF P ERFORMANCE M ETRICS
Revised: A A PPENDIX A. D EFINITIONS OF P ERFORMANCE M ETRICS

Original: As for NLDR tasks, We adopt the performance metrics used in MLDL (Li et al., 2020) and TopoAE (Moor et al., 2020) to measure topology-based manifold learning, and add a new indicator to evaluate the generalization ability of the latent space.
Revised: As for NLDR tasks, We adopt the performance metrics used in MLDL (Li et al., 2020) and TopoAE (Moor et al., 2020) to measure topology-based manifold learning, and add a new indicator to evaluate the generalization ability of the latent space.

Original: Essentially, the related indicators regard the local neighborhood of the input space as the ground truth.
Revised: Essentially, the related indicators are deﬁned based on comparisons of the local neighborhood of the input space and the latent representation.

Original: As for the invertible property, we adopted the norm-based reconstruction metrics, i.e. the L 2 and L ∞ norm errors, which are also based on the inputs.
Revised: As for the invertible property, we adopted the norm-based reconstruction metrics, i.e. the L 2 and L ∞ norm errors, which are also based on the inputs.

Original: The following notations are used in the deﬁnitions d ( l ) i,j is the pairwise distance in space Z ( l ) ; N ( l ) i,k is the set of indices to the k -nearest neighbors ( k -NN) of z ( l ) i in latent space, and N i,k is the set of indices to the k -NN of x i in input space; r ( l ) i,j is the closeness rank of z ( l ) j in the k -NN of z ( l ) i .
Revised: The following notations are used in the deﬁnitions d ( l ) i,j is the pairwise distance in space Z ( l ) ; N ( l ) i,k is the set of indices to the k -nearest neighbors ( k -NN) of z ( l ) i in latent space, and N i,k is the set of indices to the k -NN of x i in input space; r ( l ) i,j is the closeness rank of z ( l ) j in the k -NN of z ( l ) i .

Original: The evaluation metrics are deﬁned below:
Revised: The evaluation metrics are deﬁned below:

Original: (1) RMSE (invertible quality).
Revised: (1) RMSE (invertible quality).

Original: This indicator is commonly used to measure reconstruction quality.
Revised: This indicator is commonly used to measure reconstruction quality.

Original: Based on the input x and the reconstruction output ˆ x , the mean square error (MSE) of the L2 norm isdeﬁned as:
Revised: Based on the input x and the reconstruction output ˆ x , the mean square error (MSE) of the L norm is

Revised: deﬁned

Revised: as:

Original: [Equation]
Revised: [Equation]

Original: (2) MNE (invertible quality).
Revised: (2) MNE (invertible quality).

Original: This indicator is designed to evaluate the bijective property of a L layers neural network model.
Revised: This indicator is designed to evaluate the bijective property of a L layers neural network model.

Original: Speciﬁcally, taking each invertible unit in the network, calculate the L ∞ norm error of the input and reconstruction output of each corresponding layer, and choose the maximum value among all units.
Revised: Speciﬁcally, taking each invertible unit in the network, calculate the L ∞ norm error of the input and reconstruction output of each corresponding layer, and choose the maximum value among all units.

Original: If a model is bijective, this indicator can reﬂect the stability of the model:
Revised: If a model is bijective, this indicator can reﬂect the stability of the model:

Original: [Equation]
Revised: [Equation]

Original: (3) Trust (embedding quality).
Revised: (3) Trust (embedding quality).

Original: The indicator measures how well neighbors are preserved between the two spaces.
Revised: This indicator measures how well neighbors are preserved between the two spaces.

Original: The k nearest neighbors of a point are preserved when going from the input space X to space Z ( l ) :
Revised: The k nearest neighbors of a point are preserved when going from the input space X to space Z ( l ) :

Original: [Equation]
Revised: [Equation]

Original:   where k 1 and k 2 are the bounds of the number of nearest neighbors, so averaged for different k -NN numbers.
Revised:   where k 1 and k 2 are the bounds of the number of nearest neighbors, so averaged for different k -NN numbers.

Original: (4) Cont (embedding quality).
Revised: (4) Cont (embedding quality).

Original: The indicator is asymmetric to Trust (from space Z ( l ) to  space X ):
Revised: This indicator is asymmetric to Trust . It checks to what extent neighbors are preserved from the latent space Z ( l ) to the input space X :

Original: [Equation]
Revised: [Equation]

Original:   (5) K min and K max (embedding quality).
Revised:   (5) K min and K max (embedding quality).

Original: Those indicators are the minimum and maximum of the local bi-Lipschitz constant for the homeomorphism between input space and the l -th layer, with respect to the given neighborhood system:
Revised: Those indicators are the minimum and maximum of the local bi-Lipschitz constant for the homeomorphism between input space and the l -th layer, with respect to the given neighborhood system:

Original: [Equation]
Revised: [Equation]

Original: where k is that for k -NN used in deﬁning N i and
Revised: where k is that for k -NN used in deﬁning N i and

Original: [Equation]
Revised: [Equation]

Revised: ) i,j d ( l ) i,j (cid:41) .

Original: (6) l -MSE (embedding quality).
Revised: (6) l -MSE (embedding quality).

Original: This indicator is to evaluate the distance disturbance between the input space and latent space with L 2 norm-based error.
Revised: This indicator is to evaluate the distance disturbance between the input space and latent space with L 2 norm-based error.

Original: [Equation]
Revised: [Equation]

Original: (7) ACC (generalization ability).
Revised: (7) ACC (generalization ability).

Original:  To measure the generalization ability of the learned representation, logistic regression (Pedregosa et al., 2011) is performed after the  latent representation.
Revised: In general, a good representation should have a good generation ability to downstream tasks. To measure this ability, logistic regression (Pedregosa et al., 2011) is performed after the learned latent representation.

Original: We report the mean accuracy on the test set for 10 -fold cross-validation.
Revised: We report the mean accuracy on the test set for 10 -fold cross-validation.

Original: A. M ETHOD C OMPARISON 
Revised: A. M ETHOD C OMPARISON Conﬁgurations of datasets.

Original: Conﬁgurations of datasets.

Original: The NLDR performance and its inverse process are veriﬁed on both synthetic and real world datasets.
Revised: The NLDR performance and its inverse process are veriﬁed on both synthetic and real-world datasets.

Original: As shown in Table 2, we list the type of the dataset, the class number of clusters, the input dimension m , the target dimension s (cid:48) , the intrinsic dimension d which is only an approximation for the real world dataset, thesample number of train and test , and the logistic classiﬁcation performance on the latent representation.
Revised: As shown in Table 2, we list the type of the dataset, the class number of clusters, the input dimension m , the target dimension s (cid:48) , the intrinsic dimension d which is only an approximation for the real-world dataset, the number of train and test samples, and the logistic classiﬁcation performance on the raw input space.

Original: Among them, Swiss roll serves as an ideal example of information-lossless NLDR; Spheres serves as an excessive case of NLDR compared to half spheres; four image datasets with increasing difﬁculties are used to analyze complex situations in real world scenarios.
Revised: Among them, Swiss roll serves as an ideal example of information-lossless NLDR; Spheres, whose target dimension s (cid:48) is lower than the intrinsic dimension s , serves as an excessive case of NLDR compared to Half-spheres; four image datasets with increasing difﬁculties are used to analyze complex situations in real-world scenarios.

Original: Besides, the lower bound and upper bound of the intrinsic dimension of real-world datasets are approximated by (Hein & Audibert, 2005) and AE-based INN (Nguyen et al., 2019).
Revised: Additionally, the lower bound and upper bound of the intrinsic dimension of real-world datasets are approximated by (Hein & Audibert, 2005) and AE-based INN (Nguyen et al., 2019).

Original: Speciﬁcally, the upper bound can be found  the grid search of different bottlenecks of the INN, and we report the bottleneck size of each dataset when the reconstruction MSE loss is almost unchanged.
Revised: Speciﬁcally, the upper bound can be found by the grid search of different bottlenecks of the INN, and we report the bottleneck size of each dataset when the reconstruction MSE loss is almost unchanged

Original: [Table]
Revised: [Table]

Original: Hyperparameter values.
Revised: Hyperparameter values.

Original: Basically,we set α = 0 . 1 in LeakyReLU, and optimize the i-ML-Enc  with Adam  (Kingma & Ba, 2015)  for 10000 epochs. 
Revised: Basically, i-ML-Enc is trained with Adam optimizer (Kingma & Ba, 2015) and learning rate lr = 0 . 001 for 8000 epochs. We set the layer number L = 8 for most datasets but L = 6 for COIL-20.

Revised: The bound in push-away loss is set B = 3 in most datasets but removed in Spheres and Half-spheres.

Original: We set the hyperparameter based on two intuitions: (1) the implementation of sparse coordinate transformation should achieve DR on the premise of maintaining homeomorphism; (2) NLDR should be achieved gradually rather than suddenly.
Revised: We set the hyperparameter based on two intuitions: (1) the implementation of sparse coordinate transformation should achieve DR on the premise of maintaining homeomorphism; (2) NLDR should be achieved gradually from the ﬁrst to ( L − 1) -th layer because NLDR is impossible to achieve by a single nonlinear layer.

Original: Based on (1), wegradually decrease the extra heads weights γ linearly for epochs from 2000 to 8000 , while linearly increase the orthogonal loss weights α for epochs from 500 to 2000 .
Revised: Based on (1) [Table] we decrease the extra heads weights γ linearly for epochs from 2000 to 8000 , while linearly increase the orthogonal loss weights α for epochs from 500 to 2000 .

Original: Based on (2), we approximate the DR trend by exponential series.
Revised: Based on (2) [Table] we approximate the DR trend by exponential series.

Original: For the extra heads , the target dimensions l should decrease exponentially from m to s (cid:48) for the 2 -th to  L − 1 -th layer, and the push-away loss weights µ decrease linearly.
Revised: For the extra heads , the target dimension decrease exponentially from m to s (cid:48) for the 2 -th to ( L − 1) -th layer [Table] and the push-away loss weights µ increase linearly.

Original: Similarly, the padding weight β should increase linearly.
Revised: Similarly, the padding weight β should increase linearly.

Original: The layer number L is set to 6 for COIL-20, and 8 for other datasets.
Revised: Because the intrinsic dimension is different from each real-world dataset, we adjust the prior hyperparameters according to the approximated intrinsic dimension.

Original: Results on toy datasets.
Revised: Results on toy datasets.

Original: The Table 3 compares the i-ML-Enc with other methods in 9 performance metrics on Swiss roll and half spheres.
Revised: The Table 3 compares the i-ML-Enc with other methods in 9 performance metrics on Swiss roll and Half-spheres datasets in the case of s = s (cid:48) .

Original: Seven methods for manifold learning:  t-SNE (Maaten & Hinton, 2008), RR (McQueen et al., 2016) and ML-Enc (Li et al., 2020) are compared for NLDR; four AE-based methods AE (Hinton &Salakhutdinov, 2006), VAE (Kingma & Welling, 2014),
Revised: Eight methods for manifold learning: Isomap (Tenenbaum et al., 2000), t-SNE (Maaten & Hinton, 2008), RR (McQueen et al., 2016), and ML-Enc (Li et al., 2020) are compared for NLDR; four AE-based methods AE (Hinton &

Original:  TopoAE (Moor et al., 2020), and ML-AE (Li et al., 2020) are compared for reconstructible manifold learning.
Revised: Salakhutdinov, 2006), VAE (Kingma & Welling, 2014), TopoAE (Moor et al., 2020), and ML-AE (Li et al., 2020) are compared for reconstructible manifold learning.

Original: We report the L -th layer of i-ML-Enc  for the NLDR quality and the  L − 1 -th layer  for theinvertible ability.
Revised: We report the L -th layer of i-ML-Enc (the ﬁrst stage) for the NLDR quality and the ( L − 1) -th layer (the second stage) for the

Original: ML-Enc performs best in Trust, K min, K max and l -MSE on Swiss roll which shows its great NLDR ability in the low-dimensional case, while i-ML-Enc shows advantages in the high-dimensional case. It shows that i-ML-Enc outperforms other methods in its invertible property of the L − 1 -th layer
Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: 0.7644 INN 0.

Original: [Table]
Revised: 0.8471 i-RevNet 0. 0.9386 i-ResNet 0. i-ML-Enc(L-1) 0. Coilt-SNE - - 0.

Original: Results on real-world datasets.
Revised: 0.9039 ML-Enc - - 0. AE 0. 0.8958 TopoAE 0. 0.9043 ML-AE 0.

Revised: 0.9548 i-ML-Enc (L) 0.

Revised: 0.9386 INN 0.

Revised: 0.9936 i-RevNet

Revised: [Table]

Revised: [Figure]

Revised: the high-dimensional latent space are visualized by PCA.

Revised: For each row, the left cells show the NLDR and reconstruction process in the ﬁrst stage of i-ML-

Original: The Table 4 compares the  i-ML-Enc with other methods inperformance metrics on USPS, FMNIST and COIL-20.
Revised: , and the cells show 2D results for comparison. ML- and t- show great clustering effects but drop topological information. Compared to the classical DR method Isomap (preserving the global geodesic distance) and t-SNE (preserving the local geometry), the representations learned by i-ML-Enc preserves the relationship between clusters and the local geometry within clusters. [Figure]

Original: Five methods for manifold learning: t-SNE and ML-Enc are compared for NLDR; three AE-based methods AE, ML-AE and TopoAE are compared for reconstructible manifold learning. Three methods for inverse models: INN (Nguyen et al., 2019), i-RevNet (Jacobsen et al., 2018), and i-ResNet (Behrmann et al., 2019) are compared for bijective property.
Revised: invertible NLDR ability. ML-Enc performs best in Trust, K min, K max, and l -MSE on Swiss roll which shows its great embedding abilities. Based on ML-Enc, i-ML-Enc achieves great embedding results in the second stage on Half-spheres, which shows its advantages of preserving topological and geometric structures in the high-dimensional case.

Original: The visualization of NLDR and its inverse process of i-ML-Enc are shown in 
Revised: And i-ML-Enc outperforms other methods in its invertible NLDR property of the ﬁrst stage.

Original: Fig.
Revised: The Table 4 compares the i-ML-Enc with other methods inperformance metrics on USPS, FMNIST and COIL-20 datasets in the case of s > s (cid:48) .

Original: 7, together with the NLDR results of t-SNE and ML-Enc. The target dimension for visualization
Revised: Six methods for manifold learning: Isomap, t-SNE, and ML-Enc are compared for NLDR; three AE-based methods

Original: [Figure] PCA)
Revised: AE, ML-AE, and TopoAE are compared for reconstructible manifold learning.

Original: (c [Figure] [Figure] ) reconstruction 22 -6767 -: - 67 - 589 , 7 ; : - 67 - 589 , 7 ; : - 67 - <=9 , 7 ;> : - 67 - <=9 , 7 ;> : - 67 - 589 , 7 ? : - 67 - 589 , 7 ? : - 67 - 589 , 7 @ : - 67 - 589 , 7 @  reconstruction (
Revised: Three methods for inverse models: INN (Nguyen et al., 2019), i-RevNet (Jacobsen et al., 2018), and i-ResNet (Behrmann et al., 2019) are compared for bijective property.

Original: [Figure] ) 2 - 345 67 - 589 : - 67 - 589 , 7 ; : - 67 - <=9 , 7 ;> : - 67 - 589 , 7 ? : - 67 - 589 , 7 @   reconstruction reconstruction 2 - 345 67 - 589 : - 67 - 589 , 7 B : - 67 - <=9 , 7 B> : - 67 - 589 , 7 C : - 67 - 589 , 7 D (b)
Revised: The visualization of NLDR and its inverse process of i-ML-Enc are shown in Fig.

Original: MNIST (PCA) 2 - 345 67 - 589 : - 67 - 589 , 7 ; : - 67 - <=9 , 7 ;> : - 67 - 589 , 7 ? : - 67 - 589 , 7 @ reconstruction [Figure] [Figure]
Revised: 7, together with the NLDR results of Isomap, t-SNE and, ML-Enc.

Original: randomly selected 10 classes to display. is s (cid:48)
Revised: The target dimension for visualization is s (cid:48)

Original: = 2 and the high-dimensional latent space are visualized by PCA.
Revised: = 2 and the high-dimensional latent space are visualized by PCA.

Original: Compared with NLDR algorithms, the  L -th layer of i-ML-Enc nearly achieves  best NLDR results on FMNIST, but ranksthe second place on USPS andthe third place on COIL-20.
Revised: Compared with NLDR algorithms, the representation of the L -th layer of i-ML-Enc nearly achieves the best NLDR metrics on FMNIST, and ranks second place on USPS and third place on COIL-20.

Original: Chances arethat it is caused by the  linear transformation layer, since the NLDR results of the L − 1 -th layer are quite reliable.
Revised: The drop of performance between the ( L − 1) -th and L -th layers of i-ML-Enc are caused by the sub-optimal linear transformation layer, since the representation of its ﬁrst stage are quite reliable.

Original: Compared with other invertible models, i-ML-Enc outperforms in all the NLDR metrics and  the RMSE loss, which shows the good quality of its low-dimensional representation of data manifolds with a series of equi-dimensional layers.
Revised: Compared with other inverse models, i-ML-Enc outperforms in all the NLDR metrics and inverse metrics in the ﬁrst stage, which indicates that a great low-dimensional representation of data manifolds can be learned by a series of equidimensional layers.

Original: Comparing the NME of i-ML-Enc with bijective models, i-ML-Enc has larger NME on FMNIST and COIL-20, which indicates that i-ML-Enc is more unstable dealing with complex datasets.
Revised: However, i-ML-Enc shows larger NME on FMNIST and COIL-20 compared with inverse models, which indicates that i-MLEnc is more unstable dealing with complex datasets in the ﬁrst stage.

Original: Besides, we visualize the reconstruction samples of six image datasets including COIL-100 (Nene et al., 1996a) to show the inverse quality of i-ML-Enc in Fig.8.
Revised: Besides, we visualize the reconstruction samples of six image datasets including COIL-100 (Nene et al., 1996a) to show the inverse quality of i-ML-Enc in Fig.

Original: A. L ATENT S PACE I NTERPOLATION A.3. D ATASETS C OMPARISON
Revised: A. L ATENT S PACE I NTERPOLATION A.3. D ATASETS C OMPARISON

Original: Here is a brief introduction to four interpolation data sets.
Revised: Here is a brief introduction to four interpolation data sets.

Original: We analyze the difﬁculty of dataset roughly according to dimension , sample size , image entropy , texture , and the performance of classiﬁcation tasks : (1) Sampling ratio.
Revised: We analyze the difﬁculty of dataset roughly according to dimension , sample size , image entropy , texture , and the performance of classiﬁcation tasks : (1) Sampling ratio.

Original: The input dimension and sample number reﬂect the sampling ratio.
Revised: The input dimension and sample number reﬂect the sampling ratio.

Original: Generally, the sample number has an exponential relationship with the input dimension in the case of sufﬁcient sampling.
Revised: Generally, the sample number has an exponential relationship with the input dimension in the case of sufﬁcient sampling.

Original: Thus, the sampling ratio of USPS is higher than others.
Revised: Thus, the sampling ratio of USPS is higher than others.

Original: (2) Image entropy.
Revised: (2) Image entropy.

Original: The Shannon entropy of the histogram measures the information content of the image, and it reaches the maximum when the density estimated by  histogram is an uniform distribution.
Revised: The Shannon entropy of the histogram measures the information content of the image, and it reaches the maximum when the density estimated by the histogram is an uniform distribution.

Original: We report the mean entropy of each dataset.
Revised: We report the mean entropy of each dataset.

Original: We conclude that USPS has richer gray scale than MNIST(256), while the information content of MNIST(784), KMNIST and FMNIST shows an increasing trend.
Revised: We conclude that USPS has richer grayscale than MNIST(256), while the information content of MNIST(784), KMNIST, and FMNIST shows an increasing trend.

Original:  Texture.
Revised: (3) Texture.

Original: The standard deviation (std) of the histogram reﬂects the texture information in the image, and we report the mean std of each dataset.
Revised: The standard deviation (std) of the histogram reﬂects the texture information in the image, and we report the mean std of each dataset.

Original: Combined with the evaluation of human eyes, the texture features become more and more rough from USPS, MNIST to KMNIST, while FMNIST contains complex and regular texture.
Revised: Combined with the evaluation of human eyes, the texture features become rougher and rougher from USPS, MNIST to KMNIST, while FMNIST contains complex and regular texture.

Original: (4) Classiﬁcation tasks.
Revised: (4) Classiﬁcation tasks.

Original: We report the mean accuracy of 10 -fold cross validation using kNN and logistic classiﬁer (Pedregosa et al., 2011) for each data set.
Revised: We report the mean accuracy of 10 -fold cross-validation using kNN and logistic classiﬁer (Pedregosa et al., 2011) for each data set.

Original: The credibility of  neighborhood system decreases gradually from USPS, MNIST, KMNIST to FMNIST.
Revised: The credibility of the neighborhood system decreases gradually from USPS, MNIST, KMNIST to FMNIST.

Original: Combined with  visualization results of each dataset in Fig.
Revised: Combined with the visualization results of each dataset in Fig.

Original: 7, it obvious that KMNIST has the worst linear separability.
Revised: 7, it obvious that KMNIST has the worst linear separability.

Original: Above all, we can roughly give the order of the difﬁculty of manifold learning on each data set: USPS < MNIST(256) < MNIST(784) < KMNIST < FMNIST .
Revised: Above all, we can roughly give the order of the difﬁculty of manifold learning on each data set: USPS < MNIST(256) < MNIST(784) < KMNIST < FMNIST .

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: A.3. M ORE I NTERPOLATION R ESULTS
Revised: A.3. M ORE I NTERPOLATION R ESULTS

Original: kNN interpolation.
Revised: kNN interpolation

Original: We verify the reliability of the low-dimensional representation by kNN interpolation.
Revised: We verify the reliability of the low-dimensional representation by kNN interpolation.

Original: Comparing the results of different values of k , as shown in Fig.
Revised: Comparing the results of different values of k , as shown in Fig.

Original: 9, we conclude that: (1) Because the high-dimensional latent space is still quite sparse, there are some noise caused by linear approximation on the latent results.
Revised: 9, we conclude that: (1) Because the high-dimensional latent space is still quite sparse, there is some noise caused by linear approximation on the latent results.

Original: The MSE loss and noises of the latent results is increasing with the expansion of the local neighborhood on the same dataset, reﬂecting the reliability of  local neighborhood system.
Revised: The MSE loss and noises of the latent results are increasing with the expansion of the local neighborhood on the same dataset, reﬂecting the reliability of the local neighborhood system.

Original: In terms of the same sampling rate, the MSE loss and noises of the latent results grow from MNIST(784), KMNIST to FMNIST, which indicates that the conﬁdence of the local homeomorphism property of the latent space decreases gradually on more difﬁcult manifolds.
Revised: In terms of the same sampling rate, the MSE loss and noises of the latent results grow from MNIST(784), KMNIST to FMNIST, which indicates that the conﬁdence of the local homeomorphism property of the latent space decreases gradually on more difﬁcult manifolds.

Original: In terms of the similar data manifolds, USPS(256) and MNIST(256) show better latent interpolation results than MNIST(784), which demonstrates that it is harder to preserve the geometric properties on higher input dimension.
Revised: In terms of the similar data manifolds, USPS(256) and MNIST(256) show better latent interpolation results than MNIST(784), which demonstrates that it is harder to preserve the geometric properties on higher input dimension.

Original: (4) Though the latent results import some noise, the input results have unnatural transformation such as pseudo-contour and overlapping.
Revised: (4) Though the latent results import some noise, the input results have unnatural transformations such as pseudo-contour and overlapping.

Original: Thus, the latent  space results are more smooth than the input space, which validates that the latent space learned by i-ML-Enc is ﬂatter and denser than the input space.
Revised: Thus, the latent [Figure] space results are more smooth than the input space, which validates that the latent space learned by i-ML-Enc is ﬂatter and denser than the input space.

Original: In a nutshell, we infer that the difﬁculty of preserving the geometric properties based on approximation of the local tangent space by the local neighborhood is the key reason for the manifold learning fails in the real-world case.
Revised: In a nutshell, we infer that the difﬁculty of preserving the geometric properties based on approximation of the local tangent space by the local neighborhood is the key reason for the manifold learning fails in the real-world case.

Original: Geodesic interpolation.
Revised: Geodesic interpolation.

Original: We further perform the latent interpolation along the geodesic path between sample pairs when k is large to generate reliable intermediate samples.
Revised: We further perform the latent interpolation along the geodesic path between sample pairs when k is large to generate reliable intermediate samples.

Original: It might reﬂect the topological structure of data manifolds when two samples in a sample pair are in different clusters.
Revised: It might reﬂect the topological structure of data manifolds when two samples in a sample pair are in different clusters.

Original: Compared with results of MNIST, KMNIST and FMNIST, as shown in Fig.
Revised: Compared with results of MNIST, KMNIST, and FMNIST, as shown in Fig.

Original: 10, we can conclude:[Figure] (1) The latent results is more reliable than that in the input space which can generates the synthetic samples between two different clusters.
Revised: 10, we can conclude: (1) The latent results are more reliable than those in the input space which can generate the synthetic samples between two different clusters.

Original: (2) Compared with MNIST, KMNIST and FMNIST, the latent results of more complex datasets are more ambiguous and noisy, which indicates that it is more difﬁcult to ﬁnd a low-dimensional representation of more complex data manifolds with all geometric structure preserved.
Revised: (2) Compared with MNIST, KMNIST, and FMNIST, the latent results of more complex datasets are more ambiguous and noisy, which indicates that it is more difﬁcult to ﬁnd a low-dimensional representation of more complex data manifolds with all geometric structure preserved.

Original: A. A BLATION S TUDY 
Revised: A. A BLATION S TUDY A.4. A NALYSIS OF THE LOSS TERMS

Original: Analysis and conclusion.

Original: We further conduct ablation study of the extra head ( +Ex ), the orthogonal loss L orth ( +Orth ), and the zero padding loss L pad ( +Pad ) on MNIST, USPS, KMNIST, FMNIST and COIL-20.
Revised: We further conduct ablation study of the extra head ( +Ex ), the orthogonal loss L orth ( +Orth ), and the zero padding loss L pad ( +Pad ) on MNIST, USPS, KMNIST, FMNIST and COIL-20.

Original: The Table 6 reports ablation results in the 8 indicators and the r ( Z L − 1 ) .
Revised: The Table 6 reports ablation results in the 8 indicators and the r ( Z L − 1 ) .

Original: We analyze and conclude: (1) The combination of Ex and Orth nearly achieve the best inverse and DR performance on MNIST, USPS, FMNIST and COIL-20, which indicates that it is the basic factor for invertible NLDR in the ﬁrst L − 1 layers.
Revised: We analyze and conclude: (1) The combination of Ex and Orth nearly achieve the best inverse and DR performance on MNIST, USPS, FMNIST, and COIL-20, which indicates that it is the basic factor for invertible NLDR in the ﬁrst L −layers.

Original: (2) When only use Orth , the NLDR in the ﬁrst L − 1 layer[Figure] of the network will degenerate into the identity mapping, and DR is achieved with  linear project on layer L .
Revised: (2) When only use Orth , the NLDR in the ﬁrst L − 1 layer of the network will degenerate into the identity mapping, and DR is achieved with the linear project on layer L .

Original: (3) Combined with all three items
Revised: (3) Combined with all three items

Original: Ex , Orth and Pad , i-ML-Enc obtains a sparse coordinate representation, but achieves little worse embedding quality on USPS and COIL-20 than using Ex and Orth .
Revised: Ex , Orth and Pad , i-ML-Enc obtains a sparse coordinate representation, but achieves little worse embedding quality on USPS and COIL-20 than using Ex and Orth .

Original: (4) Besides the proposed loss items, ML-AE overperforms the orther combinations in the Acc metric indicating the reconstruction loss helps improve the generation ability of ML-Enc.
Revised: (4) Besides the proposed loss items, ML-AE overperforms the other combinations in the Acc metric indicating the reconstruction loss helps improve the generation ability of ML-Enc.

Original: Above all, the Ex+Orth+Pad combination, i.e. i-ML-Enc , can achieve the proposed invertible NLDR. 
Revised: Above all, the Ex+Orth+Pad combination, i.e. i-ML-Enc , can achieve the proposed invertible NLDR. A.4. O RTHOGONALITY AND S PARSITY

Original: Further discussion.
Revised: Orthogonal analysis.

Original: We notice that the s -sparse achieved by i-ML-Enc is higher than the approximate intrinsic dimension d on each dataset, e.g. 116 -sparse on USPS and 125 -sparse on MNIST. We found the following reasons: (1) Because the data manifolds are usually quite complex but sampling sparsely, the lowest isometric embedding dimension are between d to 2 d according to Nash Embedding Theorem and the hyper-plane hypothesis.
Revised: We ﬁrst analyze the orthogonality of weight matrices in the ﬁrst stage by evaluating the orthogonal loss || W Tl W l − I || .

Original: The s obtained by i-ML-Enc on each dataset is nearly failed into the region. The proposed i-ML-Enc is not optimized enough which serves as a simple network implementation of inv-ML.
Revised: Using the same experimental settings as Sec 4.1, the maximum of non-diagonal elements and the minimum of diagonal elements of each layer are calculated in the ﬁrst stage of i-ML-Enc on different datasets.

Original: We need to design a better implementation model if we want to approach the lower embedding dimension with the preservation of both geometry and topology.
Revised: We ﬁnd that the margin of the maximum value and the minimum value is at least 4 orders of magnitude apart, as shown in Fig. We can conclude that the ﬁrst L − 1 layers in i-ML-Enc are close to strict orthogonal mappings.

Original: [Table]
Revised: [Table]

Original: KMNIST ML-AE 0. 19 ML-Enc - - 0.
Revised: [Table]

Original: +Ex - - 0. +Ex+Orth 0.
Revised: [Table] [Table] [Table]

Original: 406 +Ex+Orth+Pad 0. 198 +Orth 0.
Revised: [Table] [Table] [Table]

Original: FMNIST
Revised: [Table] [Table]

Original: ML-AE 0. 19 ML-Enc - - 0.
Revised: [Table] [Table] [Table]

Original: 18 +Ex - - 0. +Ex+Orth 0.
Revised: [Table] [Table] [Table]

Original: 393 +Ex+Orth+Pad 0.
Revised: [Table]

Original: 182 +Orth 0.
Revised: 182 +Orth 0.

Original: COILML-AE 0.
Revised: COILML-AE 0.

Original: [Table]

Original: [Table]

Original: [Table]

Original: ML-Enc - - 0.
Revised: ML-Enc - - 0.

Original: [Table]

Original: [Table]

Original: [Table]

Original: 46 +Ex+Orth 0.
Revised: 46 +Ex+Orth 0.

Original: [Table] [Table] [Table] 1190 +
Revised: 1190 +Ex+Orth+Pad 0.

Original: [Table]
Revised: [Figure]

Original: Orth+Pad 0.
Revised: Discussion on the s -sparse representation. We ﬁrst provide a possible way to decompose the learned low-dimensional representation in the L − 1 -th layer of i-ML-Enc , i.e. decomposing the output matrix by PCA to construct a linear subspace. Taking the 8 -th layer of i-ML-Enc on the test set of MNIST which is 784 -D as an example, we can construct a 171 -D orthogonal base vectors in the linear subspace from the data matrix after dimension reduction and reconstruct to the original space ( 784 -D) without losing information by PCA (Pedregosa et al., 2011) and its inverse transform, as shown in

Original: [Table]
Revised: Fig.

Original: [Table]
Revised: Compared to the matrix rank 125 -D in the 784 -D space, the extra 46 -D in the subspace can be regarded as the machine error in the process of performing PCA because of the large margin between the ﬁrst 125 eigenvalues and the rest. We notice that the s -sparse achieved by the ﬁrst stage of i-ML-Enc is higher than the approximate intrinsic dimension d on each dataset, e.g. 116 -sparse on

Original: [Table]
Revised: USPS and 125 -sparse on MNIST.

Original: [Table]
Revised: We found the following reasons: (1) Because the data manifolds are usually quite complex but sampling sparsely, the lowest isometric embedding dimension are between d to 2 d according to Nash Embedding Theorem and the hyper-plane hypothesis. The s obtained by i-ML-Enc on each dataset is nearly in the interval of [ d, 2 d ] , which is not the true intrinsic dimension of the manifolds. The proposed i-ML-Enc is not optimized enough which serves as a simple network implementation of inv-ML. We need to design a better implementation model if we want to approach the lower embedding dimension with the preservation of both geometry and topology.