The ability to preserve local geometry of highly nonlinear manifolds in high dimensional spaces and properly unfold them into lower dimensional hyperplanes is the key to the success of manifold computing, nonlinear dimensionality reduction (NLDR) and visualization. This paper proposes a novel method, called elastic locally isometric smoothness (ELIS {\em   ), to empower deep neural networks with such an ability. ELIS requires that a desired metric between points should be preserved across layers in order to preserve local geometry; such a smoothness constraint effectively regularizes vector-based transformations to become well-behaved local metric-preserving homeomorphisms. Moreover, ELIS requires that the smoothness should be imposed in a way to render sufficient flexibility for tackling complicated nonlinearity and non-Euclideanity; this is achieved layer-wisely via nonlinearity in both the similarity and activation functions. The ELIS method incorporates a class of suitable nonlinear similarity functions into a two-way divergence loss and uses hyperparameter continuation in finding optimal solutions . Extensive experiments, comparisons, and ablation study demonstrate that ELIS can deliver results not only superior to UMAP and t-SNE for and visualization but also better than other leading counterparts of manifold and autoencoder learning for NLDR and manifold data generation .
Nonlinear dimensionality reduction (NLDR) plays an important role in feature extraction and visualization of high dimensional data. It transforms patterns of interest, or manifolds, in data to regions in a lower-dimensional latent space. When the manifolds are highly complicated, the NLDR transformation has to provide sufficient nonlinearity for downstream tasks. This paper proposes a novel method, called {\em deep manifold transformation (DMT ), to tackle this problem. As a multi-layer neural network, DMT can support a higher degree of nonlinearity than single-layer methods. Continuation strategy is applied during training to address the local minimum problem common in manifold learning algorithms. Such a learned DMT network can generalize to unseen data whereas traditional manifold learning methods only provide an embedding of the training data . Extensive experiments, comparisons, and ablation studies demonstrate that DMT can deliver results superior to UMAP and t-SNE , and other leading manifold-based NLDR methods .
R
The ability to preserve local geometry of highly nonlinear manifolds in high dimensional spaces and properly unfold them into lower dimensional hyperplanes is the key to the success of manifold computing, nonlinear
Nonlinear
R
and visualization.
plays an important role in feature extraction and visualization of high dimensional data. It transforms patterns of interest, or manifolds, in data to regions in a lower-dimensional latent space. When the manifolds are highly complicated, the NLDR transformation has to provide sufficient nonlinearity for downstream tasks.
D
elastic locally isometric smoothness (ELIS
A
deep manifold transformation
A
(DMT
R
empower deep neural networks with such an ability. ELIS requires that a desired metric between points should be preserved across layers in order to preserve local geometry; such a smoothness constraint effectively regularizes vector-based transformations to become well-behaved local metric-preserving homeomorphisms. Moreover, ELIS requires that the smoothness should be imposed in a way to render sufficient flexibility for tackling complicated nonlinearity and non-Euclideanity; this is achieved layer-wisely via nonlinearity in both the similarity and activation functions. The ELIS method incorporates a class of suitable nonlinear similarity functions into a two-way divergence loss and uses hyperparameter continuation in finding optimal solutions
tackle this problem. As a multi-layer neural network, DMT can support a higher degree of nonlinearity than single-layer methods. Continuation strategy is applied during training to address the local minimum problem common in manifold learning algorithms. Such a learned DMT network can generalize to unseen data whereas traditional manifold learning methods only provide an embedding of the training data
R
study demonstrate that ELIS
studies demonstrate that DMT
D
not only
R
for and visualization but also better than other leading counterparts of manifold and autoencoder learning for NLDR and manifold data generation
, and other leading manifold-based NLDR methods