Paper:
Auto-encoding and generative models have made tremendous successes in image and signal representation learning and generation. These models, however, generally employ the full Euclidean space or a bounded subset (such as [0, 1]   l ) as the latent space, whose flat geometry is often too simplistic to meaningfully reflect the topological structure of the data. This paper aims at exploring a universal geometric structure of the latent space for better data representation. Inspired by differential geometry, we propose a Chart Auto-Encoder (CAE), which captures the manifold structure of the data with multiple charts and transition functions among them. CAE translates the mathematical definition of manifold through parameterizing the entire data set as a collection of overlapping charts, creating local latent representations. These representations are an enhancement of the singlecharted latent space commonly employed in auto-encoding models, as they reflect the intrinsic structure of the manifold. Therefore, CAE achieves a more accurate approximation of data and generates realistic synthetic examples. We demonstrate the efficacy of CAEs through a series experiments with synthetic and real-life data which illustrate that CAEs can out-preform variational auto-encoders on reconstruction tasks while using much smaller latent spaces.

Review:
## Summary of the Paper
This paper introduces a new architecture for autoencoders based on the
concept of *charts* from (differential) topology. Instead of learning a
single latent representation, the paper proposes learning charts, which
serve as local latent representations. Experiments demonstrate that the
local representations perform favourably in terms of approximating  the
underlying manifold.
## Summary of the Review
This is an interesting paper with an original idea. I appreciate the use
of concepts from differential topology in deep learning and agree with
the paper that such a perspective is required to increase our
understanding of complicated manifold data sets. However, I find the
following issues with the paper in its current form, which prevent me
from endorsing it for acceptance:
1. I have doubts about the technical correctness of the proposed
   architecture; specifically, the relevance of the *initial* latent
   representation, which employs a Euclidean space, is not analysed.
2. The role of the number of charts, which needs to be specified
   before-hand, is not analysed in an ablation study.
3. The experiments do not showcase the *conceptual* improvements of the
   proposed technique.
I shall briefly comment on each of these points before discussing other
improvements.
I want to point out that I really like the ideas presented in this
paper and I think it has the potential to make a strong contribution but
the issues in their current form require substantial revisions and
additions.
# Concern 1: Technical correctness
The paper claims at multiple places that the geometry of Euclidean space
is 'trivial' or 'too simplistic' to meaningfully reflect the structure
of the data. This claim is double-edged, though: first, there are
many methods that use autoencoders based on these spaces that exhibit
sufficient reconstruction capabilities. Second, the proposed
architecture itself uses a Euclidean latent representation as its
initial encoder. The paper states that 'Ideally, this step preserves the
topology of the data [...]', but this is never analysed.
I fully agree with the idea that charts are a suitable way to describe
complicated manifolds, but the paper needs more precision when terms
such as 'topology' and 'geometry' are being used. Likewise, I disagree
with referring to Euclidean space as 'trivial'. Again, other methods
demonstrate that the space captures high-level phenomena sufficiently
well for reconstruction purposes. At the very least, the paper should
be more precise here.
Moreover, I would recommend experiments in which the dimensionality of
the initial encoder is discussed.
# Concern 2: Number of charts
Selecting the number of charts appears to me as a critical component of
the proposed method. While the appendix contains one experiment for
MNIST with different numbers of charts, this concept needs to be fleshed
out more. How do we know that we have a sufficient number of charts?
Since in differential topology, the choice of chart should not matter,
how does it behave in these cases? Is there a way to detect that the
number of charts must be increased?
I could envision something like a simple 'step size' control procedure:
if a quality measure indicates that there need to be more charts, double
the number of charts and re-run the training; if the number of charts
is too big, halve it and re-run the training.
I get the idea that increasing the number of charts will probably
decrease the reconstruction error, but this comes at the obvious expense
of even more parameters. I thus recommend another set of experiments
that shows the influence of the number of charts, maybe even on the
synthetic data sets used in the paper.
# Concern 3: Conceptual improvements
While I enjoyed the didactic approach of the paper, which first
introduces simple test data sets to illustrate the concepts, my
main question is about the conceptual improvements that the charts
provide in the end.
I see that the reconstruction error for MNIST goes down---but there are
also significantly more (!) parameters than in the comparison
architectures. The ideas of the sampling or interpolation experiments
go in the right direction, but in their present version, they are not
entirely convincing. In fact, they even raised more questions for me: 
- Figure 6 depicts individual charts but their *covering* of the space
  is highly non-uniform. The letter '0' is covered more often than the
  letter '1', for example. How can this be compatible with the claim
  that the novel architecture learns a suitable set of charts? I could
  understand some overlaps, but there seems to be a clear difference
  between the charts generated in the synthetic examples---which do
  appear to be cover everything in a uniform manner---and the charts for
  MNIST. This needs to be elucidated some more, in particular since the
  paper writes that the charts 'cover [...] in a balanced and regular
  way'.
- The digit morphing example is not not entirely convincing to me. Is
  this not something that I can do equally well with a VAE or generative
  models in general? I am *not* disputing the claims of the paper here,
  I am merely stating that *if* the new method is beneficial for this
  sort of application, a more in-depth experiment is required.
Thus, while I would like to give the paper the benefit of the doubt, it
does not show just *why* it is relevant to have a chart-based embedding.
Some suggestions for a set of experiments:
- Do charts help in separating the input space? I would hypothesise
  that this is the case---it thus might be worthwhile to study
  low-dimensional embeddings obtained based on each chart and 'stitch'
  them together.
- Do charts tell us something about the properties of a manifold? For
  example, are certain charts 'easier' to embed than others? This could
  be used to indicate different dimensions in a data set.
## Experimental setup
I have one major point of critique here, namely the way results are
presented without any measures of tendency. Instead of showing a bar
plot in Figure 7, I would suggest showing a Table with standard
deviations along multiple repetitions of the experiment. It is not clear
from looking at this to what extent this results can be replicated.
Moreover, a discussion of the number of parameters is required. To some
extent, I find it not surprising that a better reconstruction error is
achieved if more parameters are present.
This makes some of the claims in the paper hard to assess.
## Technical clarity
The papers is generally written well and has a good expository style.
Here are some cases where I find that clarity can be improved:
- To add to what I wrote above: if charts are Euclidean as well, the
  paper should elucidate why Euclidean charts do *not* suffer from being
  too simplistic.
- The discussion of homeomorphisms in the introduction is slightly
  misleading; none of the functions learned later on is a homeomorphism
  because of the latent space dimensions.
- Homeomorphic mappings of manifolds into a Euclidean space are not
  necessarily desirable---this is why the definition of a manifold uses
  the concept of neighbourhoods. I think this should be rephrased in
  a positive manner, as in: manifolds are complex, so we cannot expect
  a *single* map to suffice...
- The leading example of a torus embedding needs more details. Why is
  the structure destroyed?
- The introduction of 'topological features' on p. 2 is slightly abrupt.
  It would be sufficient to explain by means of the figure that the
  mapping obviously does not respect all properties.
- p.2: paths become invariant to _what_ exactly?
- p.2: what is the 'topological class'?
- p.2: would LLE not be a good precursor to the method proposed in this
  paper?
- p.2: is this paper to be seen as an implementation of Chen et al.  (2019)?
  This should be made more clear.
- p.3: the concept of intrinsic dimension slightly varies in literature.
  I would propose mentioning the homeomorphism of every chart to some
  $d$-dimensional space, and state that if this exists, one calls the
  manifold $d$-dimensional.
- p.3: the circle example could be explained in more detail for readers
  unfamiliar with the concepts.
- p.4: the chart prediction module requires a brief explanation at the
  point when it is first introduced (1 sentence is sufficient). The
  method plus architecture is presented but the details come very late;
  I would prefer some intuition here
- p.4: $N$ needs to be defined earlier
- p.4: how is the dimension of latent spaces chosen? Please also refer
  to my comments on the experiments above.
- p.5: Section 4.1 again mixes 'topological' and 'geometrical' concepts;
  suddenly, the concept of curvature crops up---this needs to be
  explained better!
- p.5: Distances can always be measured in connected subsets of
  real-valued spaces; whether the set is open or closed does not change
  the fact that a centre exists. Am I misunderstanding this?
- p.5: I like the 'partition of unity' approach, but to me, this reads
  like a convex combination of predictions. Am I misreading this? If
  not, I would suggest to rephrase this.
- p.5/6: the goals of the new method need to be stated more clearly; the
  paper needs to explain better to what extent *reconstruction error* is
  affected by charts (it does not seem to be, as I outlined above)---and
  this again raises the question of which quality measure the new method
  *can* preserve.
- p.6: the definition of the Lipschitz constant could be more precise;
  please specify the requirements $f$ has to satisfy
- Eq. 4 needs more details for me: it seems as if the weights appear
  twice as a kind of 'decay term' (in the second part, I see the sum
  but the product appears in both terms). This should be stated more
  clearly.
- p.6: the pre-training needs more details; how crucial is this step?
- p.6: what does the 'orientation' imply? It is not defined except in the
  appendix.
- p.6: the jump from the illustrative examples to the non-synthetic ones
  is large; the uniform sampling of the latent space does not scale to
  higher dimensions, for example. The paper should comment on this if
  possible.
- In general, I would recommend giving the employed models more
  'speaking' names. I found it hard to keep track of all of them and had
  to refer to the appendix constantly.
- For Figure 4, please show the full space, together will all charts
- p.7: please give some ideas (see above) for how to use the covering of
  the points in practice; I like that the object can be reconstructed
  with a proper set of charts, but the paper could make the necessity
  of the technique much more obvious by choosing stronger examples.
- p.7: the object arguably *also* has a complex geometry, not only
  complex topology. This should be mentioned.
- p.8: the discussion of MNIST is slightly incorrect; as outlined above,
  many digits appear to be generated by multiple charts, while some,
  such as `1` do not appear on more than one chart.
- The metrics in Section 5.3 should be introduced earlier, maybe at the
  expense of some exposition in the introduction or the simpler
  examples; it is not good style to have to refer to the appendix to
  understand a core experiment of a paper.
- p.8: I do not understand the term 'wholly pyramid'.
- p.11: the decoder should map to $x_i$, if I am not mistaken
- p.11: I would suggest a more consistent terminology to describe the
  models. The prediction function is replicated multiple times, for
  example, so why not introduce a shorthand notation for this?
- p.12: '\cup' and '\cap' need to be switched: the *intersection* of
  domains needs to be empty, not their *union*
- p.13: to what extent are the 'faithfulness' and 'coverage' established
  metrics? It seems that they are developed for this paper, so I would
  explain them in the main text and also make clear why they are
  desirable metrics---else, the metrics could be criticised as being
  fine-tuned for the proposed method.
  For example, if *coverage* can measure the phenomenon of *mode
  collapse*, this needs to be demonstrated.
## Minor comments
Some typos:
- low dimensional --> low-dimensional
- eigen-functions --> eigenfunctions
- considers manifold point --> considers a manifold point
- paring subnetwork --> pairing subnetwork
- paramterized --> parametrized
- preformed --> performed [occurs multiple times]
- chats --> charts
- Lipshitz --> Lipschitz (in Figure 4)
- evalutation --> evaluation
- seciton --> section