Paper:
Principles of Riemannian Geometry in Neural Networks
This study deals with neural networks in the sense of geometric transformations acting on the coordinate representation of the underlying data manifold which the data is sampled from. It forms part of an attempt to construct a formalized general theory of neural networks in the setting of Riemannian geometry. From this perspective, the following theoretical results are developed and proven for feedforward networks. First it is shown that residual neural networks arenite di erence approximations to dynamical systems of rst order di erential equations, as opposed to ordinary networks that are static. This implies that the network is learning systems of di erential equations governing the coordinate transformations that represent the data. Second it is shown that a closed form solution of the metric tensor on the underlying data manifold can be found by backpropagating the coordinate representations learned by the neural network itself. This is formulated in a formal abstract sense as a sequence of Lie group actions on the metric bre space in the principal and associated bundles on the data manifold. Toy experiments were run to con rm parts of the proposed theory, as well as to provide intuitions as to how neural networks operate on data.

Review:
The paper develops a mathematical framework for working with neural network representations in the context of finite differences and differential geometry.
In this framework, data points going though layers have fixed coordinates but space is smoothly curved with each layer.
The paper presents a very interesting framework for working with neural network representations, especially in the case of residual networks.
Unfortunately, taking the limit as the number of layers goes to infinity does not make practical application very easy and somewhat limits the impact of this paper.
The paper is not always completely clear. Since the goal of the paper is to present a minority perspective, clarity should be paramount.
experiments are a bit disapoiting. Their goal is not always very explicit. What the experiments do exactly is not very clear (despite their obvious simplicity). The experiments which show how a simple neural networks disentangle data points feel well known and their relation to the current paper feels a bit tenuous.
Line 62: The . in tensors (…) ensure consistency in the order of the superscrips and subscripts.
=> Not very clear. I assume the indices with a dot at the end are to be on the left of indices with a dot at the beginning (so that the dots would sort of align).
line 75, Eq 1:
Taking the limit as L -> /infty seems to pose a problem for practical applications. Wouldn’t an equation of the form bellow make sense (interpolation between layers):
x^a(x + \delta l) = x^a(l) + f^a(x^b(l); l) \delta l
instead of
x^a(x + 1 ) = x^a(l) + f^a(x^b(l); l) \delta l
6.1 & Figure1: The Harmonic oscillator
Section 6.1 and the corresponding figures are unclear.
=> The given differential equation does not indicate what the parameters are and what the coordinates are. This is a bit confusing since \xi is a common notation for coordinates in differential geometry. We are told that the problem is two dimensional so I assume that x is the two dimensional variable.
=> Figure 1 is confusing. What does the finite differencing plot show ? what do the x and y axis represent ? What about metric transformations ? What about the scalar metric values ?
The fact that the particle stands still is understandable if in the context of a network but confusing w.r.t. the given differential equation for x.
=> a, b = 1,2 is not clear in this context. Does this mean that a = 1 and b = 2 ? can the state space representation be written in the given way only if a, b = 1,2 ?
Figure 2: Every plot has the legend “layer 11”. Typo ?
section 6.3 is a bit superfluous.