Paper:
Developing deep neural networks (DNNs) for manifold-valued data sets has gained much interest of late in the deep learning research community. Examples of manifold-valued data include data from omnidirectional cameras on automobiles, drones etc., diffusion magnetic resonance imaging, elastography and others. In this paper, we present a novel theoretical framework for DNNs to cope with manifold-valued data inputs. In doing this generalization, we draw parallels to the widely popular convolutional neural networks (CNNs). We call our network the ManifoldNet. As in vector spaces where convolutions are equivalent to computing the weighted mean of functions, an analogous definition for manifold-valued data can be constructed involving the computation of the weighted Fréchet Mean (wFM). To this end, we present a provably convergent recursive computation of the wFM of the given data, where the weights makeup the convolution mask, to be learned. Further, we prove that the proposed wFM layer achieves a contraction mapping and hence the ManifoldNet does not need the additional non-linear ReLU unit used in standard CNNs. Operations such as pooling in traditional CNN are no longer necessary in this setting since wFM is already a pooling type operation. Analogous to the equivariance of convolution in Euclidean space to translations, we prove that the wFM is equivariant to the action of the group of isometries admitted by the Riemannian manifold on which the data reside. This equivariance property facilitates weight sharing within the network. We present experiments, using the ManifoldNet framework, to achieve video classification and image reconstruction using an auto-encoder+decoder setting. Experimental results demonstrate the efficacy of ManifoldNet in the context of classification and reconstruction accuracy.

Review:
Brief summary:
The paper considers a generalization of convolutional neural networks (CNNs) to data residing on Riemannian manifolds. The idea is to replace convolutions with weighted averages, which are implemented intrinsically on the manifold. It is shown that this operator is equivariant to isometric group actions. A related approach for dimensionality reduction is also proposed, but I think this only applies to Euclidean data, so I am a bit confused about that part. Empirical performance is reported on toy data with weak baselines.
Good points about the paper:
+ It is a relevant point that the intrinsic average in manifolds is a way to generalized convolutions to the Riemannian domain.
+ The paper is generally fairly easy to read.
Concerns with the paper:
- A key point of CNNs is that you learn filters that are small, i.e. only have non-zero weight assigned to a few neighboring pixels. As far as I can tell, the here proposed "filters" would be one weight per data point. 
- I am concerned about the stacking of multiple "convolution" (wFM) layers: since each layer computes the average of a set of points, then doesn't stacking multiple "convolution" (wFM) layers on top of each other correspond to computing the average of a set of averages? And can this not be computed by a single average? In other words, is a cascade of "convolution" (wFM) layers equivalent to a single layer? Seems like a complicated way of doing shallow learning unless I misunderstand.
- In section 2.2 it is argued that the weighted average (wFM) is a contraction mapping. While I think the proof is correct, I am concerned about the prerequisite assumption that that distance between a set of points X and Y is the *maximal* distance between points in the two sets. Usually, one would define this distance as the *minimal* distance (akin to the Hausdorff distance). It seems that under this more reasonable choice of distance, the proof no longer holds
- Large parts of section 2.1 is devoted to an efficient algorithm for computing weighted averages on manifolds.  Here the text is written such as to indicate that this is a novel contribution of the present paper, even if these results are readily available in the literature. I strongly encourage a re-writing to emphasize that this is a repetition of previous knowledge.
- Proposition 5 does not appear to come with a proof.
- Section 3.2 introduce a new dimensionality reducing layer based on the Grassmann average construction for subspace learning. I was quite confused when reading this. From what I can tell this layer is only applicable when the input data is Euclidean, and as such appears to be unrelated to the rest of the paper.  
- At times the paper is rather sloppy written, e.g. fonts are way too small in figures, the dot(.) notation is not defined (e.g. in def. 7), and the citation style is very difficult to read (please use \citet and \citep instead of \cite).
Other comments:
*) The assumption (page 3) that the data reside within a geodesic ball of a certain radius seem quite strong. It would be good to comment on this in the paper.
*) It is not clear to me that the weighted average is a particularly good way to generalize convolution. Yes, I agree, it is *one
 way to generalize, but why should I pick this one in particular?
*) In the experiment in sec. 3.1 the manifold comes from a particular way to extract features from data. In a deep learning context, we would usually learn such features instead of manually designing them. This seem like a general issue, that most often manifold come into play through a manual feature-design, which seem to be at odds with the end-to-end learning mindset. It might be good to have examples where this is not the case.