Paper:
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network
Recent work by Cohen et al. [1] has achieved state-of-the-art results for learning spherical images in a rotation invariant way by using ideas from group representation theory and noncommutative harmonic analysis. In this paper we propose a generalization of this work that generally exhibits improved performace, but from an implementation point of view is actually simpler. An unusual feature of the proposed architecture is that it uses the Clebsch-Gordan transform as its only source of nonlinearity, thus avoiding repeated forward and backward Fourier transforms. The underlying ideas of the paper generalize to constructing neural networks that are invariant to the action of other compact groups.

Review:
This paper proposes a generalized version of SO(3)-equivariant architectures including Spherical CNN. By utilizing the algebraic properties of Fourier transform and the tools in non-commutative harmonic analysis, the authors are able to construct (and prove) a most generalized version of SO(3)-equivariant architecture. Specifically, it only requires that, when an input image is rotated, each fragment (i.e., the output, Fourier coefficient vectors) of each layer will be multiplied by a Wigner-D matrix. To include non-linearities without performing inverse Fourier transform, the authors propose to use Clebsch-Gordon transformation. The experiments show that the proposed CG-Net can outperform Spherical CNN in several tasks.  Strength: - The paper proposes a generalized version of Spherical CNN. Plus, the paper proves that this architecture is the most generalized version of SO(3)-equivariant architecture. - A principled treatment on SO(3)-equivarient operations by looking into the algebraic properties of them. As a result, almost all operations in CG-Net are performed in Fourier space. - CG-Net works well even without traditional non-linearity activation functions like ReLU.  Weakness: - The improvement compared to Spherical CNN seems incremental. - The paper claims that it is costly to compute forward and backward Fourier transform in Spherical CNN. However, it does not compare the runtime of Spherical CNN and CG-Net.  - The experiment seems unfair as the number of parameters in CG-Net is almost 5 times the number in Spherical CNN. - The paper does not include an ablation study on the hyperparameter tau, which is the maximal number of fragments for each spherical harmonic function index l in each layer (to avoid exponential growth in number of fragments) - It would be better if the authors include a visualization to provide a graphical explanation of the non-linearity in CG-transform.  ---------------------- Authors' rebuttal only paritally addresses my concerns. I will keep the original rating