Paper:
The effectiveness of Convolutional Neural Networks (CNNs) has been substantially attributed to their built-in property of translation equivariance. However, CNNs do not have embedded mechanisms to handle other types of transformations. In this work, we pay attention to scale changes, which regularly appear in various tasks due to the changing distances between the objects and the camera. First, we introduce the general theory for building scale-equivariant convolutional networks with steerable filters. We develop scale-convolution and generalize other common blocks to be scale-equivariant. We demonstrate the computational efficiency and numerical stability of the proposed method. We compare the proposed models to the previously developed methods for scale equivariance and local scale invariance. We demonstrate state-of-the-art results on the MNIST-scale dataset and on the STL-10 dataset in the supervised learning setting.

Review:
This paper proposes a framework (SESN) for learning deep networks that possess scale equivariance in addition to translation invariance. The formulation is based on group convolution on the scale-translation group. Filters are represented as the coefficients of a set of continuous basis functions, which are sampled (once) at a discrete set of scales. The theoretical formulatioin is clear and interesting. The approach is evaluated in terms of image classification accuracy. The set of baselines is quite exhaustive, including recent papers and papers that are not widely-known.
The most significant improvement for the STL-10 dataset was obtained by the SESN-B variant. This is interesting, because it applies the same operation independently at multiple scales and periodically performs global pooling over scale.
The effectiveness of the approach was demonstrated in the low-data regime, where the inductive bias of scale equivariance is more likely to help.
Overall I found the paper to be thought-provoking and well-executed. There are a number of questions that I would still like to see investigated, but nevertheless I feel that this paper already represents a worthwhile contribution.
Most important issues and questions:
(1.1) The SESN-B architecture resembles quite closely the SI-ConvNet architecture of Kanazawa et al. (except that that paper resized the images instead of the filters). While your approach may be more computationally efficient, it's not clear what leads to the improvement in accuracy here? Can you explain the difference?
(1.2) I would have preferred to see the approach demonstrated on a task which possesses scale equivariance, such as semantic segmentation.
(1.3) To argue in favour of the continuous basis, it would have been more convincing to compare against directly learning the filters at the highest resolution and obtaining the other filters by downsampling. This would not represent a runtime cost during inference.
(1.4a) It seems that SESN-C should contain SESN-A as a special case. However, SESN-C is worse than SESN-A. Do you have any idea whether this is due to optimization difficulty or over-fitting? Could you compare the training objectives?
(1.4b) It is stated that the scale equivariance of SESN-C is worse than SESN-A and -B. However, it should still be scale equivariant, except for boundary effects in scale? What is the parameter N_S in this experiment compared to the number of scales S? And the same question for the plot on the right in Figure 2.
(1.4c) How does SESN-C have the same number of parameters as SESN-A and SESN-B? I thought that more parameters would be required to compute interscale interaction. Was the number of channels reduced?
Issues with clarity:
(2.1) The explanation of equation 10 is not clear. In particular, the diagonal structure in Figure 1 is not stated anywhere in the text, it is simply explained as an expansion from [C_out, C_in, S, V, V] to [C_out S, C_in S, V, V].
(2.2) It's not immediately apparent how multiple applications of convHH are used to provide interscale interaction. I assume it is achieved by shifting f or psi in the scale dimension for each application of convHH, or equivalently by modifying the base-scale in the basis?
(2.3) The explanation of "scalar" and "vector" variants in the experimental section was not perfectly clear. It is stated that "all the layers have now scalar input instead of vector input." However, I understood that the max-reduction was only over the scale dimension, not the channel dimension, so that the inputs are still vectors? This is confusing as a reader.
(2.4) The expansion of the filters to a diagonal structure is described in the "implementation" section. However, it seems that this would entail wasteful multiplications by zero. Nevertheless, SESN is shown to be highly efficient in the appendix. Do you avoid these pointless operations in the actual implementation?
(2.5) It was not immediately obvious how $\psi_{\sigma}(s, t)$ was related to $\psi_{\sigma}(x)$.
Other details:
(3.1) In Figure 2, how many layers does the network have which was used to construct the middle plot?
(3.2) It would have been useful to include a study of the effect of the range and resolution of the scale space.