Paper:
In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios. Our observations indicate that: (1) There exists a limit to signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size -more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio (2) in general, higher capacity models exhibit better generalization; (3) when knowing the approximate object sizes, adapting receptive field is beneficial; and (4) for very small signal-to-noise ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noise values, all tested global pooling operations exhibit similar performance.

Review:
The submission proposes an analysis of the impact of object size in images when performing classification tasks using neural networks of the BagNet family. The analysis is performed on two datasets, a large resolution cluttered MNIST and a histopathology dataset named nCAMELYON.
The paper attack interesting questions and links the size of the object in the image (O2I) to the training dataset size required. Also, showing that max-pooling is the only pooling operation that converges for very low O2I (but is the slowest to converge at higher O2I) is interesting and encourages discussion about the training (optimization) process.
In my opinion, the main issue about the submission is the limited depth in the contributions, analyzing a single family of network architectures (BagNets) over two datasets, one of which is relatively small. The family of R-CNN and its derivatives were especially designed to counter the impact of object size, it would have been interesting to include them in the analysis. Furthermore, limited insights can be carried out for tasks related to classification such as localization and segmentation. Models such as Single Shot Detectors split the image into grids and variable anchor sizes to perform their inference, are they affected to a lesser extent by object size? The limited insights provided to potential future readers prevents me from recommending the submission for acceptance.
On p. 7 (and fig. 7 (b) ), it is said that the lower performance of the larger receptive field suggests that class-relevant information is contained in the texture. I am not sure about this remark, I would have expected the network to learn to focus on the right regions, provided the receptive field is big enough to see the whole object of interest. Could the decrease in performance be attributed to the increased amount of learnable parameters that ended up too large for the “relatively small nCAMELYON dataset used for training”? (sec. 3.1, Global pooling operations)
Fig. 16 seems to suggest that multiple numbers can overlap significantly in nMNIST, yielding potentially confusing images even for humans. I am not sure if this is a desirable characteristic for such dataset.
Minor details
- Please use \cdot instead of the asterisk operator to denote multiplication (sec. 3, footnote 3, fig. 5 (b-c) and sec. 3.1;
- Fig. 4 uses two different (and non-linear) values for the x-axes, giving the impression that both curves can be compared, that might be confusing to the reader;
- Sec 1 and 5 “pixel-level level annotations”: duplicate “level”;
- p. 6 “1 and , 2”: extraneous comma.