  Neural Networks require large amounts of memory and compute to process high
resolution images, even when only a small part of the image is actually
informative for the task at hand. We propose a method based on a differentiable
Top-K operator to select the most relevant parts of the input to efficiently
process high resolution images. Our method may be interfaced with any
downstream neural network, is able to aggregate information from different
patches in a flexible way, and allows the whole model to be trained end-to-end
using backpropagation. We show results for traffic sign recognition,
inter-patch relationship reasoning, and fine-grained recognition without using
object/part bounding box annotations during training.
