  Convolutional neural networks have established themselves over the past years
as the state of the art method for image classification, and for many datasets,
they even surpass humans in categorizing images. Unfortunately, the same
architectures perform much worse when they have to compare parts of an image to
each other to correctly classify this image.
  Until now, no well-formed theoretical argument has been presented to explain
this deficiency. In this paper, we will argue that convolutional layers are of
little use for such problems, since comparison tasks are global by nature, but
convolutional layers are local by design. We will use this insight to
reformulate a comparison task into a sorting task and use findings on sorting
networks to propose a lower bound for the number of parameters a neural network
needs to solve comparison tasks in a generalizable way. We will use this lower
bound to argue that attention, as well as iterative/recurrent processing, is
needed to prevent a combinatorial explosion.
