  Contrasting the previous evidence that neurons in the later layers of a
Convolutional Neural Network (CNN) respond to complex object shapes, recent
studies have shown that CNNs actually exhibit a `texture bias': given an image
with both texture and shape cues (e.g., a stylized image), a CNN is biased
towards predicting the category corresponding to the texture. However, these
previous studies conduct experiments on the final classification output of the
network, and fail to robustly evaluate the bias contained (i) in the latent
representations, and (ii) on a per-pixel level. In this paper, we design a
series of experiments that overcome these issues. We do this with the goal of
better understanding what type of shape information contained in the network is
discriminative, where shape information is encoded, as well as when the network
learns about object shape during training. We show that a network learns the
majority of overall shape information at the first few epochs of training and
that this information is largely encoded in the last few layers of a CNN.
Finally, we show that the encoding of shape does not imply the encoding of
localized per-pixel semantic information. The experimental results and findings
provide a more accurate understanding of the behaviour of current CNNs, thus
helping to inform future design choices.
