  Classification is one of the major tasks that deep learning is successfully
tackling. Categorization is also a fundamental cognitive ability. A well-known
perceptual consequence of categorization in humans and other animals, called
categorical perception, is characterized by a within-category compression and a
between-category separation: two items, close in input space, are perceived
closer if they belong to the same category than if they belong to different
categories. Elaborating on experimental and theoretical results in cognitive
science, here we study categorical effects in artificial neural networks. Our
formal and numerical analysis provides insights into the geometry of the neural
representation in deep layers, with expansion of space near category boundaries
and contraction far from category boundaries. We investigate categorical
representation by using two complementary approaches: one mimics experiments in
psychophysics and cognitive neuroscience by means of morphed continua between
stimuli of different categories, while the other introduces a categoricality
index that quantifies the separability of the classes at the population level
(a given layer in the neural network). We show on both shallow and deep neural
networks that category learning automatically induces categorical perception.
We further show that the deeper a layer, the stronger the categorical effects.
An important outcome of our analysis is to provide a coherent and unifying view
of the efficacy of different heuristic practices of the dropout regularization
technique. Our views, which find echoes in the neuroscience literature, insist
on the differential role of noise as a function of the level of representation
and in the course of learning: noise injected in the hidden layers gets
structured according to the organization of the categories, more variability
being allowed within a category than across classes.
