  A recent series of theoretical works showed that the dynamics of neural
networks with a certain initialisation are well-captured by kernel methods.
Concurrent empirical work demonstrated that kernel methods can come close to
the performance of neural networks on some image classification tasks. These
results raise the question of whether neural networks only learn successfully
if kernels also learn successfully, despite neural networks being more
expressive. Here, we show theoretically that two-layer neural networks (2LNN)
with only a few hidden neurons can beat the performance of kernel learning on a
simple Gaussian mixture classification task. We study the high-dimensional
limit where the number of samples is linearly proportional to the input
dimension, and show that while small 2LNN achieve near-optimal performance on
this task, lazy training approaches such as random features and kernel methods
do not. Our analysis is based on the derivation of a closed set of equations
that track the learning dynamics of the 2LNN and thus allow to extract the
asymptotic performance of the network as a function of signal-to-noise ratio
and other hyperparameters. We finally illustrate how over-parametrising the
neural network leads to faster convergence, but does not improve its final
performance.
