  We study the relative power of learning with gradient descent on
differentiable models, such as neural networks, versus using the corresponding
tangent kernels. We show that under certain conditions, gradient descent
achieves small error only if a related tangent kernel method achieves a
non-trivial advantage over random guessing (a.k.a. weak learning), though this
advantage might be very small even when gradient descent can achieve
arbitrarily high accuracy. Complementing this, we show that without these
conditions, gradient descent can in fact learn with small error even when no
kernel method, in particular using the tangent kernel, can achieve a
non-trivial advantage over random guessing.
