  Discrete supervised learning problems such as classification are often
tackled by introducing a continuous surrogate problem akin to regression.
Bounding the original error, between estimate and solution, by the surrogate
error endows discrete problems with convergence rates already shown for
continuous instances. Yet, current approaches do not leverage the fact that
discrete problems are essentially predicting a discrete output when continuous
problems are predicting a continuous value. In this paper, we tackle this issue
for general structured prediction problems, opening the way to "super fast"
rates, that is, convergence rates for the excess risk faster than $n^{-1}$,
where $n$ is the number of observations, with even exponential rates with the
strongest assumptions. We first illustrate it for predictors based on nearest
neighbors, generalizing rates known for binary classification to any discrete
problem within the framework of structured prediction. We then consider kernel
ridge regression where we improve known rates in $n^{-1/4}$ to arbitrarily fast
rates, depending on a parameter characterizing the hardness of the problem,
thus allowing, under smoothness assumptions, to bypass the curse of
dimensionality.
