  We define infinitesimal gradient boosting as a limit of the popular
tree-based gradient boosting algorithm from machine learning. The limit is
considered in the vanishing-learning-rate asymptotic, that is when the learning
rate tends to zero and the number of gradient trees is rescaled accordingly.
For this purpose, we introduce a new class of randomized regression trees
bridging totally randomized trees and Extra Trees and using a softmax
distribution for binary splitting. Our main result is the convergence of the
associated stochastic algorithm and the characterization of the limiting
procedure as the unique solution of a nonlinear ordinary differential equation
in a infinite dimensional function space. Infinitesimal gradient boosting
defines a smooth path in the space of continuous functions along which the
training error decreases, the residuals remain centered and the total variation
is well controlled.
