On the numerical stability of algorithmic differentiation
In contrast to integration, the differentiation of a function is an illconditioned process, if only an oracle is available for its pointwise evaluation. That is, unrelated small variations in the value of the composite function are allowed at nearly identical arguments. In contrast, we show here that, if the function is defined by an evaluation procedure as a composition of arithmetic operations and elementary functions, then automatic, or algorithmic differentiation is backward stable in the sense of Wilkinson. More specifically, the derivative values obtained are exact for a perturbation of the elementary components at the level of the machine precision. We also provide a forward error analysis for both the forward and reverse mode. The theoretical analysis is confirmed by numerical experiments.
automatic differentiation forward-backward stability rounding error estimates ieee arithmetic ultimate rounding