  It has been found that residual networks are an Euler discretization of
solutions to Ordinary Differential Equations (ODEs). In this paper, we explore
a deeper relationship between Transformer and numerical methods of ODEs. We
show that a residual block of layers in Transformer can be described as a
higher-order solution to ODEs. This leads us to design a new architecture (call
it ODE Transformer) analogous to the Runge-Kutta method that is well motivated
in ODEs. As a natural extension to Transformer, ODE Transformer is easy to
implement and parameter efficient. Our experiments on three WMT tasks
demonstrate the genericity of this model, and large improvements in performance
over several strong baselines. It achieves 30.76 and 44.11 BLEU scores on the
WMT'14 En-De and En-Fr test data. This sets a new state-of-the-art on the
WMT'14 En-Fr task.
