ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation (2022)

Abstract: Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). This paper explores a deeper relationship between Transformer and numerical ODE methods. We first show that a residual block of layers in Transformer can be described as a higher-order solution to ODE. Inspired by this, we design a new architecture, ODE Transformer , which is analogous to the Runge-Kutta method that is well motivated in ODE. As a natural extension to Transformer, ODE Transformer is easy to implement and efficient to use. Experimental results on the large-scale machine translation, abstractive summarization, and grammar error correction tasks demonstrate the high genericity of ODE Transformer. It can gain large improvements in model performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the WMT’14 English-German and English-French benchmarks) at a slight cost in inference efficiency.

## introduction residual networks have been used with a great success as a standard method of easing information flow in multi-layer neural models @xcite . given an input y t , models of this kind define the output of a layer t to be: this paper continues the line of research on the ode-inspired method. the basic idea is to use a high-order method for more accurate numerical solutions to the ode. this leads to a larger ode block that generates a sequence of intermediate approximations to the solution. we find that the larger ode block is sufficient to take the role of several ode blocks with first-order solutions. the benefit is obvious: the use of fewer ode blocks lowers the risk of introducing errors in block switching, and the high-order method reduces the approximation error in each ode block. see figure 1 for a comparison of different models. our method is parameter-efficient because θ(t) is re-used within the same ode block. as another "bonus", the model can be improved by learning coefficients of different intermediate approximations in a block. we evaluate our method in strong transformer systems, covering both the wide (and big) model and the deep model. for machine translation tasks, ode transformer achieves 30.77 and 44.11 bleu scores on the wmt'14 en-de and en-fr test sets, setting a new state-of-the-art on the wmt'14 en-fr task. it also significantly outperforms baselines on abstractive summarization and grammar error correction tasks. ## transformer and odes we start with a description of transformer, followed by its relationship with odes. we choose transformer for our discussion and experiments because it is one of the state-of-the-art models in recent sentence generation tasks. ## the ode transformer in numerical methods of odes, we want to ensure the precise solutions to the odes in a minimum number of computation steps. but the euler method is not "precise" because it is a first-order method, and naturally with local truncation errors. the global error might be larger if we run it for a number of times. 2 this is obviously the case for transformer, especially when the multi-layer neural network arises a higher risk of instability in solving the odes @xcite . ## experiments we evaluated the ode transformer on three sequence generation tasks: machine translation, abstractive summarization and grammar error correction. the datasets we used are elaborated in the following section, and more details of experimental setups could be found in appendix a and b. ## analysis here we investigate some interesting issues. for simplicity, we call rk2-block with coefficients initialized by 1 as rk2-block-v1, and learnable coefficients (eq. ( 25 ) ) as rk2-block-v2. quantization of the truncation error in fact, we cannot obtain the "true" solution of each block output in nmt, because we mainly experimented on the encoder side. instead, we tested our system on the language modeling task, where the perplexity between the single-layer model output and the ground truth could be regarded as the truncation error with no error propagations. ## related work deep transformer models recently, deep transformer has witnessed tremendous success in machine translation, especially on wmt news tasks @xcite @xcite . a straightforward way is to shorten the path from upper-level layers to lowerlevel layers thus to alleviate the gradient vanishing or exploding problems @xcite @xcite . for deeper models, the training cost is nonnegligible. to speed up the training, an alternative way is to train a shallow model first and progressively increase the model depth @xcite . apart from the model architecture improvements, another way of easing the optimization is to utilize carefully designed parameter initialization strategies @xcite @xcite . with the model capacity going larger, one can use layerdrop @xcite or skipping sublayers @xcite to prevent deep models from the overfitting problem. note that ode transformer is orthogonal to the aforementioned methods, and we will test it on these methods in future work. ## conclusions this paper explores the relationship between transformer and odes. we propose ode transformer to help the model benefit from high-order ode solutions. experimental results on the three representative sentence generations tasks (i.e., machine