Accelerating Transformer Inference for Translation via Parallel Decoding (2023)

Abstract: Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT). The community proposed specific network architectures and learning-based methods to solve this issue, which are expensive and require changes to the MT model, trading inference speed at the cost of the translation quality. In this paper, we propose to address the problem from the point of view of decoding algorithms, as a less explored but rather compelling direction. We propose to reframe the standard greedy autoregressive decoding of MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point iteration methods for fast inference. This formulation allows to speed up existing models without training or modifications while retaining translation quality. We present three parallel decoding algorithms and test them on different languages and models showing how the parallelization introduces a speedup up to 38% w.r.t. the standard autoregressive decoding and nearly 2x when scaling the method on parallel resources. Finally, we introduce a decoding dependency graph visualizer (DDGviz) that let us see how the model has learned the conditional dependence between tokens and inspect the decoding procedure.

## introduction in recent years there have been dramatic improvements in machine translation (mt) @xcite thanks to the transition to neural models and the advent of the transformer architecture @xcite . these models can produce high-quality translations while being extremely parallelizable during training. however, transformers are used sequentially at inference time, generating one token per time (i.e., sending each token as input for the next autoregressive iteration). this process of autoregressive inference hampers the efficiency of neural machine translation systems in terms of latency, limiting applications and portability. considering that these systems are extensively used in production multiple times to produce new translations (e.g., google translate foot_0 , deepl translator foot_1 ), even a minor speedup would be beneficial in the long run, especially if the translation is done on embedded devices. to address this issue, the community proposed ad-hoc trained models specific for parallel machine translation under the umbrella term of non-autoregressive machine translation models (nat) @xcite . these models produce the translation in parallel but require (i) a complete reengineering of the mt system, (ii) extensive training resources and (iii) complex design choices like distillation from larger autoregressive models. these requirements are quite demanding and not easily satisfiable. for example, production systems are heavily optimized for hardware and software and even introducing a minimal modification requires non-trivial human effort @xcite . furthermore, training a new model from scratch is not always possible due to non-released training data or low-resource languages having few or lacking parallel corpora. in this paper, we propose to address the problem of parallel machine translation with an orthogonal approach consisting in novel decoding algorithms that work in parallel and can be used on top of existing autoregressive models for mt. we overcome previous limitations with a flexible and generic method that does not require any modification to the model or costly retraining. specifically, inspired by previous successes in speeding up feedforward computation for image generation @xcite , we reframe the greedy autoregressive decoding for mt as a system of nonlinear equations solvable in parallel. this simple formulation speeds up the decoding procedure by using fixed-point iteration methods like jacobi and gauss-seidel while having mathematical guarantees on figure 1 : on the left, the classical autoregressive decoding for mt. the target sentence is produced token-by-token sequentially, sending the partial result as input for the next autoregressive iteration up to the length m of the target. on the right parallel decoding proposed in this paper. this method changes only the decoding algorithm (orange block) and is usable on top of any autoregressive model without modifications. parallel decoding algorithms resolve the whole sentence or a block of b tokens in parallel: initial tokens (pad tokens) are gradually refined with k steps until a stopping condition is reached. crucially, k ⩽ m with quality guarantees and overall decoding speedups. the quality of the translation. a high-level description of the method is available in (fig. 1 ). our contributions can be summarized as the following: all the code is publicly released foot_2 . in the context of grammatical error correction by thinking of them as a system of non-linear equations. they parallelized the backpropagation of rnns, feedforward layers and autoregressive generative models on images. we extend the approach defined on dense pixel prediction to the discrete conditional token generation in mt. while this work was under submission and anonymity period, @xcite , @xcite and @xcite concurrently proposed decoding approaches that speed up inference of a large transformer model by using another smaller model to draft tokens. compared to these approaches our method requires just an existing autoregressive model (no matter the size) and mathematically guarantees the output quality. in the next section we describe the method. ## method in this section, we introduce notations, develop the theory behind parallel decoding, present three algorithms (fig. 2 ), and discuss the initialization and stopping conditions for the proposed approaches. ## conclusions in this paper, we showed that is possible to speed up existing machine translation models by simply changing the decoding algorithm with a parallel formulation. we introduced three parallel decoding methods which achieve consistent speedups without requiring any training, modifications, or quality loss. our solution is orthogonal to previous approaches proposed in literature which often entail demanding requirements in terms of data, computational resources, and engineering effort. although our method is not without shortcomings, it is a first valuable step toward integrating parallel decoding algorithms into any model. this is particularly relevant in limited-resource scenarios where nats are not a viable option and to speed up any transformer model, especially fine-grained or character-level models @xcite . we believe that further advancements in this area, including the exploration of optimal initialization procedures and stopping conditions, as well as the use of alternative parallel solvers for non-linear equations, will close the gap with learning-based techniques and continue to improve the efficiency and effectiveness of parallel decoding algorithms.