Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021)

Abstract: In this paper, we propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC). SAD optimizes the online inference efficiency for GEC by two innovations: 1) it aggressively decodes as many tokens as possible in parallel instead of always decoding only one token in each step to improve computational parallelism; 2) it uses a shallow decoder instead of the conventional Transformer architecture with balanced encoder-decoder depth to reduce the computational cost during inference. Experiments in both English and Chinese GEC benchmarks show that aggressive decoding could yield identical predictions to greedy decoding but with significant speedup for online inference. Its combination with the shallow decoder could offer an even higher online inference speedup over the powerful Transformer baseline without quality loss. Not only does our approach allow a single model to achieve the state-of-the-art results in English GEC benchmarks: 66.4 F0.5 in the CoNLL-14 and 72.9 F0.5 in the BEA-19 test set with an almost 10x online inference speedup over the Transformer-big model, but also it is easily adapted to other languages. Our code is available at https://github.com/AutoTemp/Shallow-Aggressive-Decoding .

## introduction the transformer @xcite has become the most popular model for grammatical error correction (gec). in practice, however, the sequenceto-sequence (seq2seq) approach has been blamed recently @xcite @xcite for its poor inference efficiency in modern writing assistance applications (e.g., microsoft office word foot_0 , google docs foot_1 and grammarly foot_2 ) where a gec model usually performs online inference, instead of batch inference, for proactively and incrementally checking a user's latest completed sentence to offer instantaneous feedback. to better exploit the transformer for instantaneous gec in practice, we propose a novel approach -shallow aggressive decoding (sad) to improve the model's online inference efficiency. the core innovation of sad is aggressive decoding: instead of sequentially decoding only one token at each step, aggressive decoding tries to decode as many tokens as possible in parallel with the assumption that the output sequence should be almost the same with the input. as shown in figure 1 , if the output prediction at each step perfectly matches its counterpart in the input sentence, the inference will finish, meaning that the model will keep the input untouched without editing; if the output token at a step does not match its corresponding token in the input, we will discard all the predictions after the bifurcation position and re-decode them in the original autoregressive decoding manner until we find a new opportunity for aggressive decoding. in this way, we can decode the most text in parallel in the same prediction quality as autoregressive greedy decoding, but largely improve the inference efficiency. in addition to aggressive decoding, sad proposes to use a shallow decoder, instead of the conventional transformer with balanced encoderdecoder depth, to reduce the computational cost for further accelerating inference. the experimental 5938 [bos] i 'm wri,ng to inform some some advice on traveling and working . [pad] i 'm wri,ng to give you advice advice on traveling and working . [eos] ﹅ ✔ ✔ ✔ ✔ ✔ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ input output [bos] i 'm wri,ng to give you [bos] i 'm wri,ng to give you some [bos] i 'm wri,ng to give you some advice one-by-one decoding for suffix match ini9al aggressive decoding (in parallel) re-decoding switch back to aggressive decoding (in parallel) bifurcation ✔ accept ✘ discard ﹅ no prediction [bos] i 'm wri,ng to give you some advice on traveling and working . [pad] ﹅ ﹅ ﹅ ﹅ ﹅ ﹅ ﹅ ﹅ on traveling and working . [eos] ﹅ ✔ ✔ ✔ ✔ ✔ ✔ ✘ decoder input output bifurcation figure 1: the overview of aggressive decoding. aggressive decoding tries decoding as many tokens as possible in parallel with the assumption that the input and output should be almost the same in gec. when we find a bifurcation between the input and the output of aggressive decoding, then we accept the predictions before (including) the bifurcation, and discard all the predictions after the bifurcation and re-decode them using original one-by-one autoregressive decoding. if we find a suffix match (i.e., some advice highlighted with the blue dot lines) between the output and the input during one-by-one re-decoding, we switch back to aggressive decoding by copying the tokens (highlighted with the orange dashed lines) following the matched tokens in the input to the decoder input by assuming they are likely to be the same. our final approach not only advances the stateof-the-art in english gec benchmarks with an almost 10× online inference speedup but also is easily adapted to other languages. ## background: transformer the transformer is a seq2seq neural network architecture based on multi-head attention mechanism, which has become the most successful and widely used seq2seq models in various generation tasks such as machine translation, abstractive summarization as well as gec. the original transformer follows the balanced encoder-decoder architecture: its encoder, consisting of a stack of identical encoder layers, maps an input sentence @xmath0, . . . , x n ) to a sequence of continuous representation @xmath1, . . . , z n ); and its decoder, which is composed of a stack of the same number of identical decoder layers as the encoder, generates an output sequence @xmath2, . . . , o m ) given z. in the training phase, the model learns an autoregressive scoring model p (y | x; φ), implemented with teacher forcing: during inference, the output sequence @xmath3, . . . , o m ) is derived by maximizing the following equation: since no ground truth is available in the inference phase, the model has to decode only one token at each step conditioning on the previous decoded tokens @xmath4. ## related work the state-of-the-art of gec has been significantly advanced owing to the tremendous success of seq2seq learning @xcite and the transformer @xcite . most recent work on gec focuses on improving the performance of the transformer-based gec models. however, except for the approaches that add synthetic erroneous data for pretraining (ge et al., 2018a; grundkiewicz et al., 2019; zhang et al., 11 the same model checkpoint also achieves the state-ofthe-art result -72.9 f0.5 with a 9.3× speedup in the bea-19 test set. 2019; @xcite @xcite , most methods that improve performance @xcite introduce additional computational cost and thus slow down inference despite the performance improvement. to make the transformer-based gec model more efficient during inference for practical application scenarios, some recent studies have started exploring the approaches based on edit operations. among them, pie @xcite and gec-tor @xcite propose to accelerate the inference by simplifying gec from sequence generation to iterative edit operation tagging. however, as they rely on many languagedependent edit operations such as the conversion of singular nouns to plurals, it is difficult for them to adapt to other languages. lasertagger @xcite uses the similar method but it is datadriven and language-independent by learning operations from training data. however, its performance is not so desirable as its seq2seq counterpart despite its high efficiency. the only two previous efficient approaches that are both languageindependent and good-performing are @xcite which uses span-based edit operations to correct sentences to save the time for copying unchanged tokens, and @xcite which first identifies incorrect spans with a tagging model then only corrects these spans with a generator. however, all the approaches have to extract edit operations and even conduct token alignment in advance from the error-corrected sentence pairs for training the model. in contrast, our proposed shallow aggressive decoding tries to accelerate the model inference through parallel autoregressive decoding which is related to some previous work @xcite in neural machine translation (nmt), and the imbalanced encoder-decoder architecture which is recently explored by @xcite and @xcite for nmt. not only is our approach language-independent, efficient and guarantees that its predictions are exactly the same with greedy decoding, but also does not need to change the way of training, making it much easier to train without so complicated data preparation as in the edit operation based approaches. ## conclusion and future work in this paper, we propose shallow aggressive decoding (sad) to accelerate online inference efficiency of the transformer for instantaneous gec. aggressive decoding can yield the same prediction quality as autoregressive greedy decoding but with much less latency. its combination with the transformer with a shallow decoder can achieve state-of-the-art performance with a 9× ∼ 12× online inference speedup over the transformer-big baseline for gec. based on the preliminary study of sad in gec, we plan to further explore the technique for accelerating the transformer for other sentence rewriting tasks, where the input is similar to the output, such as style transfer and text simplification. we believe sad is promising to become a general acceleration methodology for writing intelligence models in modern writing assistant applications that require fast online inference.