GEC - D e P en D : Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding (2023)

Abstract: Grammatical error correction (GEC) is an important NLP task that is currently usually solved with autoregressive sequence-to-sequence models. However, approaches of this class are inherently slow due to one-by-one token generation, so non-autoregressive alternatives are needed. In this work, we propose a novel non-autoregressive approach to GEC that decouples the architecture into a permutation network that outputs a self-attention weight matrix that can be used in beam search to find the best permutation of input tokens (with auxiliary <ins> tokens) and a decoder network based on a step-unrolled denoising autoencoder that fills in specific tokens. This allows us to find the token permutation after only one forward pass of the permutation network, avoiding autoregressive constructions. We show that the resulting network improves over previously known non-autoregressive methods for GEC and reaches the level of autoregressive methods that do not use language-specific synthetic data generation methods. Our results are supported by a comprehensive experimental validation on the ConLL-2014 and BEA datasets and an extensive ablation study that supports our architectural and algorithmic choices.

## introduction grammatical error correction (gec) is an important and obviously practically relevant problem in natural language processing. in recent works, gec has been usually tackled with machine learning approaches, where it has been formalized either as looking for a sequence of edits or transformation tags @xcite or, more generally, as a sequence-to-sequence text rewriting problem @xcite , a problem that is a natural fit for encoder-decoder architectures. latest encoder-decoder architectures indeed define the state of the art in grammatical error correction @xcite . however, the best current results for gec are achieved by autoregressive methods that need to produce output tokens one by one, which significantly hinders inference time and thus limits their applicability in real world solutions. this motivates the development of non-autoregressive models that can achieve results similar to autoregressive ones but with a significantly improved runtime. previously developed non-autoregressive approaches have relied on language-specific transformation tags @xcite . in this work, we develop a novel non-autoregressive and languageagnostic approach, called gec-depend (gec with decoupled permutation & decoding) that yields excellent performance on the gec task and has other attractive properties. in particular, it is able to output a ranked list of hypotheses that a potential user can choose from. the main idea of gec-depend is to decouple permutation and decoding, with one network producing a permutation of input tokens together with specially added ins tokens for possible insertions and another network actually infilling ins tokens. fig. 1 illustrates the idea: the source sentence "i be busy" is encoded as " s i be busy \s ins ", the permutation network obtains " s i ins busy \s ", and then the decoder network converts " s i msk 1 msk 2 msk 3 busy \s " into " s i am pad pad busy \s " and outputs "i am busy" as the corrected sentence. in a single run, the permutation network produces a self-attention matrix for subsequent beam search @xcite , while in the decoder network we use the step-unrolled denoising autoencoder (sundae) proposed by @xcite . we also adapt and evaluate several additional techniques including a three-stage training schedule, length normalization, and inference tweaks that improve the final performance. thus, our main contributions can be summarized as follows: (i) we propose, to the best of our knowledge, the first open-vocabulary iterative non-autoregressive gec model foot_0 based on decoupling permutation and decoding, including (ii) a novel pointing mechanism that can be implemented by a single permutation network without an additional tagger and (iii) a new algorithm for producing ground truth permutations from source (errorful) and target (corrected) sentences, leading to more adequate dataset construction for the gec task. in experimental evaluation, we show that our model outperforms previously known nonautoregressive approaches (apart from gector that uses language-specific tagging @xcite ) and operates, with similar implementations for backbone networks, several times faster than either autoregressive approaches or gector. the paper is organized as follows. section 2 surveys related work on both autoregressive and nonautoregressive approaches to gec. section 3 introduces our approach, including our idea on decoupling permutation and decoding, sundae, and new ideas for dataset construction and inference tweaks that make our approach work. section 4 shows the main experimental results, section 5 presents an extensive ablation study that highlights the contributions of various parts of our approach, section 6 concludes the paper, and section 7 discusses the limitations of our approach. ## ablation study in this section, we present a detailed ablation study, reporting both ideas that worked (section 3) and ideas that did not work. ## conclusion in this work, we have presented gec-depend, a novel method for non-autoregressive grammatical error correction that decouples permutation and decoding steps, adds the step-unrolled denoising autoencoder into the decoder network, changes the dataset construction algorithm to preserve long spans, and uses inference tweaks to improve the results. gec-depend shows the best results among non-autoregressive language-agnostic gec models and significantly outperforms other models in terms of inference speed. we hope that our approach can become a basis for real life applications of grammatical error correction.