GECT o R – Grammatical Error Correction: Tag, Not Rewrite (2020)

Abstract: In this paper, we present a simple and efficient GEC sequence tagger using a Transformer encoder. Our system is pre-trained on synthetic data and then fine-tuned in two stages: first on errorful corpora, and second on a combination of errorful and error-free parallel corpora. We design custom token-level transformations to map input tokens to target corrections. Our best single-model/ensemble GEC tagger achieves an F_0.5 of 65.3/66.5 on CONLL-2014 (test) and F_0.5 of 72.4/73.6 on BEA-2019 (test). Its inference speed is up to 10 times as fast as a Transformer-based seq2seq GEC system.

## introduction neural machine translation (nmt)-based approaches @xcite have become the preferred method for the task of grammatical error correction (gec) foot_1 . in this formulation, errorful sentences correspond to the source language, and error-free sentences correspond to the target language. recently, transformer-based @xcite sequence-to-sequence (seq2seq) models have achieved state-of-the-art performance on standard gec benchmarks @xcite . now the focus of research has shifted more towards generating synthetic data for pretraining the transformer-nmt-based gec systems @xcite . nmtbased gec systems suffer from several issues which make them inconvenient for real world deployment: (i) slow inference speed, (ii) demand for large amounts of training data and (iii) interpretability and explainability; they require additional functionality to explain corrections, e.g., grammatical error type classification @xcite . in this paper, we deal with the aforementioned issues by simplifying the task from sequence generation to sequence tagging. our gec sequence tagging system consists of three training stages: pretraining on synthetic data, fine-tuning on an errorful parallel corpus, and finally, fine-tuning on a combination of errorful and error-free parallel corpora. related work. lasertagger @xcite combines a bert encoder with an autoregressive transformer decoder to predict three main edit operations: keeping a token, deleting a token, and adding a phrase before a token. in contrast, in our system, the decoder is a softmax layer. pie @xcite is an iterative sequence tagging gec system that predicts token-level edit operations. while their approach is the most similar to ours, our work differs from theirs as described in our contributions below: dataset # sentences % errorful training sentences stage pie-synthetic 9,000,000 100.0% i lang-8 947,344 52.5% ii nucle 56,958 38.0% ii fce 34,490 62.4% ii w&i+locness 34,304 67.3% ii, iii ## datasets training data. we use the following datasets for fine-tuning stages ii and iii: national university of singapore corpus of learner english (nu-cle) 4 @xcite , lang-8 corpus of learner english (lang-8) 5 @xcite , fce dataset foot_5 @xcite , the publicly available part of the cambridge learner corpus @xcite and write & improve + loc-ness corpus @xcite 7 . evaluation data. we report results on conll-2014 test set @xcite evaluated by official m 2 scorer @xcite , and on bea-2019 dev and test sets evaluated by errant @xcite . ## token-level transformations we developed custom token-level transformations t (x i ) to recover the target text by applying them to the source tokens (x 1 . . . x n ). transformations increase the coverage of grammatical error corrections for limited output vocabulary size for the most common grammatical errors, such as spelling, noun number, subject-verb agreement and verb form @xcite . the edit space which corresponds to our default tag vocabulary @xmath2, delete and 1167 token-dependent append, 3802 replace) and 29 token-independent gtransformations. basic transformations perform the most common token-level edit operations, such as: keep the current token unchanged (tag @xmath0delete), append new token t 1 next to the current token x i (tag @xmath1replace t 2 ). to obtain the transformation suffix for the verb form tag, we use the verb conjugation dictionary foot_7 . for convenience, it was converted into the following format: token 0 token 1 : tag 0 tag 1 (e.g., go goes : v b v bz). this means that there is a transition from word 0 and word 1 to the respective tags. the transition is unidirectional, so if there exists a reverse transition, it is presented separately. the experimental comparison of covering capabilities for our token-level transformations is in preprocessing. to approach the task as a sequence tagging problem we need to convert each target sentence from training/evaluation sets into a sequence of tags where each tag is mapped to a single source token. below is a brief description of our 3-step preprocessing algorithm for color-coded sentence pair from step 2). for each mapping in the list, find tokenlevel transformations which convert source token to the target subsequence: ## tagging model architecture our gec sequence tagging model is an encoder made up of pretrained bert-like transformer stacked with two linear layers with softmax layers on the top. we always use cased pretrained transformers in their base configurations. tokenization depends on the particular transformer's design: bpe @xcite is used in roberta, wordpiece @xcite in bert and sentencepiece @xcite in xlnet. to process the information at the tokenlevel, we take the first subword per token from the encoder's representation, which is then forwarded to subsequent linear layers, which are responsible for error detection and error tagging, respectively. ## iterative sequence tagging approach to correct the text, for each input token x i , 1 ≤ @xmath4. . . x n ), we predict the tag-encoded token-level transformation t (x i ) described in section 3. these predicted tagencoded transformations are then applied to the sentence to get the modified sentence. since some corrections in a sentence may depend on others, applying gec sequence tagger only once may not be enough to fully correct the sentence. therefore, we use the iterative correction approach from @xcite : we use the gec sequence tagger to tag the now modified sequence, and apply the corresponding transformations on the new tags, which changes the sentence further (see an example in ## conclusions we show that a faster, simpler, and more efficient gec system can be developed using a sequence tagging approach, an encoder from a pretrained transformer, custom transformations and 3-stage training. our best single-model/ensemble gec tagger achieves an f 0.5 of 65.3/66.5 on conll-2014 (test) and f 0.5 of 72.4/73.6 on bea-2019 (test). we achieve state-of-the-art results for the gec task with an inference speed up to 10 times as fast as transformer-based seq2seq systems.