Improved grammatical error correction by ranking elementary edits (2022)

Abstract: We offer a two-stage reranking method for grammatical error correction: the first model serves as edit generator, while the second classifies the proposed edits as correct or false. We show how to use both encoder-decoder and sequence labeling models for the first step of our pipeline. We achieve state-of-the-art quality on BEA 2019 English dataset even using weak BERT-GEC edit generator. Combining our roberta-base scorer with state-of-the-art GECToR edit generator, we surpass GECToR by 2-3%. With a larger model we establish a new SOTA on BEA development and test sets. Our model also sets a new SOTA on Russian, despite using smaller models and less data than the previous approaches.

## introduction grammatical error correction (gec) is the task of converting the source text to its clean version with no orthographic, punctuation, lexical or other errors. it becomes increasingly popular during last years due to its applications such as second language learning. however, even for english with its numerous resources and wide research community, modern models are far from being perfect. in particular, the recall of the state-of-the-art gec-tor model @xcite on standard bea2019 development set is lower than 40%. while gector uses sequence labeling approach with linguistically motivated label inventory, most works for other languages, such as @xcite and others, simply mimic machine translation methods, training a transformer model @xcite on the pairs of source and corrected sentences. this approach suffers from left-to-right decoding: the model can make a wrong decision not observing the future context. this problem may be mitigated using reranking: the ranker observes entire corrected sequences and thus may utilize richer context. it also helps to discriminate between several possible edits with similar basic model probability . due to these reasons, it was heavily used in machine translation both in statistical @xcite and neural @xcite era. in contrast to machine translation, sequence editing in gec usually does not require complete rewriting and can be decomposed to elementary edits such as modifying a single word or a consecutive group of words. in this paper we propose to score elementary edits produced by the basic model and classify them as positive or negative on the second stage of the pipeline. than the calculated probabilities can be either used directly or combined with the scores from the first stage. we show that our scoring model achieves stateof-the-art performance on bea2019 dataset even with rather weak first stage model. its combination with gector edit generator outperforms the models of the same size by about 2 f 0.5 points. the large version of our model establishes the new sota on bea dataset among models of all size. we also improve over current sota on russian rulec-gec dataset by more than 3 f 0.5 points. we make most of our code and models freely available foot_0 . ## pipeline description as proposed by @xcite , probably the simplest approach to grammatical error correction is to generate possible edits using a rule-based model and then extract those that increase the sentence probability by a sufficient margin. this probability may be estimated using a large pretrained language model, such as gpt @xcite or bert @xcite . this approach requires no training data, only a development set for tuning the hyperparameters. as a reverse side of its simplicity, this algorithm has two main limitations: therefore the main idea of our paper is to replace the scorer by a more powerful trainable model. another key detail is that we apply the scorer not to the full corrections, but to the elementary edits. namely, given the erroneous sentence *the boy fall on floor yesterday and its correction the boy fell on the floor yesterday, our model should return true for sentences the boy fell on floor yesterday and the boy fall on the floor yesterday and false for other elementary corrections, for example, *the boy falls on floor yesterday. so, our model includes three main stages: the first part is described in 3 and the remaining two in section 4. a schematic description of our algorithm is given on figure 1 . ## edit generators in this section we describe the first stage of our pipeline -the edit generator. we seek to make our pipeline independent of particular generator selection. therefore we describe three possible variants: the rule-based generator (subsection 3.1), the seq2seq model (3.2) and the sequence labeling one (3.3), based on the well-known gector model @xcite . note that gector is available only for english and its development for languages with complex morphology is problematic since it needs a word inflection module to transform the predicted labels into surface forms. ## experiments in this section we describe our experiments. note that our main contribution is the scorer and we claim that our method is not limited to a particular edit generator. thus we do not train edit generators 8 by default we use the roberta-base variant. by ourselves and only adapt them to our pipeline as described in section 3. our main experiments are conducted for english, in subsection 6.3 we also present results for russian. we compare the models by f 0.5 score using errant @xcite for english bea data and m2scorer @xcite for other datasets. since the bea 2019 test set does not contain correct answers, we do most of the comparisons on the development data. ## related work the task of grammatical error correction has a long history. the main paradigm of recent years is to treat it as low-resource machine translation @xcite using extensive pretraining on synthetic data @xcite . synthetic data is usually generated using random replacement, deletion, insertion, spelling errors and perturbations @xcite @xcite , other approaches include training on wikipedia edits @xcite and backtranslation @xcite . another trend is incorporating pretrained transformer language models either as a part of system architecture @xcite or for the initialization of model weights @xcite . the extreme case of the latter approach is the "brute force" when one simply uses large encoder-decoder transformer that potentially is able to solve any text-to-text task @xcite . another paradigm in gec is to reduce grammar correction to sequence labeling @xcite training data p r f 0.5 transformer (náplava and straka, 2019) 10m synth., rulec-gec train, dev 63.3 27.5 50.2 mt5-xxl @xcite mc4 synth., rulec-gec train na na 51.6 rugpt-large finetune (strong baseline) 200k synth., rulec-gec train 65.7 27.4 51.3 rule-based edits 200k synth., rulec-gec train 69.4 25.9 51.9 rugpt-large edits, 'scorer-only' 200k synth., rulec-gec train 70.9 26.8 53.4 rugpt-large edits, 'combined' 200k synth., rulec-gec train 73.7 27.3 55.0 ## conclusion we have developed a two-stage algorithm for grammatical error correction based on edit classifications. our main results are the following: since our model shows competitive performance even with rule-based edit generators, it may be applied in settings that require control over possible corrections. one such field is language learning, e.g., correcting error of particular type, such as verb tense or determiner choice. in the future work we plan to address this question in more details and test the applicability of our approach on additional languages, such as german or czech. last but not the least, the main idea of ranking individual edits can be applied not only to gec, but to any task where the concept of elementary edit has meaning, for example, machine translation post-editing.