M ulti-Class Grammatical Error Detection for Correction: A Tale of Two Systems (2021)

Abstract: In this paper, we show how a multi-class grammatical error detection (GED) system can be used to improve grammatical error correction (GEC) for English. Specifically, we first develop a new state-of-the-art binary detection system based on pre-trained ELECTRA, and then extend it to multi-class detection using different error type tagsets derived from the ERRANT framework. Output from this detection system is used as auxiliary input to fine-tune a novel encoder-decoder GEC model, and we subsequently re-rank the N-best GEC output to find the hypothesis that most agrees with the GED output. Results show that fine-tuning the GEC system using 4-class GED produces the best model, but re-ranking using 55-class GED leads to the best performance overall. This suggests that different multi-class GED systems benefit GEC in different ways. Ultimately, our system outperforms all other previous work that combines GED and GEC, and achieves a new single-model NMT-based state of the art on the BEA-test benchmark.

## introduction grammatical error detection (ged) is the task of automatically detecting grammatical errors in text, while grammatical error correction (gec) is the task of also correcting these errors. both tasks have obvious pedagogical applications that can benefit both teachers and students in online language learning. ged is typically cast as a binary sequence labelling task, where each token is classified as either correct or incorrect @xcite , while gec is often considered a sequence-to-sequence translation task, where systems learn to "translate" an ungrammatical input sentence to a grammatical output sentence @xcite @xcite . recent work has also begun to treat gec as a sequence labelling task, where tokens are classified in terms of edit operations @xcite . we similarly treat ged as a sequence labelling task and gec as a sequence-to-sequence task, but additionally investigate different ways to combine and extend both approaches. in particular, we first experiment with pretrained language models, and show that simply finetuning electra @xcite leads to significant improvements in binary ged and achieves a new state of the art. given that binary detection is limited in terms of the specific error type information it can provide to downstream tasks however, we also extend our ged system to 4-class, 25-class and 55-class error detection using different error type tagsets derived from the errant framework @xcite . to illustrate the value of multi-class ged for downstream gec, we extend the transformer encoder-decoder model @xcite and employ a multi-encoder gec model @xcite . we also introduce a two-step training strategy which only requires additional input information from a small dataset for fine-tuning. specifically, we experiment with two methods that use ged to inform gec: i) we use ged predictions as auxiliary input to fine-tune a model; and ii) we use ged predictions as a means to re-rank system output during post-processing. to summarise, we present the first study using multi-class ged to improve gec for english. our main contributions are: ## previous work early approaches to ged focused on specific error types, and in particular article and preposition errors, which are among the most frequent in nonnative english learner writing @xcite . more general open-class ged systems were later developed using parse and text-based features @xcite . @xcite presented the first work using a neural approach and framed ged as a binary sequence labelling problem, classifying each token in a sentence as either correct or incorrect. subsequent improvements were obtained through auxiliary objectives @xcite , and incorporating artificial training data @xcite . recent work takes advantage of large scale pre-trained language models: bell et al. previous work has attempted to combine these two similar tasks and explore different ways of using ged in gec. @xcite and @xcite employed multi-task learning and introduced token-level and sentence-level ged as auxiliary tasks when training for gec. @xcite fine-tuned bert for binary ged and then incorporated their model into an encoder-decoder gec framework. similarly, chen et al. (2020) finetuned roberta @xcite for ged and reformatted the input to include error span information, which was used by their encoder-decoder model. our approach of using additional ged input during gec training differs from theirs in that we only use a small set of training examples with ged information for fine-tuning. the ged training and gec training are not coupled together, so ged predictions from any system can be used. binary ged has also been used in postprocessing to re-rank gec system output @xcite @xcite or filter out unnecessary corrections @xcite . similarly, @xcite and @xcite applied quality estimation approaches to re-rank gec output. additional ged and/or gec features are often used in these systems however, which also require extra tuning. in contrast, our simple re-ranking approach only uses ged predictions and does not require tuning. our work differs most centrally in that we treat ged as a multi-class problem and investigate ways to use multi-class ged predictions to inform gec. ## data following previous studies, we use the public lang-8 corpus @xcite , the first certificate in english (fce) corpus @xcite , the cambridge learner corpus (clc) @xcite , the national university of singapore corpus of learner english (nucle) @xcite the edit annotations in all these corpora were pre-processed and standardised using the errant annotation framework. one advantage of this framework is that error types are modular, and consist of "operation" + "main" type tags; e.g. r:noun for replacement noun. we hence use this modularity in our multi-class ged experiments such that 4-classes consist of operation type only (i.e. missing, replacement, unnecessary and correct), 25-classes consist of main type only (e.g. noun, noun number, verb tense, etc.) and 55-classes consist of the full tags combined. 2 an example is shown in ## grammatical error detection following @xcite , we treat ged as a sequence labelling (or token classification) task and assign a label to each token in the input sentence, indicating whether it is correct or incorrect (i.e. binary classification) or which error type it belongs to at different levels of granularity (i.e. multi-class classification). we first perform binary classification to compare our models with current state-of-the-art systems. we further extend them with multi-class error classification with the aim of improving downstream gec. ## conclusion we have shown that multi-class ged can be used to significantly improve gec. first, we showed that fine-tuning a pre-trained transformer-based language model can lead to significant improvements in binary ged. specifically, we found that fine-tuning electra, which has a discriminative pre-training objective that is conceptually similar to ged, produces a new state of the art on three benchmark datasets. we furthermore showed that our models are capable of multi-class detection, and obtain similar f 0.5 performance to binary ged. next, we employ a multi-encoder gec model and presented two methods of integrating ged predictions into gec systems: firstly during gec fine-tuning and secondly as a post-processing reranking step. results show that both methods, when applied independently, significantly improve over a strong nmt-based gec baseline. when applied together, we find the methods complement each other, yielding further performance gains. our best single-model gec system outperforms all previous systems that combine ged and gec on both test sets, and all other single-model nmt-based systems on bea-test. our results ultimately demonstrate the advantages of integrating multi-class detection into correction. in particular, different multi-class ged systems benefit gec in different ways, and we find that the 4-class ged model leads to the best performance in fine-tuning the gec system, but re-ranking using the 55-class ged model produces the best gec performance overall. finally, oracle experiments reveal that our proposed gec systems are very effective at incorporating new ged information, but that there are still significant gains to be made in terms of more accurate ged systems. a corpus statistics corpus # sentences fce-train 28,350 fce-dev 2,191 fce-test 2,695 w&i-train 34,308 bea-dev 4,384 bea-test 4,477 nucle 57,151 lang-8 574,281 clc 1,961,065