Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction (2022)

Abstract: In this paper, we investigate improvements to the GEC sequence tagging architecture with a focus on ensembling of recent cutting-edge Transformer-based encoders in Large configurations. We encourage ensembling models by majority votes on span-level edits because this approach is tolerant to the model architecture and vocabulary size. Our best ensemble achieves a new SOTA result with an F_{0.5} score of 76.05 on BEA-2019 (test), even without pre-training on synthetic datasets. In addition, we perform knowledge distillation with a trained ensemble to generate new synthetic training datasets, “Troy-Blogs” and “Troy-1BW”. Our best single sequence tagging model that is pretrained on the generated Troy- datasets in combination with the publicly available synthetic PIE dataset achieves a near-SOTA result with an F_{0.5} score of 73.21 on BEA-2019 (test). The code, datasets, and trained models are publicly available.

## introduction the purpose of the grammatical error correction (gec) task is to correct grammatical errors in natural texts. this includes correcting errors in spelling, punctuation, grammar, morphology, word choice, and others. an intelligent gec system receives text containing mistakes and produces its corrected version. the gec task is complicated and challenging: the accuracy of edits, inference speed, and memory limitations are topics of intensive research. currently, machine translation (mt) is the mainstream approach for gec. in this setting, errorful sentences correspond to the source language, and error-free sentences correspond to the target language. early gec-mt methods leveraged phrase-based statistical machine translation (pbsmt) @xcite . then this approach rapidly evolved to seq2seq neural machine translation (nmt) based on gated recurrent neural networks @xcite and recent powerful transformer-based seq2seq models. transformer-based models autoregressively capture the full dependency among output tokens; however, inference can be slow due to sequential decoding. @xcite leveraged a transformer model @xcite that was pre-trained on synthetic gec data and right-toleft re-ranking for ensemble. @xcite adopted several strategies of bert @xcite usage for gec. recently, @xcite built their system on top of t5 @xcite , a xxl version of the t5 transformer encoder-decoder model and reached new state-of-the-art results (11b parameters). while still not as widespread as mt, the sequence tagging approach for gec, which generates a sequence of text edit operations encoded by tags for errorful input text is becoming more common. lasertagger @xcite is a sequence tagging model that casts text generation as a text editing task. corrected texts are reconstructed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. lasertagger combines a bert encoder with an autoregressive transformer decoder, which predicts edit operations. the parallel iterative edit (pie) model @xcite does parallel decoding, achieving quality that is competitive with the seq2seq models. 3 it predicts edits instead of tokens and iteratively refines predictions to capture dependencies. a similar approach is presented in @xcite . the gector system achieves competitive results using various trans-formers as an encoder; and linear layers with softmax for tag prediction and error detection. by replacing an autoregressive decoder with linear output layers, it's also potentially several times faster than seq2seq systems. today, the generation of synthetic data is becoming significant for most gec models. natural languages are rich, and their grammars contain many rules and exceptions; therefore, professional linguists are often utilized to annotate highquality corpora for further training ml-based systems mostly in a supervised manner @xcite , @xcite . however, human annotation is expensive, so researchers are working on methods for augmentation of training data, synthetic data generation, and strategies for its efficient usage @xcite , @xcite , @xcite . the majority of gec systems today use synthetic data to pre-train transformer-based components of their models. in this work, we are focusing on exploring sequence tagging models and their ensembles. although most of our developments may eventually be applied to other languages, we work with english only in this study. being a resource-rich language, english is a highly competitive area for the gec task 3 . ## ensembling the gec taggers ensembling is a proven quality-boosting method for models sets that have diverse outputs. most of the recent gec solutions achieved their best results by ensembling single models (stahlberg and kumar, 2021), @xcite , @xcite . in this section we consider two ensembling methods for our gec tagging models: averaging of output tag probabilities and majority votes on output edit spans (fig. 2 ). ## knowledge distillation knowledge distillation is the method for transferring knowledge from a large model ("teacher") to a smaller one ("student") @xcite , @xcite . it has strong practical applications because large models usually have expensive inference costs and are inconvenient for deployment. in our case, the teacher model is an ensemble of trained sequence taggers, whereas the student model is a single sequence tagger. the ensemble receives errorful texts and generates their corrected versions. later these input-output pairs of sentences are used for training single models. like any synthetic annotation method, knowledge-distilled data contains a certain share of systematic errors that deteriorates the student model's quality.