Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020)

Abstract: We propose a novel language-independent approach to improve the efficiency for Grammatical Error Correction (GEC) by dividing the task into two subtasks: Erroneous Span Detection (ESD) and Erroneous Span Correction (ESC). ESD identifies grammatically incorrect text spans with an efficient sequence tagging model. Then, ESC leverages a seq2seq model to take the sentence with annotated erroneous spans as input and only outputs the corrected text for these spans. Experiments show our approach performs comparably to conventional seq2seq approaches in both English and Chinese GEC benchmarks with less than 50% time cost for inference.

## introduction due to a growing number of error-corrected parallel sentences available in recent years, sequenceto-sequence (seq2seq) models with the encoderdecoder architecture @xcite @xcite have become a popular solution to gec, which take the source (original) sentence as input and output the target (corrected) sentence. although autoregressive seq2seq models facilitate correction for various grammatical errors and perform well, they are not efficient enough for gec. as previous work @xcite points out, seq2seq models take most decoding steps to copy grammatically correct text spans from the source to the target during inference, which is the main efficiency bottleneck. if the time for the copying operations can be saved, the efficiency should be much improved. with this motivation, we propose a simple yet novel language-independent approach to improve the efficiency of gec by dividing the task into two subtasks: erroneous span detection (esd) and erroneous span correction (esc), shown in figure 1 . in esd, we use an efficient sequence tagging model to identify the text spans that are grammatically incorrect in the source sentence, as figure 1 (a) shows. then, we feed the sentence with erroneous span annotations to a seq2seq model for esc. in contrast to conventional seq2seq approaches correcting the complete sentence, esc only corrects the erroneous spans (see figure 1(b) ), which largely decreases the number of steps for decoding. experiments in both english and chinese gec benchmarks demonstrate our approach performs comparably to the state-of-the-art transformer based seq2seq model with less than 50% time cost for inference. furthermore, our approach offers more flexibility to control correction, allowing us to adapt the precision-recall trade-off to various application scenarios. ## related work recently, many approaches have been proposed to improve gec performance. however, except those adding synthetic erroneous data @xcite @xcite @xcite and wikipedia revision logs @xcite for training, most methods cause an increase in latency. for example, language model and right-to-left (r2l) rescoring @xcite not only take time to rescore but also slow down the correction model with a larger beam size during inference; multi-round (iterative) decoding @xcite needs to repeatedly run the model; bert-fuse @xcite adds extra computation for model fusion. in contrast to extensive studies on gec performance, little work focuses on improving the efficiency of gec models until the last years. sequence tagging model as i'm new to here, i'm lost encoder @xmath0. 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 as i'm @xmath1, i'm lost and don't know @xmath2. </@xmath3. </@xmath4. </@xmath5, i'm lost and don't know where is to my hotel. as i'm new here, i'm lost and don't know where my hotel is. source target figure 1: an overview of erroneous span detection (esd) and erroneous span correction (esc). the detection model is a sequence tagging model, while the correction model is a seq2seq model but only outputs the corrected texts for annotated spans (i.e., s1 to /s1 and s2 is to my hotel. /s2 ). for example, the text span "is to my hotel." is identified as incorrect and then edited into "my hotel is.". one branch of the work is language-dependent approaches, like pie @xcite and gector @xcite . they predict a sequence of token-level edit operations including a number of manually designed languagespecific operations like changing verb forms (e.g., @xmath6.g., @xmath7. however, they are difficult to be adapted to other languages. the other branch is language-independent models like lasertagger @xcite . they learn a vocabulary of edit operations from training data and thus can work for any language. however, their performance is inferior to their seq2seq counterpart. our approach combines the advantages of both branches, which is language-independent and performs comparably to the state-of-the-art seq2seq approach with efficient inference. ## erroneous span detection to identify incorrect spans, we use a binary sequence tagging model in which tag 0 means the token is in a correct span; while tag 1 means the token is in a grammatically incorrect span that needs to be edited, as shown in figure 1 @xcite . in order to train the tagging model, we align 1 tokens across the 1 alignment can be solved by dynamic programming like levenshtein distance. we here use errant ( https:// github.com/chrisjbryant/errant ) for alignment. ## erroneous span correction with esd, we can identify grammatically incorrect text spans in a sentence. if a sentence is identified as error-free, we take no further action; otherwise, we annotate the incorrect spans and use the esc model to correct them, shown in figure 1(b) . to avoid esc being misled by span detection errors from esd during inference, we randomly select text spans in the similar way to span-bert @xcite instead of only using gold erroneous spans in training data, to train the esc model. in this way, the esc model will see a large variety of span annotations and learn how to correct during training, and thus its robustness is improved: even if the detected spans during inference are not exactly accurate, the esc model will not easily fail. with token alignment across the source and target sentence in gec training data, we can generate training instances with span annotations and corrections like the example in figure 1( ## conclusion and future work we propose a novel language-independent approach to improve the efficiency of gec. our approach performs comparably to the state-of-the-art seq2seq model with a considerable reduction in inference time, and can be easily adapted to other languages and offer more flexibility to control correction behavior (e.g., trading precision for recall). through our experiments in gec, we verify the feasibility of span-specific decoding, which has been explored for text infilling @xcite ) and text rewriting. it is inspiring and promising to be generalized to more rewriting tasks, which will be studied as our future work. source sentence instead, we will post a seed and tag our friends to inform this kind of changments.