Position Offset Label Prediction for Grammatical Error Correction (2022)

Abstract: We introduce a novel position offset label prediction subtask to the encoder-decoder architecture for grammatical error correction (GEC) task. To keep the meaning of the input sentence unchanged, only a few words should be inserted or deleted during correction, and most of tokens in the erroneous sentence appear in the paired correct sentence with limited position movement. Inspired by this observation, we design an auxiliary task to predict position offset label (POL) of tokens, which is naturally capable of integrating different correction editing operations into a unified framework. Based on the predicted POL, we further propose a new copy mechanism (P-copy) to replace the vanilla copy module. Experimental results on Chinese, English and Japanese datasets demonstrate that our proposed POL-Pc framework obviously improves the performance of baseline models. Moreover, our model yields consistent performance gain over various data augmentation methods. Especially, after incorporating synthetic data, our model achieves a 38.95 F-0.5 score on Chinese GEC dataset, which outperforms the previous state-of-the-art by a wide margin of 1.98 points.

## introduction grammatical error correction (gec) is to automatically correct grammatical errors in the input sequence, which is significant for both academic research and practical applications. comparing with english grammatical error correction, chinese grammatical error correction (cgec) is less addressed. after convolutional neural network and recurrent neural network, the transformer based sequenceto-sequence models have achieved remarkable performance on gec task @xcite . to alleviate the data sparsity problem, the method of generating pseudo data is intensively studied @xcite . several models simplify the gec task from sequence generation to tokenlevel edit prediction. for example, lasertagger @xcite predicts three edit tags keep, delete and append_# for each token in the source sentence, and gector @xcite designs ample english-specific tags to predict. to keep the meaning of the erroneous sentence unchanged, only a few words could be inserted or deleted while most words remain the same. taking the cgec task nlpcc data @xcite as an example: (1) about 95.6% tokens in erroneous sentences appear in their corresponding correct sentences (namely "cue tokens" for brevity). (2) the average lengths of erroneous sentences and the longest common sub-sequence of error-corrected sentence pairs are 29.65/27.55 tokens, indicating that cue tokens are almost in the same order. (3) the average lengths of inserted and deleted spans are 1.46 and 1.49 tokens, respectively. unfortunately, there is no existing model which takes all of these three characteristics into consid-eration. the copy mechanism @xcite only considers the characteristic (1). the sequence tagging models @xcite take advantage of characteristic (3) to build a fixed-size vocabulary for append_# label but fail to explicitly model the first and the second points. in our work, we merge these three characteristics into one key point: most of tokens appear in both erroneous and correct sentences, and their positions in erroneous sentences (denoted as i th ) are identical or close to their positions in the corresponding correct sentences (denoted as j th ). as shown in figure 2 , this phenomenon could be observed in gec datasets of different languages. for instance, on nlpcc dataset, 48.2% of cue tokens are exactly in the same position (@xmath0, and about 90% of them can be moved to the right position within a length of 3 tokens (abs(i -j) ≤ 3). inspired by the key point summarized above, we propose a novel position offset label (pol) prediction task to determine which tokens in the source sentence are error-free, as well as their position movement from the source sequence to the target sentence. as illustrated in figure 1 , our method could simultaneously model different kinds of correction editing operations, including insertion, substitution, deletion and re-ordering. at the decoder, rather than using the conventional attention distribution, a new copy mechanism (p-copy) is proposed to take advantage of the result of pol prediction. overall, we adopt a multi-task learning framework named pol-pc, where the detection network is to perform pol prediction, and the correction network then generates words to fill in the blank positions where no tokens in the source sequence are mapped to. we conduct extensive experiments on cgec data nlpcc @xcite . experimental results show that without data augmentation, our pure model obtains the best result among all nonpretrained models. after integrating data augmentation methods, our method achieves a new state-ofthe-result for cgec for single models, outperforming the previous best result by 1.98 points. we also conduct experiments on english dataset conll-2014 @xcite and japanese tec-jl dataset @xcite , our pol-pc model consistently improves the performance of the transformer, validating the generalization ability of our approach. we will make our code publicly available at the github for further research. to sum, our proposed model for gec task enjoys the following advantages: ## related work recently, transformer-based sequence to sequence models are introduced to gec task and make great progress @xcite . to augment the error-corrected parallel corpus, synthetic data is generated from the wikipedia revision log @xcite or generated by applying token-level insertion, substitution, deletion or swapping on error-free corpus @xcite . several models are proposed to simplify gec task from sequence generation to sequence tagging. lasertagger @xcite predicts keep, delete or append_# tags for each token in the source sentence. pie @xcite and gector @xcite ) manually design detailed english-specific labels, regarding case and tense, which is hard to adapt to other languages like chinese. ttt @xcite adopts a non-autoregressive model to directly predict each token in the correct sentence. esd-esc @xcite detects erroneous spans and then utilizes a seq2seq model to only produce correct text for those annotated spans. different from these previous works, we build a multi-task learning framework by treating the sequence tagging as an auxiliary subtask, and more importantly, our tag strategy is novel and effective. for cgec task, most of previous works @xcite focus on the spelling error correction task on @xcite and hybrid @xcite datasets. the nlpcc-2018 dataset @xcite provides multiple types of grammatical errors and attracts much attention from participated teams, where the top-3 systems are alibaba @xcite , youdao @xcite and blcu @xcite . hrg combines language model base spelling checker, nmt-base model and sequence editing model @xcite . later, zhao and wang proposes maskgec by adding random noises to source sentences dynamically, which achieves the state-of-the-art on nlpcc-2018 dataset. this paper also conducts experiments on nlpcc dataset and considers these methods as comparing baselines. ## proposed model as shown in figure 1 , given an erroneous sentence @xmath1, x 2 , ...x m ) and its corresponding corrected sentence @xmath2, y 2 , ...y n ), our model firstly decides whether a source token x i should be deleted and if not predicts its position changing from x to y . there might exist some blank positions where no source tokens are mapped to (denoted as @ in figure 1 ). a decoder with copy mechanism is utilized to copy source tokens and generate new tokens for blank positions, where the copy operation is guided by the result of position offset label prediction. the overview of our framework is illustrated in figure 3 . ## data augmentation in order to provide more training samples for our model, we do data augmentation by generating synthetic data and introducing dynamic noises. we introduce the following types of noises to produce erroneous sentences. deletion: to discard each word with a probability of p del . insertion: to insert a [unk] symbol before each word with a probability of p ins . substitution: to replace each word using [unk] symbol / random token / its homophone with a probability of p sub . we set p @xmath3.25 in all experiments. ## conclusion we introduce a detection network to predict pol of tokens between source erroneous sentences and target correct sentences. based on the output of pol, we design a new copy mechanism p-copy. our pol-pc model exceeds both end-to-end models and sequence tagging approaches, achieving a new state-of-the-art result on cgec task for single models. for english and japanese gec tasks, our approach also obtains significant performance gain over the baseline transformer.