Multi-pass Decoding for Grammatical Error Correction (2024)

Abstract: Sequence-to-sequence (seq2seq) models achieve comparable or better grammatical error correction performance compared to sequence-to-edit (seq2edit) models. Seq2edit models normally iteratively refine the correction result, while seq2seq models decode only once without aware of subsequent tokens. Iteratively refining the correction results of seq2seq models via Multi-Pass Decoding (MPD) may lead to better performance. However, MPD increases the inference costs. Deleting or replacing corrections in previous rounds may lose useful information in the source input. We present an early-stop mechanism to alleviate the efficiency issue. To address the source information loss issue, we propose to merge the source input with the previous round correction result into one sequence. Experiments on the CoNLL-14 test set and BEA-19 test set show that our approach can lead to consistent and significant improvements over strong BART and T5 baselines (+1.80, +1.35, and +2.02 F0.5 for BART 12-2, large and T5 large respectively on CoNLL-14 and +2.99, +1.82, and +2.79 correspondingly on BEA-19), obtaining F0.5 scores of 68.41 and 75.36 on CoNLL-14 and BEA-19 respectively.

## introduction grammatical error correction (gec) aims to correct grammatical errors in the given sentence @xcite . nowadays, there are two mainstream gec approaches. sequence-to-edit (seq2edit) methods regard gec as a sequence tagging task, where the model predicts edit tags (e.g., keep, delete, insert, replace, etc.) for each token iteratively for multiple rounds until all tokens are assigned the keep tag @xcite @xcite . seq2edit methods normally require to correct for a number of correction rounds to * corresponding author: hongfei @xcite complete the correction. in contrast, sequence-tosequence (seq2seq) approaches consider the gec task as machine translation (mt) from ungrammatical texts to grammatical texts @xcite @xcite @xcite . the seq2seq model encodes the input sentence and auto-regressively decodes the corrected sentence. current methods normally utilize the pre-trained models for better performance, such as bert @xcite and xlnet @xcite for seq2edit @xcite , and bart @xcite and t5 @xcite for seq2seq @xcite . seq2seq models lead to comparable or better performance than seq2edit approaches without using language-specific edit operations. however, current seq2seq gec studies typically decode only once without aware of subsequent tokens. multi-pass decoding (mpd) may enhance the performance through iterative refinements @xcite . training mpd models to generate the gold reference given its correction results may also benefit its learning via self-correction @xcite . multi-pass decoding leads to two problems: 1) iterative decoding increases the inference computational costs, and 2) deleting or replacing in previous correction rounds may incur information loss. we propose to introduce an early-stop mechanism to alleviate the efficiency issue. it takes the hidden representation of the end-of-sentence token (<@xmath0, and stops mpd in cases: 1) the next round's correction result matches the current correction result, or 2) the next round's correction result has a larger edit distance to the reference. as for the information loss issue, we present methods to merge the source sentence and the previous round's correction output into a single sequence, as pre-trained models normally do not have multiple encoders for more than one inputs. we evaluate our approach on the conll 2014 and bea 2019 gec shared tasks, and obtain significant improvements over the strong bart and t5 baselines, showing the effectiveness of our method. ## preliminaries: sequence-to-sequence gec the seq2seq model m comprises an encoder and a decoder. it takes the input sequence x to correct, and generates the corrected sequence x. the encoder takes the input sequence x, and computes the contextual hidden state vectors h e : the decoder generates the hidden state h k d based on the encoder hidden states h e and the decoding history @xmath1: algorithm 1 multi-pass decoding with early-stop. input: input sentence to correct x, gec model m , early-stop classifier c e , maximum number of decoding rounds n, early-stop threshold τ ; output: corrected sentence y. the decoder repeats this process until the classifier produces the end-of-sequence token (<@xmath2. pre-training by reconstructing the corrupted text can compress the knowledge of large-scale corpus into model parameters. and fine-tuning pre-trained models (such as bart and t5) for gec can lead to better performance @xcite . ## conclusion we utilize multi-pass decoding to improve the performance of seq2seq grammatical error correction. we present an early-stop mechanism to alleviate the inference efficiency issue, and derive source information fusion approaches to address the source information loss issue. our experiments on the conll-14 test set and the bea-19 test set show that our approach can lead to significant improvements (+1.80, +1.35, +2.02 f0.5 scores for bart 12-2, large and t5 large respectively on conll-14 and +2.99, +1.82, and +2.79 correspondingly on bea-19) over strong baselines, showing the effectiveness of our method.