Document-level grammatical error correction (2021)

Abstract: Document-level context can provide valuable information in grammatical error correction (GEC), which is crucial for correcting certain errors and resolving inconsistencies. In this paper, we investigate context-aware approaches and propose document-level GEC systems. Additionally, we employ a three-step training strategy to benefit from both sentence-level and document-level data. Our system outperforms previous document-level and all other NMT-based single-model systems, achieving state of the art on a common test set.

## introduction grammatical error correction (gec) attempts to automatically detect and correct grammatical errors in text. with recent advances in sequenceto-sequence modelling, neural machine translation (nmt) has been widely applied to gec @xcite @xcite and state-of-the-art results have been reported @xcite . cross-sentence context has proven useful for language modelling @xcite , dialogue systems @xcite and machine translation @xcite . in error correction, we observe that certain errors can only be detected and/or corrected using wider context, which may fall outside the current sentence. however, existing gec systems typically process each sentence independently, ignoring document-level context. these sentencelevel systems may fail to correct document-level errors (e.g. verb tense errors, pronoun errors, runon sentences) or propose inconsistent corrections throughout a document: for example, all the errors (red) in example (a) could feasibly be corrected (bold) using either the present or past tense if we only consider the target sentence in isolation, but the wider context reveals the correction using the present tense is ungrammatical (strikethrough). similarly, a sentence-level system is also unable to handle cases where sentences should be merged such as in example (b). to date, gec evaluation has always been carried out at the sentence level. as a result, successful corrections of these document-level errors would not be given any credit (or even be unfairly penalised) and systems proposing inconsistent modifications would not be penalised. on the one hand, gec should look beyond the current sentence and use more context to build context-aware gec systems; on the other hand, systems should be better evaluated at the document rather than sentence level. this paper makes the following contributions. first, we compare different architectures to capture wider context for nmt-based gec and show that simple document-level approaches can be applied to improve gec performance. second, we present a three-step training strategy to effectively use both sentence-level and document-level parallel data for gec. third, we report state of the art on a publicly available test set. finally, we perform the first document-level gec evaluation and release our document-level evaluation scripts to facilitate research in the area. 1 in this work, we use the transformer sequenceto-sequence model @xcite as our baseline system and investigate three context-aware extensions for gec. ## comparison with nmt-based gec systems we perform sentence-level evaluation on the fcetest and conll-2014 test sets using the m 2 scorer @xcite . a comparison of nmt-based single model systems is made in ## conclusion we have investigated document-level approaches to nmt-based gec and presented a three-step training strategy to use both sentence-level and document-level data. we have shown that context is useful in gec but very long context is not necessary for improved performance. experiments on three test sets demonstrated the effectiveness of our document-level gec models. our best system outperforms all nmt-based single-model gec systems and achieves state of the art on fce-test. by drawing attention to this understudied area in gec, we hope to motivate future efforts to build better context-aware gec systems. we have also performed the first document-level gec evaluation and make our document-level evaluation scripts available to facilitate research in this area. viewers for their useful feedback. we would also like to thank shiva taslimipoor, christopher davis, andrew caines, and ted briscoe for feedback on early drafts of this paper. this work was performed using resources provided by the cambridge service for data driven discovery operated by the university of cambridge research computing service, provided by dell emc and intel using tier-2 funding from the engineering and physical sciences research council (capital grant ep/p020259/1), and dirac funding from the science and technology facilities council. we acknowledge nvidia for an academic hardware grant. a corpus statistics split dataset #tokens #sents #docs doc. length (avg. #sents) train fce-train 454,736 28,350 2,116 13 w&i-train 628,720 34,308 3,000 11 nucle 1,161,567 57,151 1,397 41 clc 28,988,729 1,961,065 206,418 10 dev fce-dev 34,748 2,191 159 14 test fce-test 41,932 2,695 194 14 bea-dev 86,973 4,384 350 13 conll-2014 30,144 1,312 50 26 table a.1: summary of datasets used in our experiments. all datasets were preprocessed using spacy. foot_6 b hyper-parameter settings model architecture transformer (big) max tokens 3,584 optimiser adam (β 1 = 0.9, β 2 = 0.98, = 1 × 10 -8 ) learning rate schedule same as in vaswani et al. (2017) loss function label smoothed cross entropy ( @xmath0.1) dropout 0.3 gradient clipping 1.0 beam size 12