An Extended Sequence Tagging Vocabulary for Grammatical Error Correction (2023)

Abstract: We extend a current sequence-tagging approach to Grammatical Error Correction (GEC) by introducing specialised tags for spelling correction and morphological inflection using the SymSpell and LemmInflect algorithms. Our approach improves generalisation: the proposed new tagset allows a smaller number of tags to correct a larger range of errors. Our results show a performance improvement both overall and in the targeted error categories. We further show that ensembles trained with our new tagset outperform those trained with the baseline tagset on the public BEA benchmark.

## introduction current approaches to grammatical error correction (gec) fall under two broad categories: sequence-to-sequence and sequence-tagging. the former treats gec as a machine-translation problem, "translating" from error-containing to errorfree language (yuan and briscoe, 2016; schmaltz et al., 2017; junczys-dowmunt et al., 2018; grundkiewicz et al., 2019; yuan et al., 2019; rothe et al., 2021). by contrast, sequence-tagging approaches tag each input word with an edit operation such that applying the operations produces the corrected output (yannakoudakis et al., 2017; awasthi et al., 2019; omelianchuk et al., 2020; tarnavskyi et al., 2022). the basic operations include keeping a word unchanged, deleting a word, and inserting new words (awasthi et al., 2019; malmi et al., 2019). one advantage of sequence-tagging over sequence-to-sequence approaches is computational efficiency: the former do not require expensive auto-regressive decoding, foot_0 and currently achieve competitive performance using smaller models (tarnavskyi et al., 2022; rothe et al., 2021). however, current sequence-tagging approaches require manual linguistic efforts to curate languagespecific edit tags (yuan et al., 2021). for example, awasthi et al. (2019) introduce rule-based morphological inflection tags, like replacing the "-ing" suffix with "-ion" (e.g. @xmath10. omelianchuk et al. (2020) introduce a wider range of operations including verb-form and nounnumber changes. for verb-form inflections, they use a dictionary to map between verb forms. @xcite in this paper, we focus on a sequence-tagging approach. we extend the approach of omelianchuk et al. (2020) by introducing more general transformation tags (figure 1 ). specifically, we introduce: these modifications allow a broader range of errors to be handled by a smaller number of transformation tags, which simplifies the sequence tagging problem, as well as improves the generalisation of the gec system. our results show that our modifications improve the system's performance on the bea-2019 development and test sets. our code and model weights are publicly available. @xcite ## methods we extend the system described by omelianchuk et al. (2020) by adding new tags to the model's output vocabulary and modifying the inference and dataset preprocessing code to support our new tags. our new tags perform spelling correction and morphological inflection and are described in sections 2.3 and 2.4 below. we evaluate our tagset using the roberta (liu et al., 2019), deberta (he et al., 2021b), debertav3 (he et al., 2021a), elec-tra (clark et al., 2020) and xlnet (yang et al., 2019) encoders, as well as an ensemble of three encoders (see section 2.6). ## results we report the span-based precision, recall and f 0.5 scores on the bea-2019 development and test sets (bryant et al., 2019) using the errant scorer (bryant et al., 2017). @xcite the term "basetags" indicates the tagset proposed by omelianchuk et al. (2020), and @xmath0inflect denote our proposed tagsets containing the spelling and inflection tags respectively. @xmath1inflect denotes tagsets containing both the spelling and inflection tags. we adopt the model and tagset size notation of tarnavskyi et al. (2022) which, for example, denotes a deberta-large model using a 5k vocabulary size as deberta (l) 5k . differences in error-type frequencies in the development and test sets do not provide an explanation, since the frequency of spelling errors is lower in the test set than in the development set, and the frequencies of the error types which the $inflect tagset most impacts 13 are higher in the test set than in the development set (bryant et al., 2019, ## conclusions we have motivated and described new tags for spelling correction and morphological inflection. these tags are capable of correcting a broader range of errors than previous tags, thereby improving generalisation. our results show that the new tags improve performance both in the targeted error categories and overall for both single-encoder models and ensembles. @xcite specifically the noun:num, verb:form and verb:sva error types. see section a.7 for details. our findings ultimately show there is great scope for improving gec sequence-labelling model performance by introducing tags capable of correcting more general and possibly complex classes of errors. finally, we believe our results are of immediate value to practitioners building gec applications since they offer improved performance without the use of seq-to-seq models which can require orders of magnitude more computation at inference time.