Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora (2023)

Abstract: Grammatical error correction (GEC) is the task of correcting typos, spelling, punctuation and grammatical issues in text. Approaching the problem as a sequence-to-sequence task, we compare the use of a common subword unit vocabulary and byte-level encoding. Initial synthetic training data is created using an error-generating pipeline, and used for finetuning two subword-level models and one byte-level model. Models are then finetuned further on hand-corrected error corpora, including texts written by children, university students, dyslexic and second-language writers, and evaluated over different error types and error origins. We show that a byte-level model enables higher correction quality than a subword approach, not only for simple spelling errors, but also for more complex semantic, stylistic and grammatical issues. In particular, initial training on synthetic corpora followed by finetuning on a relatively small parallel corpus of real-world errors helps the byte-level model correct a wide range of commonly occurring errors. Our experiments are run for the Icelandic language but should hold for other similar languages, and in particular to morphologically rich ones.

## introduction spelling mistakes due to typos and rushed writing, nonstandard punctuation and spelling, and grammatical and stylistic issues are common to almost everyone who writes any kind of text. this applies in any language and can distract the reader or make the communication miss its mark. this can hinder people who have difficulties writing text conforming to a particular language standard, be it due to disability, dyslexia, linguistic background, limited access to education or any other reason. prejudice against people whose writing deviates from the standard can make some shy away from communicating with others, leaving their voices out of important discussions and restricting their opportunities @xcite . grammatical error correction (gec) is the task of adjusting a text's spelling, grammar, and linguistic style to conform to an approved language standard or convention @xcite . while the latest work on gec is based on transformer models @xcite , the subword tokenization methods commonly used in these models are a source of problems when it comes to typos and other variants @xcite . subword tokenization @xcite was presented as a solution to the open vocabulary problem, as it is a compromise between character encoding and whole word encoding, enabling unknown words to be represented using known subwords. however, a significant downside of subword tokenization is how it is affected by noisy input; if a word contains a typo or other spelling variants, this can completely shift its representation. in addition, in languages with rich morphology, a word can have many different surface forms, some rarer than others, that all carry the meaning of the base word, but appear in different syntactic contexts. a subword-tokenized model may struggle to capture the nuances of such a language effectively since it may need several different subwords to represent a single word, depending on spelling and context. when an unfamiliar variant of the word appears in unseen text, the model is challenged to decode it correctly, even when it results in uncommon subword units. our motivation is that a byte or character-level approach should intuitively be more robust to spelling or morphology variations, as it is not constrained by the subword vocabulary. we explore using a byte-level architecture, byt5 @xcite , for correcting everything from typos to complex grammatical issues in text. the language studied is icelandic, a highly-inflected north germanic language. for instance, the morphological complexity in icelandic means that nouns can have up to 16 different surface forms, and adjectives over 50. gec for a morphologically complex language needs to go beyond correcting only single words or limited phrases; it needs to consider the syntax of the whole sentence. this is the case for icelandic but also for other languages with rich morphology, such as arabic, hebrew, polish, basque, lithuanian and hungarian, to name a few. we compare the performance of the byte-level architecture to two subword-based architectures; byt5's predecessor, mt5 @xcite , and an mbart @xcite model that has been pretrained further on both icelandic and english. we employ real and synthetic error data for training, and present models and a framework for error generation methods that can be adapted to other languages. for under-resourced languages such as icelandic @xcite , using synthetic training data makes neural training for gec a viable option. our main contributions include a comparison between subword tokenization and byte-level tokenization for gec when training over a combination of curated and synthesized data. we demonstrate how byte-level models not only bypass subword-related issues, but can also correct long-range errors in text. we release our error generation framework as well as models for gec using byte-level and subword tokens in icelandic. while our work focuses on the icelandic language, we have no reason to believe that similar results do not hold for other languages, particularly those similar to icelandic in terms of morphological complexity. ## related work the bulk of research on grammatical error detection and correction has been focused on english and english learner texts, due to existing training data and benchmarks, and the large market of english learners worldwide who benefit from an automatic language correction tool @xcite . while spelling and grammar errors appear in every language, each language has its own set of error types that are more common than others, due to different phonetic, morphological and syntactic characteristics. ## results different metrics exist for evaluating gec performance, but most are language-specific, and have not been adapted to icelandic. here we employ a language agnostic metric for scoring our models, the gleu score @xcite . gleu foot_5 is a variant of the bleu @xcite score used to evaluate machine-translation output. it has been modified to account for both the source and the reference, by rewarding overlap between the source and the target sentence, and penalizing n-grams that should have been changed, but were not. when evaluating gec for english, errant @xcite is commonly used. it is a spanbased correction method that uses the f 0.5 metric, where precision weighs twice as much as recall. though this metric has not been implemented for icelandic, we also report errant scores using a language-agnostic approach, disregarding error types and only reporting the span-based f 0.5 scores for each test set. these results are shown in we consider a variety of curated and synthetic test sets to get a good overview of the differences between the byte-level and subword-level approach for gec. for the real errors, we report scores over the iceec.test set, the test set from the iceec, which contains around 5000 sentences. in con-trast, the dyslexic, l2 and children test sets contain 500 held-out sentences each from the respective specialized error corpora described in section 3.1 (only 100 examples were collected for the dativitis error type, a rarer occurrence in the data). we also annotated a small dataset (163 sentences) of data from an icelandic news outlet (news), where each sentence contains at least one error; this is further described in appendix b. for the synthetic errors, we report gleu scores over the test.synth set, which contains around 4000 held-out sentences from the synthetic data. furthermore, we generated test sets of synthetic examples, each containing a particular error type in each sentence (dativitis, spaces, commas, dupl-words, mood, rand-noise, noun-case). this last group of test sets was generated using source texts that, while editorial, may include other errors, just like the synthetic training examples. the models, as they get better, learn to correct these errors as well. this may paradoxically lower the gleu score as the corrected output deviates from the erroneous reference. these generated test sets still provide valuable information about what the models learn about each error type in isolation. to understand what approach is best suited for gec we trained the models on different data combinations and using different pre-trained models. the synth-100k models are all trained for 100k updates on the same synthetic data, and the synth-100k-ec models are additionally finetuned on the curated iceec error corpus. to provide a baseline for the gleu scores, we also report no_corr scores, where the source text is not corrected. this gives some idea of the noise level of the test sets, with test.synth being the noisiest and iceec.test containing the least noise. the greynircorrect @xcite (gc) results were produced by applying greynircorrect to the test sets in its two configurations; correcting single-word errors on the one hand (spell.) and all errors found on the other (all). the gc system was developed in part to focus on the error categories and error frequencies defined in the iceec training data, and this may be reflected in the scores for the iceec test sets (asterisked in their performance on the curated error corpora is somewhat lacking. in contrast, models trained only on the curated error corpora (ec) generally produce somewhat better gleu scores on the curated error corpora than the synth 100k models, but do not generalize to the error categories presented in the synthetic test sets. they are also unable to correct multiple errors in a single sentence (test.synth). training on the synthetic data and then finetuning on the curated error corpora (synth100k/ synth550k+ec) performs best on the curated errors and retains much of the performance on the synthetic test sets. in all of these experiments, we can see that the byt5 models generally perform better than the subword counterparts. this is also reflected in the errant scores in appendix e, ## discussion & conclusion our results show that the byt5 models are the overall high-scorers on the real-world test sets, and on most of the synthetic ones. we include finetuning results on the byt5 model that has been trained for longer on the synthetic data (550k updates) to compare how performance improves with time. we see the gleu scores keep going up with time, and more importantly, when taking a close look at the actual generated output, this is the model that best corrects real-world errors. this makes it the most feasible option for use in real-world scenarios. a comparison of the output of the models trained on both data sources is shown in appendix d. an example from the test data is when the subword-tokenized model mbart-enis-synth100k+ec incorrectly changes the name of a person from a rare name ("láretta") to a more common one ("lára"). this kind of error is not seen in the byte-level model, which is quite conservative in its corrections of unknown entities. while this means byt5 occasionally misses actual errors, we find that it is much better suited for production than a subword-level model that makes serious semantic errors. these more nuanced error correction examples may not be fully captured by the automated metrics, but are crucial for real-world use. the subword regularization experiments are included as an alternative approach for mitigating the subword tokenization problem. the results are marginally better than the model without subword regularization when trained on the synthetic data, and the model performs better than the byt5-synth100k model in the case of dupli-cate words, which linguistically is a quite trivial task, and in more intricate mood errors. it however doesn't do any better than the mb-isen-synth100k trained without subword regularization on the curated datasets, and this also holds when the model is finetuned additionally on curated data. the model finetuned on only the curated data with subword regularization (mb-isen-reg-ec) however performs consistently much better than its counterpart without subword regularization, often on par with or surpassing byt5. this model has not seen any of the highly noised synthetic data, and thus has the most to gain from the subword noise. we speculate that this is one of the reasons we don't see more gains from adding subword regularization; the training examples are already so highly noised that there is not much to be learned from the added subword noise. the iceec finetuning data contain real-world errors which have been hand-corrected. these texts are somewhat different from the highly noised training examples with synthetic errors, have fewer errors on average and are more varied as they are naturally occurring. they also include stylistic edits from the reviewers, which improve the text's fluency, but in those cases the original is not necessarily incorrect as per the language standard. with these differences in mind, we expect the models to have to forget some of the synthetic error correction task in order to adapt to this "new" denoising task. we see this happen in the mbart-enis finetuning on the curated data, and to a lesser extent in the byt5 finetuning. the denoising task performance on the synthetic errors from the previous step has in part been lost, which is expected, since some of these errors are not particularly common in real texts. for the more grammatically complex error-types in the synthetic data (dativitis and changes to noun cases and verb moods), we find that the mbart-enis trained on synthetic data generally does well; for some subsets even surpassing the byt5 counterpart that was finetuned on curated corpora. we suspect that this has to do with the linguistic knowledge the model has already gained during its pretraining on icelandic texts, as explained in appendix a. the byt5 model that was trained for longer however manages to surpass it on the mood error type, indicating that it is still adapting to the icelandic language, alongside its primary denoising task. the models trained on only the finetuning data perform the worst throughout. the results show that they do not manage to correct the synthetic categories much beyond the baseline, except for mbart-enis in some cases. we expect that this has to do with their extra knowledge of icelandic and the denoising objective used in the synthetic error correction finetuning. the results for these models on the curated in-domain test sets are in fact mostly on par with the models finetuned on the synthetic data only. looking at the generated output, we see that the error types these models correct are not the same as those that the syntheticonly models are able to correct, which is expected, as they are trained on different data. we conclude that adopting a byte-level approach rather than a subword approach leads to best results for the task of gec, at the very least in the case of a morphologically rich language such as icelandic. finally, we find that the optimal way of capturing a wide range of errors is to train on a combination of synthetic and curated data, particularly when the curated data is limited.