A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languages (2024)

Abstract: Large language models excel in text generation and generalization, however they face challenges in text editing tasks, especially in correcting spelling errors and mistyping.In this paper, we present a methodology for generative spelling correction (SC), tested on English and Russian languages and potentially can be extended to any language with minor changes. Our research mainly focuses on exploring natural spelling errors and mistyping in texts and studying how those errors can be emulated in correct sentences to enrich generative models’ pre-train procedure effectively. We investigate the effects of emulations in various text domains and examine two spelling corruption techniques: 1) first one mimics human behavior when making a mistake through leveraging statistics of errors from a particular dataset, and 2) second adds the most common spelling errors, keyboard miss clicks, and some heuristics within the texts.We conducted experiments employing various corruption strategies, models’ architectures, and sizes in the pre-training and fine-tuning stages and evaluated the models using single-domain and multi-domain test sets. As a practical outcome of our work, we introduce SAGE (Spell checking via Augmentation and Generative distribution Emulation).

## introduction recent advancements in large language models (llms) have shown impressive text generation and language understanding capabilities, evident in benchmarks like superglue @xcite , gem @xcite , bigbench @xcite etc. however, these models often encounter challenges when it comes to effectively addressing text editing tasks, particularly automatic correction of misspellings and mistyping. the automatic spelling correction (sc) task is well known, with traditional approaches using rules, dictionaries, or statistical models for spelling error detection and correction. however, the emergence of llms and generative techniques has introduced new possibilities and improved the effectiveness of sc. thus, this paper addresses the task of automatic generative sc across various domains and proposes the methodology tested on english and russian languages, which could potentially be extended to any language with minor changes. our research primarily studies natural orthographic errors, text misspellings, and their emulation during model pretraining. we explore the impact of these emulations on the model's abilities across different domains and model types. we leverage two different spelling corruption techniques. the first technique applies the statistical analysis of common errors, aiming to mimic natural human behavior when making mistakes. the second technique introduces the most frequent spelling errors, keyboard miss clicks, and a set of heuristics within the texts. we conduct experiments in both russian and english languages, employing different corruption strategies and model sizes during pre-training and fine-tuning. as a practical outcome of our work, we introduce sage (spellchecking via augmentation and generative distribution emulation)a comprehensive library for automatic generative sc. sage incorporates various generative models trained with our proposed methodology and includes built-in augmentation techniques. moreover, we release the data hub within the sage project, a valuable russian language resource consisting of novel open source datasets for spelling. rule-based spell checking is one of the most common approaches that rely on predefined rules and dictionaries for detecting and rectifying misspelled words. these resources can incorporate algorithmic error models such as longest common subsequence @xcite , levenshtein distance @xcite , or phonetic algorithms @xcite . statistical spell checking approaches employ machine learning algorithms to learn from extensive text corpora. these algorithms can identify common spelling errors and their corresponding corrections. some examples of statistical approaches include n-gram models @xcite , hidden markov models @xcite , part-ofspeech tagging @xcite and noisy channel model @xcite . generative sc is a promising spell checking approach that has shown remarkable results in recent years. such systems take into account the context, due to the architecture nature of llms such as seq2seq long short-term memory (lstm) @xcite , seq2seq bidirectional lstm @xcite , and state-of-the-art transformer models like bert @xcite , @xcite , etc. the paper @xcite presents multilingual translation models for paraphrase generation task. m2m100 models @xcite (manyto-many multilingual models) effectively translate source language text into a target language that aligns with the source language. given the m2m100 models' comprehensive understanding of multiple languages, their utilization in spell checking tasks proves promising. in our research, among other investigations, we explore the suitability of the m2m approach for sc. datasets english spell checking research has received significant attention due to widespread english use, which results in the creation of spell checking datasets. evaluation datasets such as bea-2019 shared task @xcite , comprising corpora like fce @xcite , w&i+locness, lang-8 @xcite , and nucle @xcite , provide valuable resources for assessing spell checking and error correction tasks. neuspell @xcite introduced the bea60k natural test set and the well-established jfleg dataset @xcite , containing only spelling mistakes. other clean corpora, including the leipzig corpora col-lection @xcite and the gutenberg corpus @xcite , offer diverse sources such as news, web content, and books for further exploration in spell checking research. among the standard open source datasets for the russian language is ruspellru foot_2 , which emerged after the competition on automatic sc for russian social media texts @xcite . other open sources include the github typo corpus @xcite , which contains the russian section, and the recent work @xcite , which introduces a multi-domain dataset. text corruption methods for training generative sc models, building a parallel corpus is essential. there are several ways to emulate spelling errors or augment the existing datasets. the example is the gem benchmark and its associated augmentation library nl-augmenter @xcite and the work @xcite with the method for creating artificial typos. for the russian language, the rutransform framework @xcite presents adding noise into data through spelling corruption and @xcite proposes augmentation methods. ## methodology in this work, we aim to design models that meet the end users' demands. the broad application areas of sc tools, encompassing various orthographies and styles, pose additional challenges for text editing systems. we decided to enhance the conventional approach of treating standard language as the only correct spelling option. ## experiments we conducted a comprehensive series of experiments involving diverse spelling corruption strategies over the encoder-decoder generative models of different sizes throughout the pre-training and finetuning phases as well as zero-shot evaluation of the pre-trained models. the models' statistics are presented in ## conclusion in this paper, we have presented a novel methodology for generative sc. the approach involves emulating natural spelling errors during large generative model pre-training and has shown state-ofthe-art results in addressing text editing tasks. we use two augmentation techniques for text corruption to improve the results. conducting the experiments in two languages, we have demonstrated the effectiveness of these techniques and the impact of different corruption strategies across different domains. as for the research's practical impact, we proposed the library sage for automatic sc, including the russian data hub, proposed methods, and the family of generative models. the work contributes significantly to the sc field and opens routes for further exploration.