Spelling Correction for E stonian Learner Language (2023)

Abstract: Second and foreign language (L2) learners often make specific spelling errors compared to native speakers. Language-independent spell-checking algorithms that rely on n-gram models can offer a simple solution for improving learner error detection and correction due to context-sensitivity. As the open-source speller previously available for Estonian is rule-based, our aim was to evaluate the performance of bi- and trigram-based statistical spelling correctors on an error-tagged set of A2–C1-level texts written by L2 learners of Estonian. The newly trained spell-checking models were compared to existing correction tools (open-source and commercial). Then, the best-performing Jamspell corrector was trained on various datasets to analyse their effect on the correction results.

## introduction it has been proposed that tailor-made spelling error correction systems are best suited for language learning purposes because the spell-checking tools developed for proficient users often prove unable to correct specific mistakes, like real-word errors, i.e., errors that result in a valid homonym; diacritic errors; or pronunciation-induced errors possibly with a large edit distance (e.g., @xcite . whereas it is costly to develop rule-based error correction systems with learner-oriented explanations, or neural spell-checking systems that require vast quantities of training data comprising authentic or synthetic errors, statistical spelling correction algorithms which use n-gram language models to analyse context could form a simple starting point for improving error detection and correction of l2 learner writings. in this language-independent approach, only a corpus of (presumably) correct language use samples is needed to train the system. currently, the only open-source spell-checker developed for estonian language is vabamorf foot_0 . it is a lexicon-and rule-based library created by filosoft ltd. at the beginning of the 1990-s alongside a commercial speller distributed in microsoft word @xcite . the spellers make use of a lexicon and a list of typing misspellings to assess candidate corrections but they do not appear to rely on context in their suggestions. for evaluating statistical spelling error detection and correction on estonian learner language, we first used peter norvig's approach that generates all possible spelling corrections by different edits, such as character deletions, insertions, replacements, and transpositions @xcite . the procedure is repeated to get correction candidates with two edits. the probability of candidates is estimated based on a unigram language model derived from a language corpus. we used a bigram language model in addition to a unigram model to add context-sensitivity. second, we applied the compound aware version of symmetric delete spelling correction (symspell) foot_1 . the algorithm searches for candidate corrections with an edit distance of 1 or 2 based on deletions only, increasing the speed of spelling correction. a corpus-based bigram dictionary can be used, however, bigrams are only considered in ranking suggestions if no suggestions with an edit distance of 1 are found for a single token. realword spelling errors are currently not corrected by symspell @xcite . third, we tested jamspell 3 that additionally uses a trigram language model for selecting the highest-scored correction candidate. jamspell is based on a modified symspell algorithm, optimized for speed and memory usage, so that the spell-checking library can process up to 5,000 words per second. we compared the algorithms with three existing spell-checking tools: vabamorf, and the commercial spellers offered by ms word (microsoft 365) and the google docs application. the latter uses neural machine translation @xcite . ## test data and evaluation the correction output was evaluated on a set of 84 error-annotated proficiency examination writings from the estonian interlanguage corpus foot_3 . divided between four proficiency levels (a2, b1, b2, and c1), the texts contain 1,054 sentences, 9,186 words (excluding anonymized identifiers), and 309 spelling errors in total. we distinguished simple spelling errors and mixed errors, i.e., spelling mistakes co-occurring with another error such as word choice, inflectional form, or capitalization error. the error distribution is given in error detection is the first step of error correction. nevertheless, to achieve high performance in error detection, the proposed edits do not have to match the gold standard annotation, as opposed to measuring error correction performance. we evaluated both spelling error detection and correction based on three metrics: the f0.5 score was preferred to the harmonic mean (f1 score) due to the assumption that an error correction system's reliability is rather reduced by false and needless corrections than unproposed corrections (see @xcite . we verticalized the system output and automatically compared it to the test set to detect changes and correction matches. since l2 learners may not select the correct option from a list of suggestions (e.g., @xcite and such selection cannot be implemented in an automated workflow, e.g., using spell-checking as a pre-processing step of grammatical error correction, we prioritized the speller's accuracy of defining the best correction. thus, we focused on the highest-ranked suggestion. the cases of mixed errors were reviewed manually to find partial corrections fixing only the spelling of an otherwise erroneous word (e.g., *parnu∼*pärnu instead of pärnu, which is an estonian town name and should be capitalized). both full and partial word corrections were considered in calculating the evaluation metrics. ## comparison of spell-checking tools the training material for building new statistical spell-checking models came from the estonian national corpus (enc) 2019, which includes web corpora downloaded from estonian websites as well as the estonian reference corpus, wikipedia corpora and the corpus of estonian open access journals (doaj) @xcite . jamspell and norvig's spelling corrector were trained on a random sample of 6 million sentences and over 82 million words retrieved from the reference corpus that represents the "standard" varieties of estonian -mostly newspaper texts but also fiction, science and legislation texts from 1990-2008. the sample constitutes nearly half of the reference corpus; increasing the volume of the training set did not improve the correction results. symspell, on the other hand, reached the best results with a uni-and bigram frequency dictionary based on the full enc 2019 containing over 1.5 billion words. even then, it performed poorly compared to other tools, especially in terms of recall. the comparison of spelling error detection and correction by the different applications is summarized in tables 2 and 3. ## jamspell correction models we experimented with different training data to see if we can improve jamspell's efficiency in learner spelling error detection and correction. the training sets are listed in the results of spelling error detection and correction obtained on the previously used test set are presented in tables 6 and 7. the system edit distribution is provided in the comparison of the jamspell models reflects the well-known trade-off between precision and recall. the highest error detection and correction precision were achieved by the model trained on estonian web 2019. it was the least likely to make unnecessary corrections but also to detect words with a spelling error, thus having the lowest recall. at the same time, the initial model trained on a reference corpus sample scored highest in error detection and correction recall, being able to identify and correct the largest amount of spelling errors. the latter model featured the best f0.5 score in error detection, whereas the web 2019 model had a slightly better f0.5 score in error correction. in terms of spelling error detection, the 10:1 reference + web sample offered a compromise, yielding a higher precision than the reference corpus model and a higher recall than the estonian web model. this resulted in the second best f0.5 score. on the other hand, there was little variation in the error correction f0.5 score. the performance obtained with the 10:1 reference + web the choice of the most suitable model depends on the purpose -whether we want to maximize the amount of errors detected and corrected, minimize the amount of needless corrections, or find a middle ground. for this, combining a larger proportion of standard texts with a smaller proportion of web material seems the best suited. in summary, the results are promising compared to the precision and recall of learner spelling error correction accomplished in other languages (e.g., @xcite . three best-performing jamspell models have been made available for use as a part of the new estonian spelling and grammatical error correction toolkit currently in development foot_6 . ## conclusion and perspectives this study has demonstrated the benefit of statistical context-sensitive spelling correction for processing l2 learner writings. jamspell that uses trigram contexts of words for spell-checking could correct real-word errors and other learner-specific spelling errors more efficiently than other tested open-source spellers. in spelling error correction, it also outperformed ms word speller, achieving precision and recall comparable to google's corrector. in spelling error detection, its performance was similar to ms word's and better than google's. the evaluation of different jamspell correction models revealed that using a web corpus as training material increases error detection and correction precision, while using a reference corpus increases recall. we consider the current correction models a decent baseline for further development. their performance could be improved, e.g., by employing learner spelling error frequency data or namedentity recognition to avoid false name edits and enable correction of name capitalization. we acknowledge that the results might have been different if we had implemented norvig's spell-checking algorithm on a trigram language model. the tested spell-checking tools and models should also be evaluated on a larger errorannotated set of writings by l2 learners as well as native speakers. such a gold-standard dataset of approximately 8,000 sentences is in development for estonian. expectedly, context-sensitive spelling correction also benefits proficient language users, although the difference in performance may not be as outstanding.