Spelling Correction for R ussian: A Comparative Study of Datasets and Methods (2021)

Abstract: We develop a minimally-supervised model for spelling correction and evaluate its performance on three datasets annotated for spelling errors in Russian. The first corpus is a dataset of Russian social media data that was recently used in a shared task on Russian spelling correction. The other two corpora contain texts produced by learners of Russian as a foreign language. Evaluating on three diverse datasets allows for a cross-corpus comparison. We compare the performance of the minimally-supervised model to two baseline models that do not use context for candidate re-ranking, as well as to a character-level statistical machine translation system with context-based re-ranking. We show that the minimally-supervised model outperforms all of the other models. We also present an analysis of the spelling errors and discuss the difficulty of the task compared to the spelling correction problem in English.

## introduction the spelling correction task has been a fundamental natural language processing (nlp) problem ever since the origins of the field and has enjoyed a lot of attention in the nlp research. it is not surprising, since correcting spelling mistakes is of practical relevance for various higherlevel nlp tasks and downstream applications dealing with noisy data, such as named entity recognition, dependency parsing, information retrieval, topic modeling, machine translation, essay scoring, speech recognition, automatic text correction @xcite . running a spellchecker is now a common pre-processing step performed in essay scoring @xcite , grammatical error correction @xcite @xcite and numerous other applications. nevertheless, even for en-glish, performance of spellchecking tools is not as good as one would expect, especially in noisy domains @xcite . @xcite evaluate three publicly available spellcheckers on english learner data and find that the highest recall achieved is that of 69%, and the best precision is 57%, which indicates that the task is far from being solved. further, their simple in-house implementation outperforms by a large margin all of the common publicly available spellcheckers. one reason for the slow progress on the task might be the lack of common benchmark datasets. as a result, proposed methods are being evaluated either on isolated spelling errors extracted from a corpus without context foot_0 or on artificially created datasets. recently, @xcite released a dataset annotated for spelling errors in english learner essays and provided an evaluation of a minimal supervision system that combines features based on the misspelled word itself and the context in which it appears. they report strong performance on that corpus, as well as competitive results on a dataset from the medical domain. we address the problem of correcting non-word spelling mistakes in russian, a language with rich morphology. our goal is two-fold: first, to implement various established spelling methods with known results for english and determine how they perform on russian. our second goal is to perform cross-corpus comparison, by evaluating on several russian datasets that contain diverse data (texts by native russian speakers in the social media domain, as well as russian learner texts). we implement four models. first, we use two baselines that do not take context into account: aspell spellchecker and the model proposed in @xcite that was shown to outperform aspell and several other grammar check-ers for english. we then implement two models that also take into account contextual information when proposing a correction: a statistical machine translation (smt) approach and a minimally-supervised method. the minimallysupervised model follows the approach of @xcite . this model is compared against a character-level smt spellchecker that takes context into account by incorporating a word-based language model (lm) as well as other context features at the re-ranking level @xcite . we evaluate these four methods on three russian datasets that contain annotated spelling mistakes. we perform a detailed error analysis and identify the challenges pertaining to the russian language. we show that even though spelling correction of non-word errors is considered to be an easy task, performance on a morphologically-rich language is challenging and leaves a wide gap for future research. this paper makes the following contributions: (1) we implement and evaluate four established approaches to spelling correction and evaluate these on three russian datasets; (2) we show that the minimally-supervised approach outperforms the other methods and is the most robust; (3) we perform error analysis identifying challenges of spelling correction for russian. section 2 reviews related work on the spelling correction of russian and on the established methods well-studied for english. section 3 describes the three datasets of spelling errors used in this work. section 4 presents the models. in section 5, we present the results, and in section 6 we perform error analysis of the results. section 7 concludes. ## related work a non-word misspelling is a spelling error, such that the resulting string is not a valid word in the language. this is different from real-word (context-sensitive) errors, for example confusing "their", "there" and "they're" (wilcox- @xcite . context-sensitive errors also subsume grammar errors made by non-native speakers (e.g. confusing "a" and "the"), but these typically are addressed using a different set of methods @xcite . most of the spelling correction research has been focused on the english language. when dealing with a language that has rich morphology, such as russian, specific challenges may arise. for example, the rich morphology of russian, as we show, affects the candidate generation algorithm, where a substantially higher number of competing candidates is being generated, including those that are morphological variants of the same lemma. there is very little spelling work on other languages with complex and diverse morphology. for instance, @xcite ; @xcite ; @xcite address a variety of errors in arabic, including grammar and usage errors, but they do not focus on spelling. previous studies on russian spelling mainly addressed correcting spelling errors in search queries @xcite , which is a special subtask of spelling correction, as the surrounding context for candidate selection is not considered or is considered in a quite restrictive way. @xcite introduced the first competition on spelling correction for russian, which focused on correcting texts collected from russian social media websites. @xcite presents a follow-up study, where they show that the use of morphological information for candidate selection is beneficial for languages with well-developed morphology, such as russian. we use the corpus released in this competition and show that it is quite different from the other two corpora used in this work. ## datasets we use three russian datasets annotated for misspellings. the first one, rulec-gec, is a learner corpus collected at the university of oregon and consists of essays written by learners of russian as a foreign language and heritage speakers @xcite . the dataset was corrected and annotated by native russian speakers and is error-coded. it is annotated exhaustively for various grammar and usage errors, and contains a large proportion of spelling errors, especially for heritage speakers (over 42% of all errors), and over 18% of all errors in the foreign group. we only focus on mistakes that are marked as spelling errors. the corpus is partitioned into training, development, and test. since we focus on the spelling errors, we evaluate only with respect to those mistakes and ignore other annotated errors in the data. the second corpus, henceforth ru-lang8 (trinh and rozovskaya, 2021), is a dataset collected from the online language learning platform lang-8 @xcite and annotated by native speakers. the dataset contains texts by learners of a variety of foreign languages. the annotation is publicly available for research. ru-lang8 contains 54,000 tokens split up into development and test partitions. we only use the test partition in this work for evaluation, as the models are developed and tuned on the rulec-gec data. ru-lang8 differs from rulec-gec: the latter consists of essays written on a university setting in a controlled environment, while the lang-8 data was collected online; the majority of texts are short paragraphs or questions posed by language learners. ru-lang8 is thus more informal and contains data by learners of multiple first language backgrounds (unlike rulec-gec, whose authors are from the united states). the third corpus, ruspellru, is a dataset released as part of the competition on automatic spelling correction for the russian language, which focused on social media texts. the dataset is a collection of essays from russian blogs and social media. this is another unique dataset, very distinct: it contains a lot of colloquialisms, slang expressions and social media spelling conventions @xcite . since the corpus contains social media texts, the misspellings include, in addition to typos, a lot of slang and colloquial forms common in social media spelling, such as the use of digits inside the words or unconventional spellings, e.g. using phonetic spelling instead of standard one. statistics on the datasets, including the total number of tokens as well as the spelling error rates (percentage of tokens containing a spelling error), are shown in finally, in ## the models in this section, we describe the minimallysupervised model, the character-level smt speller, and the two baselines that do not use context. ## results in all cases, the models are trained on the rulectrain corpus and tuned on the rulec development data. all results are reported on the test partitions of the three datasets. key results of the minimally-supervised model on the three datasets are shown in the results of the minimally-supervised model and of the other models implemented in this work are shown in ## error analysis candidate re-ranking we perform error analysis of the candidate re-ranking component that is part of the minimally-supervised approach. from each dataset, we analyze 100 errors, on which an incorrect candidate is preferred. results are shown in ## conclusion in this paper, we implement four models for spelling correction for russian and evaluate these on three diverse datasets that contain spelling mistakes. we present a comparative analysis of spelling mistakes contained in the three datasets. evaluation results show that the minimallysupervised model outperforms two baseline models that do not use context when selecting a candidate correction, and another model that uses a character-level smt and a language model in re-ranking. we perform feature ablation of the minimally-supervised model showing that contex-tual information contributes to the performance. we also carry out error analysis that reveals that one common source of errors in russian in selecting the appropriate correction candidate is the presence of morphological variants. this study should provide insight into the spelling correction problem for languages with rich morphology.