Identifying and analyzing ‘noisy’ spelling errors in a second language corpus (2025)

Abstract: This paper addresses the problem of identifying and analyzing ‘noisy’ spelling errors in texts written by second language (L2) learners’ texts in a written corpus. Using Python, spelling errors were identified in 5774 texts greater than or equal to 66 words (total=1,814,209 words), selected from a corpus of 4.2 million words (Authors-1). The statistical analysis used hurdle() models in R, which are appropriate for non-normal, count data, with many zeros.

## introduction the problem of 'noisy data' addressed in this paper is how to automatically identify and analyze spelling errors in texts written by speakers of english as a second language. this issue is important in automated scoring of written texts in high-stakes tests such as the internet based toefl (ibt) and duolingo english test (det). tests such as these use models that include numerous features, and these features may produce different values depending on whether they are considering correctly or incorrectly spelled tokens. thus, this paper reports on one method of identifying the rate of spelling errors in the written output of learners of english as a second language in an intensive english program (iep; juffs 2020) over time and addresses the optimal statistical method for measuring those errors. the first languages (l1s) of these learners varied in their orthographies from abjads (arabic), alphabets (spanish, korean), logographic characters (chinese), and a mix of logographic and syllabaries (japanese). at the time of data collection, the iep had three levels of proficiency with approximate equivalent cefr levels (common european framework of reference; council of europe, 2001) as follows: level 3 low-intermediate, cefr a2-b1; level 4 intermediate, cefr b1+-b2; and advanced, c1 and above. therefore, this corpus is representative of the population of iep students across the usa at the time of data collection between approximately 2007-2012. @xcite we note, however, that international student populations in us ieps vary somewhat by region and have varied over time from the 1960s until present.) english spelling poses special challenges because it uses a 'deep' orthography, meaning that the spoken sounds of english do not closely match their written forms and vice-versa. for example, the same sound /i/ is represented by different letters in 'ea' as in 'eat', 'e' as in the first 'e' in 'scene', 'ee' as in 'see @xcite specifically, our research questions were the following. in terms of the rate of spelling errors in learners' written texts: among open-access models used for spell checking, neuspell is trained on diverse datasets and uses contextual embeddings such as bert and elmo @xcite . symspell, though often considered a rule-based system, incorporates bigram look-ups to enhance context awareness, allowing it to resolve some ambiguous cases where single-word spell checkers might fail. similarly, jamspell incorporates a 3-gram language model to refine corrections based on surrounding words @xcite . unlike deep learning models, which infer spelling corrections from large corpora, symspell and similar models use a pre-compiled frequency dictionary to determine valid words and generate correction candidates efficiently @xcite ). the spell checker oriented word list (scowl; atkinson 2019) is one of the most widely used resources, providing a hierarchical lexicon of words categorized by frequency and linguistic validity. other resources, such as hunspell and aspell, also use wordlist-based approaches, making them highly efficient for misspellings but limited when handling real-word errors @xcite . for l2 learner errors, the choice of a spelling correction system is particularly important. rulebased systems offer a more conservative approach, as they avoid over-correcting errors that might be intentional learner choices or non-standard but comprehensible variants @xcite . in contrast, deep learning models, while highly accurate, may introduce unwanted corrections that mistake the intended choices in learners' interlanguage @xcite , particularly when trained on l1-english corpora. other proprietary systems, such as google's spell checker and microsoft's bingspell, remain inaccessible for customization, though they benefit from large-scale user data and adaptive correction mechanisms. therefore, in settings or applications focusing on learner data, a hybrid approach using open-source tools (e.g., using wordlist-based methods to avoid excessive intervention, supplemented by context-aware models for ambiguous cases) may be the most effective strategy @xcite . in high-stakes english proficiency assessments that implement automated scoring of writing, spelling accuracy is explicitly listed as a dimension of the scoring models (e.g., toefl, det, pte). however, details about the spelling error identification methods are scarce. although the problem of correcting spelling with computers has a long history, as far as we are aware, spelling errors in a second language written corpus in l2 english with various l1s have not been addressed. the pittsburgh english language institute corpus (pelic) is unusual in that it contains longitudinal data in addition to a variety of l1s. in addition, the appropriate statistical models for analyzing the rate of errors has not been determined. applied linguists are not just interested in computational detection and correction, but also in the potential qualitative impact of spelling errors on human graders, along with pedagogical implications. while the cited on-line spelling checking resources are coded in a variety of computer languages, for applied linguists who work with l2 data, python is the main programming language, and so python was used to provide accessibility to such researchers. a complete description of pelic spelling error identification and correction is provided at a public github repository and jupyter notebook @xcite , where links include the following resources which were used in this paper: related work in applied second language read-ing and spelling research has noted that for l2, the challenges of english orthography are compounded by the influence of their l1 writing systems and limited vocabulary size @xcite @xcite ). an important construct in this domain is lexical quality perfetti and hart 2002, which established the importance of strong links in the mental lexicon among sound (phonology), orthography, and meaning. poor links among sounds, graphemes, and semantics in any direction in lexical representations pose problems in both reading comprehension @xcite and writing production @xcite . moreover, baker and hawn 2022 raise the problem that computer-automated grading may unfairly disadvantage some groups, known as 'algorithmic bias' in education. thus, this work is innovative because it is a rare(?) example of explicitly interdisciplinary work drawing on computational linguistics in automatic spell-checking and correction, applied statistics, with insights from applied linguistics research on literacy and instruction. ## spelling identification spelling errors were identified using the following steps. first, scowl was consulted, and a scowl file was created and used to decide whether a word in the iep texts was 'real' or not (scowl list for pelic). all items were included from the lists except the abbreviations dictionary. words that had previously been considered 'non-words' by dictionaries were added to our list, for example, 'southside', which is a neighborhood of pittsburgh, 'frisbee', which is a toy/game, and 'onsen', which is a japanese loanword for 'hot spring'. all hyphenated words were included as real words, forexample, 'prize-winning'. after running the revised dictionary, a list of misspellings with their adjacent words was created, followed by a dictionary of misspelled items. examples in the dictionary of common misspellings included '*alot', '*becouse', '*sould', etc. a python module, symspell (garbe 2021a), was used that included the spelling errors and corrections for those errors. examples, of corrections made are 'beccuase' -> 'because', 'nise' -> 'nice', 'friendlly' -> 'friendly'. only word and bigram frequencies, but not full sentence context, were considered in resolving the appropriate target. this practice is consistent with other spellcheckers (hun-spell, pyspell, etc.). thus, following this common practice the accuracy of corrected tokens will not be 100%. nevertheless, the accuracy was inspected by random sampling and found that where the word is accurately spelled, the checker correctly does nothing 100% of the time @xcite , that is, there are no false positives. an important caveat is that incorrect spellings that are actual words, for example, the pronoun 'him' misspelled as the real word 'hem', are not corrected. such 'clang associates' @xcite are not counted as spelling errors in this automated process because it is difficult to automatically identify and correct misspellings that are real words. it might be possible to differentiate clang associates based on part of speech, for example, noun 'hem' vs. pronoun for 'him', or frequency of clang associate based on phonology, for example, 'ship' vs. 'sheep', but these possibilities have not yet been explored. nevertheless, based on the corrections, it was possible to programmatically count and tally the misspellings in each text in the database. these text-based counts were the basis of the data in the study. to control for number of errors by text length, the spelling errors were calculated by dividing the number of errors by the number of tokens in each text and multiplying by 100. because the appropriate statistical analysis requires whole numbers (no decimals), 0.5 was added to the result of all calculations before the number was converted to an integer. thus, 0.287 errors in a text remains 0, but a score of 0.847 became 1.347, and was converted to the integer 1. ## statistical models this section addresses the problem of the appropriate statistical model for non-normal count data with many zeros. models that permit inferential quantitative investigation of count data include the poisson family of analyses. zeileis, kleiber, and jackman 2008 provide a detailed review of poisson models that are both suitable and unsuitable for count data such as the spelling error data under consideration. two points about our spelling error data are relevant here for poisson analysis. first, standard poisson analysis for count data is inappropriate for over-dispersed data, that is, data with very large numbers of outliers. second, these data contain very large numbers of zeros, that is, texts without spelling errors -in fact over 50% for each l1 and level. in this context, crawley 2013 (chapter 14) also raised the problem of high frequency of '0' in count data. zeileis, kleiber, and jackman 2008 recommended the 'hurdle()' procedure for data with these characteristics. the hurdle() procedure is available in the (r package 'pscl') and is discussed in greater detail in the next section. ## results in addition to l1 and level, other available student information that relates to the data includes standardized proficiency scores of a placement test and writing sample on entry to the iep as well as self-reported biological gender. neither the placement score nor the writing sample scores correlated at higher than @xmath0.07 to the number of errors and were therefore not included in any model even though these correlations were reliable at p<.0001 due to the large n sizes. gender was also nonsignificant as a predictor. the percentage of texts with 0 errors are displayed in the results are provided in a reliable chi-square statistic for the interaction of l1 and level_id in the model (@xmath5, @xmath6.37, @xmath7.016) revealed that it contains frequencies of errors that are contingent on l1 and level. in addition, compared to an (inappropriate) standard poisson model, the akaike information criterion (aic) confirmed that the hurdle() model with negative binomial was a better fit. the significant log(θ) = -2.37, @xmath8.019 in the count data section in the hurdle model, having selected the count data with a lower limit of 1, then proceeds to model the number of texts by l1 and level with 0 errors, that is, the zero data. for zero hurdle model co-efficients, [p. 523]hoftstetter) state that "the zero model represents the probability of observing a positive count". in this case, the arabic level 3 intercept with errors is reliable: exp(-0.456) = 0.63, which means the odds of arabic speakers making a spelling error compared to zero is reliably negative 0.63, with to the right part of the chinese speakers' estimates at level 4 and level 5 compared to the (intercept) show a reliable difference, except this time in a positive direction. this result means that compared to texts with an error for arabic levels 4 and 5, the odds of the chinese level 4 and level 5 learner producing a text with even one error increases by odds of 1.52 and 2.23 respectively. at level 5, the spanish speakers also reliably increase the odds of making an error by 2.32 compared to the intercept, with only 73% of their texts at level 5 being error-free. no other comparisons with arabic-speakers' level 3 in the model are reliable. 1 ## discussion the differences by l1 are statistically reliable according to the chi square test on the entire model. thus, while spelling mistakes by all iep learners 1 it is possible to make multiple pairwise comparisons by changing the reference level from arabic to other l1s. however, given the limited number of errors and the similar means and dispersion statistics in with access to spell-checkers in word processing software are relatively low, they are noticeably and reliably different by l1. moreover, it is important to note that overall the learners improved in their accuracy over time. returning to the research questions, the results first demonstrated an effect for l1. when a text contains errors, arabic-speaking learners make more spelling errors than chinese-speaking, japanese-speaking, and korean-speaking (but not spanish-speaking) learners at level 3, but these differences disappear at levels 4 and 5. regarding texts with 0 errors, the pattern is similar, but the odds of chinese-speaking learners making an error remains somewhat higher at level 4 and level 5 compared to level 3 arabic speakers. thus, there is an interesting interaction and difference between the arabic-speaking and chinese-speaking learners such that arabic speakers decrease their proportion of errors in texts, while chinese speakers seem to be more stable compared to the arabic-speaking learners. taken together, a cautious interpretation of these results suggests the most reliable difference is between the arabic-speaking in contrast to the chinese-speaking learners as there are differences between these groups in both the count and zero hurdle models. while japanese-speaking, and korean-speaking learners at level 3 differ from arabic speaking learners producing texts with fewer errors, the zero model showed no differences among these three l1s. the spanish speakers make errors at a statistically similar rate to the arabic speakers. second, as to the effect of level of iep at 4 and 5, we can see that for texts with errors there is no effect. this means that the rate of errors in texts varies little across levels overall. however, numbers of texts with zero errors increases from level 3 to level 4 and remains steady at level 5. we may infer that use of the spell-checker with word processing skills improved along with knowledge of orthography, and especially for the arabic-speaking learners. third, interactions exist in the rate of errors among arabic-speaking learners. a decrease occurred from level 3 to level 4, but not for other l1s, indicated by the interaction of level for level 4 with numbers of zero error texts. in addition, for the zero-count data, chinese speakers showed an interaction at levels 4 and 5, indicating that the number of zero error texts remained more constant compared to arabic level 3. figure 1 shows that errors by chinese speakers at levels 4 and 5 remain higher, while other l1 error rates declined. to some extent, this outcome is reassuring for automated scoring of many features, for example those related to lexical sophistication @xcite , because automated measures of lexical sophistication, for example, advanced guiraud @xcite , based on word-processed texts will not unduly penalize one group at intermediate and high-intermediate levels, for example, arabic-speaking learners, by excluding misspelled low-frequency words, that is, words with a frequency band higher than 2000 at a higher rate than other l1s. this possibility had been suggested by @xcite but now seems to be less of a concern based on this analysis. this confidence is possible due to the low number of statistically significant differences among the groups and the low numbers of errors per text overall. the group most at risk would be the arabic level 3 learners, who made the most errors. specifically, automatic scoring of lexical sophistication measures derived from frequencies of lemmas in an external corpus will not be affected by learners losing credit for too many misspelled words above the 2k frequency band at intermediate levels and above. however, arabic-speaking learners' errors may be salient to human raters compared to other l1 groups. this impression arises from the visualization of the data, even if it is not statistically robust, because of the numbers of texts that contain outlier tallies of spelling mistakes. although the l1 effect is only statistically reliable at level 3, the tendency is very noticeable on a qualitative level at levels 4 and 5 also. such a pattern of errors could cause human raters to negatively perceive arabic speakers' writing, when only 61.2% of their texts at level 3 are error free compared to chinese speakers' 76.5%. thus, in high stakes testing where both human and computer-based automatic scoring are deployed, spelling errors have spelling errors have the potential to create bias against one group, even though those learners 'know' the items in question. moreover, these spelling errors (even when using word processing software), and the evidence from the reading studies cited in the introduction, are indicative of wider problems with lexical quality perfetti and stafura 2013 at the low-intermediate level (level 3). thus, these data support interventions with spelling for all l1s, perhaps especially at the low-intermediate stage at the early stages of learning. when spellcheckers highlight many words -including proper names not frequent in english -it may be difficult for learners to guess which words are misspelled and, perhaps more importantly, which ones are the correct replacements. in fact, due to the saliency of spelling errors reported previously by dunlap 2012 in student transcriptions of their own speech, one iep instituted a dictation component as part of its curriculum to address lexical quality. this decision is given additional support by these data and other studies such as those reported in humaidan and martin 2019. ## qualitative review of 'noisy' errors this section provides a qualitative review of the type and frequency of orthography mistakes in these word-processed data to complement the quantitative analysis based on automatic tagging in the previous section. this review provides additional insights into these 'noisy' data that vary by l1 and proficiency. the process through which this was done was that the first author, an experienced english as a second language teacher, reviewed all the spelling errors in the texts. thus, the list is not exhaustive but provides some indication of the challenges that learners face. the mistakes fall generally into four categories: (i) mistakes many learners make with frequent words; (ii) errors across l1s with the use of english spelling conventions; (iii) those forms influenced by l1 morpho-phonology; (iv) forms flagged as errors even though they are correct, for example, the (now sadly outdated) blend 'brangelina' or abbreviations, for example, nhk, cbs (japanese and us tv stations), and rmb (= renminbi, the chinese currency). in the first category, regardless of l1 and level, many learners made mistakes with some frequent words, for example, 'because' (the range of misspellings of this word is very large) and 'studying', with the 'y' plus 'ing' creating uncertainty. errors flagged due to spelling conventions of english double consonants were also frequent across learners both at morphological boundaries, for example, *writting, *eightteen, *eatting, vs. *regreting, and within words, for example, *recomendation, *profesion vs. *bussiness. unsurprisingly, given the different double consonant spelling rules in spanish, spanish-speaking learners made many errors with double consonants. second, errors influenced by l1 morpho-phonology seemed especially frequent at level 3. caution is in order as some could be simply 'typos', and others of these errors could be spacing problems that are influenced by english chunks (e.g., many learners made a mistake with *alot) or reduced stress on functors such as indefinite articles. however, arabic speakers seemed to produce more with pronouns such as *iowe, and *idid, in addition to more frequent lack of spacing between indefinite articles and nouns (e.g., *anest and *abeach). in addition, trilled /r/ pronunciation could plausibly have produced *bearrd. omitted vowels by arabic speakers at the lower levels are also quite frequent, even in common words, but especially with liquids /l/ and /r/, for example, *evry and *evrybady, but also other words *amrica and *cmfortable. such omission may be attributed to the influence of the abjad orthography, which only marks some vowels in arabic and which also affects reading l2 reading in english @xcite . because arabic lacks the phoneme /p/, there is also an occasional, predictable voicing error in orthographic 'p' vs. 'b' (e.g., *laptob). for chinese speakers, it is possible to identify errors due to syllable structure constraints in mandarin, which disallows consonant clusters (some possible with glides) in onsets and permits only alveolar and velar nasals in syllable final position. potential examples of such influence include epenthesis (e.g., *@xmath9. in general, chinese, japanese (e.g., *toraditional 'traditional'), and korean speakers (e.g., *zebara 'zebra') seemed more likely to insert a vowel compared to the arabic speakers from the examples reviewed in the data. however, such qualitative observation would need to be quantitatively confirmed with inferential statistical analysis. finally, all l1s showed influence of vowel quality pronunciation on spelling, especially the [ae] as in 'cat' vs. @xcite in 'ketchup', for example, korean level 3, *@xmath10. (we note that a merger between these two sounds may be occuring in some english varieties in australia and in the northern cities of the usa near the great lakes.) schwa [@] in unstressed syllables also caused learners to have problems in identifying the correct grapheme, for example, chinese level 4 *@xmath11. some diphthongs (e.g., @xcite for japanese *@xmath12, but less so. the impression from this review of specific errors is that the influence of l1 morpho-phonology on spelling accuracy was more evident at level 3. this finding suggests that because learners do not control the pronunciation of the word, it is harder for them to evaluate choices provided by the spellchecker. this difficulty is due to their own representation of meaning to sound, being influenced by l1 phonology, is stronger than the link from meaning to orthography. thus, this qualitative review suggests a possible developmental trajectory of orthographic accuracy from influence of l1 pronunciation on spelling accuracy at lower proficiency progressing to greater challenges based on proper nouns, abbreviations, and longer, less familiar technical words at the higher levels. such an impression requires careful review of the whole data set to be supported quantitatively and faces the challenge of reconstructing the source of each error, which is a non-trivial task. it is worth re-emphasizing that a limitation to this analysis is that we do not account for 'clang' effects in the spelling data (e.g., *sees, *case, and *scene for a target such a 'cease') which we found in responses to prompts in reading vocabulary test data @xcite or 'hem' vs. 'him', which actually occurred in the corpus. automated spelling correction cannot correct words that are actually in the dictionary without further refinement of the correction algorithms. it is possible that were clang effects included in the analysis as spelling errors, the results would be different. ## conclusion this paper considered the problem of identifying and measuring orthographic errors in a written iep corpus by five different groups of l1 speakers across three levels of proficiency. python coding enabled the identification and enumeration of errors. using statistical models for overdispersed count data, the findings are that at the low-intermediate level, arabic speakers make errors at a significantly higher rate per text than peers from some, but not all, l1 groups at the same level of proficiency. these l1 differences are reduced at intermediate and high-intermediate/advanced levels, with chinese-speaking learners changing some-what less than other groups in the proportion of texts with spelling errors. gender was not found to be significant, perhaps because so many of the arabic-speaking learners were male and hence it is difficult to tease apart this factor from l1 influence. the statistical models did not demonstrate important differences in the groups in spelling errors at higher proficiency levels, which should be seen as a positive outcome for automatic assessment. however, the large numbers of errors by arabic l1 students in some texts could create a perception that they are worse than other l1 groups, when in fact they are not at levels 4 and 5.