Recognizing Learner Handwriting Retaining Orthographic Errors for Enabling Fine-Grained Error Feedback (2023)

Abstract: This paper addresses the problem of providing automatic feedback on orthographic errors in handwritten text. Despite the availability of automatic error detection systems, the practical problem of digitizing the handwriting remains. Current handwriting recognition (HWR) systems produce highly accurate transcriptions but normalize away the very errors that are essential for providing useful feedback, e.g. orthographic errors. Our contribution is twofold:First, we create a comprehensive dataset of handwritten text with transcripts retaining orthographic errors by transcribing 1,350 pages from the German learner dataset FD-LEX. Second, we train a simple HWR system on our dataset, allowing it to transcribe words with orthographic errors. Thereby, we evaluate the effect of different dictionaries on recognition output, highlighting the importance of addressing spelling errors in these dictionaries.

## introduction early l1 learners typically write by hand, even in the digital age, and handwriting remains important @xcite @xcite . automatic feedback on error types in learner language is available (laarmann-quante, 2017; @xcite , but faces the practical problem of having to digitize the handwriting first. current handwriting recognition (hwr) systems yield very good results @xcite @xcite with one crucial problem: they typically normalize away the orthographic errors @xcite that are important for giving useful feedback to learners. in figure 1 , when humans read this handwritten word, they look at the shapes of the letters to form hypotheses. the first letter(s) could be a d or a cl and we decide about this informed by a hypothesis about the whole word. in this case, we see that it is probably supposed to be dounut, so the first letter is a d. we see that there is an extra letter u at the third position which we ignore for forming our hypothesis about the word, but still recognize so that we could give a learner appropriate feedback about it. automatic handwriting recognition systems are typically trained and evaluated on handwritten text along with transcripts that do not contain orthographic errors. many hwr systems contain a language model component @xcite that is used to further normalize the output. as a result, hwr systems yield 'clean' transcripts without any orthographic errors (right branch in figure 1 ) that cannot be used to give feedback on orthographic errors. instead, we need hwr systems outputting transcripts that retain orthographic errors (middle branch in figure 1 ). in this paper, we tackle this problem by first creating a dataset of handwritten text with transcripts retaining orthographic errors. for that purpose, we created comprehensive transcription guidelines @xcite that precisely define our transcription goal. this is necessary as handwritten text contains other artifacts beyond orthographic errors, such as strikethroughs or inserts that we need to transcribe. in total, we transcribe 1,350 handwritten pages from german learners and thus create a dataset that is comparable in size to widely used english datasets like iam @xcite and cvl @xcite . given this dataset, we are then able to quantify to what extent existing baseline systems are unable to transcribe handwritten text, especially if we only use the underlying character recognition probabilities. we compare this with training the hwr system on parts of our data, enabling it (in theory) to learn to correctly transcribe words with orthographic errors. furthermore, we change the dictionary used in the hwr system to also include systematic learner errors created by an automated generator. note that providing the actual feedback is outside the scope of this paper. here, we focus on analyzing the problem of turning an image of handwritten text into a digitized transcript, which is currently the main obstacle to applying existing feedback methods on a scale. ## existing datasets for training and evaluating a handwriting recognition system that retains orthographic errors, we need a dataset combining images of learner handwriting with transcripts containing orthographic errors. to our knowledge, no such dataset exists. iam and cvl are mostly in english and are often used to evaluate handwriting recognition systems. iam in its version 3.0 is an extensive dataset and consists of about 1,500 pages with more than 13,000 text lines written by 650 adults, with different segmentation levels and corresponding transcripts. cvl is comparable to iam with about 1,600 pages from 310 adult writers. the set consists of six english and one german text and thus has a slightly increased alphabet as the german umlauts (ä, ö, and ü) are included. in comparison to iam, it is only transcribed word-wise, ignoring most punctuation marks or strikethrough words, although a segmentation of text lines is available. the growth-in-grammar gig dataset @xcite is a learner dataset that retained orthographic errors. however, the corresponding image data is not available. in contrast to gig, fd-lex @xcite is another learner dataset with published image data. in comparison to iam and cvl where the participants copied a presented text by hand, this dataset consists of texts that were freely written based on a picture or a short story, and thus, more errors were made. albeit, the transcripts from the fd-lex dataset normal- in conclusion, none of the existing datasets fulfills our need for available image data and a transcript containing orthographic errors. ## dataset creation as no suitable dataset is available, we need to build one. we decided to use the german learner corpus fd-lex as a starting point, as it already contains scans of learner handwriting with a sufficient number of orthographic errors. looking at the example in figure 2 , we can see additional typical challenges for automatic handwriting recognition e.g. strikethroughs and inserts. fd-lex was built as a corpus for analyzing the writing competence of learners. it covers two different german school types: gymnasium (gym) ('academic track school') and integrierte gesamtschule (igs) ('comprehensive school') from two grades (5 th and 9 th ) each. it has about 5,600 scanned color pages from about 940 children and is thus exceeding the iam (1,500 pages) and cvl (1,600 pages) datasets in size. a detailed listing can be seen in ## baseline experiments to track our recognition performance improvements, we create a baseline by training a straightfor-ward handwriting recognizer on our dataset. commonly, the performance of the recognizer is evaluated with two metrics, namely character error rate (cer) and word error rate (wer). while cer gives numerical feedback on how many characters have been misread by the recognizer, wer measures how many words are different from the gold-standard transcription. this means that lower values indicate better recognition performance. for the purpose of this paper's focus on word-level analysis, we will concentrate on wer rather than cer. ## decoding with dictionary constraint most research and publicly available databases for hwr pertain to adults. in these cases, spelling errors are typically ignored because they are estimated to be rare and not important to be kept in the output. therefore, the predicted words can be mapped to a large dictionary of possible words, which has been shown to yield better recognition rates, as recognition errors can be eliminated this way @xcite . ## spelling error generator to approximate the lower bound (see section 5.2), spelling variants must be added to the dictionary. thus, we generate possible (systematic) spelling errors based on the procedure described in laarmann-quante (2016). we generate possible misspellings for all words in the childlex and fd-lex dictionaries. the error generation procedure works as follows: a correctly spelled word is automatically enriched with linguistic information such as phonemes, syllables, and morphemes, based on the web service g2p of the bavarian archive of speech signals (bas) (reichel, 2012; reichel and kisler, 2014) 9 , see also @xcite for more information about these annotations. the information is then used to analyze (via a set of rules) which systematic errors could be made on this word. by systematic we mean that particular principles of german orthography are violated, e.g. consonant doubling (*komen for kommen, eng.: 'to come') foot_8 , a syllabic principle, or final devoicing (*walt for wald, eng.: 'forest'), a morphological principle (see @xcite for the theoretical framework). we also generate errors reflecting the overuse of such principles, e.g. *walld for wald. errors that cannot be explained via such principles (such as a seemingly random omission of a letter as in *wad for wald) are not generated because there is an infinite number of ways in which a word could be misspelled. we assume, however, that using the systematic errors in the sense described above, should capture most of the errors that the pupils commit because they are the major obstacles when learning how to spell in german. in total, 57 different error categories can be generated (not all apply to each word, though, while some words may contain multiple instances of the same error category, e.g. when there are two doubled consonants in one word such as wasserfall, eng.: 'waterfall'). the error categories that can be generated can be found in laarmann-quante et al. (2019b). 11 of course, more than one error can be committed within a word. we account for this by including all possible combinations of up to 2 systematic errors that apply to a word. including all possible error combinations would lead to an exponential increase of misspellings to consider, most of which will be highly unlikely, though. ## conclusion and further work in this paper we tackled the issue of retaining orthographic errors when automatically recognizing learner handwriting. this is a prerequisite for giving automated feedback on spelling performance based on handwritten texts. we created a handwriting recognition dataset of german learner texts based on the fd-lex dataset by transcribing 1,350 pages using new transcription guidelines. the utilization of a dictionary to restrict the output resulted in an improvement of our baseline. furthermore, our results indicate that incorporating generated spelling errors leads to an improvement in recognition performance at the word level, with the error rate decreasing from 35% to 25%, representing a decrease of 10 percentage points. although we were able to cover 94% of the originally used words using a spelling error generator, the huge number of words in the dictionary raises questions about its practicality. therefore, one of the next goals should be to allow more probable errors while avoiding overwhelming the dictionary. therefore, further analysis is necessary to determine which errors were made by learners in fd-lex and which ones were addressed by the generated errors. this information can be used to reduce the size of the error set by eliminating unnecessary or rare errors. additionally, an analysis of com-mon error combinations can aid in generating more targeted errors while avoiding redundant ones. furthermore, the focus of this study was not on improving the recognition model itself. however, recognition improvements could be made by implementing a more sophisticated model like full page recognition as introduced by @xcite .