Preserving the Authenticity of Handwritten Learner Language: Annotation Guidelines for Creating Transcripts Retaining Orthographic Features (2023)

Abstract: Handwritten texts produced by young learners often contain orthographic features like spelling errors, capitalization errors, punctuation errors, and impurities such as strikethroughs, inserts, and smudges. All of those are typically normalized or ignored in existing transcriptions. For applications like handwriting recognition with the goal of automatically analyzing a learner’s language performance, however, retaining such features would be necessary. To address this, we present transcription guidelines that retain the features addressed above. Our guidelines were developed iteratively and include numerous example images to illustrate the various issues. On a subset of about 90 double-transcribed texts, we compute inter-annotator agreement and show that our guidelines can be applied with high levels of percentage agreement of about .98. Overall, we transcribed 1,350 learner texts, which is about the same size as the widely adopted handwriting recognition datasets IAM (1,500 pages) and CVL (1,600 pages). Our final corpus can be used to train a handwriting recognition system that transcribes closely to the real productions by young learners. Such a system is a prerequisite for applying automatic orthography feedback systems to handwritten texts in the future.

## introduction when looking at the educational landscape, particularly with children, handwriting remains a prevalent mode of writing. as shown in figure 2 , handwritten texts contain various features such as strikethroughs, inserts, spelling errors, and smudges, which can provide additional information beyond the actual text about the writing process and the writer's skills. when handwritten texts are transcribed, e.g. to make them accessible to digital analysis, there is always a loss of information involved, as we need to abstract from the source depending on the intended use. different applications may require different levels of abstraction, depending on the focus of the analysis. this is similar to the transcription of spoken language, where depending on the application it may or may not be necessary to retain e.g. filler words or pauses. in the case of handwriting, a quite common abstraction is the normalization of orthographic errors. for example, if the texts are analyzed for aspects like vocabulary, thematic coherence, or reader-orientedness @xcite , retaining spelling errors in the transcripts is not necessary and may even hamper the analyses. in contrast, preserving spelling errors in the transcripts would be crucial to assess orthographic competence and yet other analyses may require even more information from the handwriting, e.g. what pieces of information were added to a sentence after it was finished (see figure 2 for examples of such inserts). another task with special requirements concerning the transcripts is handwriting recognition (hwr). to achieve accurate hwr, it is crucial to have reliable training data that closely resembles real handwriting transcriptions which are directly linked to the corresponding image. the requirements of the different tasks may be conflicting. for example, for analyzing text coherence, inserted pieces of text should be transcribed where the writer intended them to appear. in contrast, to serve as training data for hwr, the inserts have to be transcribed at the position where they were written in the text. furthermore, transcribers often need to make decisions that affect later analyses. see for example figure 1 , where two letters are written on top of each other ('s' and 's', where schüler 'student' with a captial 's' would be the correct spelling). transcripts of handwritten texts are often produced in the context of a particular project with specific goals. however, it is a very time-consuming task requiring a lot of manual effort. it would be much more sustainable to provide a transcript that is broad enough to cover multiple use-cases. ## related work over the last years, numerous datasets of texts produced by language learners have been compiled. for example, some datasets aim to provide authentic records that do not normalize orthographic deviations, especially if the frequency of these deviations is negligible. others aim at normalizing orthographic deviations to facilitate semantic analysis of the texts. a good illustration of the approach to preserving the authenticity of handwritten manuscripts can be found in the transcription guidelines outlined in @xcite , which serves as a (comprehensive) exemplary model for the transcription of historical documents. the guidelines prioritize a detailed transcription of the handwriting, without any amendments to obvious errors in spelling or punctuation that might result in changes to the meaning. the detailed and timeconsuming nature of these guidelines allows the preservation of a significant amount of information. furthermore, they enable the creation of a transcript that can be analyzed with a focus on specific aspects, such as the differentiation between comments from individual authors or the use of different writing tools. for handwritten learner content, the grow in grammar (gig) corpus, which is documented in @xcite , comes with transcripts and a detailed transcription manual. although not focusing on hwr, the main goal was to create an authentic record 1 https://github.com/catalpa-cl/ learner-handwriting-recognition of what the learner wrote. however, annotations are often not precise enough to be usable for hwr. for example, in the case of strikethroughs, the complete sentence was flagged instead of indicating the exact position of the crossed-out words. furthermore, the image data is not available. becker-mrotzek and grabowski (2018) released the fd-lex dataset comprising images and their corresponding transcripts, i.e. the two key components for hwr. however, the transcripts have been orthographically normalized to focus on diagnosing and promoting sub-components of writing competence. in a recent work @xcite , the datasets gig and fd-lex were both comparably used to analyze the development of writing in english and german children across school grades. although these extensive datasets were created, as orthographical errors were not present in both data, a deeper analysis of these differences could not be made. in contrast to these datasets, several datasets targeting hwr exist. iam @xcite and cvl @xcite are widely adopted in the hwr community and are frequently utilized for comparing recognition performance across various methods. they consist of image data with different segmentation levels such as text-line or word level and align with the corresponding transcripts. however, these datasets are non-learner datasets, as the texts were written by skilled writers and merely transcribed from provided texts, resulting in minimal amounts of orthographic errors. none of the datasets had all three components -image data, a properly aligned transcript, and a transcript that retained orthographic errors -available, despite the wide range of datasets that were examined. ## handwritten learner data for our objective of exploiting a learner handwritten dataset for hwr, as described in @xcite , we choose the dataset of fd-lex @xcite . the data set consists of texts from two different german school types (gymnasium and integrierte gesamtschule) foot_1 at two different learner levels (5th and 9th grade). the fd-lex corpus consists of 5,628 texts from 938 learners (i.e. on average 6 texts per student). the images in fd-lex are colored scans of white din-a4 paper with ruled lines and a header that includes the writer's id. this layout is consistent throughout the entire dataset, with only a few exceptions such as rare writings on the backside or a blank white page. figure 2 shows an example scan from this dataset. the data from fd-lex were collected in compliance with the relevant data protection regulations. thus, the data were processed in such a way that the privacy and anonymity of the participating schools, classes, and students were preserved. no individual or group can be identified from the processed data, except for the fact that certain cases belong to the same school class or educational level. the anonymized transcripts provided by becker-mrotzek and grabowski (2018) normalize orthographic errors so that they cannot be directly used for our purposes. we thus had to re-transcribe the data according to our developed guidelines (preserving spelling errors, punctuation errors, and other orthographic peculiarities) as described in the next section. ## transcription guidelines the main goal of the guidelines is to ensure that the transcription reflects exactly what is written by the learner -i.e. orthography is not corrected -and where. in cases of doubt, it is necessary to reconcile what the child has written or intended to write with what a machine transcription would read. this involves careful consideration of the context and a deep understanding of the learner's level of proficiency. the transcription process should prioritize preserving the integrity of the original text and capturing the nuances of the learner's writing style, while also ensuring that the final output is legible for the handwriting recognition task. in order to ensure consistency in the transcription process, transcribers are required to write the transcription in excel. it is mandatory to turn off automatic error correction and automatic capitalization correction for the beginning of a text. the transcript should contain the following columns: name of the image, line number, status, content, and comment. the status column should be set to either 'ok', 'dis' (discussion), or 'err' (error). the 'dis' status indicates that the transcription requires further review, while the 'err' status indicates that the line should be disregarded. next, we will provide more specific guidelines on how to transcribe certain elements which are accompanied by examples in figure 3 : spacing inexperienced learners often struggle with producing consistent spacing in their writing. it is not uncommon to find instances where a particular letter is spaced differently from the rest of the word, necessitating the use of curly brackets for the transcription. moreover, it is crucial to identify whether the letter is at the beginning or end of the word. this is represented by placing a space character within the curly brackets too. compounding words can present further challenges, as learners may inadvertently leave excessive gaps between the constituent words or use insufficient spacing. (example c: undzwar) strikethrough if learners did not want a particular part of their content evaluated, they crossed it out. these strikethrough elements are transcribed with hashes (#). in the transcript, the number of hashes represents approximately the number of letters that were struck through. (example d: ###### #...) insert when a learner wanted to add content afterwards, the person used inserts. a small number of words or letters to be inserted are usually located at the targeted position and are transcribed in curly brackets with a "less than" symbol on the left of the content (example e1: weil {<er} zu). if an insert is dislocated, the targeted location is tagged using the word "insert" in curly brackets, followed by the number of the indirect insert on the page and the signaling character (often asterisks are used), if there is one ({insert1}). the insertion content is tagged likewise with the preceding insert1 and if present, a signaling character. (example e2: sep.{insert1} ein unfall passierte {insert1 wie}) regular punctuation mark in accordance with grammatical rules, regular punctuation marks such as stops (.), commas (,), and exclamation marks (!) are placed directly adjacent to the last written word. however, it should be noted that learners may sometimes place them differently, e.g. with more spacing, which is then ignored. tally marks in some cases, the learner had to count the written words and marked them with tally marks '|'. these are transcribed in curly brackets according to the direction of the stroke, followed by an ampersand. (example f: nur {/&} das) separation of a word into two words one type of correction made by the writer is adding a separator between two words that were originally written together because the learner intended them to be separate afterwards. both words are transcribed separately and a separation sequence '|-' is placed into curly brackets. (example g: zu {|-} sehen) overlay another correction made by a learner is the overlaying of letters. in this case, both letters are placed in curly brackets and connected by a plus sign: {f+f}. the correct letter is written to the left of the plus and to the right is the incorrect one. (examples h: {f+f}enster; au{f+e}; auß{ß+ss}erdem) irregular letter we found some special letters like letters with additional artifacts or even unusual versions of letters. these are transcribed with a plus sign to the right within curly brackets like: {d+}. (examples i: {d+}ie; {e+}s) emoticon / smiley despite a large number of different emoticons, we decided to transcribe every emoticon in curly brackets with the same icon: ('u+1f642'). drawing a few learners put down larger drawings extending over several lines. if there is text before as well as after the drawing, each of the drawn lines are given an error status, and they are transcribed as three hashes (###) and a comment with a reference to the drawing. in the same style, if no text follows below the drawing, only one line is added to the transcript. ## transcription analysis in this transcription project, a total of 1,350 handwritten learner pages were transcribed, resulting in about 13,300 lines of text in total. a subset of about 90 pages was transcribed by two annotators and a gold transcription was created by an adjudicator for improved accuracy. ## summary and related research findings in order to make handwritten texts available to automatic analyses such as an automatic feedback system for spelling errors, the texts need to be transcribed first, whereby all necessary features such as spelling errors need to be retained. a hwr system that automates such transcriptions needs images and corresponding transcripts as training data. since no such dataset yet existed, we manually re-transcribed 1,350 pages of the learner dataset fd-lex, while maintaining the authenticity of the handwritten texts and preserving nonnormative writing practices. we developed comprehensive transcription guidelines to address issues such as spelling errors, indistinct characters, word separation, drawings, and special signs like tally marks. the transcription process resulted in a corpus that can be transformed using two converters into a version for hwr and a continuous text for content assessment. to ensure consistency, about 90 pages were doubletranscribed, yielding a high iaa of about .98 at the character level. we also investigated the frequency of certain nonnormative writing practices and highlighted the benefit of having an authentic record of young learners' texts. based on this work, we were able to investigate handwriting recgonition of learner texts when orthographical errors are supposed to be retained @xcite . in this subsequent study, we used 1,350 of the transcribed pages of the fd-lex dataset for training a handwriting recognizer and tested it on the gold transcription of the double-transcribed pages. by incorporating a language model and a dictionary that we automatically enriched with possible spelling errors, we were able to improve the recognition performance and to retain spelling errors in the transcripts.