A tailored Handwritten-Text-Recognition System for Medieval L atin (2023)

Abstract: The Bavarian Academy of Sciences and Humanities aims to digitize the Medieval Latin Dictionary. This dictionary entails record cards referring to lemmas in medieval Latin, a low-resource language. A crucial step of the digitization process is the handwritten text recognition (HTR) of the handwritten lemmas on the record cards. In our work, we introduce an end-to-end pipeline, tailored for the medieval Latin dictionary, for locating, extracting, and transcribing the lemmas. We employ two state-of-the-art image segmentation models to prepare the initial data set for the HTR task. Further, we experiment with different transformer-based models and conduct a set of experiments to explore the capabilities of different combinations of vision encoders with a GPT-2 decoder. Additionally, we also apply extensive data augmentation resulting in a highly competitive model. The best-performing setup achieved a character error rate of 0.015, which is even superior to the commercial Google Cloud Vision model, and shows more stable performance.

## introduction the medieval latin dictionary (mlw) 1 deals with latin texts that were created between 500 and 1280 in the german-speaking region. the foundations for this project have been developed from 1948 onwards and since then, the dictionary has been continuously published in individual partial editions since 1959. the basis of the dictionary consists of 50 selected texts that have been fully transcribed onto din-a6 record cards (cf. fig. 1 ) constituting about 40% of the note material. later, another 2,500 texts were excerpted and transcribed manually onto din-a6 record cards, using a typewriter. in addition, there are so-called "index cards", a type of record card, that helps to uncover often 1 in german: mittellateinisches wörterbuch (mlw) hundreds of additional references. in total, it is estimated that 1.3 million reference points have been recorded for the mlw. these record cards were sorted alphabetically by the first letter of the keyword (lemma), and serve as the foundation for creating a dictionary. around 200,000 record cards have been scanned and annotated with their respective lemma. the accurate extraction and transcription of the lemma present a challenge, which is further compounded by the limited resources available for medieval latin. our contributions are as follows: (1) we present a novel end-to-end htr pipeline specifically designed for detecting and transcribing handwritten medieval latin text. notably, it surpasses commercial applications currently considered sota for related tasks. ## related work we provide an overview for htr, which is the main challenge of this work. for object detection, which is an intermediate step of this work, we refer to @xcite for a detailed overview. connectionist temporal classification (ctc) @xcite is a technique in which a neural network -initially a recurrent neural network (rnn) but other networks might also be used @xcite -is trained to predict a matrix of conditional transition probabilities. the input image, represented as a vector representation through a convolutional neural network (cnn), is fed to the network, and for each input (i.e. the activation maps of the cnn) the network predicts the character. ctc, combined with cnns and rnns, often yielded competitive results, such as shown by @xcite and @xcite . furthermore, approaches applying only cnns and ctc also exist @xcite . easter2.0 achieved competitive results on iam @xcite , a frequently used htr data set consisting of english handwritten text. a recent work that achieved sota results on iam is trocr @xcite , based on the transformer @xcite , consisting of a vision encoder and a text decoder. this deviated from previous approaches where primarily cnns and rnns were used. this development is closely linked to the emergence of the transformer in the vision domain @xcite . @xcite introduce another transformer model also using ctc, with the main difference to trocr being a different embedding technique for visual features based on a cnn. the results have also been shown to be competitive on the iam data set. @xcite compare encoderdecoder models' performance on htr, using different models in the encoder and decoder parts, e.g. a transformer encoder plus a ctc-based decoder. furthermore, they found that enriching this architecture with a language model yields sota results on iam. the trocr framework has already been successfully applied to historical data akin to our task. @xcite fine-tune a trocr instance to handwritten latin from the 16th century @xcite to as gwalther), achieving competitive results. ## data our data set comprises 114,653 images, holding 3,507 distinct lemmas. all images are in rgb, but not uniform in size. the information on the corresponding lemma is available on the image level. most lemmas start with the letter "s", followed by a large number of lemmas starting with the letters "m", "v", "t", "u", "l", and "n". we observe lemmas from a length of one character up to 19 characters, with an average length between five and six characters. a total of 2,420 lemmas (69%) appear on ten record cards or less; 854 lemmas (24.4%), on 10 to 100 record cards, and just 233 lemmas (6.6%) on more than 100 record cards. 1,123 lemmas (36.7%) only occur on one card. ## conclusion and outlook since the record cards include much more information than the one we extracted, we recommend further research into various extraction techniques. with the recent publication of segment anything model, @xcite introduce a model that might be able to extract further features from the record cards with much higher accuracy. we present a novel end-to-end pipeline for the medieval latin dictionary. our library includes an image-detection-based model for lemma extraction and a tailored htr model. we experiment with training different configurations of transformers using the vit, beit, and swin encoders while using a gpt-2 decoder. employing data augmentation, our best model (swin+gpt-2) achieves a cer of 0.015. the evaluation of the results exhibits a weaker performance on longer lemmas and on lemmas that appear less frequently in the training data. further experiments with generative models to produce synthetic data (not reported in the paper) were not successful, however, we recommend further research into this direction. to conclude, our approach presents a promising htr solution for medieval latin. future research can build upon our work, and explore its generalizability to other languages and data sets by making use of our pip-installable python package.