Uncovering the Handwritten Text in the Margins: End-to-end Handwritten Text Detection and Recognition (2024)

Abstract: The pressing need for digitization of historical documents has led to a strong interest in designing computerised image processing methods for automatic handwritten text recognition. However, not much attention has been paid on studying the handwritten text written in the margins, i.e. marginalia, that also forms an important source of information. Nevertheless, training an accurate and robust recognition system for marginalia calls for data-efficient approaches due to the unavailability of sufficient amounts of annotated multi-writer texts. Therefore, this work presents an end-to-end framework for automatic detection and recognition of handwritten marginalia, and leverages data augmentation and transfer learning to overcome training data scarcity. The detection phase involves investigation of R-CNN and Faster R-CNN networks. The recognition phase includes an attention-based sequence-to-sequence model, with ResNet feature extraction, bidirectional LSTM-based sequence modeling, and attention-based prediction of marginalia. The effectiveness of the proposed framework has been empirically evaluated on the data from early book collections found in the Uppsala University Library in Sweden. Source code and pre-trained models are available at Github.

## introduction libraries and archives across the globe are in possession of rich cultural heritage collections to be digitized for preservation and preventing degradation over time. for example, the uppsala university library in sweden maintains several early book collections, dating back to the 1400s. an example of such a collection is the walleriana book collection, encompassing medicine and science (uub). ## related work since the advent of deep neural networks, the htr research has witnessed significant advancement in method design and development, with popular approaches such as transformers based architectures trocr @xcite and attention-based sequence to sequence models @xcite @xcite . our previous work @xcite introduced an end-to-end htr system based on attention encoderdecoder networks, where the attention-based architecture is simple, modular, and reproducible, allow-ing more data to be added in the pipeline. there have been other attempts at automatically reading historical handwritten documents @xcite . for example, (aradillas et al., 2020) discusses the challenge of dealing with different styles of handwriting. since two documents can be from completely different centuries and countries, there is bound to be a lot of variability in the handwriting. the solution that is presented by the authors is to use transfer learning. first, a network is trained on a large set containing handwritten text of modern english. this base network can then be tweaked, depending on what kind of text will be processed. for example, if the network is to be used for a certain collection, a subset of this collection can be used to further train the network. doing this can make the network significantly better at making predictions on that collection. the benefit of this technique is that since some collections can be very small, it can be difficult to train a network solely on that collection without running the risk of over-fitting. transfer learning handles this problem by first creating a good general model, that is then tweaked and specialized to a certain dataset. another related work @xcite presents a joint line segmentation and transcription approach for end-to-end handwritten paragraph recognition. to do so, handwritten text is automatically segmented into individual lines by using multi-dimensional long short-term memory recurrent neural networks, or mdlstm-rnn for short. however, the problem of reading the text in the margins is relatively under-explored @xcite . the work (bold and wagstaff, 2017) highlighted the importance of marginal annotations for the community and how it benefits the readers and historians. for example, for historians working with texts, marginalia can provide insights into earlier readers and their perspectives. there have been some attempts at studying the marginalia notes and drafts by moby-dick author herman melville in @xcite @xcite . these works focus on the visualization of marginalia, gender research and knowledge exploration for instance. on the other hand, our proposed work focuses on developing advanced computational methods for automatic detection and recognition of marginalia, leveraging modern deep learning models and our reproducible research. ## methodology the methodology is divided into three parts: localizing the marginalia (using r-cnn and faster r-cnn); segmenting the found text; and attentionbased text recognition pipeline. in the following section, the methods that have been used to accomplish these functionalities will be presented. ## discussion in general, the r-cnn model works well if the input images have been well-cut. but the segmenta-tion is not always perfect because the layout of each image is different. for instance, some have multiple marginalia outside of the printed text, some have a few marginalia inside the printed part, some have graphics besides handwritten and printed text, etc. the difference between each image requires adjustment of segmentation algorithm parameters for each image to make a good segmentation. therefore, only with a better way to do the segmentation for an input image, can this model perform better. in contrast to this, faster r-cnn does not require the pre-segmentation operation, so it has a good chance to address the problem of not marking all potential marginalia before the images are input to the model. instead, the regional proposal network of the faster r-cnn does not only speed up computation, but also seems to propose more relevant sections of the image. combined with the fast r-cnn, the coordinate predictions for the marginalia are much more accurate than the ones of the r-cnn. the accuracy, together with the more efficient computation, lets the faster r-cnn outperform the r-cnn. this is also reflected by a high iou score (0.82) of the faster r-cnn. with regard to the marginalia segmentation, it was observed that the word segmentation algorithm struggles with lines and words that intersect with each other. these problems are very hard to solve with the current implementation, and requires more sophisticated algorithms. it would be interesting to try a r-cnn approach similar to the method for localizing the marginalia. however, this would require a good amount of time dedicated to creating training data for the algorithm by manually labeling figure 15: handwriting with intersecting words figure 16: attentionhtr word recognition results. while the faster r-cnn was found to be well performing for our use-case, there also exists other object detection methods that were not explored in this study, such as mask r-cnn @xcite and yolo @xcite . however, the dataset is prepared in a format easily adaptable for different architectures, and other architectures such as mask r-cnn and yolo can be investigated as a potential future work. for text recognition, transformer based methods such as trocr @xcite also exists, but this study focused on atten-tionhtr due to a powerful in-house general purpose htr multi-writer model, developed as part of our previous work @xcite . the experimental evaluation can be further extended with a comparison with other text recognition methods as future work. the results show that the text recognition accuracy with attentionhtr heavily depends on the quality of the segmented input image. if the image is clear and well segmented, i.e., only one word is visible, attentionhtr provides accurate results, without the need for training from scratch. however, some output images of the marginalia segmentation algorithm were multiple lines long, or the quality of the given image was just not precise enough. in that latter case, even a human cannot correctly classify the text. overall, attentionhtr is highly accurate when the input was clear and well segmented. in future work, words in the dataset can be labeled, which would allow for training of the attentionhtr network on the given data. also, the present study will be extended further to include annotated marginalia such as melville @xcite and mill (pionke, 2020).