Automatic Transcription of Handwritten Old O ccitan Language (2023)

Abstract: While existing neural network-based approaches have shown promising results in Handwritten Text Recognition (HTR) for high-resource languages and standardized/machine-written text, their application to low-resource languages often presents challenges, resulting in reduced effectiveness. In this paper, we propose an innovative HTR approach that leverages the Transformer architecture for recognizing handwritten Old Occitan language. Given the limited availability of data, which comprises only word pairs of graphical variants and lemmas, we develop and rely on elaborate data augmentation techniques for both text and image data. Our model combines a custom-trained Swin image encoder with a BERT text decoder, which we pre-train using a large-scale augmented synthetic data set and fine-tune on the small human-labeled data set. Experimental results reveal that our approach surpasses the performance of current state-of-the-art models for Old Occitan HTR, including open-source Transformer-based models such as a fine-tuned TrOCR and commercial applications like Google Cloud Vision. To nurture further research and development, we make our models, data sets, and code publicly available.

## introduction old occitan, also known as old provençal, was a language widely spoken in the 11th-16th centuries, in particular in southern france, northeastern spain, and northwestern italy. it occupies a prominent position in both the linguistic and cultural legacy of romance languages, primarily due to its role as a precursor to french lyric and the dissemination of works by troubadours throughout europe. despite its well-established historical importance, the linguistic research of old occitan remains relatively limited. in contrast to old french, it lacks comprehensive collections of digitized manuscripts with scanned images and annotated corpora, which are essential resources for conducting detailed morpho-syntactic or syntactic analyses @xcite . more recently, however, there have been efforts to ramp up the availability of resources for this language. an example of this is the creation of a digital version of the old occitan dictionary foot_0 , led by a team of researchers at the bavarian academy of sciences. the project aims to establish an openaccess database for scientific research, featuring a curated collection of vocabulary. a crucial step for this is the digitization of handwritten material of (non-standardized) words. for our research, we had access to a collection of 600,000 handwritten cards (93% of these being unlabeled) containing graphical variants alongside their respective lemmas. in this work, we explore and combine transformer-based architectures with data augmentation techniques specifically tailored to address the challenges posed by htr for low-resource languages. the experimental outcomes demonstrate the effectiveness of our method, as it achieves stateof-the-art (sota) results in old occitan. contributions this work provides the following contributions: a comparative study by @xcite examined various attention mechanisms and positional encodings to address the alignment between input and output sequences in the context of htr, a field of study that was traditionally dominated by recurrent neural network (rnn) encoders combined with connectionist temporal classification (ctc) decoders @xcite @xcite . subsequently, a systematic literature review conducted by @xcite @xcite most of the reported approaches employed architectures based on convolutional recurrent neural network (crnn), long short-term memory (lstm) models, and ctc, applied to well-studied languages such as english, chinese, urdu, and arabic. with the popularization of the transformer model by @xcite and its multimodal applications, other approaches deviating from the conventional convolutional neural network (cnn) and rnn architectures have emerged and shown sota performances in htr. one notable example is transformer ocr (trocr) @xcite , which adopts a transformer model as its backbone. this end-to-end htr model consists of a vision encoder and an autoregressive text decoder. subsequently, @xcite proposed a more compact transformer-based architecture that employs different visual feature embedding techniques and combines ctc and crossentropy loss during training. both transformerbased approaches achieved competitive results on the widely-used iam data set @xcite , which contains overall 115,320 handwritten english words on 13,353 images. additionally, diaz et al. (2021) compared various encoderdecoder models and found that using self-attention in the encoder and a ctc-trained decoder enriched with a language model yielded sota performance on the same data set. furthermore, researchers have explored the application of transformer-based models to lowresource languages. for instance, @xcite successfully fine-tuned a trocr instance for handwritten medieval latin and surpassed the widely-used transkribus @xcite , a commercial platform for historical document transcription. the results highlight the potential of custom transformer-based models to address htr challenges in languages with a limited amount of available data. ## experimental setup model selection we explore combinations of four vision encoder models: beit @xcite , deit @xcite , vit @xcite , and swin @xcite with two language decoders: gpt-2 @xcite and bert @xcite . to investigate the impact of synthetic data, we train each combination using three different training setups: synthetic data only ("pre-train"), real data only ("only real"), and fine-tuning the pre-trained model with real data ("fine-tune"). this resulted in a total of 4 × 2 × 3 = 24 experiments. to evaluate the performance of the models, we used our validation data set. tables 5 and 7 provide a comprehensive overview of the model combinations and the corresponding training regimes. final model training after selecting the bestperforming model on the validation set, we use enhanced data augmentation and incorporate 900,000 synthetic images in total during pre-training. seven random rotations and dilations were applied to the real images, as outlined in to assess the effectiveness of our optimized model, we conducted a benchmarking study. we compared its performance on the test set against popular open-source tools that support handwritten occitan. in addition, we included a commercial alternative, google cloud vision foot_3 , known for its proven high performance in practical applications @xcite . a short description of the models is presented in ## conclusion in this paper, we present a transformer-based model for the htr of old occitan, a low-resource language. our approach involves training various configurations of vision encoders in conjunction with language decoders. through our experiments, we have determined that the most effective combination is a swin + bert model, as it achieves superior performance in terms of weighted cer and share of correctly predicted labels. furthermore, it has fewer parameters, requires less training time, and utilizes fewer computational resources. to improve the performance of our model, we applied elaborate data augmentation techniques, reducing the weighted cer from 0.026 to 0.005 and improving the share of correctly predicted labels from 81.1% to 96.5%. as a result, our approach surpasses various open-source and commercial htr and ocr tools. during the evaluation of our best model, we observe that it tends to be less effective on images that contain long character sequences, annotation irregularities, multiple lines of text, and rare characters. however, it demonstrates competency in handling complex cases (cf. fig. 13 in appendix h). overall, our approach presents a promising htr solution for old occitan, and nurtures further research to explore applications to other languages and data sets.