OCR Processing of S wedish Historical Newspapers Using Deep Hybrid CNN – LSTM Networks (2021)

Abstract: Deep CNN–LSTM hybrid neural networks have proven to improve the accuracy of Optical Character Recognition (OCR) models for different languages. In this paper we examine to what extent these networks improve the OCR accuracy rates on Swedish historical newspapers. By experimenting with the open source OCR engine Calamari, we are able to show that mixed deep CNN–LSTM hybrid models outperform previous models on the task of character recognition of Swedish historical newspapers spanning 1818–1848. We achieved an average character accuracy rate (CAR) of 97.43% which is a new state–of–the–art result on 19th century Swedish newspaper text. Our data, code and models are released under CC-BY licence.

## introduction making previously inaccessible historical data available by converting them into digital resources through optical character recognition (ocr) techniques is important for preserving our cultural heritage and thereto provide greater insight into the past. today there are several leading off-theshelf ocr engines such as the commercial ab-byy finereader, foot_0 open source ocropus @xcite , tesseract @xcite and calamari @xcite , offering a comprehensive set of pre-trained models that could be applied during recognition for different languages. these models, in particular models for historical data, vary in accuracy for some languages mainly because they were trained on very limited amount of data. to overcome this limitation researchers have been training language specific character recognition models for different time periods @xcite @xcite . there have also been attempts to improve the accuracy of the models after recognition by applying post-correction methods @xcite . in this paper we examine to what extent deep cnn-lstm hybrid neural networks can improve the character accuracy rate (car) on 19th century swedish newspaper text during recognition. following @xcite approach we trained a character model for swedish in calamari and achieved an average car of 97.43% which is a new state-of-the-art result for historical swedish newspaper text. 2 2 related work mainly due to the introduction of recurrent neural networks (rnns) in particular with the long short-term memory (lstm) architecture @xcite great progress has been made in the field of ocr in recent years. today most of modern ocr systems leverage deep learning algorithms for training models and to improve performance. for instance some state-of-the-art ocr systems use shallow lstm neural networks, consisting of one or two hidden layers @xcite . lstm models became a standard in 2013 @xcite and lstm-based networks have proven to be one of the most effective approaches for complex natural language processing (nlp) tasks @xcite . likewise convolutional neural networks (cnns) have shown outstanding results for image data processing tasks @xcite including feature extraction @xcite . hybrids of these two network structures are used in a high diversity of fields and have achieved state-ofthe-art results in many cases @xcite . breuel (2017) presented a groundbreaking deep hybrid cnn-lstm implementation for text recognition, that outperformed previously state of the art methods based on shallow lstms. @xcite show that transfer learning drastically improves character accuracy rates on early printed books, compared to when training models from scratch. a year later this have become the default approach for training models. in their continued research @xcite utilize a combination of cross-fold training and confidence voting and succeed to significantly reduce character error rates on early printed books, compared to training a single model on a single fold. along the same lines, in our work we also train an ensemble of models on a cross-fold of the same gt data and then combine the models through voting. drobac and lindén (2020) and @xcite approach the complex task of ocr for historical prints by utilizing these types of deep cnn-lstm hybrid networks. to improve ocr results for historical finnish and swedish newspapers and journals drobac and lindén (2020) train mixed-language models and after post-processing of the ocr output achieve 1.7% cer for the finnish, and 2.7% cer for the swedish test set from 1771 to 1874. unlike drobac and lindén (2020) we do not apply any post-processing method but rather focusing on increasing the accuracy of the swedish character model. in @xcite error-rates are successfully reduced to a factor of up to 55% for digitised historical texts from the icdar 2017 dataset, achieving an average cer of 1.5%. to further improve these results, confidence voting was applied, resulting in cer below 0.5%. moreover, @xcite show these types of deep neural networks significantly outperform shallow networks in terms of both recognition capabilities and speed. ## calamari deep cnn-lstm hybrid networks calamari is a high-performance tensorflow-based package for line based recognition using state of the art deep neural networks (dnns) @xcite . 3 the advantage of calamari is that the soft-3 https://www.tensorflow.org/ ware supports customized deep network architectures composed by convolutional neural networks (cnns) and long short-term memory (lstm) layers, trained by the connectionist temporal classification (ctc) algorithm described in @xcite . it uses the tensorflow 3 framework for deep neural network computations, and consequently supports training and recognition on the graphics processing unit (gpu), which is proven to significantly reduce the overall computation time @xcite . calamari also provides additional features that might improve accuracy rates, such as pre-training, early stopping, cross-fold training, data argumentation, and confidence voting of different predictions. however, while data augmentation has been shown to improve accuracy for small data sets, it impairs the accuracy for larger data sets according to @xcite . moreover, calamari with its deep network architecture has proven to significantly outperforms the shallow one-dimensional lstm based neural network approach, such as the one used by ocropus @xcite in terms of both recognition capabilities and speed @xcite . the default neural network structure consists of two consecutive cnn blocks containing a convolution and a max-pooling layer, both connected with a relu-activation function. each of the two convolution layers has a kernel size of 3 x 3, the first one consists of 40 filters, and the second one of 60 filters. the pooling layer has a kernel size and stride of 2 x 2. this is followed by a bidirectional lstm layer, and finally an output layer with a drop-out rate of 0.5 in order to prevent overfitting. given the predictions of the output-layer and the gt labels, the ctc-loss is computed. ## data and preprocessing the reference data we experimented with forms part of the kubhist data @xcite . it is a large collection of approximately 300 thousand swedish historical newspaper editions, spread over roughly 200 years. the collection has been digitised and ocred by the national library of sweden. part of the kubhist, more specifically a reference data of 400 pages spanning between 1818 and 2018, was processed through an enhanced ocr-process @xcite by combin-ing two ocr engines: abbyy finereader, 4 and the open source system tesseract. 5 this two-ocr engine system was originally developed in cooperation with the norwegian software company zissor in 2017 and is based on the principle of evaluating and comparing the ocr results from multiple engines. the approach has been proven to improve the character recognition accuracy for some newspapers @xcite . many errors however remain. the reference data comes with a ground truth (gt) data available in plain text files. 6 it was first segmented down to paragraph level, 7 and further manually transcribed through double-keying. 8 this reference material has been previously used to evaluate the two-ocr engine system, focusing on the time period 1818-2018. the evaluation results show that accuracy varies from 56,65% to 98,41%, on character level, depending on the newspaper edition. these results further constitute the very baseline of this project. ## experiments and results our strategy for dividing the dataset into training and test sets has been to randomly selecting lines from each newspaper page. 80% has been allocated for training and 20% for testing. this way of randomizing generates a good diverse training data and hence a good representation of the data that is important for constructing a model able to generalize well @xcite . in total, 6 742 lines containing 53 963 words and 338 701 characters have served as training set, 1 671 lines containing 13 478 words and 84 713 characters as test set. for training of a single model (single voter) the final training set has been further split into training and validation subsets, again with a split ratio of 80:20. in total the generated training subset comprises 5 400 lines and the validation set 1 349 lines. more details can be found in once the line images have been prepared we use them together with their ground truth to train models with calamari. during training early stopping has been applied in order to avoid overfitting and improve generalization. if performance on the held out validation dataset after each epoch has not improved after 5 epochs or if validation loss has begun increasing training was stopped. one approach for reducing overfitting is to add more examples to the training data. however, since segmenting newspaper pages into image lines and linking them to their corresponding gt lines has turned out to be relatively time consuming. adding more data in order to investigate whether results can be improved has not been done. to improve performance and find the optimal neural network structure for our data a series of experiments have been performed using different network configurations. these include testing out different combinations and dimensions of the cnn and lstm layers and in this way expanding the network. these experiments are described in the following sections. in all our experiments both single models are trained as well as using a combination of cross-fold training and subsequent confidence voting. training was performed using a 5-fold over the same training data resulting in 5 models with different characteristics. these 5 models are then used to recognize the held out test lines. for each test line a total of 5 output sequences are generated, 1 from each model. these 5 output sequences later serve as the input for the voting process in order to determine the final output by using confidence scores. after experimenting with different network configurations both by training single models and by combining cross-fold training and confidence voting we found that the best results were achieved using the following neural network, seen in equation 1. @xmath0: 3 x 3, @xmath1, @xmath2: 3 x 3, @xmath3, @xmath4, @xmath5.5, @xmath6, @xmath7.5. the optimized neural network consists of two consecutive cnn blocks containing a convolution layer followed by a max-pooling layer. both connected with an relu-activation function. each of the two convolution layers have a kernel size of 3 x 3. the first one consists of 80 filters and the second one of 100 filters. the pooling layer has a kernel size and stride of 2 x 2. the two cnn blocks are followed by two lstm layers containing a total of 400 nodes divided over two layers with 200 nodes each. ## conclusion and future work in this paper we have investigated how deep cnn-lstm hybrid neural networks can be utilized in order to improve current ocr results for 19th century swedish newspaper text. mixed deep cnn-lstm hybrid models have been successfully trained in calamari for the task of character recognition of swedish historical newspaper texts spanning 1818-1848. initial testing with the calamari default network revealed a significant improvement in accuracy compared to our baseline (avg. 82.37% car) resulting in a car of 96.06%. this test alone showed the advantage of training individual mixed models over pre-trained models from commercial systems such as abbyy finereader. highest car of 97.43% was achieved through voting with 5 best models using the optimized network. thus, the combination of cross-fold training and confidence based voting significantly improve accuracy rates compared to training a single model using the same neural network. furthermore, our best results show an significant improvement over the baseline results for the specific time period. a promising future direction is to incorporate active learning @xcite . active learning is based on the principle of maximal disagreement. an ensemble of voters (or models) are first trained on a set of training lines with their corresponding gt. then, the voters are given unseen text lines and make predictions. those lines where voters disagree the most on is then added to the training data for subsequent training enabling a maximal learning effect.