A Benchmark and Dataset for Post- OCR text correction in S anskrit (2022)

Abstract: Sanskrit is a classical language with about 30 million extant manuscripts fit for digitisation, available in written, printed or scanned-image forms. However, it is still considered to be a low-resource language when it comes to available digital resources. In this work, we release a post-OCR text correction dataset containing around 218,000 sentences, with 1.5 million words, from 30 different books. Texts in Sanskrit are known to be diverse in terms of their linguistic and stylistic usage since Sanskrit was the ‘lingua francua’ for discourse in the Indian subcontinent for about 3 millennia. Keeping this in mind, we release a multi-domain dataset, from areas as diverse as astronomy, medicine and mathematics, with some of them as old as 18 centuries. Further, we release multiple strong baselines as benchmarks for the task, based on pre-trained Seq2Seq language models. We find that our best-performing model, consisting of byte level tokenization in conjunction with phonetic encoding (Byt5+SLP1), yields a 23% point increase over the OCR output in terms of word and character error rates. Moreover, we perform extensive experiments in evaluating these models on their performance and analyse common causes of mispredictions both at the graphemic and lexical levels. Our code and dataset is publicly available at https://github.com/ayushbits/pe-ocr-sanskrit.

## introduction post-ocr text correction is a crucial postprocessing step employed for correcting errors from the predictions of optical character recognition (ocr) systems @xcite . a post-ocr corrector leverages the distributional information encoded in language models that aims to not only handle the systemic errors introduced by the ocr engine but also to predict meaningful and fluent sequences based on the context @xcite . for a language like sanskrit, the sources of ocr errors are diverse, owing to the availability of printed historical documents that vary vastly on a number of factors such as scan quality, book layout, typefaces, the orthographic similarity of letters in the alphabet, etc. (see figure 1 ). moreover, processing texts in sanskrit is often challenging as the language is morphologically rich, lexically productive, follows relatively free-word order and is a low-resource language with limited available machine-readable corpora @xcite . in this work, we release a large sanskrit postcorrection dataset of more than 218,000 manually verified sentences, consisting of 1.5 million words. our dataset consists of sentences from 30 books from domains as diverse as philosophy, literature, astronomy, medicine, mathematics etc.. figure 1 shows a sample of the scanned images from these books, from which we obtained our dataset. further, the sample clearly demonstrates the diversity in some of the aforementioned factors affecting the quality of the ocr predictions. histori-cally, depending on the region or the time period in which it was used, several writing systems and scripts were employed for writing sanskrit. however, the advent of the printing press largely standardized the use of the 'devanāgari' script as the default writing system for sanskrit. we additionally release a set of strong seq2seq baselines to benchmark for the task, including a copynet based lstm model @xcite and four pretrained seq2seq systems (lms). we find that all the pretrained-lm based baselines improve over the predictions from the original ocr. the best model which invokes byte level tokenization, viz., byt5 @xcite , in conjunction with phonetic encoding (slp1), among these benchmarks (all described in § 3) reports a character and word error rates of 2.98% and 23.19% respectively, as against that of 3.89% and 30.23% from the original ocr. this is primarily due to the ability of byte-level tokenizers to learn arbitrarily longer text in a setting where the frequency of words is low and out-of-vocabulary words are high ( §2). moreover, this goes well with the fact that the writings in sanskrit follow a phonemic orthography, i.e. phonemes have a direct one-to-one correspondence with the orthographic symbols. we identify that most errors arising from the original ocr are from mispredictions in word boundary detection, diacritics and orthographically similar characters. further, as the performance of the post-ocr text correction system is highly dependent on the predictions of the ocr, we also release the test dataset used for testing our current ocr. the test dataset consists of 500 images and their corresponding text, which can be used to benchmark an ocr, prior to using its predictions for post-ocr text correction. ## dataset sanskrit used to be the 'lingua franca' for scholarly discourse in the indian subcontinent for about three millennia and the classical language is still in sustenance in the region. it is estimated that as many as 30 million extant documents, more than that in greek and latin combined, are fit for digitisation in sanskrit @xcite . the current corpus is released as part of our attempt at large scale digitisation of old manuscripts in sanskrit. our corpus contains about 30 books, a subset of 103 books in our digitisation pipeline. these books were originally devanāgari क् क का $क क% क ृ 'क अ romanized k ka kā ki kī kr̥ kka a split # sentences # words train 208,173 1,444,913 validation 5000 34,762 test 5000 34,705 prior work in post-ocr text correction in sanskrit @xcite focused on texts written in iast or romanized sanskrit (monier-williams et al., 1899), while the current dataset is focused on devanāgari. all the books we consider here use devanāgari script as the writing system. while devanāgari is in existence since the fourth century ce (state), 1896), it has become the primary standard for writing sanskrit, and several other languages, with the advent of the printing press in india. the script consists of 47 primary characters and is a left-to-right abiguda, where contiguous consonant-vowel sequences are treated as unitary units. as shown in figure 2 , the vowels following the consonants (k in the figure), are treated as secondary units. these units are expressed as 'mātras' in the writing system, which essentially are diacritic markers. these markers may appear before, after, above or below an orthographic consonant symbol as shown in figure 2 . the same vowel, say 'a' in the romanised script, is written as a different character when used independently as a primary unit. similarly, conjunct consonants, like 'kka' in the figure, would also result in a different orthographic unit, leading to an increase in possible output units for the original ocr. moreover, these orthographically similar units can also be confusing to an ocr system. ## system descriptions ocr post-correction is a text correction task which can be formalised as a monotone seq2seq model @xcite . we use an encoderdecoder framework that takes predictions from an ocr as its input. while we use multiple pretrained seq2seq models as our baselines, none of these has sanskrit as one of their languages. however, devanāgari script is employed in other languages, such as hindi, which are present in these models. secondly, unicode encoding of devanāgari often poses several challenges owing to the variable byte length employed per character for encoding. hence, we losslessly transliterate the text into slp1, an ascii-based case-sensitive transliteration scheme in our experiments. baseline ocr model : our baseline ocr model is an ocr engine that uses the tesseract ocr @xcite . the model is fine-tuned upon 20,000 synthetically-created images with the sanskrit language flag. we release our ocr test set along with the post-ocr correction dataset. copynet @xcite : uses a copying mechanism in an lstm-based seq2seq framework to leverage the (partial) overlap between input and output strings. the model consists of 3 lstm modules stacked on top of each other for both the encoder and decoder. following @xcite , we use bpe for learning the vocabulary, which has shown the ability to handle corpora with 'rare words' @xcite . mbart @xcite ) is a multilingual variant of the bart, both of which are seq2seq models. it has an autoregressive decoder and a bertbased encoder. we used mbart-50 @xcite , specifically its huggingface implementa-tion (large), in our experiments which has been trained on large monolingual corpus of 50 languages. here, we use text in its original form as well as in the transliterated slp1 form. mt5 @xcite is a multilingual variant of t5 @xcite , trained on 107 languages. t5 is a seq2seq text generation model, pretrained on a mixture of supervised and unsupervised tasks using a span-corruption objective. in experiments, we employ the mt5 base model from hugging-face along with bpe tokenization and use both devānagari and slp1 encoded text. byt5 @xcite : given that we have a corpus with mostly 'rare words' (c.f., section 2), any unseen set in sanskrit will suffer from out-ofvocabulary words. a natural solution is to tokenize words at a character level where each character is represented by utf-8 bytes. byt5 is a variant of mt5 except that model is fed with a fixed 256 byte values. in experiments, we use byt5 small model from huggingface with byte tokenizer and slp1 encoded text. indicbart @xcite : it is a multilingual bart-based model trained on 11 indicfamily languages in devānagari script. the model is roughly half the size of mt5 model. ## experiments and results as a further analysis, we collect the most frequent 300 tokens in the corpus, with at least three letters for a word, and find a total of 2875 occurrences in the ground truth corpus. among these frequent tokens, ocr predicts each of them correctly at least once. further, to identify possible similar tokens predicted instead of the correct token, we use the ratcliff pattern recognition algorithm @xcite , with a matching ratio of 0.6. here, we find that 8.27% of the token occurrences among the most frequent tokens do not have a corresponding prediction that satisfies the matching criteria. with byt5, this number is reduced by 2.09%. moreover, in both ocr predictions and byt5 based predictions, we mostly find unique patterns in mispredictions for each token and are not able to find any consistent or systemic patterns for each token. ## conclusion we release a dataset consisting of 218,000 sentences from 30 books for sanskrit post-ocr text correction. we also release a set of strong baselines as a benchmark, which currently shows consistent and significant improvements from the ocr predictions, both on the in-domain test data and out-of-domain test data. all our baselines, in spite of not seeing sanskrit during pretraining, have shown to generalise well for the task. while using devanāgari unicode encoding for our experiments has shown to perform better than using slp1 for multiple baselines, slp1-based encoding on byt5 gives the best performance overall.