Recovering Text from Endangered Languages Corrupted PDF documents (2022)

Abstract: In this paper we present an approach to efficiently recover texts from corrupted documents of endangered languages. Textual resources for such languages are scarce, and sometimes the few available resources are corrupted PDF documents. Endangered languages are not supported by standard tools and present even the additional difficulties of not possessing any corpus over which to train language models to assist with the recovery. The approach presented is able to fully recover born digital PDF documents with minimal effort, thereby helping the preservation effort of endangered languages, by extending the range of documents usable for corpus building.

## introduction endangered languages usually have extremely scarce linguistic resources available, and even less in a directly usable unicode-encoded text format. often the only available resources are pdf documents produced by language preservation efforts or for proselyte purposes by different organisations. despite the efforts of such organisations to provide text content in endangered languages, it is often the case that the documents they produced are only printable or displayable on screen but totally unusable for automatic text processing purposes. sometimes the only documents available in these language are contained in such documents, thereby preventing a wider exposure (impossibility to find them with search engines) and study of that language (impossibility to build corpora). the problem this paper tackles is how to recover usable texts from such documents. there are two main drives behind pdf documents not being exploitable by natural language processing (nlp) systems: 1) the document has a corrupted font to unicode value translation tablein which case the text can not be extracted as the content of the pasted text is either unavailable or gibberish; 2) the pdf actually contains scanned images -and therefore it is impossible to copy/paste from that document. in this paper we tackle only the first, simpler, problem of born digital pdf document recovery, leaving the second more complex problem for future works. the alternative of using our approach for such corrupted document is a much more time consuming manual correction of incorrectly ocred text, or an even more time consuming plain manual transliteration into unicode of the document content. as such it can be viewed as a lightning fast alternative to manual recovery. such a work can help speedup corpus creation efforts for endangered languages such as in @xcite . despite the general applicability of our approach, we will specifically restrict our attention to the universal declaration of human right corpus (udrh), as it is -outside the bible -the most translated documents whose documents are openly accessible on the internet @xcite . it has to be noted that corrupted pdf documents for endangered languages is a common phenomena: there exists no support for these languages, and very often they posses unique symbols or even use their own writing system. in order to write in these languages the creators of the documents must design their own ad-hoc fonts, which are not publicly available. for all these reasons, the approach we propose in this article particularly relevant for recovering text in endangered languages. ## proposed system the system we designed is an interactive tool that allows the user to recover the text of corrupted document, requiring from a few minutes to a few hours depending on the quantity of symbols to recover, on the knowledge of the user of the target language, and on its ability to input the required characters. our system proceeds in two phases: a first fully automatic one that recovers the symbols for space and dot; and a second interactive one that helps the user gradually build the minimal quantity of resources in order to decipher the text. ## experiments in the experiments we consider only corrupted documents that do not have an unicode version even on the unicode consortium website. out of the dozen such documents, we focus our effort on four languages in order to demonstrate the capacity of language niv yrk unique characters 81 68 text length (words) 1430 1530 input length (words) 57 76 in ## discussion our approach allowed us to quickly and efficiently recover the udhr unicode text for two languages, requiring less than an hour of work: nivkh and nenets. these two documents have been sent to the udhr page of the unicode consortium, and are now already publicly available. one additional advantage of our approach, is that when dealing with a document collection using the same corrupted font, it is necessary to recover it only once in order to process the other documents, thereby yielding additional time gains for linguists and experts striving to create corpora. when recovering mongolian, we have been confronted to the problem that this language is written top down, because pdf documents consider that lines are going exclusively from left to right or right to left, the text extraction tool is totally unable to recover the lines. consequently, our method can not be applied directly for that language, and instead of relying on an external library, it requires a further ad-hoc vertical segmentation step. this is left for future work. at the time of the writing, another document is in the process of being recovered: the udhr in mundari, which is written in devanagari script. devanagari presents one specific difficulty: the long vowel "i" is written in a unicode text string after the character of the consonant it is attached to, but it is displayed before it when the string is visually rendered. this is because the cid sequence of the symbols of a line is extracted in increasing order of the bounding box coordinates of the characters it contains. in order to deal with that, the user is constrained to input the characters in the same order as the one expected by the cid sequence, and a post-processing step is required to swap the corresponding unicode characters before rendering the final text. ## conclusion we present an approach that is able to quickly guide a human expert in recovering text from corrupted born digital pdf documents containing text in rare or endangered languages. such languages impose severe constraints, because often there exists no preexisting corpus to train on, or to compare to the extracted text. our approach has been designed specifically to operates within these constraints and consists in reconstructing the cid to unicode translation maps by efficiently leveraging user input in an interactive way. we applied this approach to 4 documents of the udhr corpus for which there exists no unicode text, 2 of which where fully recovered, the other ones needing some additional development related to particularities of the writing system they use. the tool is not yet available, but will be released on this repository foot_3 . future work will deal in applying the approach to more languages of the udhr, and to deal with the harder problem of recovering text of endangered language existing solely as pictures. to that intent, we consider exploring image augmentation techniques @xcite in order to train ad-hoc ocr system for any scripts solely based on the symbols present in the unicode charts, with the aim of interactively presenting the user with potential choices. our approach is useful to help protect and study endangered languages for which document base exists in born digital pdf format, but for which some of documents, or all of them, are corrupted.