Word-Level Alignment of Paper Documents with their Electronic Full-Text Counterparts (2021)

Abstract: We describe a simple procedure for the automatic creation of word-level alignments between printed documents and their respective full-text versions. The procedure is unsupervised, uses standard, off-the-shelf components only, and reaches an F-score of 85.01 in the basic setup and up to 86.63 when using pre- and post-processing. Potential areas of application are manual database curation (incl. document triage) and biomedical expression OCR.

## introduction even though most research literature in the life sciences is born-digital nowadays, manual data curation (international society for biocuration, 2018) from these documents still often involves paper. for curation steps that require close reading and markup of relevant sections, curators frequently rely on paper printouts and highlighter pens @xcite . figure 1a shows a page of a typical document used for manual curation. the potential reasons for this can be as varied as merely sticking to a habit, ergonomic issues related to reading from and interacting with a device, and functional limitations of that device @xcite @xcite . whatever the reason, the consequence is a two-fold media break in many manual curation workflows: first from electronic format (either pdf or full-text xml) to paper, and then back from paper to the electronic format of the curation database. given the above arguments in favor of paper-based curation, removing the first media break from the curation workflow does not seem feasible. instead, we propose to bridge the gap between paper and electronic media by automatically creating an alignment between the words on the printed document pages and their counterparts in an electronic fulltext version of the same document. our approach works as follows: we automatically create machine-readable versions of printed paper documents (which might or might not contain markup) by scanning them, applying optical character recognition (ocr), and converting the resulting semi-structured ocr output text into a flexible xml format for further processing. for this, we use the multilevel xml format of the annotation tool mmax2 foot_0 @xcite . we retrieve electronic full-text counterparts of the scanned paper documents from pubmedcentral ® in .nxml format foot_1 , and also convert them into mmax2 format. by using a shared xml format for the two heterogeneous text sources, we can capture their content and structural information in a way that provides a compatible, though often not identical, word-level tokenization. finally, using a sequence alignment algorithm from bioinformatics and some pre-and post-processing, we create a word-level alignment of both documents. aligning words from ocr and full-text documents is challenging for several reasons. the ocr output contains various types of recognition errors, many of which involve special symbols, greek letters like µ or sub-and superscript characters and numbers, which are particularly frequent in chemical names, formulae, and measurement units, and which are notoriously difficult for ocr @xcite . if the printed paper document is based on pdf, it usually has an explicit page layout, which is different from the way the corresponding full-text xml document is displayed in a web browser. differences include double-vs. single-column layout, but also the way in which tables and figures are rendered and positioned. finally, printed papers might contain additional content in headers or footers (like e.g. download timestamps). also, while the references/bibliography section is an integral part of a printed paper and will be covered by ocr, in xml documents it is often structurally kept apart from the actual document text. given these challenges, attempting data extraction from document images if the documents are available in pdf or even full-text format may seem unreasonable. we see, however, the following useful applications: 1. manual database curation as mentioned above, manual database curation requires the extraction, normalization, and database insertion of scientific content, often from paper documents. given a paper document in which a human expert curator has manually marked a word or sequence of words for insertion into the database, having a link from these words to their electronic counterparts can eliminate or at least reduce error-prone and time-consuming steps like manual re-keying. also, already existing annotations of the electronic fulltext foot_2 would also be accessible and could be used to inform the curation decision or to supplement the database entry. ## data for the alignment of a paper document with its electronic full-text counterpart, what is minimally required is an image of every page of the document, and a full-text xml file of the same document. the document images can either be created by scanning or by directly converting the corresponding pdf into an image. the latter method will probably yield images of a better quality, because it completely avoids the physical printing and subsequent scanning step, while the output of the former method will be more realistic. we experiment with both types of images (see section 2.1). we identify a document by its doi, and refer to the different versions as doi xml (from the full-text xml), doi conv , and doi scan . whenever a distinction between doi conv and doi scan is not required, we refer to these versions collectively as doi ocr . printable pdf documents and their associated .nxml files are readily available at pmc-oai. 5 in our case, however, printed paper versions were already available, as we have access to a collection of more than 6.000 printed scientific papers (approx. 30.000 pages in total) that were created in the sabio-rk foot_5 biochemical reaction kinetics database project @xcite . these papers contain manual highlighter markup at different levels of granularity, including the word, line, and section level. transferring this type of markup from printed paper to the electronic medium is one of the key applications of our alignment procedure. our paper collection spans many publication years and venues. for our experiments, however, it was required that each document was freely available both as pubmedcentral ® full-text xml and as pdf. while this leaves only a fraction of (currently) 68 papers, the data situation is still sufficient to demonstrate the feasibility of our procedure. even more importantly, the procedure is unsupervised, i.e. it does not involve learning and does not require any training data. ## outline of the alignment procedure the actual word-level alignment of the doi xml version with the doi ocr version of a document operates on lists @xmath0, @xmath1. these lists are characterized by longer and shorter stretches of tuples with matching tokens, which just happen to start and end at different list indices. these stretches are interrupted at times by (usually shorter) sequences of tuples with non-matching tokens, which mostly exist as the result of ocr errors (see below). larger distances between stretches of tuples with matching tokens, on the other hand, can be caused by structural differences between the doi xml and the doi ocr version, which can reflect actual layout differences, but which can also result from ocr errors like incorrectly joining two adjacent lines from two columns. the task of the alignment is to find the correct mapping on the token level for as many tuples as possible. we use the align.globalxx method from the bio.pairwise2 module of biopython @xcite , which provides pairwise sequence alignment using a dynamic programming algorithm @xcite . while this library supports the definition of custom similarity functions for minimizing the alignment cost, we use the most simple version which just applies a binary (=identity) matching scheme, i.e. full matches are scored as 1, all others as 0. this way, we keep full control of the alignment, and can identify and locally fix non-matching sequences during post-processing (cf. section 3.2 below). the result of the alignment (after optional pre-and post-processing) is an n-to-m mapping @xmath2, @xmath3. 13 ## related work the work in this paper is obviously related to automatic text alignment, with the difference that what is mostly done there is the alignment of texts in different languages (i.e. bi-lingual alignment). @xcite align not words but entire sentences from two languages based on statistical properties. even if words were aligned, alignment candidates in bi-lingual corpora are not identified on the basis of simple matching, with the exception of language-independent tokens like e.g. proper names. scanning and ocr is also often applied to historical documents, which are only available in paper @xcite @xcite . here, ocr postcorrection attempts to map words with word-and character-level ocr errors (similar to those found in our doi ocr data) to their correct variants, but it does so by using general language models and dictionaries, and not an aligned correct version. many of the above approaches have in common that they employ specialized ocr models and often ml/dl models of considerable complexity. the idea of using an electronic and a paper version of the same document for creating a characterlevel alignment dates back at least to @xcite , who worked on ocr ground-truth data generation. like most later methods, the procedure of @xcite works on the graphical level, as opposed to the textual level. @xcite use latex to cre-ate what they call 'ideal document images' with controlled content. print-outs of these images are created, which are then photocopied and scanned, yielding slightly noisy and skewed variants of the 'ideal' images. then, corresponding feature points in both images are identified, and a projective transformation between these is computed. finally, the actual ground-truth data is generated by applying this transformation for aligning the bounding boxes in the ideal images to their correspondences in the scanned images. since @xcite have full control over the content of their 'ideal document images', extracting the groundtruth character data is trivial. the approach of @xcite is similar to that of @xcite , but the former use more sophisticated methods, including canny edge detection @xcite for finding corresponding sections in images of the original and the scanned document, and rast (breuel, 2001) for doing the actual alignment. another difference is that van @xcite use pre-existing pdf documents as the source documents from which groundtruth data is to be extracted. interestingly, however, their experiments only use synthetic ground-truth data from the uw3 data set foot_7 , in which bounding boxes and the contained characters are explicitly encoded. in their conclusion, @xcite concede that extracting ground-truth data from pdf is a non-trivial task in itself. @xcite work on automatic ground-truth data generation for camera-captured document images, which they claim pose different problems than document images created by scanning, like e.g. blur, perspective distortion, and varying lighting. their procedure, however, is similar to that of @xcite . they also use pre-existing pdf documents and automatically rendered 300 dpi images of these documents. ## conclusions in this paper, we described a completely unsupervised procedure for automatically aligning printed paper documents with their electronic full-text counterparts. our point of departure and main motivation was the idea to alleviate the effect of the paper-to-electronic media break in manual biocuration, where printed paper is still very popular when it comes to close reading and manual markup. we also argued that the related task of document triage can benefit from the availability of alignments between electronic full-text documents (as retrieved from a literature database) and the corresponding pdf documents. apart from this, we identified yet another field of application, biomedical expression ocr, which can benefit from ground-truth data which can automatically be generated with our procedure. improvements in biomedical expression ocr, then, can feed back into the other use cases, by improving the ocr step and thus the alignment, thus potentially establishing a kind of bootstrapping development. our implementation relies on tried and tested technology, including tesseract as off-the-shelf ocr component, biopython for the alignment, and mmax2 as visualization and data processing platform. the most computationally complex part is the actual sequence alignment with a dynamic programming algorithm from the biopython library, which we keep tractable even for longer documents by using a simple pre-compression method. the main experimental finding of this paper is that our approach, although very simple, yields a level of performance that we consider suitable for practical applications. in quantitative terms, the procedure reaches a very good f-score of 86.63 on converted and 85.20 on printed and scanned pdf documents, with corresponding precision scores of 94.90 and 93.47, respectively. the negligible difference in results between the two types of images is interesting, as it seems to indicate that converted pdf documents, which are very easy to generate in large amounts, are almost equivalent to the more labour-intensive scans. in future work, we plan to implement solutions for the identified use cases, and to test them in actual biocuration settings. also, we will start creating ocr ground-truth data at a larger scale, and apply that for the development of specialised tools for biomedical ocr. in the long run, procedures like the one presented in this paper might contribute to the development of systems that support curators to work in a more natural, practical, convenient, and efficient way.