Reconstructing Manual Information Extraction with DB -to-Document Backprojection: Experiments in the Life Science Domain (2020)

Abstract: We introduce a novel scientific document processing task for making previously inaccessible information in printed paper documents available to automatic processing. We describe our data set of scanned documents and data records from the biological database SABIO-RK, provide a definition of the task, and report findings from preliminary experiments. Rigorous evaluation proved challenging due to lack of gold-standard data and a difficult notion of correctness. Qualitative inspection of results, however, showed the feasibility and usefulness of the task

## introduction research results from the life sciences are mainly published in the form of written journal or conference papers, even though these results often take the form of measurements of experimental parameters, which would more appropriately be stored in a structured, machine-readable form. while there is some tendency towards directly publishing experimental data, e.g. on sourcedata @xcite or (for environmental data) pangaea 1 , this is not the norm yet, and does not help with the huge body of data already published in the conventional literature. it is common practice in the life sciences, therefore, to manually extract information (including measurements and the experimental conditions underlying them) from natural language documents, and to use it to populate biological databases. this process is called biocuration (international society for biocuration, 2018) and comprises, for every document, 1) identification and mark-up of curatable information, 2) data extraction, normalization, and consolidation, and 3) database insertion. despite constant improvements in nlp technology, biocuration involves significant human labor (mostly reading) (oughtred 1 www.pangaea.de @xcite @xcite , because data quality (i.e. correctness and integrity) has priority over quantity (i.e. more quickly available, but potentially less reliable, data), and the error rates of current nlp systems are still considered too high @xcite . for reasons of ergonomics and ease of handling @xcite @xcite , the identification and mark-up step often involves paper printouts and highlighter pens, foot_0 like in the example page in figure 1 . as mere intermediate products of the curation process, the manually highlighted printouts are only required until all data from the respective document has been curated, and they will normally be archived afterwards. we argue, however, that the printouts contain even more information which curation simply does not make full use of: first, some document sections, although containing highlighting, will not lead to the creation of a record in the biological database (see our results in section 5.3). yet, this highlighting can still be regarded as a kind of relevance annotation, produced by life science domain experts through attentive, task-oriented reading. obviously, this information should be useful, e.g. for the analysis of how important information is dispersed over a scientific document. second, for those database records that are created from highlighted document sections, the reference to that section is normally not preserved. again, an obvious way to use this information is to allow users of the biological database to visually trace the record to its source in the document, including the original context. in this paper, we describe our approach towards re-purposing scientific document printouts which were manually highlighted during biocuration. more precisely, our research question is: given records of curated information from the database and the original, scanned source document, (to what degree) can we recover the document section that a particular record was extracted from? we consider this to be a novel scientific document processing task, and propose to refer to it as dbto-document backprojection. the remainder of the paper is structured as follows. in section 2 we describe the data basis of our work. section 3 introduces the highlighted text extraction task, which we consider as self-contained and only loosely linked to the main task. section 4 deals with the actual db-to-document backprojection task, provides a precise definition, and describes our processing steps. section 5 presents some preliminary experiments, results, and error analysis.initially, this section will also discuss our approach to evaluation. in section 6 we discuss some related work, and section 7 contains our conclusions and directions for the future. note that, although our data is from the life sciences, the task is relevant for all domains where manual information extraction is performed on natural language documents (like e.g. in @xcite , where information is extracted from ir research papers in the form of machine-readable 'nanopublications'). ## data the work in this paper is based on two related data sets, which have been collected in the sabio-rk biochemical reaction kinetics database project foot_1 . sabio-rk is a curated database containing structured information about biochemical reactions and their corresponding kinetics @xcite . the document data set is an electronic version of our archive of 6, 000+ manually highlighted printouts of documents from the life science domain, which have been curated in the 10+ years of our database's existence. over the years, numerous different curators were involved in the manual mark-up. different highlighter colors were used, sometimes even within the same document (see figure 1 ). in case of equivalent information appearing repeatedly in the same document, curators generally attempted to be economical and to avoid redundancy by highlighting only the most appropriate appearance, which is often, but not always, the first appearance. while the mark-up was performed in a completely unrestricted manner (cf. below), in the vast majority of cases, highlighting was applied directly to words or lines (cf. figure 1 ), which greatly helped in extracting the highlighted text (cf. section 3). in some rare cases, curators selected whole sections by drawing a vertical line at the section's margin. also, data in tables was sometimes highlighted on the cell level, while in other cases, only the column header, the table header, or even the table caption was highlighted. we created an electronic version of the document collection by scanning and ocr-processing all papers 4 , which resulted in a sandwich pdf for each document with the (partially highlighted) background superimposed with the extracted text. ocr was performed with commercial software (alaris capture pro), which was used out-of-the-box. the total number of tokens in the 98 documents is 630, 153, with 6, 430 tokens/document on average. the second data set is the record data set which contains measurements of kinetic parameters that were extracted from individual documents from the document collection in the course of manual curation. each of the 2, 916 records in this data set is linked to exactly one source document (via its pubmed id), but no lower-level links (to pages or lines) exist. each document, in turn, can be linked to an arbitrary number of records (29.76 records/document on average). it has to be noted that the above count of 2, 916 records contains a considerable number of multiple counts. this is true in particular for records of type experimental condition (cf. below), and is due to the fact that often, several measurements are performed under identical experimental conditions. for scoring and evaluation, however, this does not make a differ-ence, because we conflate semantically identical records before analysis. there are two main types of records, experimental condition and parameter. each record consists of three to six attribute-value (a-v) pairs. figures 2 and 3 show one example of each type of record. figure 2 : record of type experimental condition with three a-v pairs featuring one numeric, one atomic string, and one complex string value. note that we only consider a subset of all a-v pairs available for each record: some attributes have unspecific values (e.g. role:'variable') which are not useful for searching. also, most attributes have a variant with a normalized value, which does not appear in the text. with the exception of the experimental conditions' buffer attribute, all values are atomic. therefore, the buffer attribute will be handled differently in the second phase of backprojection (see section 4.2). ## highlighted text extraction highlighted text extraction comprises 1) extracting from each sandwich pdf both the searchable plain text and a background image for every page, 2) detecting highlighted areas in the background images, and 3) mapping the detected image areas to the extracted text. the workflow is shown in figure 4 . we use both pdftohtml and pdftotext from the poppler foot_3 library to extract data from the scanned and ocr-processed sandwich pdf documents from our collection. the only task of pdftohtml is to extract, from each page, a png image with the non-textual background, which also includes the color-marked areas. these images were already generated during ocr processing and consist of document pages from which pixels that were detected as belonging to text were removed by inpainting (see 'page background image' in the lower left part of figure 4 ). pdftotext, on the other hand, is used to extract the text that was previously recognized by ocr. it produces one xml file for the document, incl. bounding boxes on the token-level. these tokens reflect the original document layout, but come in correct reading order even for multi-column documents. the second step makes use of some simple image processing. as described above (section 2), document highlighting can come in any color, so searching for areas of any particular color (like e.g. yellow) is not an option. instead, our algorithm combines the facts that 1) highlighting is always non-grey and 2) shades of grey in the rgb color model are characterized by identical, or at least highly similar, values in the r, g, and b components. 6 we create a binarized version of each page by going over all pixels in a copy of the original image and setting each pixel to 'black' if the difference between the r, g, and b components is above a threshold of 50 (i.e. if the pixel is non-grey), and setting it to 'white' otherwise. the resulting image, then, contains regions with higher and lower density of black pixels (see 'binarized page background image' in figure 4 ). in the last step, text tokens are labelled as highlighted if their bounding boxes (from the xml file), when projected to the binarized image, cover an area that is at least 50% black. while this process is very simple, we found it to work surprisingly well, at least for the very frequent cases where the highlighting was applied directly to words or lines, yielding almost perfect extraction accuracy on most of the images we inspected. of the 630, 153 tokens in our data set, only 39, 071 (6.2%) were detected as containing highlighting. the following is worth noting. first and foremost, the above definition reflects the fact that there is no simple notion of a correct backprojection of a database record to a document section, neither in our data sets nor, arguably, in reality. in part, this is because the same (or highly similar) information can appear in more than one section of a document. second, the value of sec size is important because, by specifying the number of tokens that are considered at the same time, it might penalize records with a comparably large number of values. at the same time, however, an excessively large sec size will undermine the whole endeavour because it will be difficult to locate the actual matched values within the section. also, with increasing sec size, there is a growing risk of clustering values which are actually completely unrelated, creating spurious backprojection results. finally, the role of automatically detected highlighting for db-to-document backprojection is still unclear. since obtaining this highlighting information was the prime reason for scanning the paper printouts in the first place, a rather strong contribution of this feature is desirable. one obvious role of highlighting is that of a filter for preventing non-highlighted tokens from being potential backprojection targets. ## conclusions & future work in this paper, we introduced, defined, and performed some preliminary experiments with dbto-document backprojection, a novel scientific document processing task. our motivation for attempting this task comes from the requirements of a biocuration project, and from our idea to re-purpose previously unused (or rather under-used) data to advance biocuration methods. the focus of this initial paper was mostly on motivation, on a definition of the task and its functional parameters, and on the development of a better understanding of the effects and interactions of these parameters. for the latter, we performed some simple experiments and analysed the results. rigorous evaluation, however, was not attempted, and, what is more, our results showed that defining what it means for a backprojection to be correct is difficult. while a quantitative evaluation remains difficult, a qualitative inspection of backprojection results clearly showed that the answer to our original research question is a positive one, which is the main result of this paper. our findings regarding the role of color highlighting for backprojection, on the other hand, are somewhat mixed: while our system is able to detect highlighted tokens with high accuracy, appropriate ways to integrate this information into backprojection still need be be explored much further. the approach of requiring highlighting for matching, while not disproved yet, might be too strict, and alternative strategies will be evaluated in future work, for which our system and data sets provide a valuable basis. additional future work includes the following: the optimization heuristics against ocr errors, although already shown to be effective in practice, are far from complete, and should be improved by handling additional cases of ocr errors, and other spelling variants. also, as suggested by one reviewer, xml versions of papers from e.g. pubmed could be used to inform the backprojection task, which might also include automatic correction of ocr errors. future work will also include the creation of an annotated dataset by inspecting the automatic results and storing correct highlighting in the extracted xml. ideally, this should be done with the help and feedback of domain experts. finally, the system will be applied to our full data set of 6, 000+ documents, which will yield a stronger data basis for analysis. apart from the obvious use cases like quality assurance in biocuration (and other fields where information is manually extracted from documents), and support for users of biological databases, both by means of visualizations, we also envisage several other potential applications for db-to-document backprojection. these include creation of multimodal training data for page-topology-based document understanding systems like @xcite , creation of input for empirical studies on document structure (distribution of information in scientific documents), and others.