D eep P aper C omposer: A Simple Solution for Training Data Preparation for Parsing Research Papers (2020)

Abstract: We present DeepPaperComposer, a simple solution for preparing highly accurate (100%) training data without manual labeling to extract content from scholarly articles using convolutional neural networks (CNNs). We used our approach to generate data and trained CNNs to extract eight categories of both textual (titles, abstracts, headers, figure and table captions, and other texts) and non-textural content (figures and tables) from 30 years of IEEE VIS conference papers, of which a third were scanned bitmap PDFs. We curated this dataset and named it VISpaper-3K. We then showed our initial benchmark performance using VISpaper-3K over itself and CS-150 using YOLOv3 and Faster-RCNN. We open-source DeepPaperComposer of our training data generation and released the resulting annotation data VISpaper-3K to promote re-producible research.

## introduction texts, figures, tables and their associated captions are used in leveraging key concepts, data, and inferences to improve accessibility of knowledge @xcite , to offer succinct content summaries @xcite , to understand visual literacy, to tell data stories, and to improve research workflow, e.g., citeseerx @xcite foot_2 , microsoft academic @xcite 4 , google scholar @xcite 5 , semantic scholar @xcite 6 , and ibm science summarizer @xcite 7 . in these applications, extracting textual and nontextual content is often a necessary first step before any subsequent uses of these components are possible. however, the vast majority of published scholarly articles are available only in pdfs or scanned bitmaps. even though recent deeplearning-based algorithms using convolutional neural networks (cnns) provide considerably better performance @xcite @xcite @xcite , the quality of the labeled training data often determines the success of these cnn-based algorithms. the lack of large-scale labeled document datasets has been recognized as a major hindrance in deep-learning research for structure analyses @xcite @xcite . training data for the cnn-based algorithms are typically prepared manually by crowdsourcing (e.g., cs-150 @xcite ) or by automated tag extraction in xml (e.g., cs-large @xcite ). recently, @xcite designed a most successful and least labor-intensive approach to align and modify l a t e x syntax-based documents to automatically extract labels of over 4-million pages and achieve training data label accuracy of up to 94%. inspired by these recent advances, we designed deeppapercomposer, a simple data-preparation method to create 100% accurate training samples of any scale for content extraction in large numbers of scientific documents, by simply "rendering" papers to paste non-textual and textual content onto a white page to assemble the look of a real document. we introduce the workflow (figure 1 ), the resulting real-world case study to construct a new annotated dataset vispaper-3k, and two benchmark tests using this new dataset. the main contributions of this work include: 1. deeppapercomposer, a simple data- rule-based postprocessing human curation union and voting preparation method to synthesize dummy papers for generating accurate annotated labels, grounded upon scholarly articles' structural heuristics, without human intervention, in particular without manual labeling. 2. vispaper-3k, a new scholarly dataset with eight categories of ground-truth annotation of 2916 ieee-vis conference papers (24,660 pages). our goal is to extract textual and non-textual content from research papers. the essence of our approach is to couple the new cnn-based solutions and the heuristic-based method: we use heuristics to produce the structures of dummy papers as the training set and then let cnns perform classification tasks before feeding the results to postprocessing (figure 1 ). ## case study: curating the vispaper-3k dataset we applied the proposed deeppapercomposer framework to ieee vis publications over the past 30 years. training and validation data from dummy papers. in total, we used 13k dummy paper pages (10k for training and 3k for validation), each of dimensions 1075 × 1400 pixels and labeled by eight class tags (the five text content types, figure, table, and captions in deeppapercomposer modeling process. the two cnn models are trained using the automated dummy paper generation. the output using deep-papercomposer contains the annotated pages. preprocessing of test data. the collection consists of articles from a single narrow conference: ieee vis. the test dataset contains the 2916 fullpaper pdfs for the years 1990-2019 @xcite validation test data iou precision recall f1 10k dummy pages 3k dummy pages ieee vis 24,660 pages 0.8 0.94 0.84 0.89 validating deeppapercomposer. since we must have ground-truth in order to quantify the performance of our automatic pipeline using deep-papercomposer, we first curated the ground-truth data: 10 coders checked figure and table tags of 2916 papers. given the groundtruth, we followed the evaluation metrics of @xcite to measure the overall performance obtained by our approach on vispaper-3k. a predicted bounding box is compared to a ground truth based on the jaccard index or intersection over union (iou), and is considered correct when iou exceeds 0.8. extracted figures with identifiers that did not exist in the ground truth were considered incorrect. the results are in ## quantitative evaluation to assess the utility of the vispaper-3k dataset, we conducted two experiments aimed at understanding whether the dataset can be used to extract figures, tables, and captions. ## conclusion we present in this short work-in-progress paper a new training data preparation approach to generate accurate ground-truth labels. our preliminary results showed that our dummy paper composer could be a viable solution to train cnns to extract several semantic and graphical entities. we have released our source code for training data generation online. we plan to diversify the structure and content our paper generator can compose and enable researchers to upload their own data to train models and run the predictions.