VILA : Improving Structured Content Extraction from Scientific PDF s Using Visual Layout Groups (2022)

Abstract: Accurately extracting structured content from PDFs is a critical first step for NLP over scientific papers. Recent work has improved extraction accuracy by incorporating elementary layout information, for example, each token’s 2D position on the page, into language model pretraining. We introduce new methods that explicitly model VIsual LAyout (VILA) groups, that is, text lines or text blocks, to further improve performance. In our I-VILA approach, we show that simply inserting special tokens denoting layout group boundaries into model inputs can lead to a 1.9% Macro F1 improvement in token classification. In the H-VILA approach, we show that hierarchical encoding of layout-groups can result in up to 47% inference time reduction with less than 0.8% Macro F1 loss. Unlike prior layout-aware approaches, our methods do not require expensive additional pretraining, only fine-tuning, which we show can reduce training cost by up to 95%. Experiments are conducted on a newly curated evaluation suite, S2-VLUE, that unifies existing automatically labeled datasets and includes a new dataset of manual annotations covering diverse papers from 19 scientific disciplines. Pre-trained weights, benchmark datasets, and source code are available at https://github.com/allenai/VILA .

## introduction scientific papers are usually distributed in portable document format (pdf) without extensive semantic markup. extracting structured document representations from these pdf files-i.e., identifying title and author blocks, figures, references, and so on-is a critical first step for downstream nlp tasks @xcite and is important for improving pdf accessibility @xcite . recent work demonstrates that document layout information can be used to enhance content extraction via large-scale, layout-aware pretraining @xcite @xcite . however, these methods only consider individual tokens' 2d positions and do not explicitly model high-level layout structures like the grouping of text into lines and blocks (see figure 1 for an example), limiting accuracy. further, existing methods come with enormous computational costs: they rely on further pretraining an existing pretrained model like bert @xcite on layout-enriched input, and achieving the best performance from the models requires more than a thousand @xcite to several thousand @xcite gpu-hours. this means that swapping in a new pretrained text model or experimenting with new layout-aware architectures can be prohibitively expensive, incompatible with the goals of green ai @xcite . in this paper, we explore how to improve the accuracy and efficiency of structured content extraction from scientific documents by using visual layout (vila) groups. following @xcite and @xcite , our methods use the idea that a document page can be segmented into visual groups of tokens (either lines or blocks), and that the tokens within each group generally have the same semantic category, which we refer to as the group uniformity assumption (see figure 1(b) ). given text lines or blocks generated by rule-based pdf parsers @xcite or vision models @xcite , we design two different methods to incorporate the vila groups and the assumption into modeling: the i-vila model adds layout indicator tokens to textual inputs to improve the accuracy of existing bert-based language models, while the h-vila model uses vila structures to define a hierarchical model that models pages as collections of groups rather than of individual tokens, increasing inference efficiency. previous datasets for evaluating pdf content extraction rely on machine-generated labels of imperfect quality, and comprise papers from a limited range of scientific disciplines. to better evaluate our proposed methods, we design a new benchmark suite, semantic scholar visual layout-enhanced scientific text understanding evaluation (s2-vlue). the benchmark extends two existing resources @xcite and introduces a newly curated dataset, s2-vl, which contains high-quality human annotations for papers across 19 disciplines. our contributions are as follows: the h-vila model performs group-level predictions and can reduce model inference time by 47% with less than 0.8% loss in macro f1. the benchmark datasets, modeling code, and trained weights are available at https://github .com/allenai/vila . ## benchmark suite to systematically evaluate the proposed methods, we develop the the semantic scholar visual grotoap2 docbank s2-vl train / dev / test pages 83k / 18k / 18k 398k / 50k / 50k 1.3k 1 annotation method automatic automatic human annotation scientific discipline life science math / physics / cs 19 disciplines visual layout group pdf parsing vision model gold label / detection methods number of categories 22 12 15 average token count 2 1203 (591) 838 (503) 790 (453) average text line count 90 (51) 60 (34) 64 (54) average text block count 12 (16) 15 (8) 22 (36) layout-enhanced scientific text understanding evaluation (s2-vlue) benchmark suite. s2-vlue consists of three datasets-two previously released resources that we augment with vila information, and a new hand-curated dataset s2-vl. key statistics for s2-vlue are provided in ## conclusion in this paper, we introduce two new ways to integrate visual layout (vila) structures into the nlp pipeline for structured content extraction from scientific paper pdfs. we show that inserting special indicator tokens based on vila (i-vila) can lead to robust improvements in token classification accuracy (up to +1.9% macro f1) and consistency (up to -32% group category inconsistency). in addition, we design a hierarchical transformer model based on vila (h-vila), which can reduce inference time by 46% with less than 0.8% macro f1 reduction compared to previous sota methods. these vila-based methods can be easily incorporated into different bert variants with only fine-tuning, achieving comparable performance against existing work with only 5% of the training cost. we ablate the influence of different visual layout detectors on vila-based models, and provide suggestions for practical use. we release a benchmark suite, along with a newly curated dataset s2-vl, to systematically evaluate the proposed methods. our study is well-aligned with the recent exploration of injecting structures into language models, and provides new perspectives on how to incorporate documents' visual structures. the approach shows how explicitly modeling task structure can help achieve ''green ai'' goals, dramatically reducing computation and energy costs without significant loss in accuracy. while we evaluate on scientific documents, related visual group structures also exist in other kinds of documents, and adapting our techniques to those domains could offer improvements in corporate reports, historical archives, or legal documents, and this is an item of future work.