P aper M age: A Unified Toolkit for Processing, Representing, and Manipulating Visually-Rich Scientific Documents (2023)

Abstract: Despite growing interest in applying natural language processing (NLP) and computer vision (CV) models to the scholarly domain, scientific documents remain challenging to work with. They’re often in difficult-to-use PDF formats, and the ecosystem of models to process them is fragmented and incomplete. We introduce PaperMage, an open-source Python toolkit for analyzing and processing visually-rich, structured scientific documents. PaperMage offers clean and intuitive abstractions for seamlessly representing and manipulating both textual and visual document elements. PaperMage achieves this by integrating disparate state-of-the-art NLP and CV models into a unified framework, and provides turn-key recipes for common scientific document processing use-cases. PaperMage has powered multiple research prototypes of AI applications over scientific documents, along with Semantic Scholar’s large-scale production system for processing millions of PDFs. GitHub: https://github.com/allenai/papermage

## vignette: building an attributed qa system for scientific papers how could researchers leverage papermage for their research? here, we walk through a user scenario in which a researcher (lucy) is prototyping an attributed qa system for science. system design. drawing inspiration from @xcite , @xcite , and @xcite , lucy is studying how language models can be used to resolve questions that arise while reading a paper (e.g. what does this mean? or what does this refer to?). in her prototype interface, a user can highlight a passage in a pdf and ask a question about it. a retrieval model then finds relevant passages from the rest of the paper. the prototype then uses the text of the retrieved passages along with the user question to prompt a language model to generate an answer. when presenting the answer to the user, the prototype also visually highlights the retrieved passages as supporting evidence to the generated answer. getting started quickly. as a researcher proficient in python, it only takes lucy minutes to install papermage using pip and successfully process a local pdf file by following the example code snippet for corerecipe in §3.2. in an interactive session, she familiarizes herself with the provided layers by following the traversal, cross-referencing and querying examples in §3.1. she makes sure she can serialize and re-instantiate her document ( §a.2). formatting input. before using papermage, lucy has prior experience building qa pipelines, but has only dealt with documents as sentencesplit text data (e.g., <list[str]>). lucy realizes that she can reuse her prior text-only code with papermage by implementing a couple of wrappers to gain additional capabilities: first, she converts a user's highlighted passage from a visual selection to text following the example in figure 3f . next, she converts document to her required text format by following the traversal examples in §3.1 (e.g., using [s.text for s in doc.sentences]). within a few lines of code, lucy has everything she needs for text-only input to her qa pipeline. formatting output. lucy runs her qa system on her newly acquired text data and now has (1) a model-generated answer and (2) several retrieved evidence passages. she realizes that she already has access to the evidences' bounding boxes via a similar call to how she defined the model input context (e.g., [s.boxes for s in doc.sentences]). she can easily pass this to the user interface to enable linking to and highlighting of those passages. defining a predictor. the pattern lucy has followed is used in our many predictor implementations: (1) gain access to text by traversing layers (e.g., sentences), (2) perform all usual nlp computation on that text, and (3) format model output as entities. this simple pattern allows users to reuse familiar models in existing frameworks and eschews lengthy onboarding to papermage. lucy wraps her prompting and retrieval code in new classes: apipredictor and snippetretrievalpredictor (see ## conclusion in this work, we've introduced papermage, an open-source python toolkit for processing scientific documents. papermage was developed to supply high-quality data and reduce friction for research prototype development at semantic scholar. today, it is being used in the production pdf processing pipeline to provide data for both the literature graph @xcite and the paper-reading interface @xcite . it has also been used in working research prototypes which have since contributed to research publications @xcite . 6 we open-source papermage in hopes it will simplify research workflows that depend on scientific documents and promote extensions to other visuallyrich documents like textbooks @xcite and digitized print media @xcite .