PAWLS : PDF Annotation With Labels and Structure (2021)

Abstract: Adobe’s Portable Document Format (PDF) is a popular way of distributing view-only documents with a rich visual markup. This presents a challenge to NLP practitioners who wish to use the information contained within PDF documents for training models or data analysis, because annotating these documents is difficult. In this paper, we present PDF Annotation with Labels and Structure (PAWLS), a new annotation tool designed specifically for the PDF document format. PAWLS is particularly suited for mixed-mode annotation and scenarios in which annotators require extended context to annotate accurately. PAWLS supports span-based textual annotation, N-ary relations and freeform, non-textual bounding boxes, all of which can be exported in convenient formats for training multi-modal machine learning models. A read-only PAWLS server is available at https://pawls.apps.allenai.org/ , and the source code is available at https://github.com/allenai/pawls .

## introduction scholars of natural language processing technology rely on access to gold standard annotated data for training and evaluation of learning algorithms. despite successful attempts to create machine readable document formats such as xml and html, the portable document format (pdf) is still widely used for read-only documents which require visual markup, across domains such as scientific publishing, law, and government. this presents a challenge to nlp practitioners, as the pdf format does not contain exhaustive markup information, making it difficult to extract semantically meaningful regions from a pdf. annotating text extracted from pdfs in a plaintext format is difficult, because the extracted text stream lacks any organization or markup, such as paragraph boundaries, figure placement and page headers/footers. existing popular annotation tools such as brat @xcite focus on annotation of user provided plain text in a web browser specifically designed for annotation only. for many labeling tasks, this format is exactly what is required. however, as the scope and ability of natural language processing technology goes beyond purely textual processing due in part to recent advances in large language models @xcite , the context and media in which datasets are created must evolve as well. in addition, the quality of both data collection and evaluation methodology is highly dependent on the particular annotation/evaluation context in which the data being annotated is viewed @xcite . annotating data directly on top of a html overlay on an underlying pdf canvas allows naturally occurring text to be annotated in its original context -that of the pdf itself. to address the need for an annotation tool that goes beyond plaintext data, we present a new annotation tool called pawls (pdf annotation with labels and structure). in this paper, we discuss some of the pdf-specific design choices in pawls, including automatic bounding box uniformity, freeform annotations for non-textual image regions and scale/dimension agnostic bounding box storage. we report agreement statistics from an initial round of labelling during the creation of a pdf structure parsing dataset for which pawls was originally designed. ## design choices as shown in figure 1 , the primary operation that pawls supports is drawing a bounding box over a pdf document with the mouse, and assigning that region of the document a textual label. pawls supports drawing both freeform boxes anywhere on the pdf, as well as boxes which are associated with tokens extracted from the pdf itself. this section describes some of the user interface design choices in pawls. ## implementation pawls is implemented as a python-based web server which serves pdfs, annotations and other metadata stored on disk in the json format. the user interface is a single page application implemented using typescript and relies heavily on the react web framework. pdfs are rendered using pdf.js. pawls is designed to be used in a browser, with no setup work required on the behalf of annotators apart from navigating to a web page. this makes annotation projects more flexible as they can be distributed across a variety of crowd-sourcing platforms, used in house, or run on local machines. pawls development and deployment are both managed using the containerization tools docker and docker compose, and multiple pawls instances are running on a google cloud platform kubernetes cluster. authentication in production environments is managed via google account logins, but pawls can be run locally by individual users with no authentication. ## case study pawls enables the collection of mixed-mode annotations on pdfs. pawls is currently in use for a pdf layout parsing project for academic papers, for which we have collected an initial set of gold standard annotations. this dataset consists of 80 pdf pages with 2558 densely annotated bounding boxes of 20 categories from 3 annotators. several pdf based datasets exist for document parsing, such as docbank @xcite , pub-lenet @xcite and tablebank @xcite . however, both docbank and pub-lenet are constructed using weak supervision from latex parses or pubmed xml information. table-bank consists of 417k tables extracted from mi-crosoft word documents and computer generated pdfs. this approach is feasible for common elements of document structure such as tables, but is not possible for custom annotation labels or detailed figure/table decomposition. the pawls interface is similar to tools which augment pdfs for reading or note taking purposes. along with commercial tools such as adobe reader, sidenoter @xcite augments pdfs with rich note taking and linguistic annotation overlays, directly on the pdf canvas. scholarphi @xcite augments the pdf reading experience with equation overlays and definition modals for symbols. as a pdf specific annotation tool, pawls adds to the wider landscape of annotation tools which fulfil a particular niche. slate @xcite provides a command line annotation tool for expert annotators; @xcite provides an annotation interface specifically designed for cross-lingual annotation in which the annotators do not speak the target language. textual annotation tools such as brat @xcite , pubtator @xcite , @xcite , or textanno @xcite are recommended for annotations which do not require full pdf context, or for which extension to multi-modal data formats is not possible or likely. we view pawls as a complimentary tool to the suite of text based annotation tools, which support more advanced types of annotation and configuration, but deal with annotation on extracted text removed from its originally published format. in particular, we envisage scholarly document annotation as one of the key use cases for pawls, as pdf is a widely used format in the context of scientific publication. several recently published datasets leave document structure parsing or multimodal annotation to future work. for example, the scirex dataset @xcite use the text-only latex source of arxiv papers for dataset construction, leaving table and figure extraction to future work. multiple iterations of the evidence inference dataset @xcite use textual descriptions of interventions in clinical trial reports; answering inferential questions using figures, tables and graphs may be a more natural format for some queries. ## conclusion in this paper, we have introduced a new annotation tool, pawls, designed specifically with pdfs in mind. pawls facilitates the creation of multimodal datasets, due to its support for mixed mode annotation of both text and image sub-regions on pdfs. additionally, we described several user interface design choices which improve the resulting annotation quality, and conducted a small initial annotation effort, reporting high annotator agreement. pawls is released as an open source project under the apache 2.0 license.