ATLAS : A System for PDF -centric Human Interaction Data Collection (2024)

Abstract: The Portable Document Format (PDF) is a popular format for distributing digital documents. Datasets on PDF reading behaviors and interactions remain limited due to the challenges of instrumenting PDF readers for these data collection tasks. We present ATLAS, a data collection tool designed to better support researchers in collecting rich PDF-centric datasets from users. ATLAS supports researchers in programmatically creating a user interface for data collection that is ready to share with annotators. It includes a toolkit and an extensible schema to easily customize the data collection tasks for a variety of purposes, allowing collection of PDF annotations (e.g., highlights, drawings) as well as reading behavior analytics (e.g., page scroll, text selections). We open-source ATLAS1 to support future research efforts and review use cases of ATLAS that showcase our system’s broad applicability.

## introduction collecting high-quality datasets from humans across varied domains has been one of the core driving factors for advances in artificial intelligence (ai) @xcite . recent progress in ai only makes the importance of such data more pronounced; as a canonical example, methods such as reinforcement learning from human feedback (rlhf) @xcite @xcite , one of the core technology for large language model (llm) fine-tuning and alignment @xcite , critically depends on large-scale, high-quality human data such as chat, preference ranking, and question answering for both model training and evaluation @xcite @xcite . in addition, the capabilities of ai systems are highly context-dependent and subjectively interpreted depending on the context being used and users' backgrounds @xcite . thus, datasets within varied interaction contexts and collected from diverse users can help address these challenges by revealing the possibilities and limitations of ai systems. for example, datasets are central for evaluating language model's (lms) capabilities in natural language processing (nlp) research @xcite . in human-computer interaction (hci) research, interaction datasets can help designers understand the capabilities of ai technology and inform interaction design and user experience design choices @xcite @xcite . one of the most critical ingredients to human data collection is a tool that supports such deeds. the past few years have seen a surging need to collect such data, stimulating the rapid development of data collection tools and systems that cover various tasks and diverse modalities. 2 however, currently and notably missing from the landscape are tools that support human data collection on digital documents in the form of the portable document format (pdf). indeed, the pdf is one of the most popular digital document formats with an estimated 2.5 trillion pdfs in the world today @xcite . it is extensively employed across various industries such as healthcare, government, education, finance, legal, and e-commerce as the de facto standard for transactions, documentation, and communication. however, few, if any, datasets exist that comprehensively capture users' interactions with pdf documents. we posit one reason for this scarcity of pdf datasets is the challenging nature of processing and instrumenting pdf readers to capture rich interaction data while maintaining good usability. recently, several tools have been developed specifically to enable pdf annotations. pdfanno @xcite and pawls (neumann et al., 2021) are two representative tools, both of which enable annotating the pdf content, such as drawing and labeling bounding boxes for various pdf elements including texts, charts, headings, and captions. however, these existing tools only support annotating the pdf content and cannot support collecting human data based on the pdf such as readers' interactions with the pdfs including highlights, comments, and questions, nor can they support evaluating ai systems' outputs based on pdfs such as question answering quality and hallucination @xcite @xcite . to enable the next-generation ai systems that collaborate with human users and readers on digital documents, we first need a tool that can aid the collection of large-scale and rich human data based on pdfs for training and evaluating document-based models. ## prior work most existing data annotation tools @xcite @xcite @xcite support data collection for many data modalities and tasks except for pdfs. a few tools @xcite support annotating content in pdfs but do not support collecting human interaction data with the pdfs. atlas bridges the gap with a suite of features to support exactly these two scenarios, complementing the already vast landscape of existing annotation tools and software. because pdfs are designed to be read-only and immutable, providing support for native pdf annotations can be challenging. therefore, a related line of efforts focuses on "morphing" the pdfs into other formats for easier annotation and processing. for example, @xcite proposed a method to parse pdfs into html format. however, converting pdfs to other formats usually loses some aspects of its original appeal, such as the persistence of its visual elements and layouts. it is well known the usability of the data annotation interface and presentation of data can impact user perception and thus data quality @xcite @xcite @xcite . therefore, when collecting human interaction data with pdfs, using pdfs directly as part of the data collection and evaluation process is highly preferable. atlas enables this by providing a consistent ui for the annotator with a native, integrated pdf viewer capable of collecting fine-grained user interactions. many applications claim to perform intelligent tasks on pdfs such as search and retrieval, question answering, and summarization. 3 however, these applications are typically not transparent in evaluating how well they perform in these tasks and compare to competitors. publicly available evaluation datasets and benchmarks are indispensable. many existing datasets and benchmarks are textonly @xcite @xcite , overlooking pdf documents. authors of a few recent works collect pdfs as part of the dataset @xcite @xcite @xcite , but the focus of these works are either annotations on the content of the pdfs and rarely on the human interactions with the pdfs @xcite . atlas provides an opportunity to enable easier and larger-scale collection of pdf-based human interaction data to benefit future developments of ai systems for documents and benchmark the progress. some of the existing works are already empowered by atlas. for example, @xcite developed a model for question answering over long, structured pdfs, in which atlas was central to collecting human data for evaluation. ## atlas design choices atlas supports researchers in programmatically creating a user interface for data collection that is ready to share with annotators. atlas scaffolds frontend creation and backend database management so that researchers can focus on the content needed for their data collection project. figure 1 shows an overview of using atlas. the next sections describe each of atlas's user interface and design choices. a demo video is available in the project github repository foot_3 and in the supplementary material. ## implementation backend: atlas is implemented using the webapp framework, ruby on rails foot_4 . postgresql is used for the database. there are three key data models: 1) a document table handles file metadata and nomenclature. 2) a project table encapsulates comprehensive details about each project. 3) a project assignments table tracks completed annotator responses. ## conclusions in this paper, we presented atlas, an open-source system for collecting pdf-centric human interaction data. atlas complements existing data collection tools by focusing on pdf-based user interactions and supports a wide range of interaction data collection tasks, such as question-and-answer pairs, qa attributions, and reading annotations and behaviors. it features a programmatically generated user interface, a toolkit for creating data collection tasks, and a suite of utilities for processing and analyzing the collected data. we demonstrated at-las's capabilities and applicability through several real-world use cases. we believe that atlas will be a valuable tool for researchers and practitioners working with pdf-based human interaction data, and we hope that it will enable new and exciting research in this area.