DOCMASTER : A Unified Platform for Annotation, Training, & Inference in Document Question-Answering (2024)

Abstract: The application of natural language processing models to PDF documents is pivotal for various business applications yet the challenge of training models for this purpose persists in businesses due to specific hurdles. These include the complexity of working with PDF formats that necessitate parsing text and layout information for curating training data and the lack of privacy-preserving annotation tools. This paper introduces DOCMASTER, a unified platform designed for annotating PDF documents, model training, and inference, tailored to document question-answering. The annotation interface enables users to input questions and highlight text spans within the PDF file as answers, saving layout information and text spans accordingly. Furthermore, DOCMASTER supports both state-of-the-art layout-aware and text models for comprehensive training purposes. Importantly, as annotations, training, and inference occur on-device, it also safeguards privacy. The platform has been instrumental in driving several research prototypes concerning document analysis such as the AI assistant utilized by University of California San Diego’s (UCSD) International Services and Engagement Office (ISEO) for processing a substantial volume of PDF documents.

## introduction documents and forms are omnipresent within enterprises encompassing financial bills like invoices, purchase records, financial statements, and official communications such as notices and announcements. the application of machine learning for automating document processing stands to significantly accelerate processing times @xcite . visually rich document understanding has recently attracted much attention from researchers. a ♠ corresponding authors simple approach involves parsing text from pdfs and leveraging established natural language processing (nlp) models @xcite @xcite . however, these methods overlook the valuable layout information embedded within pdfs. a series of works have been done to incorporate the layout features into the pre-training framework. layoutlm @xcite first proposes to encode the spatial relationships of words by embedding their position coordinates in an embedding layer. following this direction, @xcite ; @xcite move beyond basic embedding techniques and specifically adapt the attention layers of the transformer architecture to model the relative positional relationships within the 2d space of document pages. @xcite ; @xcite , on the other hand, purse a more comprehensive understanding of the layout structure. they achieve this by encoding the hierarchical relation in the documents. despite the availability of these models, the persistent challenge lies in training them with custom business data due to particular obstacles. firstly, working with the intricacies of the pdf format proves to be a nontrivial task @xcite . pdfs store text as character glyphs along with their positions on a page, necessitating complex operations to convert this data into usable text for nlp models. operations like inferring token boundaries and managing white spacing are error-prone and add to the complexity. secondly, organizations frequently handle sensitive documents that demand in-house tools for annotating and curating training data. addressing these obstacles, we introduce docmaster, a unified platform designed for annotating pdf documents, model training, and inference for the question-answering (qa) task, as shown in figure 1 . the annotation interface is designed to maintain the layout integrity, requiring users to upload pdfs, provide questions, and highlight their specific text spans as answers in the pdfs. once identified, it processes the pdf content, saving both textual and layout details. privacy measures involve on-device processing, eliminating reliance on third-party services, and securely storing annotations within a local database. docmaster accommodates an extensive array of models, encompassing both layout-aware models like layoutlm @xcite ) and text-only ones such as roberta @xcite . the inference interface is user-friendly and accepts a pdf document and a trained model. it simplifies the task of locating answers to specific questions by highlighting relevant spans within the pdf document. we deployed docmaster in a practical scenario within the iseo at ucsd, addressing the processing of hundreds of supporting documents for students to issue their work permits. previously, staff members engaged in the manual review of each document before approving work permits. through docmaster, they annotated and trained a qa model seamlessly, leading to a remarkable seven-fold increase in the average number of documents processed per hour. we present video demonstration, live demo website, and code on the project webpage. xu et al. (2020); hong et al. (2022); garncarek et al. (2021) propose to use the coordinates of words in the page as the representation for the layout structure. they embed the coordinates in the embedding layer and add relative weights in the self-attention layers. xu et al. (2021); huang et al. (2022) incorporate the visual features from the document images. following the previous works, tang et al. (2023); lv et al. (2023); perot et al. (2023) enlarge the scale of pretraining and improve language models capability in understanding pdfs of various formats. systems for document ai document ai is drawing significant interest from both academia and industry. in addition to various language modeling techniques, major companies have also launched their proprietary document ai services, including google cloud foot_1 , microsoft azure foot_2 , amazon web services foot_3 , etc. although proprietary systems offer convenient and stable services, they are primarily business-oriented and lack transparency for those outside the company. additionally, there are non-commercial document ai systems available, such as lo et al. (2023); bryan et al. (2023). however, none of these systems comprehensively enable users to combine annotation, training, and inference within a single system. in contrast, docmaster allows users to navigate the entire pipeline of document-qa task, successfully eliminating programming barriers that hinder general users from utilizing document ai tools. ## docmaster: design this section delves into the design aspects of our platform. docmaster has three interfaces: (1) the annotation interface, which processes a zip file containing pdf documents, enabling user annotation through text highlighting. ## docmaster: building your document qa system how can an organization utilize docmaster to implement a document qa system tailored to their use case? in this section, we illustrate a hypothetical scenario where the hr department of a company seeks to improve its onboarding process through the integration of a qa system. privacy-preserving shang data lab, inc. has an hr team aiming to implement a qa system for new hires to address queries related to various onboarding documents. however, due to the sensitive nature of these documents, the hr team is cautious about utilizing third-party services considering potential data leakage @xcite . their preference is to ensure internal documents never leave their servers. recognizing the opensourced system docmaster for its emphasis on privacy, shang data lab, inc. finds it to be a suitable solution meeting their specific requirements. ease of deployment setting up docmaster is straightforward, involving the cloning of source code and the execution of a single command: "docker compose up". leveraging docker, a widely adopted containerization software, shang data lab, inc. can swiftly have their own docmaster operational within a few minutes. parallel annotation intending to train a qa model to aid in comprehending onboarding documents, the hr team at shang data lab, inc. allocates tasks to each team member, requiring them to generate questions for a subset of documents to curate training data. utilizing docmaster, each team member logs in and uploads a few onboarding pdfs to the annotation interface. within this interface, they can annotate the answers to their questions by highlighting relevant text in the pdf. working concurrently, the hr team successfully compiles a training dataset containing multiple questions and corresponding answers relevant to each onboarding document. training & inference with the newly curated dataset, shang data lab, inc. initiates the training of qa models seamlessly through the training interface. utilizing the platform's features, they have the flexibility to opt for training either a textonly or layout-aware model. once the model is trained, they can deploy it using the inference interface, enabling new hires to leverage its capabilities. new hires can easily upload a pdf and input a set of questions, receiving not only accurate answers but also benefiting from the ability to precisely locate the answers within the document through highlighted references. this scenario highlights the versatility of docmaster and its aptitude to address specific needs within the ai-as-a-service ecosystem. ## public deployment: takeaways & testimonials the iseo at ucsd 7 oversees immigration services for international students. this responsibility encompasses tasks such as certifying students' admission to full-time study programs, issuing work permits, and managing various other related processes. each certification request undergoes a meticulous manual review of its accompanying supporting documents. during peak periods, the volume of applications can reach into the thousands. presently, each request is processed manually, involving a staff member who reviews supporting 7 https://ispo.ucsd.edu/ microsoft corporation is an equal opportunity employer. cid: cas-450528-s7l8x1 re: internship offer for to whom it may concern: founded in 1975, microsoft corporation is the worldwide leader in software, services and hardware that deliver new opportunities, greater convenience, and enhanced value to people's lives. microsoft corporation develops, licenses, and supports a wide range of software products and services, by designing and selling hardware, and by delivering relevant online advertising to a global customer audience. the corporation produces software in over 40 languages and has operations in more than 100 countries. microsoft employs approximately 182,268 individuals worldwide, including about 103,894 us workers, primarily at its corporate headquarters in redmond, washington. revenues for the most recent fiscal year were @xmath061.3 billion. the purpose of this letter is to confirm microsoft corporation's offer to of a full-time position as a research intern, beginning 6/6/2022 and ending 9/2/2022. during the internship, will work under the supervision of microsoft senior researcher, job duties and responsibilities will include a focus on analyzing and improving performance of advanced algorithms on large-scale datasets and cutting-edge research in machine intelligence and machine learning applications. implementing prototypes of scalable systems in ai applications will be a part of his job duties. he will be expected to collaborate closely with team members on developing systems from prototyping to production level. his duties will involve developing solutions for real world, largescale problems. to tackle this issue, we deployed docmaster to streamline the review process, focusing on a specific scenario: the issuance of work permits for internships, as a prototype use case. traditionally, iseo staff manually verifies essential fields and grants approval for work permits upon the submission of supporting documents by students. the eight key fields subject to review include employer details, salary information, job description, supervisor name & email address, weekly work hours, internship location, and start & end dates. we formulate this as a qa task and train a model to extract necessary fields @xcite . to generate training data, four individuals without a machine learning background utilize docmaster's annotation interface to annotate five documents each. each annotator formulates a question for every required field and highlights the relevant answer span within the pdf @xcite . the collected annotations, encompassing both text and layout information, are aggregated. subsequently, we train two models, roberta-base and layoutlm-base, for three epochs using this annotated dataset. during the inference phase, new student documents are uploaded to the interface, and the staff member inputs questions corresponding to the required fields. the user then selects the trained model, and answers for each field are high- ## conclusion this work introduces docmaster, a unified document-qa platform designed for annotation, training, and inference while prioritizing privacy preservation. docmaster aims to empower users to train and deploy their models for document qa purposes. despite the availability of various models, there is a scarcity of open-sourced annotation platforms. addressing this gap, docmaster is presented as an open-source solution where users can annotate pdfs effortlessly by simply highlighting relevant text. the platform has demonstrated its efficacy in constructing ucsd iseo's ai assistant, contributing to a noteworthy seven-fold reduction in document processing time. the open-sourcing of docmaster is intended to empower businesses that necessitate in-house document qa platforms.