D oc P ilot: Copilot for Automating PDF Edit Workflows in Documents (2024)

Abstract: Digital documents, such as PDFs, are vital in business workflows, enabling communication, documentation, and collaboration. Handling PDFs can involve navigating complex workflows and numerous tools (e.g., comprehension, annotation, editing), which can be tedious and time-consuming for users. We introduce DocPilot, an AI-assisted document workflow Copilot system capable of understanding user intent and executing tasks accordingly to help users streamline their workflows. DocPilot undertakes intelligent orchestration of various tools through LLM prompting in four steps: (1) Task plan generation, (2) Task plan verification and self-correction, (3) Multi-turn User Feedback, and (4) Task Plan Execution via Code Generation and Error log-based Code Self-Revision. The primary goal of this system is to free the user from the intricacies of document editing, enabling them to focus on the creative aspects and enrich their document management experience.

## introduction digital documents, particularly pdfs, play a crucial role in business workflows, facilitating communication, documentation, and collaboration. handling pdf documents involves a wide array of functionalities. these include tasks such as understanding content, annotating, editing content (e.g., comments, redaction, highlights), organizing pages (e.g., crop, rotate, extract), adding signatures or watermarks, and form-filling. several document processing applications provide standalone tools and apis to help users complete these tasks. however, accomplishing complex workflows involving numerous tools can be tedious and time-consuming. additionally, unfamiliar users may face challenges in understanding and navigating the various tools available. hence, there is a need for an ai-assisted copilot system that can comprehend the user's intent, clarify un- user request task plan code solution file output docpilot initial pdf file 1.task planning 2.plan verification 3.code generation 4.task execution tool documentation few shot prompting retrieval-based tool selection syntax hallucination argument validity tool hallucination dependency checks guardrails error-log code self revision api encapsulation few-shot prompting code compiler response generation tool name input args output copilot system capable of understanding user intent and executing pdf actions to help users achieve their editing needs. to address these issues, we present docpilot (fig. 1 ), an llm-based framework for automating editing workflows in pdf documents. inspired by recent work like hugginggpt, @xcite and controllm @xcite , docpilot takes the user's requests along with the pdf documents as inputs and leverages llms to infer the user's intent and transforms it into a task plan consisting of a sequence of pdf action tools. the task plan undergoes thorough verification checks to ensure accuracy and reliability. any errors in the task plan are self-corrected by the llm, and the final task plan is then presented to the user in easyto-understand language, inviting feedback through conversation. once the plan is approved by the user, docpilot converts the task plan into a software program that can orchestrate external tool api calls using llm's code generation capabilities. the generated program is simulated using a code interpreter and detected error logs are passed as feedback to the llm for code revision. the resultant error-free code solution executes seamless cooperation between diverse tools and provides users with a modified document that meets their expectations. to assess docpilot's performance in supporting users, we collected user feedback on diverse workflows completed with the help of docpilot. we find that docpilot is effective in improving user productivity by automating repetitive tasks and simplifying complex processes. the main contributions of docpilot are: ## related work recent research informs us how llms can act as autonomous agents for task automation in various application domains @xcite . ai-powered llm agents: frameworks like agentgpt and hugginggpt @xcite leverage llms as a controller to analyze user requests and invoke relevant tools for solving the task. audiogpt @xcite solves numerous audio understanding and generation tasks by connecting llms with input/output interface (asr, tts) for speech conversations. tpu @xcite proposes a structured framework tailored for llm-based ai agents for task planning and execution. @xcite introduced the ghost in minecraft (gitm), a framework of generally capable agents (gcas) that can skillfully nav-igate complex, sparse-reward environments with text-based interactions and develop a set of structured actions executed via llms. assistgpt @xcite proposed an interleaved code and language reasoning approach called plan, execute, inspect, and learn (peil) for processing complex images and long-form videos. recmind @xcite designed an llm-powered autonomous recommender agent capable of leveraging external knowledge and utilizing tools with careful planning to provide zero-shot personalized recommendations. frameworks like autodroid @xcite and appagent @xcite presented smartphone task automation systems that can automate arbitrary tasks on any mobile application by mimicking human-like interactions such as tapping and swiping leveraged through llms like gpt-3.5/gpt-4. adaplanner @xcite allows llm agents to refine their self-generated plan adaptively in response to environmental feedback using few-shot demonstrations. @xcite proposed a tool-augmented chain-of-thought reasoning framework that allows chat-based llms (e.g., chatgpt) to indulge in multi-turn conversations to utilize tools in a more natural conversational manner. creator @xcite built a novel framework that enables llms to create their own tools using documentation and code realization. controlllm @xcite proposed a thoughts-on-graph (tog) paradigm that searches the optimal solution path on a pre-built tool graph to resolve parameter and dependency relations among different tools for image, audio, and video processing. lumos @xcite trained open-source llms with unified data to represent complex interactive tasks. datacopilot @xcite built an llm-based system to autonomously transform raw data into visualization results that best match the user's intent by designing versatile interfaces for data management, processing, and visualization. @xcite connects llms with rest software architectural style (restful) apis, conducts coarse-to-fine online planning, and executes the apis by meticulously formulating api parameters and parsing responses. @xcite explores the use of self-instruct fine-tuning and retrieval to enable llms to accurately select from a large, overlapping, and changing set of apis. llm-grounder @xcite created a novel open-vocabulary llm-based 3d visual grounding pipeline to decompose complex natural language queries into semantic constituents for spatial object identification in 3d scenes. @xcite put forth the autoact framework that automatically synthesizes planning trajectories from experience to alleviate the reliance of copilot systems on largescale annotated data. toolken @xcite addresses the inherent problems of context length constraints and adaptability to a new set of tools by proposing llm tool embeddings. recent work has shown that descriptive tool documentation can be more beneficial than simple few-shot demonstrations for tool-augmented llm automation @xcite . ## docpilot fig. 2 shows docpilot, a chat-based ai assistant framework that uses llm as a controller to translate a user's pdf editing request into an actionable task plan and orchestrates numerous software tools to realize the document editing tasks into modified pdf outputs. docpilot undertakes intelligent orchestration of various llm capabilities into an executable workflow, which includes four steps: (1) task plan generation, (2) task plan verification and self-correction, (3) multi-turn user feedback, and (4) task plan execution via code generation and error log-based code self-revision. ## implementation details backbone llm: we use gpt-4 api through the microsoft azure platform for all our experiments. we also tried gpt-3.5 model but it performed consistently worse than gpt-4 owing to its limited context length and weak code generation abilities. rag architecture: we utilized faiss to construct the data stores for the retrieval-augmented tool selection module. we used sentencebert @xcite as the embedding model. we used scikit learn's knn library to get topk request-task plan pairs. we used gradio for the demo ui hosted on the aws cloud platform. ## user evaluation we conducted a user evaluation to assess the efficacy of docpilot in supporting users' pdf workflows. the research goals were as follows: r1 measure docpilot's performance in suggesting a reasonable plan in response to a userprovided multi-step workflow. relatedly, we wanted to understand how well our system handled ambiguity in user requests. r2 understand when/how breakdowns happen and whether users are able to refine their plan through conversation with docpilot. our data collection focused on a case study with one expert pdf user who works on pdf processing tasks daily as part of his professional work. our evaluator was hired through upwork with expertise in pdf workflows. the evaluator interacted with the docpilot app (fig. 5 ) to complete several workflows and provided feedback through a survey form (methodology details in appendix a.7.1). ## discussion, limitations & future work our evaluation results show that docpilot's task planning step is effective for most workflows as the proposed task plan captures the user's intent well and requires few clarifications by the user. only 10% of the evaluated workflows required more than one llm iteration to self-correct the generated task plans. the majority of breakdowns we observed occurred due to a mismatch in the user's expectations between the plan suggested by docpilot and how it was executed. our current interface with docpilot primarily uses a conversational ui. leveraging interactions from graphical uis can help lessen this gap by providing the user affordances for direct manipulation in content selection when executing a workflow @xcite . additionally, docpilot may allow users to edit action parameters (e.g., page number, password) directly rather than requiring the user to type a new request. both of these future works could increase user control and understanding of the system plan @xcite . quantitatively, we observe that most workflows (69/ 80) required at least one llm-based code revision to produce an error-free program, thus introducing latency and impacting the utility of the tool. self-reported ratings indicate more failures in the task execution step for complex workflows. hence, our future work will focus on instruction-tuning llms on pairs of ground truth task plans and python code. ## conclusion we present docpilot, an llm-powered copilot for automating document workflows. our copilot helps novices plan document workflows by selecting the appropriate tools and executing the task plan autonomously. docpilot benefits the users by enhancing their accessibility, being extensible to include more tools, and being consistently reliable.