D oc E dit-v2: Document Structure Editing Via Multimodal LLM Grounding (2024)

Abstract: Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user’s requests. Past works have shown that multimodal grounding of user requests in the document image and identifying the accurate structural components and their associated attributes remain key challenges for this task. To address these, we introduce the DocEditAgent, a novel framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs). It consists of three novel components – (1) Doc2Command to simultaneously localize edit regions of interest (RoI) and disambiguate user edit requests into edit commands. (2) LLM-based Command Reformulation prompting to tailor edit commands originally intended for specialized software into edit instructions suitable for generalist LMMs. (3) Moreover, DocEditAgent processes these outputs via Large Multimodal Models like GPT-4V and Gemini, to parse the document layout, execute edits on grounded Region of Interest (RoI), and generate the edited document image. Extensive experiments on the DocEdit dataset show that DocEditAgent significantly outperforms strong baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12%) tasks.

## introduction digital documents are widely used for communication, information dissemination, and business productivity. language-guided document editing entails modifying the textual, visual, and structural components of a document in response to a user's open-ended requests related to spatial alignment, component placement, regional grouping, replacement, resizing, splitting, merging, and applying special effects @xcite . document editing is inherently a generative task as it involves the creation of a new edited output from an existing document. the last entry into this register was made on 31 january 2000. text "delta electricity" with the numbering 9 and 9a are converted to main heading "9" and sub heading to "a" and "b" @xcite highlights three key challenges in the end-to-end document editing task -(1) multimodal grounding of ambiguous user requests in the document image, (2) identifying the precise components and their corresponding attributes to be edited, and (3) generating faithful edits without distorting the semantic or spatial coherence of the original document. by interpreting the visual-semantic cues from user requests, multimodal grounding can bridge the gap between natural language instructions and the spatial intricacies of the document's content. sophisticated edit commands, like those found in the docedit dataset @xcite , are usually ambiguous in nature and tailored for use in software-specific applications. disambiguation of such edit commands can help to serve as refined editing instructions for generalist generation models. we hypothesize that directly editing the parsed html/xml document structure can overcome the limitations of pixel-level image generation. prior works like doceditor @xcite ) performed edit commands generation for languageguided document editing but was limited to software-specific applications. generative methods such as diffusion models have shown promise in the visual domain but pose challenges in recreating complex textual and visual elements while preserving the structural information of documents @xcite . unlike natural images, documents contain a combination of text, images, formatting, and layout intricacies @xcite ) that necessitate a more nuanced approach to generative editing. recently, large multimodal models (lmms) like gpt-4v (openai, 2023) and gemini @xcite have demonstrated remarkable capabilities in document understanding, object localization, dense captioning, and code synthesis. prior work has also explored llm program synthesis to compose vision-and-language queries into code subroutines @xcite @xcite . our work aims to solve end-to-end editing of html representation of documents by leveraging the emergent capabilities of lmms to infer the semantic context of edit requests, visually reference them to the region of interest in the document image, determine the spatial elements to be modified, and generate the final document. main results: we present docedit-v2 (fig. 1 ) -an lmm-based end-to-end document editing framework. given a user request on a document, it utilizes a novel doc2command module to ground the edit location in the document image and generate edit commands. doc2command is a transformer-based image encoder-text decodermask transformer model that is jointly trained to perform masked semantic segmentation and ground edit regions of interest (roi) for disambiguating user edit requests into modularized commands. doc2command starts with visually integrating the edit request with the document image, processing them as a unified visual modality through a vision encoder-text decoder backbone to generate the command text. it redefines bounding box detection as a segmentation task by incorporating a mask-attention transformer over the image encoder. further, we propose command reformulation prompting to customize the edit commands into an lmm-specific editing instruction by leveraging the zero-shot in-context learning ability of llms. lastly, docedit-v2 leverages lmms such as gpt-4v and gemini to edit the html structure of the document using a multimodal prompt formed by combining the edit instruction and grounded roi. we design two new metrics -css iou, and dom tree edit distance to evaluate the final edited documents for presentation quality and structural similarity with the ground truth. experiments on the docedit dataset reveal that \texttt{docedit-v2} significantly outperforms strong baselines in edit command generation (by 2-33%), roi bounding box detection (by 12-31%), and overall document editing tasks (by 1-12%). our main contributions are: user request appendix b. tables table b.1. deviations of var estimates for bond indices table b.2. stable var estimates for bond indices with fixed α table b.3. coefficients of ols regressions table b.4. garch-normal coefficients 11 document appendix b. tables table b.1. deviations of var estimates for bond indices table b.2. stable var estimates for bond indices with fixed α table b.3. coefficients of ols regressions table b.4. garch-normal coefficients 11 changed page number in the footer from 11 to 12 render user request on the document image text decoder mask transformer segmentation map visual grounding appendix b. tables table b.1. deviations of var estimates for bond indices table b.2. stable var estimates for bond indices with fixed α table b.3. coefficients of ols regressions table b.4. garch-normal coefficients 11 changed page number in the footer from 11 to 12 appendix b. tables table b.1. deviations of var estimates for bond indices table b.2. stable var estimates for bond indices with fixed α table b.3. coefficients of ols regressions table b.4. garch-normal coefficients 11 changed page number in the footer from 11 to 12 upsample and argmax edit command region of interest user request document changed page number in the footer from 11 to 12 ## related work past works in the domain of language-guided image editing have predominantly centered on natural image datasets @xcite , overlooking the distinctive characteristics of documents, which typically exhibit text-rich content alongside a diverse array of structured elements arranged in various layouts. these datasets often lack representations of localized edits and indirect edit references, crucial facets for effective document editing. notably, contemporary ganbased @xcite @xcite and diffusion methods @xcite @xcite @xcite have gained traction for natural image manipulation tasks due to their capacity for endto-end pixel-level image synthesis. however, their applicability to digital documents, characterized by rich textual content and complex layouts, remains limited. these techniques are ill-equipped to grasp the spatial and semantic intricacies inherent in embedded textual components within documents. consequently, prior endeavors in language-guided document editing have primarily pivoted towards multimodal grounding of edit requests through textual and visual cues into actionable commands and visual localization @xcite . despite these efforts, the absence of efficient generative frameworks tailored for document image editing remains a significant challenge in this domain. ## document editing evaluation we perform system output evaluation as follows: automated metrics: apart from the document metrics reported by @xcite for command text generation (exact match, rouge-l, word overlap f1, action and component accuracy %) and roi bounding box prediction (top-1 accuracy %), we adapt two novel metrics, specific to html document editing: human evaluation: every edited document html is evaluated by three human evaluators on our three proposed metrics: (1) style replication assesses whether the styles of the original document are preserved, (2) content replication evaluates if the textual content of the region of noninterest in the original document html is conserved, (3) edit correctness: judges whether the user's editing intent has been faithfully fulfilled. each of these metrics yields a binary score, which is averaged across evaluators and then summed to compute a unified score for each document. sr cc ec tree_edit_distance css_iou sr cc ec tree_edit_distance css_iou 1 0.26 0.31 -0.001 0.73 0.26 1 0.29 -0.3 0.023 0.31 0.29 1 0.03 0.36 -0.001 -0.3 0.03 1 -0.11 0.73 0.023 0.36 -0.11 1 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 (a) correlation heatmap: tree edit distance and css iou with human evaluation metrics (style replication, content replication, edit correctness). the heatmaps in figure 4 compare our proposed metrics, tree edit distance and css iou, against both human and automated evaluation metrics. in human evaluations, css iou shows a strong correlation (0.73) with style replication, highlighting its sensitivity to visual presentation. tree edit distance, however, compares html document structures, which do not directly relate with any human evaluation parameters, showing no significant correlation results. these results demonstrate the importance of human evaluation supplementary to metric based evaluation. when compared to automated metrics like word overlap f1 and rouge-l, tree edit distance moderately correlates negatively (-0.49, -0.50), as expected, since a higher tree edit distance reflects dissimilar documents. tree edit distance, has a moderate positive correlation with both metrics (0.45), implying that presentation style partially influences text-based overlap. ## baselines command grounding baselines: we investigate several command generation baselines to establish performance benchmarks. initially, we employ seq2seq text-only models, including gpt2 @xcite , bart @xcite , and t5 @xcite , which exclusively process user text descriptions. subsequently, we explore the generator-extractor paradigm, integrating bert @xcite and detr @xcite with autoregressive decoding for command generation. additionally, we examine transformer encoder-decoder architectures, such as layoutlmv3-gpt2 and bert2gpt2 @xcite , which combine gpt2 decoders with layoutlmv3 and bert encoders, respectively. furthermore, we investigate prefix encoding @xcite , utilizing learned rep-resentations from pre-trained encoders like clip @xcite and dit @xcite as a prefix to the gpt2 decoder network. additionally, we consider the multimodal transformer @xcite , which incorporates multimodal input from user descriptions, visual objects, and document text to generate commands. moreover, we explore doceditor @xcite , a taskspecific baseline employing a transformer-based multimodal model that decomposes document images into ocr content and object boxes, utilizing multimodal transformers to generate commands. finally, we compare against gpt3.5 @xcite and gpt4 (openai, 2023), employing incontext learning by providing three examples of each command type as context to the model for evaluation.visual grounding baselines: we consider several baselines for bounding box detection in the context of visual grounding for document editing. firstly, resc-large @xcite presents a method for direct coordinates regression in the region of interest (roi) bounding box prediction task. similarly, transvg @xcite offers an alternative approach for direct coordinates regression in roi bounding box prediction. additionally, we investigate doceditor @xcite , which employs a comprehensive methodology. doceditor initially encodes the document image by extracting text through optical character recognition (ocr) and utilizes object detection to capture visual features. subsequently, transformerencoded features are fed into a gated relational graph convolutional network (r-gcn) to generate a layout graph-aware representation. this representation is then leveraged downstream to perform bounding box regression, facilitating accurate localization of document elements. document editing baselines: certain experimental configurations are employed to investigate the effectiveness of command reformulation and multimodal grounding in harnessing the capabilities of gpt-4v and gemini as document editing tools. specifically, visual grounding, command grounding, and command reformulation are selectively excluded from our experiments. in this context, command grounding is supplanted by the unstructured user request, while visual grounding is eliminated by presenting the original document image as the input, thus eliminating the need for explicit visual cues (rendered bounding boxes). moreover, command reformulation is eliminated by directly utilizing the command generated by the doc2command model. notably, ## results edit request grounding: ## conclusion we introduce the docedit-v2 framework for endto-end document editing. docedit-v2 draws on doc2command, a multi-task multimodal model that visually localizes user requests in the document image and generates edit commands, which are further refined using command reformulation prompting. docedit-v2 uses lmms multi-