D oc KD : Knowledge Distillation from LLM s for Open-World Document Understanding Models (2024)

Abstract: Visual document understanding (VDU) is a challenging task that involves understanding documents across various modalities (text and image) and layouts (forms, tables, etc.). This study aims to enhance generalizability of small VDU models by distilling knowledge from LLMs. We identify that directly prompting LLMs often fails to generate informative and useful data. In response, we present a new framework (called DocKD) that enriches the data generation process by integrating external document knowledge. Specifically, we provide an LLM with various document elements like key-value pairs, layouts, and descriptions, to elicit open-ended answers. Our experiments show that DocKD produces high-quality document annotations and surpasses the direct knowledge distillation approach that does not leverage external document knowledge. Moreover, student VDU models trained with solely DocKD-generated data is not only comparable to those trained with human-annotated data on in-domain tasks but also significantly excel them on out-of-domain tasks.

## introduction visual document understanding (vdu) requires extracting and analyzing both textual and non-textual information from a document. the textual information is usually obtained via optical character recognition (ocr), which only provides unstructured or naïvely ordered text. the non-textual information is visually-rich, demanding a solution to directly process the document image. earlier studies of vdu @xcite @xcite primarily focused on identifying certain parts of a document using heuristics or simple networks. recent approaches @xcite have shifted towards pretraining multi-modal document understanding models to address the model's comprehension of textual, visual, and layout features. however, the existing vdu methods are limited by training on a small-scale, curated document dataset, compromising the generalizability of vdu models to diverse documents. thus, their performance heavily relies on the annotated training document set for downstream tasks. in this study, we aim to improve the generalizability of vdu models by distilling knowledge from large language models (llms). in particular, we introduce an open-world document understanding problem, where the model needs to address the downstream task with a broader scope of documents than covered by the available annotations. llms, given instructions to elicit open-ended answers, can create rich and diverse annotations, as illustrated in fig. 1 . for instance, we might instruct the llm to "generate question-answer pairs from this document", along with document text extracted from ocr. however, this approach entails a critical challenge, since llms often struggle to comprehend unstructured ocr text @xcite , leading to its generation of low-quality annotations. moreover, there is a variety of non-textual information within documents which is not included in the llm prompt. to overcome these challenges, we present dockd, a document knowledge distillation frame-work that leverages external document information to enhance llm data generation. in this framework, we extract various document elements (e.g., key-value pairs, layout, and descriptions) along with text and formulate a generation prompt for llms with this visual information. the llm outputs then serve as annotations to train a small-scale vdu model. while large multimodal models like @xcite are also recognized for their visual-language capabilities, they still lag behind state-of-the-art ocr systems @xcite , but llms that utilize well-structured ocr text excel in document processing and understanding. thus, we employ llms aided with visual tools for data generation. we demonstrate the efficacy of dockd on three document understanding tasks: visual question answering, entity extraction, and classification. in each task, we introduce new tools for incorporating external document knowledge. our experiments reveal that dockd allows student models to attain open document understanding abilities, generalizing to unseen documents, questions, entities, or categories. our contributions are as follows: leveraging llms for data generation. knowledge distillation (kd) from llms has been explored across various natural language processing tasks @xcite . llms like gpt-3 @xcite are utilized for guided annotation of unlabeled data @xcite @xcite or for distilling reasoning capabilities @xcite @xcite which is then used to fine-tune smaller language models. among these, targeted distillation @xcite has demonstrated that identifying and amplifying the llm's knowledge to a high-quality dataset enables student models to attain task-specific knowledge. it has the potential to make specialized language models that outperform in specific tasks, at the expense of generic performances @xcite . in visual instruction tuning research @xcite , llms are employed to generate visual-language instruction-following data. for instance, llava @xcite is trained on the instruction-following dataset for conversation, description, and complex reasoning, created by prompting the llm with bounding box coordinates of objects along with image captions. instructblip @xcite incorporates diverse tasks, such as image question generation and video question answering. closest to our work is the extension of visual instruction tuning to the domain of vdu, generating data with documentspecific knowledge to fine-tune downstream models. @xcite proving the vdu performance of llms or lmms by introducing new designs of encoding document images @xcite @xcite , which are closely related and complementary to our work that focuses on distilling knowledge from strong llms for vdu. our work is the first to extract knowledge from llms for open document understanding tasks, exploring methods to inject visual documentspecific knowledge into llm and produce highquality data for training vdu models. ## document knowledge distillation problem formulation. similar to prior work @xcite @xcite , we formulate document understanding problem under a sequence-to-sequence (seq2seq) generation framework. that is, we design a taskspecific prompt p task which asks a vdu model to solve the task and output an answer a task . dockd involves an llm teacher f t to generate these prompt and answer pairs. given an image of a document page, we apply a pre-built ocr engine to extract its words and word bounding boxes. for simplicity, we represent a document input as d. the overall pipeline of the dockd approach is described in fig. 2 . in fig. 2 (a) , we first construct a generation prompt p gen for the task. then, given p gen and document text d text as inputs, the llm generates a gen , i.e., f t (d text , p gen ) → a gen . this can be readily parsed into (p task , a task ) by postprocessing. here, we can inject document-specific knowledge into the llm inputs, so that it can better understand the document content and generate more accurate (p task , a task ) pairs. in fig. 2 (b), we train a student model f s to output an answer a task given d and p task , i.e., f s (d, p task ) → a task . we exemplify the application of our training pipeline on three document understanding tasks: visual question answering (vqa), entity extraction, and document classification. to summarize each section, we leverage document knowledge by using the ocr linearization model to improve d text (sec. 3.1), using the key-value detection model to guide p gen (sec. 3.2), and introducing the document description into p gen for better class candidates ( sec. 3.3 ). refer to appx. b for the full templates of p gen in each task. ## conclusion we address the open-world document understanding problem by instructing the llms to generate document annotations, given the generation prompt and ocr text. to successfully achieve this, we suggest dockd framework, designing task prompts and answers that llms can easily generate, and incorporate external document knowledge from various sources. consequently, the student models distilled by dockd annotations demonstrate remarkable performance improvements compared to the plain kd approach in various document tasks. the integration with human-labeled annotations further enhances model performance.