“What is the value of templates ?” Rethinking Document Information Extraction Datasets for LLM s (2024)

Abstract: The rise of large language models (LLMs) for visually rich document understanding (VRDU) has kindled a need for prompt-response, document-based datasets. As annotating new datasets from scratch is labor-intensive, the existing literature has generated prompt-response datasets from available resources using simple templates. For the case of key information extraction (KIE), one of the most common VRDU tasks, past work has typically employed the template “What is the value for the key?”. However, given the variety of questions encountered in the wild, simple and uniform templates are insufficient for creating robust models in research and industrial contexts. In this work, we present K2Q, a diverse collection of five datasets converted from KIE to a prompt-response format using a plethora of bespoke templates. The questions in K2Q can span multiple entities and be extractive or boolean. We empirically compare the performance of seven baseline generative models on K2Q with zero-shot prompting. We further compare three of these models when training on K2Q versus training on simpler templates to motivate the need of our work. We find that creating diverse and intricate KIE questions enhances the performance and robustness of VRDU models. We hope this work encourages future studies on data quality for generative model training.

## introduction visually rich document understanding (vrdu) is a core research field at the intersection of natural language processing (nlp) and computer vision. the field aims to develop methods to process information from documents and perform natural language inference on them. vrdu is crucial for automation in industries such as finance, legal, healthcare, and 1. what is the {key} with customer number {cust_id}? 2. what is the {key} of the {line_item_pos} item? 3. how much is the {key} for {line_item_desc}? 4. is "{value}" the {key} in this document? ## key information extraction in vrdu we focus on kie in this work due to the abundance of existing datasets in the vrdu literature and the importance of information extraction in industry. we hope future work will examine other popular structured vrdu tasks such as document classification @xcite or document structure prediction @xcite @xcite . kie is the task of identifying important entities within a document. as such, many datasets have naturally examined invoices and receipts @xcite @xcite , where entities are clearly defined and often connect through line items (e.g., the name and price of an item appear on the same "line"). similarly, forms and legal documents have also been the focus of a plethora of datasets @xcite @xcite @xcite . we provide details regarding available kie datasets in ## the k2q dataset we present key information extraction transformed to visual question answering datasets, or k2q for short. this is a collection of five kie datasets transformed into prompt-response type datasets (similar to vqa): ad-buy, cord, docile, kleister charity, and reg. form. these datasets were chosen as they cover a wide range of domains within the field of vrdu. k2q contains more than 300,000 questions over 12,000 documents. figure 2 provides several examples from k2q along with their respective source dataset. the questions are designed to reflect the intricacies of document understanding, such as co-references (e.g., "how much did the charity with number 1154288 earn annually in british pounds?"), disambiguation of similar entities (e.g., "how much is the total amount with tax of the 19th item?"), and questions spanning multiple pages. ## conclusion this paper introduced k2q, a new publicly available collection of five transformed kie datasets for generative vrdu models. k2q provides a large and diverse set of template-based questions that better capture the intricacies of kie and the variety of questions that users can ask in real-world applications. we present a middle ground between llmgenerated and manually-curated questions for instruction tuning, which trades off the time to craft such data against diversity. our approach can be extended to domains such as general vqa, multi-turn multimodal conversation, and video qa, which we leave to future work. our experiments demonstrate that training generative models on k2q instead of data from simple templates improves generalization to held-out types of instructions. in addition, our error analysis suggests that questions in k2q provide enhanced contextualization compared to simple ones, resulting in more grounded answers from models, regardless of correctness. in future work, we plan to create templates covering a wider range of question types, few-shot instances, chainof-thought answers with layout-informed explanations, and multi-round instructions. we hope this work encourages researchers to carefully consider the data used for generative modeling.