B u DDIE : A Business Document Dataset for Multi-task Information Extraction (2025)

Abstract: The field of visually rich document understanding (VRDU) aims to solve a multitude of well-researched NLP tasks in the multi-modal domain. Several datasets exist for research on specific tasks of VRDU, such as document classification (DC), key entity extraction (KEE), entity linking, visual question answering (VQA), inter alia. These datasets cover documents like invoices and receipts with sparse annotations such that they support one or two co-related tasks (e.g., entity extraction and entity linking). Unfortunately, only focusing on a single specific type of documents or task is not representative of how documents often need to be processed in the wild – where variety in style and requirements is expected. In this paper, we introduce BuDDIE: Business Document Dataset for Information Extraction, the first multi-task dataset of 1665 real-world business documents that contains rich and dense annotations for DC, KEE, and VQA. Our dataset consists of publicly available business entity documents from US state government websites. The documents are structured and vary in their style and layout across states and types (e.g., forms, certificates, reports, etc.). We provide data variety and quality metrics for BuDDIE as well as a series of baselines for each task. Our baselines cover traditional textual, multi-modal, and large language model approaches to VRDU.

## introduction document images are ubiquitous in the real world, especially in the financial industry. reports, receipts, forms, certificates, inter alia, are integral throughout the business pipeline. for example, during the know your customer (kyc) process in banking, officers must conduct due diligence dataset types tasks docs labels ocr cord receipts k 1,000 30 ✓ deepform receipts k 1,100 5 ✓ docile receipts k 7,000 55 ✓ docvqa varied q 12,767 -✓ dude varied q 4,973 -✓ funsd forms k, l 199 4 ✓ kleister char. reports k 540 8 ✓ kleister nda legal k 2,778 4 ✓ naf forms k, l 860 14 ✓ rvl-cdip varied c 400,000 16 ✗ sroie receipts k 1,000 4 ✓ vrdu ad-buy receipts k 641 10 ✓ vrdu reg. forms k 1,915 6 ✓ buddie varied c, k, q 1,665 69 ✓ by reviewing documents such as government registration forms, financial reports, organizational charts, and other relevant materials to verify the customers' identities. this kind of process is usually conducted manually, which is extremely challenging due to massive data volumes and widely varying data formats. modern systems thus need to efficiently and accurately capture and understand information from digital or scanned documents. as a result, computer vision, machine learning, and nlp researchers have focused on creating models for vrdu @xcite @xcite @xcite . with rising interest in the field, the necessity for publicly available, large, and robust datasets is becoming ever-more evident. numerous datasets have been created to support the modeling of individual document understanding tasks such as document classification (dc), key entity extraction (kee), entity linking, and visual question answering (vqa) @xcite @xcite . datasets often contain ground-truth annotations, based on optical character recognition (ocr), that support a single or two co-related document understanding tasks. for example, rvl-cdip @xcite contains annotations for dc, and funsd @xcite provides annotations for kee and entity linking. the majority of vrdu datasets, specifically those targeting forms and receipts, are designed for kee. @xcite @xcite @xcite . in this paper, we introduce buddie, a new dataset comprised of 1,665 publicly available structured business documents from us state government websites. our dataset is unique in that it tackles multiple distinct vrdu tasks over the same documents: dc, kee, and vqa. such a dataset is particularly beneficial to assess document processing in the wild, where requirements may necessitate models to perform several tasks on the same input. we created a hierarchical ontology of 69 key entity classes over seven super categories that can be augmented with even more entity types in the future. these provide a semantically rich and annotation-dense kee dataset which enables us to construct a varied vqa set. while similarly sized or larger vrdu datasets exist @xcite @xcite , they tend to focus on a single sort of document (e.g., receipts) and task. this may be insufficient for general purpose models that may be required in industry to accurately infer on a plethora of document types. therefore, buddie contributes a new large and varied dataset to the field. our contributions are summarized below: and docllm @xcite , where do-cllm incorporates multi-modal information into the language model. the best baseline across all tasks, docllm, achieves a dc f1 of 99.15, kee f1 score of 89.97, and vqa anls score of 89.58. ## related work in this section, we describe past datasets from the vrdu community as well as models. ## the business document dataset for information extraction in this paper, we introduce a new dataset for vrdu, buddie, which consists of 1,665 publicly available business documents. in particular, we searched documents from us state websites (or their department of business website) which were under one of five document classes of interest shown in in the remainder of this section, we describe the data collection, annotation, and processing steps for each of the three tasks. our annotation instructions are provided in app. a. ## conclusion in this paper, we introduced a new vrdu dataset for the finance domain, buddie, consisting of 1,665 annotated documents. buddie is unique in its varied document styles, sizes, and annotations for three distinct tasks. we use a variety of language models, multi-modal language models, and llms to provide comprehensive baselines for our dataset. while we note docllm's impressive performance across the tasks, vrdu model performance is still not comparable to human performance on the tasks of kee and vqa as of the date of publication @xcite , and zero-shot prompted llms still have room for improvement. we hope that our dataset can be a valuable resource to encourage the research community to seek more robust vrdu models that help on processes such as kyc, and will spur further research in this domain. future work on buddie will include multipage annotations, multi-turn vqa, and instruction tuning benchmarks.