Towards a new research agenda for multimodal enterprise document understanding: What are we missing? (2024)

Abstract: The field of multimodal document understanding has produced a suite of models that have achieved stellar performance across several tasks, even coming close to human performance on certain benchmarks. Nevertheless, the application of these models to real-world enterprise datasets remains constrained by a number of limitations. In this position paper, we discuss these limitations in the context of three key aspects of research: dataset curation, model development, and evaluation on downstream tasks. By analyzing 14 datasets and 7 SotA models, we identify major gaps in their utility in the context of a real-world scenario. We demonstrate how each limitation impedes the widespread use of SotA models in enterprise settings, and present a set of research challenges that are motivated by these limitations. Lastly, we propose a research agenda that is aimed at driving the field towards higher impact in enterprise applications.

## introduction multimodal document understanding, also known as visually rich document understanding (vrdu), and the tasks that it encompasses-including visual information extraction, visual question answering, and image document classification-constitute a major operational bottleneck in enterprise settings @xcite . due to their rich structure, length, domain-specific language, and spatio-visual complexity, enterprise documents such as reports, memos, invoices, forms, and contracts, are often processed using human supervision. this manual process is cumbersome and therefore error prone, so much so that some institutions are compelled to adopt a dual review protocol for the task of information extraction and data entry (cbs). this, coupled with the large volume of documents in many enterprise settings 1 has led to a substantial 1 as an example, in 2024 j.p. morgan chase served nearly 80 consumers @xcite . considering each customer's and growing demand for document intelligence services @xcite . against this backdrop, adoption of multimodal document understanding models remains constrained by certain limitations in sota models and benchmarks. in this position paper, we discuss these limitation in the context of three main components of research: datasets, models, and evaluation strategies. we will contextualize our discussion with a common real-world example of a document ai task in enterprise settings. ## sota models and datasets vrdu spans a suite of tasks, which, together comprise a complete picture of a given document. these include document classification (cls), page segmentation (seg), key entity extraction (kie), tabular reasoning (tr), and visual question answering (vqa). note that we exclude the optical character recognition (ocr) task, given the increasing availability of open-source and commercial solutions that can process enterprise documents. 2 sota vrdu models fall into three categories based on their core architecture: multimodal transformer-based models (tr), graph-based models (gr), and multimodal large language models (mllm). in the following sections, we will discuss the limitations of these datasets, models, and evaluation methods used within the vrdu literature, and how they impact adoption in the enterprise domain. we will end each section with one or more research challenges (rcs) that arise from the discussion, opening up new opportunities for vrdu researchers and practitioners to engage in high-impact research. ## a utility-focused research agenda given the challenges laid out in previous sections, we will now discuss the opportunities that they offer to researchers and practitioners in the field. we will present these opportunities in the context of several high-level focus areas, each covering one or more of the research challenges discussed in previous sections. consistent with previous sections, we will present the focus areas in the context of the main components of research: datasets, models, and evaluation methods. ## conclusion in this position paper, we provided a detailed breakdown of the challenges of operationalizing current vrdu models in enterprise applications. we argued that these challenges can be embedded within three key components of research: datasets, models, and evaluation strategies. using a fictional scenario that was inspired by real-world use cases, we contextualized each challenge by demonstrating how it could hinder end-user adoption. lastly, we proposed a set of research questions that can be investigated to address each challenge and facilitate the adoption of sota models in enterprise settings. we hope that the agenda put forward in this paper can inspire new directions in the vrdu literature that accommodate downstream applications within operational pipelines.