D oc A ssistant: Integrating Key-region Reading and Step-wise Reasoning for Robust Document Visual Question Answering (2025)

Abstract: Understanding the multimodal documents is essential for accurately extracting relevant evidence and using it for reasoning. Existing document understanding models struggle to focus on key information and tend to generate answers straightforwardly, ignoring evidence from source documents and lacking interpretability. In this work, we improve the visual encoder to focus on key information relevant to the question and address the shortcomings of existing document visual question-answering datasets to provide the model with the ability to answer questions step-wise, dubbed DocAssistant. Specifically, for the visual side, we propose an effective vision-language adaptation that fuses text into visual encoders without compromising the performance of the original model. For the language side, we use Multimodal Large Language Models (MLLMs) as data generators and checkers to produce high-quality step-wise question-and-answer pairs for document images. We then use the generated high-quality data to train our enhanced model, specifically designed to solve complex questions that require reasoning or multi-hop question answering. The experimental results demonstrate the effectiveness of the model.

## introduction question base dvqa model answer the question straightforwardly region-attn dvqa model answer the question step-by-step templates or exemplars our method previous methods data generator data checker ## method given a document image and a question, the goal is to enable the model to process and reason similarly to a human. first, we design a data generator and a data checker to create high-quality data. next, we enhance the vision encoder by introduc-ing a mixture-of-modality adaptation mechanism, which directs the model to focus on the visual regions relevant to the question when processing the document image. finally, these generated data are then utilized to train the multimodal dvqa model we have developed. ## conclusion this paper introduces docassistant, designed to perform document understanding and reasoning. specifically, we first propose a mixture of modality adaptation that can be inserted into the vision encoder layers to make the model pay more attention to the region related to the question keywords. then we transform and expand the data of the existing document training set to include the intermediate analysis process in answering questions. the results surpass larger-scale models. furthermore, we conducted extensive experiments, comparing different strategies of the model, performance before and after data expansion and filtering, and various types of questions to validate our work.