\textrm{DuReader}_{\textrm{vis}} : A C hinese Dataset for Open-domain Document Visual Question Answering (2022)

Abstract: Open-domain question answering has been used in a wide range of applications, such as web search and enterprise search, which usually takes clean texts extracted from various formats of documents (e.g., web pages, PDFs, or Word documents) as the information source. However, designing different text extraction approaches is time-consuming and not scalable. In order to reduce human cost and improve the scalability of QA systems, we propose and study an \textbf{Open-domain} \textbf{Doc} ument \textbf{V} isual \textbf{Q} uestion \textbf{A} nswering (Open-domain DocVQA) task, which requires answering questions based on a collection of document images directly instead of only document texts, utilizing layouts and visual features additionally. Towards this end, we introduce the first Chinese Open-domain DocVQA dataset called \textrm{DuReader}_{\textrm{vis}} , containing about 15K question-answering pairs and 158K document images from the Baidu search engine. There are three main challenges in \textrm{DuReader}_{\textrm{vis}} : (1) long document understanding, (2) noisy texts, and (3) multi-span answer extraction. The extensive experiments demonstrate that the dataset is challenging. Additionally, we propose a simple approach that incorporates the layout and visual features, and the experimental results show the effectiveness of the proposed approach. The dataset and code will be publicly available at https://github.com/baidu/DuReader/tree/master/DuReader-vis .

## introduction open-domain question answering (open-domain qa) is a task that requires answering questions based on a collection of document texts. it has been used in a wide range of applications, such as web search @xcite ; chen et al., 2017a), enterprise qa @xcite , biomedical qa @xcite , etc. the typical procedure of an open-domain qa system can be summarized in figure 1 @xcite . it needs first to design specific text extraction methods for real-world documents in different formats (e.g., pdfs, web pages, scanned documents, etc.), and extract certain text contents from them (e.g. the main body of web pages). since there is no universal method for text extraction, it is expensive to build a unified qa system that can process documents in different formats as the information source. this greatly limits the scalability of qa systems, where a scalable qa system should process various formats of documents at a low cost, and not be restricted by the document format. in addition, the visual layouts (e.g., font size, list format, and table format) and the visual features (e.g., text color, pictures, and figures) will be lost after text extrac-tion, which are of great significance to comprehend documents. to tackle the above limitations, we propose and study an open-domain document visual question answering (open-domain docvqa) task, which takes a collection of document images (converted from real-world documents) as the information source to answer questions, as shown in figure 1(b) . in this task, we apply a universal document extractor (e.g., ocr) to extract all the texts and layouts from the document images and then utilize them along with the visual features to perform the following procedures, including document visual retriever (docvre) to retrieve relevant document images, and document visual question answering (docvqa) to extract answers from retrieved document images. the open-domain docvqa task encourages us to design an open-domain qa system that can be applied to various data sources in a scalable way, leveraging text, layout, and visual information simultaneously. in open-domain qa, it is intuitive to build the corresponding datasets from the ones for machine reading comprehension (that requires answering questions based on one or a few documents), e.g., natural questions open @xcite , squad-open @xcite . with the development of document intelligence research, several datasets of visual machine reading comprehension (or question answering) have been created, such as visualmrc @xcite , infographicvqa @xcite and docvqa foot_0 @xcite . however, the questions are collected in a crowd-sourced way rather than from real-users' information seeking questions which makes them not suitable for open-domain docvqa research. besides, most document images in existing datasets are short documents with simple layouts and few visual features, but we often need to take more complex documents in open-domain scenarios. furthermore, the answers in existing datasets are mainly short (e.g., entities, numbers, etc.). in contrast, we have longer answers in various formats like paragraphs, lists, and tables in real applications. except the above limitations, there are very few chinese datasets to the best of our knowledge. the experimental results show that there are three main challenges (section 5.4.1) in dureader vis : 1) long document understanding, where the document images are converted from long documents with rich visual features and complex layouts; 2) noisy texts, such as the advertisements and related links in web-pages, increasing the difficulty of understanding the documents; and 3) multi-span answer extraction, where the actual answers could be multispan texts, lists, and tables. furthermore, the additional zero-shot study (section 5.4.2) on real-world documents in different formats (including pdfs, word documents, and scanned images) demonstrates the scalability of our approach and the good transferability of models trained on dureader vis . our main contributions are as follows: ## dureader vis this section defines the task formally, then shows the data collection and annotation process, and finally conducts the statistics and analysis. ## proposed model in this section, we propose a simple baseline for dureader vis . the approach contains three parts: 1) the universal document extractor to obtain textual contents, layout and visual information as the input, 2) the document visual retriever (docvre) to retrieve relevant documents, and 3) the document visual question answering (docvqa) to extract answers from retrieved documents. ## experiment in this section, we firstly describe the experiments we conduct on dureader vis dataset and then conduct further analysis and discussion. case studies are shown in appendix a.2. ## conclusion we propose an open-domain document visual question answering task to encourage scalable qa applications. we introduce dureader vis to move toward the open-domain docvqa research. there are three challenges: long document understanding, noisy text, and multi-span answer extraction. we propose a baseline and the results show that there is still a huge gap compared to human performance. we show the scalability of our approach by a zeroshot study. finally, we give error cases and future directions.