Paper: ADOPD: A Large-Scale Document Page Decomposition Dataset (ICLR-2024)

Reviewer comment (Experiments):
From the segments reviewed, while the paper introduces a new dataset, there seems to be a limited in-depth comparison with existing datasets. A deeper comparative analysis highlighting the specific advantages of ADoPD over existing datasets would solidify its significance. Quantitative benchmarks against datasets like DocBank, if not already included in the deeper sections, would be beneficial.

Author response:
Thank you for your feedback. Following your suggestions, we conducted comparative testing on two document datasets: DocLayNet (Text Detection) and RVL-CDIP (Document Image Classification).

**[1]** *DocLayNet*

We investigated three settings: (a) ADoPD fine-tuning with zero-shot testing on DocLayNet. (b) ImageNet-based initialization followed by fine-tuning on DocLayNet. (c) ADoPD fine-tuned model initialization followed by fine-tuning on DocLayNet. The results are presented below (also updated in **Table 4 (b)**).

| Method              | Backbone           | Zero-Shot (ADoPD) | Zero-Shot (ADoPD) | Finetune (ImageNet) | Finetune (ImageNet) | Finetune (ADoPD) | Finetune (ADoPD) |
|---------------------|--------------------|--------------------:|-----------------------:|-------------------:|-------------------:|------------------------:|------------------------:|
|                     |                    | **mAP**             | **mAR**                | **mAP**           | **mAR**           | **mAP**               | **mAR**               |
| Faster R-CNN        | ResNet$_{50}$       | 0.9                 | 58.5                   | 43.0              | 55.8              | 44.5                   | 60.4                   |
|                     | ResNet$_{101}$      | 1.0                 | 56.6                   | 46.0              | 58.5              | 47.0                   | 60.7                   |
| Deformable-DETR     | ResNet$_{50}$       | 2.2                 | 80.4                   | 74.7              | 87.2              | 75.4                   | 88.9                   |
|                     | ResNet$_{101}$      | 2.6                 | 79.0                   | 75.4              | 85.9              | 77.2                   | 88.1                   |


**[2]** *RVL-CIDP*

We compared CLIP vision backbones: (a) Laion pretrained backbone + RVL-CDIP fine-tuned classifier. (b) ADoPD +*Doc2Seq* pretrained CLIP backbone fine-tuned on RVL-CDIP (updated in **Figure 6 (b)** and **Table 6**).

| Model                  | Type        | Letter | Form  | Email | Hw    | Ad    | SR    | SP    | SP    | FF    | NA    | Bgt   | Inv   | Prsnt | Qnr   | Rsm   | Memo  | Avg   |
|------------------------|-------------|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| ViT$_{\text{G-14}}$     | Supervised  | 86.20  | 76.70 | 94.28 | 93.48 | 91.81 | 71.94 | 91.10 | 89.72 | 94.97 | 83.68 | 81.24 | 87.44 | 78.26 | 84.97 | 92.94 | 85.87 | 86.57 |
| ViT$_{\text{G-14}}$+ADoPD | Supervised | 90.87  | 84.48 | 96.98 | 95.34 | 93.76 | 82.39 | 93.51 | 93.00 | 95.61 | 89.81 | 89.62 | 92.85 | 84.29 | 90.23 | 96.45 | 92.62 | 91.38 |

Significant ADoPD-based improvements were observed across all types, which highlight ADoPD's advantages for finetuning or pretraining.

Reviewer comment (Novelty):
The choice of the "human-in-the-loop" approach for outlier detection is novel, but the paper might not sufficiently justify why this method was chosen over others. Delve deeper into the advantages and potential limitations of this method, comparing it with purely automated outlier detection techniques.

Author response:
Thank you for your suggestion. Below, we answer your question from two perspectives: outlier detection and human-in-the-loop:

**[1]** *Outlier Detection*

The document presents a challenge for feature extraction, combining both images and text. A potential drawback may be in feature extraction, particularly in the initial stages relying solely on CLIP trained on natural images. While introducing additional data enhances robustness, this remains an area for improvement. Additional outlier detection methods, such as textual features, will be explored in the Appendix.

**[2]** *Human-in-the-loop*

During the data selection, we found that sensitive images, such as those depicting violence, might present within the in-distribution. However, recognizing their undesirability, we deemed them as failure cases. Depending solely on outlier detection to identify such outliers might yield unpredictable results. Therefore, We opted for a human-in-the-loop approach for this process.

We engaged diverse annotators from various countries and genders to check selected data. Providing only basic keywords without explicit sensitive references, annotators were tasked with removing documents they deemed sensitive. Ultimately, we achieved a drop percentage of 1.77%, and manual checks confirmed the presence of sensitive information in these images.

Reviewer comment (Evaluation):
While the use of 5 pre-trained models to pre-label images is quite innovative, it would also be interesting to indicate the cost of applying these different models to an image (and even to millions of images).

Author response:
Good question! Our method involves multiple pretrained models. The costs (model and annotation)  can be primarily categorized as follows:

**[1]** *Data Filtering*

Costs in this part include GPU and GPT-4 API calling expenses. In data selection, our challenge was identifying valid document images from the 175M Laion web image pool. We employed models like Document Finding Classifier, Watermark Detection, Language Detection, OCR tools, and CLIP zero-shot tagging. Except for CLIP, other models are lightweight. We utilize GPT-4 ($0.03/1k prompt tokens) for all data during collection and *Doc2Seq* & *Doc2Tag*, with costs limited to selected images.

**[2]** *Data Annotation Cost*

In our model-assisted data annotation approach, model inference and training occur for each batch of data sent to annotators, incurring relatively high GPU costs. However, this cost is still smaller than human labeling expenses. Additionally, periodic model refinement was observed to accelerate annotation efficiency and reduce costs. Refer to **Fig. 12** (*Appendix, Page 17*) for the annotated cost, displaying a decrease with the increase in data.

Reviewer comment (Reproducibility):
I found section 4.1 a little unclear. Although efforts have been made to explain the image selection process as fully as possible, I'm not sure that this methodology can be easily reproduced from these explanations.

Author response:
The section has been revised in the updated PDF. In addition, **Sec. 3.3** provides detailed information on data collection and annotation processes, while **Sec. 4.1** covers baseline model implementation, hyperparameters, and evaluation metrics. In **Sec. 4.2**, we analyze experimental results, including new experiments in **Tables 4 & 6**. We'll provide models and annotation guidelines for replication. The revised *Appendix* includes detailed data analyses and taxonomy analysis.

Reviewer comment (Experiments):
In the results tables, the authors show how the different models compare once they've been trained on ADOPD. It would have been interesting to compare these models with and without fine-tuning on ADOPD to see the real impact of this dataset.

Author response:
Great question.  in **Table 4 (a)**, we compared models fine-tuned on EntitySeg and fine-tuned on ADoPD Doc2Mask. The zero-shot performance of EntitySeg-tuned models is lower than ADoPD-tuned ones, given EntitySeg lacks document data. Despite excelling in natural image segmentation according to **Fig. 5 (a)**, EntitySeg-trained models are unfit for documents due to significant composition differences, where layout elements outweigh content significance.

Reviewer comment (Evaluation):
The metrics used to evaluate the caption performance should probably be a bit more explained in the text.

Author response:
We've enhanced caption evaluation in our revised paper. In **Sec. 4.2**, we used 5K test data to assess Doc2Seq's effectiveness. GPT-4 captions achieve commendable CIDEr scores, but a notable disparity persists compared to human annotations. Models fine-tuned on *Doc2Seq* match GPT-4 on B@n but exhibit significantly lower CIDEr scores. Additionally, in **Fig. 6 (left)**, human-annotated captions are notably longer than machine-generated ones, influencing the evaluation results.

Reviewer comment (Reproducibility):
This dataset would be very interesting to explore and obviously to use for pre-training. I think it would also be great to see what generalization capabilities a model trained on this dataset would have, once applied to another use case. This is not described in the paper, but do you plan to make this dataset public? If so, how do you plan to distribute it?

Author response:
We release ADoPD to support foundational models in document understanding, aiming to inspire progress in document analysis.

Reviewer comment (Reproducibility):
The detection and segmentation annotation quality are not well studied, they all come from pretrained model and corrected by human annotators. It’s better to give more detail about the procedure to correct the annotation. Especially there are some other document datasets like DocBank -- the box annotation is extracted from PDF by automatic tools and it should be much more precise than detection model.

Author response:
Great question. We've updated an explanation of our data annotation process in **Sec. 3.3**. Extracting box annotations from PDF is convenient. However, PDF extraction tools (e.g., *PyMuPDF*) may not fully parse structural information for certain PDFs, such as scanned documents in PDF format. ADoPD aims to avoid bias toward purely digital PDFs, as many documents are in image format.

For text detection, another reason we chose to have annotators label directly (instead of model-assisted) is that we observed that even when the model is trained with *DocLayNet* or *DocBank*, their prediction results introduce noisy errors and are not robust to real-world document images (especially visually-rich documents). This increases the cost, as annotators need to clean lots of redundant error bounding boxes. Therefore, we did not adopt *DocBank*'s method of parsing digital PDFs but instead used human annotation. Meanwhile, we outline the cost per instance of our manual annotation. The cost of box annotation is low compared to mask annotation. 

|  | Add Text Bbox | Modify Mask  |   Add Mask  |
|----------------------------------|-------------- |-----------------------|--------------------------------|
| Cost | 0.2*$x$ | 0.7*$x$  | $x$ |

We can see that masks are the most time-consuming and costly part of the entire process. We present quantitative and visual results in **Fig. 12, 13, 14** (*Appendix, Page 17, 18, 19*), demonstrating that as data annotation increases, the cost decreases when fine-tuning the mask prediction models for assistance. Additionally, we will release our data annotation guidelines.

Reviewer comment (Experiments):
Although the proposed dataset is claimed to be suitable for caption, but the caption annotation is generated by BLIP-2 and there is no human correction. BLIP-2 is mainly trained for universal visual-language task and it might not suitable for document caption.

Author response:
We address this question from two perspectives:

**[1]** *Our caption is not just generated by BLIP-2* 

We apologize for any misunderstandings that may have arisen due to the writing issues in the previous article. We would like to highlight that our captions are not solely based on BLIP2. Instead, they result from *Prompt-Guided Context-Aware Captioning* (updated **Sec. 3.2**). In practice, our captions are rewritten by *GPT-4* (**Eq. 1&2**) based on BLIP-2 captions, zero-shot labels (CLIP + Document taxonomy), OCR results, visual tags (using Recognize Anything Model), as well as rules. Additionally, in **Sec. A.1.1** (*Appendix, Page 14*), we provide a detailed explanation of prompt design and conduct human evaluations to validate the effectiveness of our prompt designs.
 
**[2]** *Prompt-Guided Context-Aware Captioning Benefits Vision-Language Modeling*

We attempted to collect human captions but faced challenges in controlling their quality due to variations in annotators' understanding and summarization of documents stemming from diverse educational backgrounds. Hence, we only collected a human-annotated test set (5k) for evaluation, presented in **Fig. 6 (left)** and **Table 5**. We also find that GPT-4 rewritten captions can cover multimodal information of documents and summarize effectively with the combination of human-adjusted prompts and heuristic rules.

Reviewer comment (Experiments):
The caption annotation is just generated by BLIP-2 and there are not further correction or filter. I think this model-annotation might be meaningless for the area.

Author response:
We address this question from two perspectives:

**[1]** *Our caption is not just generated by BLIP-2* 

We apologize for any misunderstandings that may have arisen due to the writing issues in the previous article. We would like to highlight that our captions are not solely based on BLIP2. Instead, they result from *Prompt-Guided Context-Aware Captioning* (updated **Sec. 3.2**). In practice, our captions are rewritten by *GPT-4* (**Eq. 1&2**) based on BLIP-2 captions, zero-shot labels (CLIP + Document taxonomy), OCR results, visual tags (using Recognize Anything Model), as well as rules. Additionally, in **Sec. A.1.1** (*Appendix, Page 14*), we provide a detailed explanation of prompt design and conduct human evaluations to validate the effectiveness of our prompt designs.
 
**[2]** *Prompt-Guided Context-Aware Captioning Benefits Vision-Language Modeling*

We attempted to collect human captions but faced challenges in controlling their quality due to variations in annotators' understanding and summarization of documents stemming from diverse educational backgrounds. Hence, we only collected a human-annotated test set (5k) for evaluation, presented in **Fig. 6 (left)** and **Table 5**. We also find that GPT-4 rewritten captions can cover multimodal information of documents and summarize effectively with the combination of human-adjusted prompts and heuristic rules.

Reviewer comment (Experiments):
The section 4.2.3 Doc2Seq claims that 5000 samples are manually annotated, but after that, the paper says “utilized 80,000 samples for training, 20,000 for testing, and another 20,000 for validation“. It’s confusing how the manually annotation are collected and used, and why it has 20,000 validation and test samples – not consistent with 5,000 manual annotation.

Author response:
For *Doc2seq* data, we utilized prompt-guided context-aware captioning on all ADoPD images, resulting in 80k/20k/20k splits. 

For *Doc2seq* model evaluation, we selectively annotated a subset of document images (5k) with human input. **Table 5** employs human-written document captions for evaluation. Generally, BLIP/BLIP2 zero-shot captions exhibit lower quality compared to models refined with Doc2Seq data.

To further validate the effectiveness of ADoPD captions, we conducted the experiment in **Table 6** (*Please also refer to the response to Reviewer umJE regarding the validity of captions*.). Specifically, we fine-tuned CLIP based on ADoPD captions, then fine-tuned only CLIP's vision backbone on RVL-CDIP. We observed a significant improvement in CLIP's performance through refinement with ADoPD on RVL-CDIP (see **Fig. 6(b)**). This indicates that rewriting document captions is better than using the noisy captions from Laion.  Detailed analysis in **Sec. 4.2** highlights the importance of caption rewriting for handling noisy data.

Reviewer comment (Reproducibility):
Will the dataset be open-source?

Author response:
Yes, the motivation behind creating ADoPD is to accelerate developments in document understanding. ADoPD represents a small step towards establishing a large-scale, high-quality document image dataset. There are still numerous directions to explore, including the analysis of multilingual documents and sensitive data. We are releasing ADoPD to support the development of future foundational models for document understanding.

Reviewer comment (Novelty):
It is unclear what the exact contribution in the paper is compared to the data source in Laion. The reviewer questions whether the labels of Doc2Mask, Doc2Box, and Doc2Seq were just used as-is from existing data or labeled in this work. The reviewer considers it a significant weakness that they cannot answer this question.

Author response:
The major contributions of this paper can be categorized into two main points: the collected dataset and the proposed methods for dataset collection and evaluation.

**[1]** *Data Contributions*

The main contribution is our extensive document dataset, detailed in **Table 1**. Notably, *Doc2Mask* and *Doc2Box* are human-annotated, while *Doc2Seq* and *Doc2Tag* involve pre-trained models (e.g., CLIP, GPT-4) with human assistance. *Doc2Mask* is the costliest part (as for costs, please refer to the annotation cost response to Reviewer aArm). Unlike natural images with fixed object sets, understanding and identifying areas to annotate in different document images requires prior comprehension.

We release the ADoPD to support foundational model development in document understanding, fostering advancements in document analysis.

**[2]** *Method Contributions*

Another contribution is our proposed data collection pipeline, detailed in **Sec. 3.2** and **Sec. 3.3**. Data-driven document taxonomy discovery is a cornerstone, determining the diversity and distribution of our data. To the best of our knowledge, our paper is pioneering in leveraging large-scale pretrained models for taxonomy discovery and data filtering in the document domain.

Reviewer comment (Evaluation):
The evaluation repeatedly claims that the authors "carry out experiments using various baseline models and parameter configurations, confirming the effectiveness of ADOPD for document domain". The reviewer questions how evaluating the models confirms the effectiveness of the dataset. They ask for more explicit explanation, including how much of the data was confirmed to be of high quality and what kind of error analysis was conducted, particularly for any labels (other than taxonomy, which is well-analyzed) that were generated automatically.

Author response:
Thank you for your suggestions. We answer this question from two aspects:
 
**[1]** *Data Effectiveness*

ADoPD stands out with fine-grained visual entity masks and text bounding boxes. Our experiments on data effectiveness focus on evaluating mainstream models, backbones (**Table 2&3**), and assessing the generalization ability of models trained on ADoPD.

In updated **Sec. 4.2**, we study the generalization ability with two new experiments: (a) Training models on ADoPD and testing on DocLayNet, results in **Table 4 (b)**. (b) Testing ADoPD fine-tuned CLIP image encoder on RVL-CDIP, results in **Fig. 6 (a)** and **Table 6**. Below, we show the RVL-CDIP results, highlighting the benefits of our data.

| Model                  | Type        | Letter | Form  | Email | Hw    | Ad    | SR    | SP    | SP    | FF    | NA    | Bgt   | Inv   | Prsnt | Qnr   | Rsm   | Memo  | Avg   |
|------------------------|-------------|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| ViT$_{\text{G-14}}$     | Supervised  | 86.20  | 76.70 | 94.28 | 93.48 | 91.81 | 71.94 | 91.10 | 89.72 | 94.97 | 83.68 | 81.24 | 87.44 | 78.26 | 84.97 | 92.94 | 85.87 | 86.57 |
| ViT$_{\text{G-14}}$+ADoPD | Supervised | 90.87  | 84.48 | 96.98 | 95.34 | 93.76 | 82.39 | 93.51 | 93.00 | 95.61 | 89.81 | 89.62 | 92.85 | 84.29 | 90.23 | 96.45 | 92.62 | 91.38 |

From above results and **Table 4,6**, we can see that benefiting from the diverse and fine-grained dense annotations in ADoPD, the trained models exhibit good generalization across various document datasets (DocLayNet and RVL-CDIP). 

**[2]** *Data Quality*

During data annotation, ensuring data diversity and quality is the original intention of ADoPD. For human-annotated data, we ensure that annotators and reviewers are separate to allow a fair confirmation of the quality and reliability of *Doc2Mask* and *Doc2Box* annotations.

Additionally, annotators from diverse backgrounds and countries were employed to assess the sensitivity of document images. With basic rules provided, they removed documents deemed sensitive. Data distribution details are in **Fig. 9** (*Appendix, Page 16*).

We will provide detailed data quality statistical results (with human review) along with the annotation guidelines.