Paper: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding (ICLR-2024)

Reviewer comment (Experiments):
The dataset introduced in this paper only brings limited improvement. In Table 2, LLAVAR achieves comparable performance with mPLUG-Owl, a model that was not trained on a text-rich dataset. The paper also does not provide a detailed comparison with other state-of-the-art models in the field of text-rich image understanding, which would help to better understand the relative performance of the proposed method.

Author response:
It is hard to conclude that mPLUG-Owl is a model not trained on a text-rich dataset, as many of the captions in LAION datasets are equivalent to incomplete OCR results (Texts in an online image will sometimes appear in the captions). In the scale of our experiment, we observe similar improvement that just training on captions of text-rich images can help with text recognition capability (In Table 3, variant (4) is better than variant (1)). However, training on captions only (variant (4)) is not as good as training on OCR-based data (variant (2)(6)), at least in the scale of our experiments. We assume training on captions can be powerful enough for the scale of mPLUG-Owl (1000M+ text-image pairs). However, we believe our data is lightweight and effective, and our pipeline is more customizable.

Reviewer comment (Experiments):
The fragmented OCR results in a few words may also exist in real-world text-rich data such as poster, table, directly removing this kind of data may be also different from the real-world distribution.

Author response:
The reason why we tried to remove fragmented OCR results from prompting GPT-4 is to remove repetitive and meaningless questions generated by GPT-4. If there are only a few unrelated words presented to GPT-4, it is hard to generate meaningful question-answer pairs for the texts. Note that (1) our noisy instruction-following data based on raw OCR results still contain such fragmented OCR results. (2) Our experiments show that the learned text recognition capability transfers well to scenarios like ST-VQA and textVQA, where the texts in images are usually fragmented words.

Reviewer comment (Novelty):
The most important contribution of the paper is the collected dataset. It succeeds in showing the data improves LLaVA's OCR capabilities, but does not demonstrate it is superior to other visual instruction datasets. For example, mPLUG-Owl has comparable OCR performance to LLaVAR under the same resolution in Table 2. This raises the question of whether OCR-specific data is needed, or if the scale of data in the paper is insufficient.

Author response:
Compared to hundreds of millions of text-image pairs used in mPLUG-owl, we acknowledge that the scale of our data is relatively limited, which is also relatively affordable for most academic labs. We presume that training on large-scale non-OCR data improves OCR performance, as many of the captions in LAION datasets are equivalent to incomplete OCR results (Texts in an online image will sometimes appear in the captions). In the scale of our experiment, we observe similar improvement that just training on captions of text-rich images can help with text recognition capability (In Table 3, variant (4) is better than variant (1)). However, training on captions only (variant (4)) is not as good as training on OCR-based data (variant (2)(6)), at least in the scale of our experiments. We assume such a training signal can be strong enough for the scale of mPLUG-Owl. In general, we believe OCR-specific data is necessary for data efficiency.

Reviewer comment (Experiments):
The evaluation is limited, mostly relying on 4 OCR QA datasets. As the authors admit in Fig 4(5), this evaluation may be unreliable. More scenarios like the LLaVA benchmark would be expected, especially in ablation studies.

Author response:
|                        | conversation | detail | complex |
|------------------------|--------------|--------|---------|
| LLaVA                  | 83.6         | 78.1   | 95.2    |
| LLaVA + $R_{pretrain}$ | 86.7         | 79.3   | 95.1    |
| LLaVA + $R_{finetune}$ | 79.0         | 79.4   | 98.0    |
| LLaVAR                 | 84.4         | 78.9   | 96.6    |

We find that including pretraining data improves the conversation capability, probably because longer training data leads to generating longer responses (Table 1). On the other hand, including finetuning data only hurts the conversation capability but increases complex reasoning. Combining pretraining and finetuning data improves the trade-off between conversation and complex reasoning. Generally speaking, GPT-4-based evaluation is not very robust, as there are some clear clues that it favors long responses [1]. By providing results on the LLaVA benchmark, we prove that incorporating our data will at least not harm the performance of interacting with natural images.

Reviewer comment (Experiments):
Why did the authors collect data based on LAION, rather than some well-annotated OCR dataset?

Author response:
The well-annotated OCR datasets are usually restricted to certain domains such as black-and-white documents, book covers, etc. As shown in Appendix Figure 10, the text-rich images in the LAION dataset, which contains all kinds of text-image pairs from the internet, are diverse and usually interleaved with natural images. We believe our instruction-following dataset based on LAION suffers from a relatively small domain shift compared to previously collected instruction-following data based on COCO, thus beneficial for potential knowledge transfer (Section 5.4). Also, we believe that collecting data based on real-world documents and well-annotated OCR datasets is an important next step to extend the scope of the data.

Reviewer comment (Evaluation):
From the results in Table 2, LLaVAR has no significant performance advantage compared with existing methods under the same resolution (224²), such as mPLGU-Owl (2023). Besides, it is better to evaluate the parameter sizes of these comparison models.

Author response:
__Table of language model parameter size and training data size.__

|                  | Language Model Parameter | Training data size |
|------------------|--------------------------|--------------------|
| BLIP-2-FlanT5XXL | 11B                      | 129M               |
| OpenFlamingo     | 7B                       | 15M                |
| MiniGPT4         | 13B                      | 5.0M               |
| LLaVA            | 13B                      | 0.6M               |
| mPLUG-owl        | 7B                       | 1112M              |
| LLaVAR           | 13B                      | 1.0M               |

As all baseline models use ViT CLIP 224 as their visual encoder, we list the parameter size of their language model above, together with their training data size.

In our experiments, we find that the language model size has a trivial effect (see the table below) on the performance related to text-based VQA. We believe data efficiency is a more important factor to consider while comparing performance.

|            | ST-VQA | OCR-VQA | TextVQA | DocVQA |
|------------|--------|---------|---------|--------|
| LLaVAR 7B  | 28.9   | 25.6    | 37.8    | 6.2    |
| LLaVAR 13B | 30.2   | 23.4    | 39.5    | 6.2    |

Reviewer comment (Experiments):
In Section 5.3, only one case study is carried out, so the derived conclusion is hardly convincing.

Author response:
__More comprehensive case study:__

The case study is now conducted on 825 examples from OCR-VQA. We have updated Figure 6 in the draft. In this large-scale case study, we still observe a threshold for recognizable texts for both $224^2$-based and $336^2$-based LLaVAR as the accuracy sharply decreases when the height is smaller than 7 pixels.

Reviewer comment (Presentation):
In Figure 7, the notations are not clearly explained, and the implementation details cannot be visually reflected in this figure.

Author response:
__On Figure 7:__

We have provided an updated Figure 7 with detailed captions. A sketch of the implementation details is as follows: Given an image, it is simultaneously processed by visual encoders $V_1$ and $V_2$. $V_1$ features are transformed by transformation matrix $W$ and directly used as input embeddings to the language model. For $V_2$ features, they are transformed by transformation matrix $K$ and $V$ and used as keys and values to calculate the cross attention in every transformer layer (assume there are $N$ layers), which uses the transformed hidden states (through $Q$) from the self-attention module as queries.

Reviewer comment (Experiments):
As can be seen in Figure 4 and Figure 5, OCR errors are inevitable, e.g., “Boynton” vs. “Byington”. Can you provide some results to analyze the impact of OCR errors?

Author response:
__The impact of OCR errors:__

We take 1673 examples from OCR-VQA, which have ground truth answers with more than 10 characters, to study such OCR errors. We (i) define “correct” as the ground truth answers that are exactly in the predictions, and (ii) define “partially correct” as there exists a substring in the prediction that has high enough similarity with the ground truth but not the same. Specifically, we look at all substrings with the same length of the ground truth in the prediction to calculate ANLS (Average Normalized Levenshtein Similarity) and regard the prediction as “partially correct” if the highest ANLS is greater or equal to 0.5 but smaller than 1.

|           | Correct % | Partially Correct% |
|-----------|-----------|--------------------|
| LLaVA224  | 1.6%      | 8.7%               |
| LLaVAR224 | 6.8%      | 22.8%              |
| LLaVA336  | 2.2%      | 11.2%              |
| LLaVAR336 | 9.0%      | 26.8%              |

We find that a considerable amount of predictions can be considered partially correct, which indicates the actual performance of tested models is better than the reported accuracy numbers. However, the percentage of partially correct predictions is highly correlated with the percentage of correct predictions. Therefore, we believe that the current metrics can effectively compare the performance of different models.

Reviewer comment (Writing):
What does “temperature” refer to in the first paragraph of Section 5?

Author response:
__Clarification on the term “temperature”:__

The temperature used to sample examples from language models.