Paper: IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks (ICLR-2024)

Reviewer comment (Experiments):
It would be more convincing to have more quantitative experiments on more tasks, e.g., one-shot or few-shot tasks. Although it might be challenging, it will give readers a bigger picture.

Author response:
We provide results of more Image-to-X tasks in Table 5 and visualization results in Figure 7 and 8.

Reviewer comment (Experiments):
The authors use ViT-L be default. Did you tried backbone at different scales?

Author response:
We tried a larger network (ViT-H) as encoder, but there was no significant improvement. We suspect that scaling up the decoder instead of the encoder may help, but we did not have sufficient computing budget to try it. We plan to explore it as a future work.

Reviewer comment (Experiments):
If we want to further scale up, how can we get more data like Semantic Scholar Computer Vision dataset? It would be interesting to have more discussion.

Author response:
One potential way to scale up the dataset is by collecting instructional videos and incorporating them during the training process (e.g. videos of academic talks, oral presentations from conferences, and paper summary videos). We will add it to the discussion and explore this direction in future work.

Reviewer comment (Experiments):
Obviously, there is a large difference between figure captions from papers and the prompts used at inference time. For example, the text prompt may not appropriately describe the task during training. In the paper, I do not see any discussions regarding this issue.

Author response:
**A**: Although the training and testing prompts are different, the model is able to generalize to the testing prompts by training on large-scale datasets. During the pre-training, the model learns to extract meaningful representation from the noisy data.  For example, CLIP’s image encoder and text encoder are pre-trained on noisy data as well, but it can zero-shot transfer to different images and texts at inference time. And according to Figure 17, the captions of our datasets include terms like “segmentation” “sketch”, “colorization” etc. So the gap between training and inference is not large.

Reviewer comment (Experiments):
In Table 3, methods are trained with different datasets, thus leading to an unfair comparison. The authors could report the performance of IMProv trained on CCVF.

Author response:
**A**: In Table 4, we report the accuracy of MAE-VQGAN and IMProv trained on the same dataset. 
CCVF just extends CVF by extracting the captions of the figures. CCVF and CVF share the same set of images. MAE-VQGAN only uses images, and IMProv is trained on both texts and images. So row 1 and 2 in Table 4 is a fair comparison between MAE-VQGAN and IMProv, indicating that text prompt can improve the model performance.

Reviewer comment (Evaluation):
For the tasks of X-to-images and images-to-X, the evaluation is not rigorous. For example, LPIPS cannot assess the performance of semantic segmentation. The reported results do not support the claim that IMProv can well generalize to these standard computer vision tasks. Commonly used metrics for these tasks should be adopted for evaluation. In addition, the results of X-to-images look poor, where the generated images are blurry and lack details.

Author response:
**A**: In [2*], Wang et al. only reported MSE for all the vision tasks. We follow them to use a single metric for all the tasks. The images from InstructPix2Pix are generated from Stable Diffusion, there is no ground-truth annotation for the depth/edge/segmentation. We will report both LPIPS and MSE in the updated version.

Reviewer comment (Experiments):
For the ablation study, I have a few concerns. Dataset ablation: This is not a valid ablation. IMProv trained with different datasets can indicate the benefits brought from larger datasets, not different methods trained with different datasets.

Author response:
**A**: We report the results of IMProv(S2CV+LAION) in the below table under the same “Random Class” setting. CCVF just extends CVF by extracting the captions of the figures. CCVF and CVF share the same set of images. MAE-VQGAN only uses images, and IMProv is trained on both texts and images. IMProv(CCVF) improves over MAE-VQGAN (CVF) by 2 points, indicates text prompt helps when visual prompt is randomly sampled from different classes. IMProv(S2CV + LAION) outperforms IMProv(CCVF + LAION) by 1.7 points, which justifies the effectiveness of our proposed S2CV dataset. 


| Model                | Avg.  |
|----------------------|-------|
| MAE-VQGAN (CVF)      | 23.52 |
| IMProv(CCVF)         | 26.13 |
| IMProv(CCVF + LAION) | 36.29 |
| IMProv(S2CV + LAION) | 38.07 |

Reviewer comment (Experiments):
For the ablation study, I have a few concerns. The authors claim that Figure 4 suggests a trade-off between visual and text prompts. However, the visual prompts are the same for a query in Figure 4.

Author response:
**A**: In Figure 4, we keep the query the same, and change the support from “without support” (first 2 columns) to “with support” (last 4 columns). We provide more examples of the trade-off between visual and text prompts in Figure 9, with different class random sample prompts, same class random sample prompts, and nearest neighbor prompts.

Reviewer comment (Experiments):
There are no comparisons with other SOTA in-context learning methods such as Painter [1*] (only a small comparison for the task of colorization is given in the appendix which is far less than enough) and Prompt Diffusion [2*]. Overall, I think the paper presented an interesting idea for multimodal in-context learning. However, significant flaws in terms of experiments (see above points 2-5) make the claims of this paper very unconvincing. [1*] Images speak in images: A generalist painter for in-context visual learning. In CVPR, 2023. [2*] In-context learning unlocked for diffusion models. arXiv, 2023.

Author response:
**A**: We acknowledge that our approach performs worse than them on their benchmarks because they are trained in a fully supervised way with ground truth annotations. In contrast, our model is trained on noisy data from the paper. We plan to explore the combination of supervised data and unsupervised in future works.

Reviewer comment (Evaluation):
Using only LPIPS as a quantitative metric for extending applications in image-to-X and X-to-image tasks may not be particularly convincing. While there may be significant variations in metrics for image-to-X tasks, the use of standardized metrics like FID for X-to-image tasks would provide more solid experimental results.

Author response:
**Q**: Metrics other than LPIPS for image-to-X and X-to-image tasks.

**A**: We generally agree, however, in [2*] we followed Wang et al. that used the images from InstructPix2Pix. These images are generated using Stable Diffusion, and there is no ground-truth annotation for the depth/edge/segmentation. Therefore, like Wang et al. we report MSE and LPIPS (see the table below).

|                                              | Depth -> Image |      | Image -> Depth |      | HED -> Image |      | Image -> HED |      | Seg -> Image |      | Image -> Seg |      | Normal -> Image |      | Image -> Normal |      |
|----------------------------------------------|----------------|------|----------------|------|--------------|------|--------------|------|--------------|------|--------------|------|-----------------|------|-----------------|------|
|                                              | LPIPS          | MSE  | LPIPS          | MSE  | LPIPS        | MSE  | LPIPS        | MSE  | LPIPS        | MSE  | LPIPS        | MSE  | LPIPS           | MSE  | LPIPS           | MSE  |
| Supervised ICL (InstructPix2Pix)             | 0.65           | 4.48 | 0.60           | 3.91 | 0.59         | 4.36 | 0.62         | 3.26 | 0.64         | 3.65 | 0.61         | 3.11 | 0.62            | 3.91 | 0.55            | 2.55 |
|IMProv~(CCVF+LAION)                 | 0.61           | 4.60 | 0.52           | 2.69 | 0.51         | 3.83 | 0.46         | 1.54 | 0.59         | 3.35 | 0.50         | 2.61 | 0.56            | 3.63 | 0.48            | 2.03 |
| IMProv~(CCVF+LAION+InstructPix2Pix) | 0.55           | 3.67 | 0.43           | 2.63 | 0.47         | 3.26 | 0.37         | 1.59 | 0.54         | 3.16 | 0.46         | 2.38 | 0.51            | 3.05 | 0.44            | 1.90 |

Reviewer comment (Experiments):
Do large language models that exclusively handle text, such as T5, outperform models that encode text using CLIP?

Author response:
**Q**: Do large language models that exclusively handle text, such as T5, outperform models that encode text using CLIP?

**A**: We tried both CLIP and T5 text encoder. And we found the larger T5 text encoder doesn't improve the performance. Under “Random Class” visual prompt setting, T5 vs CLIP is 33.26 vs 36.29 mIoU. We choose to use CLIP for all the other experiments to save computation costs.

Reviewer comment (Novelty):
This paper exhibits limited novelty since the proposed method is primarily a straightforward extension of Bar et al.'s work.

Author response:
**Q**: This paper exhibits limited novelty since the proposed method is primarily a straightforward extension of Bar et al.'s work.

**A**: In this work, our goal is to explore the tradeoff between text and image prompts based on Bar et al. work. Nevertheless, we show that text prompts can be used to improve the model performance, and we also collect a large dataset for multimodal in-context learning.

Reviewer comment (Evaluation):
Qualitative analyses are relatively scarce within the paper.

Author response:
**Q**: Qualitative analyses are relatively scarce within the paper.

**A**: We provided extensive qualitative analysis in Figures 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 13, 14, both in the main paper and in the supplementary material.

Reviewer comment (Evaluation):
It appears that there may be a trade-off between textual and visual modalities, with the advantages of a text prompt being less pronounced when paired with better visual prompts.

Author response:
**Q1**: We would like to clarify that as Table 8 and Figure 15 show, as the number of support examples increases, the mIoU improves for both with and without the text prompt. Moreover, adding the text prompt will further improve the mIoU by 4 points on top of 7 visual prompts. And when there is only one visual prompt, with and without the text prompt yield very similar results. One reason for low mIoU is that most input pixels are just black, so the model tends to predict black pixels and ignore the input support and query images.

Reviewer comment (Experiments):
The authors explore various visual prompts, such as switching from a random class to a nearest neighbor (NN) class. It raises the question of how performance would be affected if the visual prompt is incorrect (for example, an example for a different task), and how this compares to using a textual prompt alone.

Author response:
**Q**: It raises the question of how performance would be affected if the visual prompt is incorrect, and how this compares to using a textual prompt alone.

**A**: There are no guarantees whether the visual or text prompt would affect the model more. In Figure 6, we provide an example of inconsistent visual and text prompts, and in this example, the model is more likely to follow the text prompt (generate a letter "T").

Reviewer comment (Experiments):
In the appendix, the authors compare their method to SD. However, it may not be a fair comparison. It would be more equitable if the authors also fine-tuned on CCVF and S2CV datasets to assess whether the SD model can correctly inpaint in those scenarios.

Author response:
**Q**: In the appendix, the authors compare their method to SD. However, it may not be a fair comparison. It would be more equitable if the authors also fine-tuned on CCVF and S2CV datasets to assess whether the SD model can correctly inpaint in those scenarios.

**A**: We agree that the off-shelf SD is not directly comparable with our method. Our motivation for including the failure of SD is to motivate the necessity to collect S2CV for large-scale multimodal in-context learning.

Reviewer comment (Experiments):
Bar et al. discussed the limitations of their methods, including task ambiguity, non-aligned input-output, and out-of-distribution decoding. It would be interesting to explore whether these issues can be mitigated by using a textual prompt (for addressing task ambiguity and non-aligned input-output) and by employing more data (for handling out-of-distribution decoding).

Author response:
**Q**: Whether the limitation of Bar et al. can be mitigated by using a textual prompt.

**A**: We show the results of failure cases in Figure 16. For some of the failure cases in Bar et al, our IMProv could successfully address them with text prompts, e.g. the cat colorization and bottle segmentation of “Task ambiguity”. And our model could successfully generate the image that moves the orange to the center, though fails the other Non-aligned input-output example.

Reviewer comment (Experiments):
Have the authors considered how different text encoders might affect the model's performance?

Author response:
**Q**: Have the authors considered how different text encoders might affect the model's performance?

**A**: We tried both CLIP and T5 text encoder. And we found the larger T5 text encoder doesn't improve the performance. Under the “Random Class” visual prompt setting, T5 vs CLIP is 33.26 vs 36.29 mIoU. We choose to use CLIP for all the other experiments to save computation costs.

Reviewer comment (Experiments):
The authors examine the quality of visual prompts versus textual prompts. Both the authors and Bar et al. have investigated how performance changes with the number of support pairs versus performance. It would be valuable to understand the trade-off between the number of support pairs and the use of textual prompts.

Author response:
**Q**: The trade-off between the number of support pairs and the use of textual prompts.

**A**: We plot the mIoU w.r.t the number of support in Figure 15. We run the experiments under grid 4x4 setting in Table 8, with “Random Class” support image. Similar to Bar et al., as we increase the number of support visual examples, mIoU goes up.  It’s worth noting that when there are only 1 or 2 examples, the text prompt doesn’t help. It is because the model couldn't generate meaningful segmentation with a small number of visual supports due to the resolution (see Figure 13 for details). Under the setting that visual support number greater than 3, the text prompt consistently improves the results. It implies that our text prompt is orthogonal to the number of visual support pairs.

Reviewer comment (Experiments):
It appears that the authors conclude S2CV+LAION results in better performance from Table 3. Although this conclusion is likely true, it is not rigorous to say so since IMProv benefits from both textual prompts and data. An interesting experiment could involve retraining MAE-VGGAN with CCVF+LAION data and S2CV+LAION data for a fair comparison.

Author response:
**Q**: Fair comparison for S2CV+LAION results.

**A**: In Table 4, we report the accuracy of MAE-VQGAN and IMProv trained on the same dataset.
CCVF just extends CVF by extracting the captions of the figures. CCVF and CVF share the same set of images. MAE-VQGAN only uses images, and IMProv is trained on both texts and images. So row 1 and 2 in Table 4 is a fair comparison between MAE-VQGAN and IMProv, indicating that text prompt can improve the model performance. To compare S2CV with CCVF, we additionally report the results of IMProv(S2CV+LAION) in the below table (Table 10 in supplementary material) under the same “Random Class” setting. We report the average mIoU over 4 splits. IMProv(S2CV + LAION) outperforms IMProv(CCVF + LAION) by 1.7 points, which justifies the effectiveness of our proposed S2CV dataset.

| Model                | Avg.  |
|----------------------|-------|
| IMProv(CCVF + LAION) | 36.29 |
| IMProv(S2CV + LAION) | 38.07 |