Paper: InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR-2024)

Reviewer comment (Novelty):
The novelty is limited. Formulating computer vision tasks as generative tasks is not novel and has been used in previous works[18]. The overall pipeline highly resembles InstructPix2Pix.

Author response:
While it is true that the idea of creating vision generalists has been explored before, this does not mean that our method is not novel. The paper in [18] cited by the reviewer uses visual prompting for in-context learning and does not involve any natural language component. While it is true that the architecture of our model resembles that of InstructPix2Pix due to the shared Pix2Pix formulation, the goal of our model, the way its training data is generated and the evaluation are completely different.

Reviewer comment (Experiments):
The performance is not impressive at all. The results lag far from the SOTA methods in Table 1. Note that SOTA methods of depth estimation are not listed for comparisons and other bold results (VOC) are computed on very old benchmarks. The authors might argue that their method is not task-specific, but I think this is not the reason for such poor results.

Author response:
The results in Section 4 indicate that the performance of our model surpasses the state-of-the-art vision generalists [13, 15, 17, 18, 19]. For tasks such as depth estimation and segmentation, our model outperforms other vision generalists on external datasets (Table 1) as well as in open dictionary setups (Table 2). We have also added a new comparison with the Painter model in [18] as well as an additional comparison with a task specific depth estimation model (See Tables below), both showing a competitive performance by our model. Note that our goal is not to outperform specialized models optimized on the target datasets, but rather to show the feasibility of instruction-guided execution of vision tasks and compare its performance and generalization abilities with comparable vision generalists.

| Model         | Dep. Est. (RMSE↓)  |  Sem. Seg. (mIOU↑)| Cls. (Acc↑) | Det. (mAP\at0.5↑) | 
|:---------------:|:------------------:|:-----------------------:|:----------------:|:------------------:|
|   Painter  | **0.288** (NYUv2) 0.285 (SUNRGB-D)  | 49.9 (ADE20k) 52.5 (VOC) |  - | -  |
|   InstructCV-FP   | 0.275 (NYUv2) **0.268** (SUNRGB-D)  | **52.3** (ADE20k) **53.5** (VOC) |  80.4 (Pets) 75.1 (Imagenet-Sub) | 49.1(COCO) 62.0 (VOC) |
|   InstructCV-RP  | 0.297 (NYUv2) 0.279 (SUNRGB-D)  | 47.2 (ADE20k) 52.1 (VOC) |  82.1 (Pets) 74.7 (Imagenet-Sub) | 48.5(COCO) 61.7 (VOC) |

| Model         | Dep. Est. (RMSE↓) ｜
|:---------------:|:------------------:|
|   DINOv2 (SOTA)  | 0.279 (NYUv2)  | 
|   InstructCV-FP   | **0.275** (NYUv2)  |

Reviewer comment (Experiments):
While generalist visual models are the frontier in Computer Vision, there should be more justification why such a system should rely on a single backend visual model versus an ensemble of experts (i.e. prompts to use Mask-RCNN for segmentation, ResNet for classification, etc.)

Author response:
An ensemble of experts, where Mask-RCNN is used for segmentation, ResNet for classification, etc., necessitate training each specialized component separately on a predefined set of categories. This type of model lacks the ability to generalize to new categories not covered in the training data. In contrast, our approach employs a single unified model, eliminating the need for training separate models for each task. This unified model can effectively transfer knowledge across various tasks and categories.

Reviewer comment (Experiments):
Please add more justification in the paper specifically for the use of a single diffusion model versus an ensemble of experts.

Author response:
An ensemble of experts, where Mask-RCNN is used for segmentation, ResNet for classification, etc., necessitate training each specialized component separately on a predefined set of categories. This type of model lacks the ability to generalize to new categories not covered in the training data. In contrast, our approach employs a single unified model, eliminating the need for training separate models for each task. This unified model can effectively transfer knowledge across various tasks and categories.

Reviewer comment (Presentation):
Minor suggestion: Moving Section 3, Related Work, to come right after the introduction might help the paper flow better.

Author response:
Thank you for the suggestion, we will make the necessary modifications in the final version of the paper.

Reviewer comment (Novelty):
The advantage of instruction tuning lies in its ability to enable the model to generalize to unseen tasks, which sets it apart from simple multi-task learning. However, in the paper titled "InstructCV," it seems that the approach primarily identifies instructions as task indicators within a predefined set of tasks. This may not demonstrate an advantage over traditional multitask learning. Could the author provide further explanation regarding the unique advantages of this approach?

Author response:
Thank you for this thought-provoking comment. We realize that the way we implement instruction-tuning in our setup may seem different from its application in NLP, where the primary goal is zero-shot generalization to entirely unseen tasks. However, we attribute this distinction to the structural differences between language and vision tasks. Vision tasks inherently involve more structure, where the anatomy of a vision task combines a canonical task type (e.g., segmentation) with a specific category or object. **In this narrower sense, we believe that InstructCV exhibits generalization to new tasks by effectively handling segmentation, classification, and detection for categories not encountered during the instruction-tuning dataset training.** This capability arises from the T2I formulation of InstructCV, allowing it to comprehend both the intended tasks and the associated categories and apply the learned task to new categories in an open dictionary fashion. **We consider this a unique advantage of our approach**—the ability to leverage semantic correspondence between instructions and images to establish a shared representation capable of generalizing the four tasks to new categories within an open dictionary framework.
Furthermore, we believe that our model takes initial steps toward zero-shot generalization to entirely new tasks beyond the four considered. The rephrased instruction training equips the model to understand language descriptions of tasks, which fundamentally differs from the rigid task indicators of traditional multi-task learning— this flexibility is another key advantage of our model. For instance, one could envision implementing a panoptic segmentation task in a zero-shot manner by providing a natural language description that combines semantic segmentation and classification tasks. Although such capabilities may fully emerge in a future version of InstructCV trained on a larger number of tasks, the rephrased model represents a foundational step towards achieving this goal.

We will incorporate the discussion above in the final version of the paper.

Reviewer comment (Writing):
After reading the paper, I am a bit confused about the visually encoded outputs for different tasks.

Author response:
**Q1: Clarification of the visual encoding of outputs for different tasks.**

**Q1.1:** *Distinction between semantic segmentation and referring segmentation*

A1.1: We appreciate the reviewer's comment. It is accurate to note that, like other baselines such as "Unified-IO" and "Inpainter," our approach to encoding and implementing segmentation is analogous to referring image segmentation. Note that this is a consequence of our framing of *all* vision tasks within a common text-to-image generation setup, which means that all segmentation tasks are prompted by natural language instruction. Semantic segmentation can be implemented by querying classes one-by-one. We focused on the more standard semantic segmentation task in this paper since many of the baselines (e.g. the specialized architectures) cannot condition segmentation on natural language inputs. We will add a discussion on this issue in the final version of the paper to clear up any confusion.

Reviewer comment (Writing):
Object detection: What do the authors mean by saying “cross-reference the bounding boxes with the dataset annotations” in Appendix A.3?

Author response:
**Q1.2:** *What do the authors mean by "cross-reference the bounding boxes with the dataset annotations" in Appendix A.3?*

A1.2: The post-processing steps for the object detection task involve obtaining the coordinates of the bounding boxes displayed in the visual output of InstructCV using out-of-the-box post-processing functions in the OpenCV library. After that, we calculate the mAP@0.5 metric using the standard procedure in the COCOeval library. The “cross-referencing” statement was meant to describe the evaluation and not the post-processing procedure, but was misplaced in the Appendix. We will fix this error in the final version of the paper.

Reviewer comment (Evaluation):
What type of instructions is used during inference? Fixed templates or random rephrased ones? Does it remain the same for InstructCV-FP and InstructCV-RP during inference? As reported in Table 3, InstructCV-FP outperforms InstructCV-RP if both use template prompts during inference, so why bother training InstructCV using rephrased prompts? I understand this could improve the robustness of variations in wording, but one can just adopt a fixed template prompt to address a standard computer vision task. In addition, the rephrased prompts for each task are not that diverse (only 7 variations according to Table 4), why do we need to leverage LLMs as this could be readily done by humans? I would like to see more discussions about the benefits brought by utilizing rephrased prompts.

Author response:
**Q2: Utility of training using rephrased instructions**

**Q2.1:** *What type of instructions is used during inference?*

A2.1: In Tables 1 and 2, we use randomly generated rephrased prompts during inference for evaluating the InstructCV-RP model. These rephrased prompts were sampled at inference time and were not drawn from the prompts used for training.

**Q2.2:** *Why training InstructCV using rephrased prompts and why do we need to leverage LLMs as this could be readily done by humans?*

A2.2: Please note that the InstructCV-RP model was not trained only on the 7 prompts in Table 4. The 7 rephrased prompts in this table are illustrative examples of the sampled prompts from the LLM (as highlighted in the Table caption). To achieve good generalization performance, the model was trained on a large number of rephrased prompts repeatedly sampled from the LLM. We will emphasize this in the final version of the paper.
The usage of the LLM is useful for both creating a scalable framework for training InstructCV on diverse rephrased prompts, as well as evaluating the model on truly diverse and seen rephrasing of the original instructions by repeatedly sampling new prompts at inference time. To further illustrate the generalization performance of InstructCV-RP to new instructions, we conducted further experiments of the segmentation performance in the ADE20k dataset as highlighted below.

|              Instructions               | InstructCV-FP (mIOU Score) | InstructCV-RP (mIOU Score) |
|---------------------------------------|:-------------------------:|:-------------------------:|
| Template prompt: Segment the %category%. |   45.503    | 42.793 |
| In what parts of the image can the %category% be found? | 21.344 | 45.600 |
| Can you please segment the %category%? | 40.300 | 45.810 |
| Hi, InstructCV, segment the %category%. | 39.980 | 46.450 |
| Which portions of the image encompass the %category%?| 15.223 | 42.345 |
| Could you point out the areas in the image that have the %category%? | 25.400 | 41.560 |
| What regions of the image feature the %category%? | 19.523 | 45.600 |
| I would appreciate it if you could help segment the %category%.  | 36.712 | 45.800 |
| I would be grateful if you could assist in segmenting the %category%. | 26.764 | 43.260 |

Additionally, we tested the robustness of InstructCV-RP to prompt rephrasing through a different LLM (GPT-4), and calculated the average and variance of the results. The findings highlighted below indicate the robustness of InstructCV-RP to unseen instructions.

|              Our model               | Mean (mIOU Score)  | Variance (mIOU Score) |
|---------------------------------------|:-------------------------:|:-------------------------:|
| InstructCV-FP. |   25.618    | 127.850 |
| InstructCV-RP. |   45.503    | 5.85 |

**Regarding the utility of rephrased prompts:** We would like to stress that the primary goal of this paper is to create a model capable of performing standard computer vision tasks based on natural language instructions. This approach aims to enhance the model's ability to generalize to both **(1)** previously unseen categories and **(2)** novel user-generated instructions.

The importance of generalizing to unseen categories lies in the model's capacity to apply knowledge gained from training on specific categories in one dataset to perform tasks on entirely new categories in a different dataset. In the context of InstructCV, its ability to achieve this (**Goal 1**) is attributed to its text-to-image formulation, a quality applicable to both fixed and rephrased prompts.

While it is true that the same vision tasks can be executed using fixed prompts, this is not unique to InstructCV and is true for any multi-task model not reliant on natural language inputs. The utilization of rephrased prompts serves a different purpose: it does not aim to enhance performance on standard tasks but rather to improve the model's adaptability to diverse instructions and expressions from end users (**Goal 2**).

These capabilities are envisioned to be particularly beneficial for achieving zero-shot generalization to entirely new tasks beyond the four fixed tasks used in training. For instance, one could envision implementing a panoptic segmentation task in a zero-shot manner by providing a natural language description, combining semantic segmentation and classification tasks. While such capabilities are only likely to emerge in a version of InstructCV that is trained on a larger number of tasks, the rephrased model represents an initial step towards this goal.

Reviewer comment (Experiments):
More ablations should be conducted to compare InstructCV-FP with InstructCV-RP. For example, the authors could report the results of InstructCV-FP in Table 1.

Author response:
**Q3: More ablations should be conducted to compare InstructCV-FP with InstructCV-RP. For example, the authors can report the results of InstructCV-FP in Table 1.**

A3: We have added the performance of InstructCV-FP in Table 1.

| Model         | Dep. Est. (RMSE↓)  |  Sem. Seg. (mIOU↑)| Cls. (Acc↑) | Det. (mAP\at0.5↑) | 
|:---------------:|:------------------:|:-----------------------:|:----------------:|:------------------:|
|   InstructCV-FP   | 0.275 (NYUv2) 0.268 (SUNRGB-D)  | 52.3 (ADE20k) 53.5 (VOC) |  80.4 (Pets) 75.1 (Imagenet-Sub) | 49.1(COCO) 62.0 (VOC) |
|   InstructCV-RP  | 0.297 (NYUv2) 0.279 (SUNRGB-D)  | 47.2 (ADE20k) 52.1 (VOC) |  82.1 (Pets) 74.7 (Imagenet-Sub) | 48.5(COCO) 61.7 (VOC) |

We will also include both versions of InstructCV to all the performance comparison Tables.

Reviewer comment (Presentation):
For the organization of this paper, I suggest the authors move the introduction of diffusion models in Section 2.2 (which takes up most of this section) to a “Prerequisites” subsection or Appendix.

Author response:
**Q4: For the organization of this paper, I suggest the authors move the introduction of diffusion models in Section 2.2 (which takes up most of this section) to a “Prerequisites” subsection or Appendix.**

A4: Thanks for the suggestion. We will make this change in the final version of the paper.

Reviewer comment (Experiments):
The proposed method claims to be a vision generalist but it is validated only in some visual recognition tasks. What about other vision tasks such as low-level tasks (e.g., denoise, deblur, and derain) and generation (as done in Prompt Diffusion [1*])?

Author response:
**Q5: The proposed method claims to be a vision generalist but it is validated only in some visual recognition tasks. What about other vision tasks such as low-level tasks (e.g., denoise, deblur, and derain) and generation (as done in Prompt Diffusion)?**

A5: The InstructCV pipeline is not limited to the 4 tasks considered in the paper, but can be applied to a much broader range of vision tasks including low-level tasks as well. Following [18], we fine-tuned InstructCV to conduct 3 more low-level vision tasks: Denoising (SIDD dataset), Deraining (Rain14000, Rain800, Rain100H, Rain100L, Rain1200 datasets) and Enhance (LOw-Light dataset). (Due to the time constraints, we were only able to evaluate the InstructCV-FP model.) Results are provided below.

|               | Denoising (SIDD) | Deraining (5 datasets) | Enhance |
| --------------- |:---------------:|:---------------:|:---------------:|
| Painter  | 38.88 (PSNR$\uparrow$)  **0.954** (SSIM$\uparrow$)  | 29.49 (PSNR$\uparrow$)  0.868 (SSIM$\uparrow$) | 22.40 (PSNR$\uparrow$)  0.872 (SSIM$\uparrow$)  |
| InstructCV-FP   | **39.20** (PSNR$\uparrow$)  0.949 (SSIM$\uparrow$)  | **31.47** (PSNR$\uparrow$)  **0.872** (SSIM$\uparrow$)  | **24.82** (PSNR$\uparrow$)  **0.895** (SSIM$\uparrow$) |

We will add the results on low-level tasks in the final version of the paper.

Reviewer comment (Experiments):
Further comparisons with other SOTA unified models such as Painter [2*] should be conducted. Painter is currently performing the best in prediction tasks as a vision generalist.

Author response:
**Q6: Further comparisons with other SOTA unified models such as Painter should be conducted. Painter is currently performing the best in prediction tasks as a vision generalist.**

A6: Thanks for this suggestion. We have added Painter as a baseline in Table 1 (See below). On tasks supported by Painter, our InstructCV-FP model achieved higher performance on external datasets, and on non-external datasets, InstructCV-FP also achieved higher performance except for NYUv2.

| Model         | Dep. Est. (RMSE↓)  |  Sem. Seg. (mIOU↑)| Cls. (Acc↑) | Det. (mAP\at0.5↑) | 
|:---------------:|:------------------:|:-----------------------:|:----------------:|:------------------:|
|   Painter  | **0.288** (NYUv2) 0.285 (SUNRGB-D)  | 49.9 (ADE20k) 52.5 (VOC) |  - | -  |
|   InstructCV-FP   | 0.275 (NYUv2) **0.268** (SUNRGB-D)  | **52.3** (ADE20k) **53.5** (VOC) |  80.4 (Pets) 75.1 (Imagenet-Sub) | 49.1(COCO) 62.0 (VOC) |
|   InstructCV-RP  | 0.297 (NYUv2) 0.279 (SUNRGB-D)  | 47.2 (ADE20k) 52.1 (VOC) |  82.1 (Pets) 74.7 (Imagenet-Sub) | 48.5(COCO) 61.7 (VOC) |

Reviewer comment (Evaluation):
In terms of the computational cost, the authors claim that InstructCV exhibits much faster inference speed than previous methods such as Unified-IO. However, InstructCV needs multiple inferences for some tasks (e.g., semantic segmentation and classification), which could consume N×times where N is the number of semantic classes. Besides, the reported inference time of InstructCV on a single NVIDIA A100 GPU for an image is 5 seconds in the main paper but 10 seconds in the Appendix.

Author response:
**Q7: Comparing inference speeds with baselines.**

A7: We believe that our claim on InstructCV exhibiting much faster inference speed than previous methods such as Unified-IO is fair, because Unified-IO also infers one class at a time, and the inference times we correspond to speed of inference per class. The inference time is 5 seconds and not 10 seconds, the discrepancy arises from an earlier version of our model. We will fix this error in the Appendix of the final paper.

Reviewer comment (Writing):
In the second paragraph of Section 1, the authors claim that “However, these prompts consist of (uninterpretable) numerical values derived from specific training datasets, which may limit their ability to generalize to new datasets, tasks, or categories”. Why do the prompts used in inpainting-based methods (which are visual examples) contain uninterpretable numerical values?

Author response:
**Q8: In the second paragraph of Section 1, the authors claim that “However, these prompts consist of (uninterpretable) numerical values derived from specific training datasets, which may limit their ability to generalize to new datasets, tasks, or categories”. Why do the prompts used in inpainting-based methods (which are visual examples) contain uninterpretable numerical values?**

A8: We appreciate the reviewer for bringing this to our attention. The statement in question was intended to address **prompt tuning** approaches. These approaches involve learning task-specific "codes," such as input tokens or pixels, to guide Seq2Seq models in executing the desired task. This specifically pertains to the first category of methods discussed in paragraph 2 on page 2. It's important to note that this doesn't apply to [18] and [19], as they learn the vision task in-context using interpretable visual examples. To avoid any confusion, we will explicitly state that this statement is referring to **prompt tuning** and not **in-context** approaches.

Reviewer comment (Presentation):
The format of references is messy and inconsistent. For example, a bunch of references do not list the conference/journal title.

Author response:
**Q9: The format of references is messy and inconsistent. For example, a bunch of references do not list the conference/journal title.**

A9:  Thanks for pointing this out. We will revise the formatting of references in the final version of the paper.

Reviewer comment (Novelty):
Just a reminder that there is a very similar work, InstructDiffusion [3*], to this submission. The authors could consider citing that paper and provide comparisons and discussions.

Author response:
**Q10: Just a reminder that there is a very similar work, InstructDiffusion, to this submission. The authors could consider citing that paper and provide comparisons and discussions.**

A10: Thank you for your suggestions. We were not aware of InstructDiffusion at the time of submission, as it is concurrent with our work. We appreciate your diligence in bringing this to our attention. To address this, we will include a citation for InstructDiffusion and incorporate discussions about it in the paper.