Paper: UniVis: A Universal Framework for Computer Vision Tasks (ICLR-2024)

Reviewer comment (Evaluation):
Constrained by the pretraining model (i.e., SD), UniVis performs better in image generation (Table 3) while average in visual understanding and low-level image processing (Table 1&2). This indicates that the framework is coupled with the training format of the pretraining model. Furthermore, the authors claim that there is no need for high training costs, but this method requires more parameters.

Author response:
**A1:** (1) We would like to clarify that one major contribution of this paper is to build a  unified framework for vision tasks through **generative modeling** rather than commonly adopted contrastive learning or masked image modeling. Different from the well-known dictum by Richard Feynman, "What I cannot create, I do not understand", we find it very challenging for the generative models (like PromptDiffusion) to perform well on other vision tasks, especially the visual understanding tasks. Hence, the main aim of this work is making the generative model simultaneously work well on three distinct categories of vision tasks. To this end, we resort to a SOTA text-to-image model, which is one of the very few trained on web-scale data, and we carefully devise an instruction tuning pipeline to adapt it to downstream tasks. As you stated, we achieve competitive results on visual understanding and low-level image processing and exhibit SOTA performance on image generation (**see Figure 5 in the revised paper**). In contrast, our competing methods (Painter and PromptDiffusion) experience a clear collapse or near breakdown on one of the three categories of vision tasks (**see first table in general response #1**). Please refer to our general response #1 for more details.

(2) We are a bit confused about the comment saying "requires more parameters". If you mean UniVis-st requires distinct model parameters to tackle different tasks (please let us know if you did not mean this), we would like to clarify that we also present models (UniVis-sc and UniVis-mc) that are jointly trained on multiple tasks (see Tables 2, 3, and 4), which use shared model parameters. The main contribution of our paper is a unified framework for visual task learning, and it can be employed to produce three types of models under different practice scenarios (e.g., the computing power at hand).

Reviewer comment (Experiments):
Fig4 is significant as I personally believe one of the benefits of unifying computer vision tasks should be the ability to generalize to other unseen tasks (analogous to LLM), but unfortunately, only one figure is provided to demonstrate this. How to further enhance generalizability is an interesting point.

Author response:
**A3:** We have added more results in the updated paper (**please refer to Figure 17**). They verify the generalization capability of the proposed framework. We totally agree that performing out-of-distribution inference is one of the intriguing properties of unified models. Gathering more diverse data for training UniVis could be promising to enhance generalizability and we plan to investigate this in future work.

Reviewer comment (Experiments):
The testing scheme in the paper is similar to one-shot, how can it be extended to few-shot (>1)?

Author response:
**A4:** Few-shot in-context inference can be conducted by establishing the grid image with more input-output pairs. Following this, we made some attempts under two-shot and four-shot settings during inference and reported the results below (also added to Table 11 in the revised paper). We observe that there is a slight gain by introducing more visual instructions, and we think this could be further explored during training, which we leave to future work.

|                                 |   |                  | Depth estimation |                        |   |                |    Denoising   |                   |
|---------------------------------|:-:|:----------------:|:----------------:|:----------------------:|:-:|:--------------:|:--------------:|:-----------------:|
| **# of shots during inference** |   | RMSE$\downarrow$ |  REL$\downarrow$ | $\delta_{1}$$\uparrow$ |   | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| one-shot                        |   |       0.420      |       0.135      |          0.857         |   |      34.55     |      0.907     |       0.095       |
| two-shot                        |   |       0.416      |       0.131      |          0.863         |   |      34.73     |      0.911     |       0.095       |
| four-shot                       |   |       0.413      |       0.130      |          0.863         |   |      34.75     |      0.911     |       0.095       |

Reviewer comment (Evaluation):
The results in the tables cannot be understood without knowing the variation that might be produced from one repetition of an experiment to the next. It is crucial to state whether each experiment was only run once, or whether the reported numbers are averages across several trials. If the latter, it is essential to provide some measure of variation (standard deviation, confidence interval). If there is no estimation of experiment-to-experiment variation, it is unclear how to know whether the observed differences in the tables are not simply noise.

Author response:
**A1:** We have updated some tables in the paper to include the average scores and standard deviations across three trials. From the added results, we could find that the proposed UniVis is very stable with small stds. Due to limited time and computing resources, we are unable to run all experiments multiple times during rebuttal. We promise to update all tables in the final version if this paper is accepted.

Reviewer comment (Writing):
At p. 6, the statement that "patch-level inpainting that resembles word completion training schemes in NLP is not adequate for a holistic and profound understanding of vision tasks because images _have much lower semantic density than texts_" is vague. The intended meaning, possibly about redundancy and spatial correlation in images compared to text, should be made more precise.

Author response:
**A2-2:** We have updated the description as follows (see last paragraph of Section 3.2): "...due to the fact that the correlation between pixels is much stronger than that between words (e.g., this redundancy presented in images makes the model readily inpaint a patch with neighboring patches)."

Reviewer comment (Writing):
DDIM is mentioned at p. 16 (B.2) without any explanation.

Author response:
**A3-1:** We have added a brief introduction to DDIM in the revised paper as follows (see Appendix B.2): "By setting the random noise in the reverse diffusion process to 0 (i.e., a  deterministic sampling), DDIM manages to generate an image with fewer sampling steps compared to DDPM."

Reviewer comment (Writing):
Although U-Net is a well-understood term-of-art, it could still use a brief explanation of its purpose, given that it is the component being trained to fit the distribution of latent codes.

Author response:
**A3-2:** We have added a brief explanation of the purpose of U-Net in the revised paper as follows (see first paragraph of Section 3.2): "Specifically, a denoising U-Net is trained to fit the distribution of latent codes, which models the reverse diffusion process. Taking the noisy latent and the time step as input, this U-Net is further conditioned on the textual embeddings extracted through a text encoder CLIP via cross-attention to produce the output at the current time step."

Reviewer comment (Writing):
The phrase "spatial-wise concatenation" at p. 4 (first paragraph of 3.1) is unclear and should be described. If it refers to stitching images together in a grid as visualized in Figure 2, this should be explicitly stated.

Author response:
**A3-3:** Yes it is, we have appended a brief explanation of "spatial-wise concatenation" in the revised paper as follows (see first paragraph of Section 3.1): "we can implement spatial-wise concatenation of any input, out, and query samples (i.e., stitching all the images together into a grid as illustrated in Figure 2)."

Reviewer comment (Writing):
At p. 3, the use of the term "three _implementations_" of UniVis is questionable. These are all the same implementation trained in three different regimes (single-task, single-category, and multi-category). The later-used phrase "three types of model training" is preferable.

Author response:
**A4-1:** We have changed "three implementations of UniVis" to "three types of UniVis trained in different regimes" (at the top of page 3).

Reviewer comment (Writing):
In the first sentence of the abstract, the word "tam" appears to be a typo.

Author response:
**A4-2:** Thanks for pointing this out. It should be "tame". We are not allowed to make changes to the title and abstract during rebuttal per ICLR policy. We will fix this in the final version if our paper gets accepted.

Reviewer comment (Writing):
In the introduction, the sentence "The Challenges are in three aspects, elaborated in the following" is awkward and could be improved to "There are three main challenges."

Author response:
**A4-4:** Following your suggestion, we have simplified that sentence in the revised paper (see second paragraph of Section 1).

Reviewer comment (Experiments):
The proposed method is limited to dense image tasks, where the output is at a high-dimensional image level. It does not demonstrate feasibility on a wide range of computer vision tasks where the output is low-dimensional, such as image classification, image captioning, VQA, etc.

Author response:
**A1:** (1) We have evaluated UniVis on keypoint detection, where the output is low-dimensional (please refer to Figures 3, 8, and 17). (2) Other computer vision tasks with sparse outputs could be tackled by our framework as long as one can transfer the output to RGB images. Taking image classification as an example, a class label can be turned into an RGB image filled with a unique color determined by its semantic class. Due to limited time during rebuttal, we cannot provide the results on image classification at this time, but we promise to include them in the final version if the paper gets accepted. (3) We would like to clarify that this paper focuses on standard computer vision tasks instead of vision-language ones including image captioning and VQA, which are also not explored by our competing methods (Painter and PromptDiffusion).