Paper: POSITION EMBEDDING INTERPOLATION IS ALL YOU NEED FOR EFFICIENT IMAGE-TO-IMAGE VIT (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the proposed framework is limited. The method in this paper consists of guided diffusion, ViT, and position embedding. There have already been multiple transformer-based SR methods proposed in the previous 3 years, e.g., IPT and SwinIR. Therefore, there is no clear motivation or explanation for designing the framework.

Author response:
It is possible that the contribution of our method may be underestimated just in terms of the framework of the method. In this paper, we pay more attention to the framework and training schedule design. To improve our performance, we use the data augmentation and modified loss function in the diffusion model. We also use multiple loss functions to obtain the photo-realistic performance of SR. To address the multi-resolution inferring in ViTSR, we apply position embedding interpolation to break the resolution limit of ViT without performance degradation.

Although architecturally our method is not as complex as some, in fact, just by simple positional interpolation we have made an interesting finding that we believe will help to inspire more work especially ViT-based image-to-image methods.

Reviewer comment (Evaluation):
The diffusion model is trained with low-resolution data. In image inpainting, it generally relies on the quality of the generated prior, which may result in a performance loss or detail loss. There is a question whether subsequently applying a super-resolution model will cause an amplification of distortion issues, and the author needs to explain this clearly.

Author response:
As we use low-resolution images for inpainting, the restored result has fewer details than the high-resolution images. But we do not think the SR module is an amplification of distortion. We use SR for reconstructing the details more accurately, although SR cannot get the same 100% details and texture as the original. In Fig. 3, the results of our framework are all the direct output of the ViTSR, and we do not replace the unmask area with the original images. It obtains competitive performance with the original resolution results.

Reviewer comment (Novelty):
Lack of novelty. The paper combines existing modules such as guided diffusion, ViT, and position embedding.

Author response:
It is possible that the contribution of our method may be underestimated just in terms of the framework of the method. In this paper, we pay more attention to the framework and training schedule design. To improve our performance, we use the data augmentation and modified loss function in the diffusion model. We also use multiple loss functions to obtain the photo-realistic performance of SR. To address the multi-resolution inferring in ViTSR, we apply position embedding interpolation to break the resolution limit of ViT without performance degradation.

Although architecturally our method is not as complex as some, in fact, just by simple positional interpolation we have made an interesting finding that we believe will help to inspire more work especially ViT-based image-to-image methods.

Reviewer comment (Writing):
There are grammar errors that need attention. For example, in the introduction section, the phrase "super-resolution the image" should be "super-resolve the image".

Author response:
Thanks for your comments. We will further enhance the readability of our manuscript by proofreading.

Reviewer comment (Experiments):
The paper is titled 'image-to-image' tasks, but the experiment section only shows image inpainting and super-resolution tasks. It is suggested to conduct a wider range of experiments to demonstrate the effectiveness of the proposed method.

Author response:
Limited by space, some extra experiments of image colorization and deblur using a multi-task VIT autoencoder are presented in Appendices B and C. Our manuscript lies more in this elaboration and presentation of our interesting finding, with appropriate experiments to verify its validity. Subsequent more image-to-image applications can similarly be inspired.

Reviewer comment (Experiments):
The paper lacks an ablation study.

Author response:
The results of some ablation experiments are in the Appendix limited by space. Some experiments on the different preprocessing models and the position embedding types are carried out. We also show the effect of the different input sizes when inferring.

Reviewer comment (Writing):
In section 4.3, ViTSR+ needs more specific clarification. There is ambiguity here because in super-resolution methods, the "+" symbol generally signifies self-ensemble.

Author response:
Thank you for your suggestion. We will add a description of the difference between ViTSR and ViTSR+ in the title of Table 3.

Reviewer comment (Presentation):
The motivation of this paper is vague. The reviewer is not sure about the main focus of the paper: the inpainting / restoration task in higher resolution, or the position embedding interpolation trick for ViT. The authors emphasized on the position embedding in the title, while even the ablation study related to that is moved to the appendix.

Author response:
We apologize for the ambiguity of our existing title and appreciate for your suggestion. Based on this, the title is corrected to “POSITION EMBEDDING INTERPOLATION IS ALL YOU NEED FOR EFFICIENT VIT-BASED IMAGE-TO-IMAGE METHODS”, and we hope that the new title will more accurately reflect the contribution of our paper.  

Our initial motivation is to propose a framework for image inpainting and using the super-resolution for training efficiency; however, as more experiments carried out on ViTSR, we find it can be a general image-to-image translation model for a list of low-level tasks. Position embedding interpolation can be used in most of the tasks especially ViT-based image-to-image methods. Although both mentioned are core contributions to this manuscript, we feel that this finding is far more valuable than the initial motivation of building an efficient framework, and hence we choose it as our title in the hope that it will inspire more work especially ViT-based image-to-image methods.

The ablation study of PEI is moved to the appendix limited by space.

Reviewer comment (Evaluation):
The concept of efficiency is not clear enough. Do the authors mostly mean the training efficiency or the inference time? It seems the authors try to claim the proposed tricks enable the training to be done on lower-res image, while the model can be equally good when being applied to high-res images. While it can only be called efficient training tricks, but the entire ViT cannot be claimed as efficient. If the authors want to claim the pipeline of low-res inpainting + high-res ViT is efficient, then what is the purpose of showing image restoration results, and why the pipeline is better than LDM in efficiency?

Author response:
For efficiency, the paper focuses more on the new training framework that we proposed. The new training and inferring procedures can be more efficient in some extend. We noted the ambiguity of “for efficient ViT” in the title and has corrected it to reflect our contribution more accurately.

For the LDM, as we mentioned, our diffusion model can be exchanged with other SOTA diffusion models, but we think that training a LDM in high-resolution will cost more computations compared to low-resolution. The reason to show the image deblur and colorization results is that we want to present more ability of ViT-based autoencoder. It can be a general image-to-image model for many low-level image tasks.

Reviewer comment (Experiments):
Inpainting task may require the high-res masked image when doing super resolution since it needs to resolve the boundary seam issues. While this paper (Fig.4) did not show the composited results and the proposed ViT also did not include the original high-res image as the input. So it was not designed for inpainting.

Author response:
We have thought about using the unmasked part of the high-resolution image in the SR model so that we can get more texture details from the original image. However, the freeform mask is hard to directly apply to ViT, and we want it to be a general SR model that can be used in real-world SR tasks. At last, we give up fusing the prior information with the SR model. Actually, we have tried many different models like CNN or Diffusion to fuse the prior information when super-resolving the inpainting image, but the output is not as good as supposed. VITSR is the best model in terms of performance even without high-resolution information.

Reviewer comment (Experiments):
Table 1 missed many results even though LaMa can work on any resolution input. Not sure whether the authors have controlled the testing dataset when showing these numbers.

Author response:
It is a great struggle for us to do the comparison. Because of the uniqueness of our proposed method, it is difficult to find the suitable methods for considerations of fairness. In fact, we also carried out many experiments by using a different resolution that is not the same as the training for the compared models, but their outputs are too terrible. That is not fair either.

Reviewer comment (Experiments):
How does the network perform if using the same image augmentation as the employed guided diffusion (Dhariwal & Nichol)?

Author response:
The original DDPM does not use many image augmentations. Here, the multi-augmentation inspired by [1] (also cited in the paper) is adopted to improve the generalization of the network. We find the performance in [1] is improved after augmentation, hence our model achieves better performance compared to the baseline model.  

[1] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565–26577,2022.

Reviewer comment (Evaluation):
In Table 1, the reviewer finds that the proposed network is inferior to DiffIR in many metrics.

Author response:
Our method is inferior in some metrics for the reason that we inpaint the image in low resolutions which will lose some details compared to the original image. However, it should be emphasized that our framework pays more attention to training and inferring efficiency, and our ViTSR has the outstanding performance on image SR. Therefore, even with the low-resolution image as input, we can get a competitive performance with other methods.

Reviewer comment (Experiments):
For complexity verification in Table 2, the proposed framework is only compared with diffusion models. Is that because the authors only employ diffusion models for low-resolution inpainting?

Author response:
Here, we propose a framework to deal with the high-resolution training cost of the diffusion model. Our aim is to design this framework for a diffusion-based model so that our framework can obtain the advantage of the diffusion model in image generation with low training costs. Compared to the non-diffusion model, it is usually hard to inpaint the low-resolution image with large missing regions. Hence, we care more about the performance compared to the diffusion-based models instead of using a CNN as the inpainting model of our framework.

Reviewer comment (Experiments):
Why is there no performance of ViTSR+ in Table 4 for CelebA-HQ? Why do the authors conduct the experiments on this dataset while other papers do not?

Author response:
In all the experiments, we just trained a ViTSR+ with the DF2K dataset. The DF2K dataset has a big margin compared to CelebA-HQ, so we do not test CelebA-HQ on it. Although other methods were not trained on CelebA-HQ, we think SR of human face is also a significant real-world SR task and another aim is to eliminate any concerns that some readers might have about the performance depending on the dataset. Therefore, the results on CelebA-HQ are presented to further verify the effectiveness of our method. And the fact is that our method does have the outstanding performance on 8x SR on CelebA-HQ.

Reviewer comment (Novelty):
The reviewer finds that in the contribution part and conclusion section, the author aims to propose an efficient network for high-resolution image inpainting. In the title, the related keywords are "Image-to-image ViT". The reviewer thinks that this is not suitable since the authors do not propose an efficient ViT, just a combination method to achieve high efficiency.

Author response:
We apologize for the ambiguity of our existing title and appreciate for your suggestion. Based on this, the title is corrected to “POSITION EMBEDDING INTERPOLATION IS ALL YOU NEED FOR EFFICIENT VIT-BASED IMAGE-TO-IMAGE METHODS”, and we hope that the new title will more accurately reflect the contribution of our paper.