Paper: SAIR: LEARNING SEMANTIC-AWARE IMPLICIT REPRESENTATION (ICLR-2024)

Reviewer comment (Novelty):
I believe the technical novelty of the approach is limited since the improvement mainly comes from the rich representations of the clip embeddings. CLIP embeddings have been extensively used in many zero-shot tasks that exploit the strong semantics learned by the clip embeddings e.g. ZegClip (CVPR 2023), Hierarchical Text-Conditional Image Generation with CLIP Latents (arXiv 2022), NUWA-LIP (CVPR 2023).

Author response:
Our contribution extends beyond the utilization of CLIP embeddings. In the image inpainting task, the input image is masked and the CLIP embedding is damaged.
We address this challenge by introducing SIR, which can reconstruct and resize the damaged and low-resolution embeddings simultaneously. Previous methods  only need to resize undamaged embeddings.

Reviewer comment (Experiments):
Lack of comparison with recent approaches like NUWA-LIP.

Author response:
Several recent CLIP-based approaches, such as NUWA-LIP (CVPR 2023), necessitate a 'Guidance Text' to describe the input image for generation. However, datasets like CelebAHQ and ADE20K lack such descriptions.
Those methods rely on the image-level semantic descriptions, while we build the continuous pixel-wise semantic representation.

Reviewer comment (Evaluation):
One main concern comes from the application of this stream of methods. In other words, the current evaluation benchmark is made manually and may be too theoretical. I wonder if any real-life application cases can be shown, e.g. recovering objects that are occluded or blurred via dramatic camera motions. If there are more proper real application cases, there is no need to be limited to the ones I list.

Author response:
A1. Our method can be used in many real-life applications, such as object removal. To illustrate its effectiveness, we present several examples in the newly added section 'Real-life Applications'.

Reviewer comment (Evaluation):
Another concern lies in the computation cost. It is suggested to compare the inferring and training cost with previous methods, as it seems the two-module framework may be costly.

Author response:
A2. Our model takes 0.043s to process one image. The one-module framework LIIF's inference time is 0.027s. The incurred increase in time cost remains within an acceptable range.

Reviewer comment (Novelty):
What is the main advantage of the coordinate-based implicit representation method over diffusion-model ones for image inpainting? Diffusion models have shown great power in recent generation tasks, also including inpainting. It is suggested to discuss this question and include necessary related works, which will determine the significance of the contribution.

Author response:
A3. **For image inpainting, our proposed method outperforms diffusion-based methods in three aspects.** **First, inference speed.** The inference speed of our method (0.043s/image) is faster than that of the diffusion-based method, such as stable diffusion [1] (12s/image). **Second, arbitrary resolution.** Our method can generate images at arbitrary resolutions during inference, a capability lacking in diffusion-based models. **Third, high fidelity.** Diffusion-based models prioritize naturalness over fidelity, resulting in generated results that deviate significantly from the ground truth. We compare our method with stable diffusion [1] on the CelebAHQ dataset under the same setting, the results are PSNR$\uparrow$ 37.97 (ours) vs. 37.60 (stable diffusion) and L1$\downarrow$ 0.01 (ours) vs. 0.032 (stable diffusion).  

[1] Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models. (CVPR 2022)

Reviewer comment (Experiments):
Can the proposed method apply to any in-the-wild images, not limited to the used datasets? If yes, it is better to show some samples.

Author response:
A4. Our method can be used to process the in-the-wild images. The results are shown in section 'Real-life applications'.

Reviewer comment (Experiments):
The result in Table 5 is confusing. Authors trying to study the effect of SIR block, but after removing SIR, the network is just AppEncoder ConvNet. Authors didn't explain clearly how to evaluate ADE20K mIoU with AppEncoder alone.

Author response:
A1. In Table 5 'with/without SIR', we want to shighlight the reconstruction proficiency of the SIR model in semantic segmentation task.  
We do not use APPENCODER in this section.  
The model only using APPENCODER is similar to the baseline LIIF.

Reviewer comment (Experiments):
One key ablation I would suggest authors add in both Table 1 and Table 2, is that compare SAIR without CLIP and with other networks other than CLIP, e.g. ImageNet pre-trained models.

Author response:
A2. In the section 'Using other semantic embeddings', we substitute the CLIP encoder with a pre-trained SAM encoder [1]. Tthe text-aligned embeddings produced by CLIP yield superior results.  

[1] Kirillov et al. Segment anything.

Reviewer comment (Experiments):
In the dataset section, authors states CelebA and ADE20K have 19 and 150 classes respectively. Are these semantic labels of the dataset used during training and testing?

Author response:
A3. We do not use the dataset semantic labels, as the text-aligned embeddings are generated by CLIP encoder directly.

Reviewer comment (Writing):
In Section 5.3, authors state " we used CLIP_T to filter the image feature CLIP_I". The term filter is not very clear or straightforward. Are authors trying to imply "dot product"?

Author response:
A4. In this section, we have image pixel features and text features for all dataset semantic labels.  
We assess the similarity between individual pixel features and semantic label features, and assign a label to each pixel based on the similarity.

Reviewer comment (Reproducibility):
Is there a loss for semantic feature reconstruction? If not, how could SIR reconstruct the semantic features, as stated in Section 5.3.

Author response:
A5. **Yes, we use a loss function to guarantee that SIR does not change the text-aligned feature space.**  
We choose L1 loss to quantify the dissimilarity between the unmasked image's text-aligned feature and the SIR reconstructed feature  without changing the resolution.

Reviewer comment (Experiments):
Is there any comparison with mask ratio 0? Just compare to the original LIIF on super-resolution tasks.

Author response:
A6: We test the super-resolution ability of proposed method against LIIF.  
PSNR/SSIM results under different upsample ratios are shown in the table.  
Our method still gets promising results in SR task.  

|Method |x2| x4 |x6|x8|  
|-|-|-|-|-|  
|LIIF | 34.32/0.963 | 30.98/0.882 | 30.13/0.838 | 29.83/0.819|  
|Ours | $\textbf{34.61}$/$\textbf{0.972}$ | $\textbf{31.24}$/0.881 | $\textbf{30.39}$/$\textbf{0.840}$| $\textbf{30.03}$/$\textbf{0.825}$|

Reviewer comment (Theory):
How to understand the claim that the $f_\theta$ of SIR learns the **text-aligned** embeddings (Eq (4))? Though the operation similar with MaskCLIP does not alter the text-aligned feature space, these features are then processed by learnable $\theta$, and there is no guarantee that the embedding space is text-aligned.

Author response:
A1.To ensure the preservation of the text-aligned feature space in SIR, we choose L1 loss to quantify the dissimilarity between the unmasked image's text-aligned feature and the SIR reconstructed feature without changing the resolution.

Reviewer comment (Theory):
Why do the authors highlight the **text-aligned** embeddings? If I understand correctly, the embedding space is just an enhanced pixel-wise semantic feature space.

Author response:
A2. 
Our text-aligned embedding seamlessly aligns with semantic label features, other semantic features lack this inherent compatibility. In the 'Using other semantic embeddings' section, we substantiate the superior efficacy of our embedding. And our contribution is not only using the clip representations, we also propose SIR to reconstruct the damaged and low-resolution embeddings for image inpainting task.

Reviewer comment (Evaluation):
Why mapping the original CLIP feature space by $f_\theta$ performs better than the original CLIP feature space? Furthermore, the details about how to implement "*models without SIR block*" are not clear.

Author response:
A3. In Fig. 4, the initial CLIP feature derived from the pre-trained CLIP encoder encounters challenges in effectively handling the masked regions. Our introduced SIR module $f_θ$ proficiently reconstructs the semantic feature, showcasing a notable enhancement in performance.

Reviewer comment (Experiments):
Is the proposed SAIR robust/generalizable for other degraded images, like raining or noised images?

Author response:
A5. We conduct experiments on adversarial defence task following DISCO [1]. To create training pairs, PGD ($\epsilon_\infty$ = 8/255 with
step size is 2/255 and the number of steps 100 ) is used to attack a WideResNet28 on Cifar10 dataset.  We evaluate both Standard Accuracy (SA) and Robust Accuracy (RA), where SA measures classification accuracy on clean examples, and RA measures accuracy on adversarial examples. The results in terms of $L_\infty$ norm affirm that our model demonstrates robust adversarial defense capabilities.

|Method | SA | RA | Avg|
|-|-|-|-|
|No Defence   |$\textbf{94.78}$ | 0|47.39|
|Bit Reduction [2] | 92.66|1.04|46.85|
|Jpeg [3] |83.9|50.73|67.32|
|Input Rand. [4] | 94.3|8.59|51.54|
|AutoEncoder|76.54|67.41|71.98|
|STL [5] |82.22|67.92|75.07|
|DISCO [1]|89.26|85.86|87.41|
|Ours | 90.0|$\textbf{86.21}$|$\textbf{88.10}$|


[1] Ho et al. Disco: Adversarial defense with local implicit functions. (Neurips 2022)

[2] Xu et al. Feature squeezing: Detecting adversarial examples in deep neural networks.

[3] Dziugaite et al. A study of the effect of compression on adversarial images.

[4] Xie et al. Mitigating adversarial effects through randomization. (ICLR 2018)

[5] Sun et al. Adversarial defense by stratified convolutional sparse coding. (CVPR 2019)