Paper: Rethinking RGB Color Representation for Image Restoration Models (ICLR-2024)

Reviewer comment (Presentation):
In the paper, the performance of the proposed loss are demonstrated on perceptual SR task. The results in table 1 are confusing. The PSNR and SSIM of RRDBNet are the highest among all the settings, but they are not bolded. The SSIM of the last setting is worse than most of settings for DIV2K-Val dataset, but it is bolded as better score.

Author response:
- We appreciate your constructive feedback. We have changed Table 1 accordingly to clarify the main points we wanted to make with it.

- In Table 1 and Section 3.2 of the main manuscript, our goal was to excel in perceptual metrics in perceptual image super-resolution. In particular, we would like to note that we focus on the improvement brought by replacing the L1 loss in the RGB space with the L1 loss in our $a$RGB space.

- The first row stands for the PSNR-oriented reference model just for the reference. This is not trained under the perceptual super-resolution setting. The rows two to five represent methods to enhance perceptual metrics. Here, we intended to make comparisons between the second and third rows, while comparing the fourth and fifth rows.

- We hope this change removes the original confusion.

Reviewer comment (Experiments):
For perceptual SR and image deblur tasks, there are considerable baselines perform better than ESRGAN and MPRNet. For example, restormer and NAFNet could be used for deblurring evaluation. In this way, we can test whether the proposed loss could consistently boost performance and lead to a new SOTA.

Author response:
- Regarding state-of-the-art methods, we agree with your concerns that the goal of designing a loss function is to work with the best models we have at that time. However, please understand that due to the strict resource constraint upon us, we had to find the optimal combination of experiments to best demonstrate the versatility of our method in various tasks and model types.

- Regarding our experiments for perceptual super-resolution in Section 3.2 of our manuscript, due to heavy computation of Restormer and NAFNet, many previous works (ESRGAN [5*], Real-ESRGAN [8*], SPSR [9*], LDL [10*], CAL-GAN [11*]) adopt the RRDB network as their baseline. Moreover, we would like to note that NAFNet-width64, the state-of-the-art model you have mentioned is used for the denoising task, yet not for the deblurring task, in Section 3.3 of our manuscript.

- From the experiments in Section 3 of the manuscript, we can observe consistent performance gain in various tasks and model architectures. The improvements were achieved using the same pre-trained $a$RGB autoencoder. Therefore, we would like to argue that our $a$RGB space suggests that there can be alternatives to the RGB space for the guidance of training image restoration models.

- As a final remark, though it is not a state-of-the-art, we hope we can justify our approach more clearly with an additional experiment. Here, as an answer to your regard on our small performance gain, we apply our method on another deblurring model [4*], as presented in the table below. Consistent with Table 2 in the main manuscript, our method demonstrates superior performance. The numbers are calculated from the author’s official code repository.


| Loss | GoPro PSNR | GoPro SSIM |
| --- | --- | --- |
| L1 | 30.40 | 0.9018 |
| L1-$a$RGB | 30.85 (+0.45 dB) | 0.9096 (+0.0078) |

Reviewer comment (Evaluation):
The performance gains are too small, which can hardly verify the effectiveness of the proposed loss.

Author response:
- Regarding state-of-the-art methods, we agree with your concerns that the goal of designing a loss function is to work with the best models we have at that time. However, please understand that due to the strict resource constraint upon us, we had to find the optimal combination of experiments to best demonstrate the versatility of our method in various tasks and model types.

- Regarding our experiments for perceptual super-resolution in Section 3.2 of our manuscript, due to heavy computation of Restormer and NAFNet, many previous works (ESRGAN [5*], Real-ESRGAN [8*], SPSR [9*], LDL [10*], CAL-GAN [11*]) adopt the RRDB network as their baseline. Moreover, we would like to note that NAFNet-width64, the state-of-the-art model you have mentioned is used for the denoising task, yet not for the deblurring task, in Section 3.3 of our manuscript.

- From the experiments in Section 3 of the manuscript, we can observe consistent performance gain in various tasks and model architectures. The improvements were achieved using the same pre-trained $a$RGB autoencoder. Therefore, we would like to argue that our $a$RGB space suggests that there can be alternatives to the RGB space for the guidance of training image restoration models.

- As a final remark, though it is not a state-of-the-art, we hope we can justify our approach more clearly with an additional experiment. Here, as an answer to your regard on our small performance gain, we apply our method on another deblurring model [4*], as presented in the table below. Consistent with Table 2 in the main manuscript, our method demonstrates superior performance. The numbers are calculated from the author’s official code repository.


| Loss | GoPro PSNR | GoPro SSIM |
| --- | --- | --- |
| L1 | 30.40 | 0.9018 |
| L1-$a$RGB | 30.85 (+0.45 dB) | 0.9096 (+0.0078) |

Reviewer comment (Novelty):
The proposed loss is similar to Fourier loss, which is also decomposed the image upon pre-defined basis. Can authors discuss the difference between them and compare their performance?

Author response:
- We find three key differences between the Fourier space loss and the per-pixel loss in our $a$RGB space.

    1. Fourier space loss is typically combined with its primary loss, i.e., the RGB space-based loss, without removing the necessity of the RGB loss. This is the common practice of the Fourier loss we can found in other notable works [6*, 7*], too. However, in our work, **we suggest an alternative to the RGB space** with our $a$RGB space.
    
    2. Fourier space loss calculates distances in the discrete dual of the image space by transforming the pair of images using FFT algorithm. FFT is known to be a linear operation, and therefore a Fourier space loss can be thought of as a linearly weighted distances between two image patches of fixed size. On the other hand, **our $a$RGB encoder is nonlinear**, and therefore the distance metrics are contracted and expanded based on the local patch structure. This unique feature of ours helps our $a$RGB-based loss functions adaptively weigh important structures. The evidence of the metric warping of our $a$RGB space is shown in Figure 5d.
    
    3. Although the Fourier space loss can handle long range nonlocal correlations via resonance in Fourier domain, their resonance frequency is strictly fixed because the linear weight of the FFT algorithm is a constant. In contrast, our $a$RGB encoder **only cares for the local structure** within its receptive fields, yet it can **encode more complex structures adaptively** thanks to its Mixture-of-Experts architecture. Figures 5b, 5c and 20 in our manuscript show that pixel assignments to each of $a$RGB encoder’s expert depends on the local structures of its neighborhood.

- To demonstrate the effectiveness of our $a$RGB loss over the Fourier space loss, we design an experiment to train a deblurring model [4*] using two types of losses: (1) the combination of the primary (the L1-RGB) and the dual (the L1-Fourier space loss) losses following recent works [6*, 7*] and (2) the L1-$a$RGB loss without using the L1-RGB loss. Please understand that the time and resources permitted to us is limited, so we are temporarily reporting the intermediate scores. However, the difference between the two method clearly demonstrates the superirority of our $a$RGB loss.

| DeepDeblur / GoPro (100 epochs) | PSNR (dB) | SSIM |
| --- | --- | --- |
| Fourier space loss + L1-RGB | 27.0007 | 0.8241 |
| L1-$a$RGB | 27.7348 (+0.73 dB) | 0.8416 (+0.0175) |

Reviewer comment (Evaluation):
The performance improvement of the proposed aRGB space in the denoising and deblurring tasks seems insignificant. In Table 2, comparing the first two rows, and the last two rows, the PSNR gains are only 0.02 dB and 0.03 dB, respectively. In Table 3, the PSNR improvements between the last two rows are 0.07 dB on GoPro and 0.02 dB on HIDE dataset.

Author response:
- We acknowledge your concerns. We have conducted additional experiment on another deblurring model [4*], changing its original L1-RGB loss with our L1-$a$RGB loss. The results in the table below shows greater improvement, consistent with Table 2 in the main manuscript.

| Loss | GoPro PSNR | GoPro SSIM |
| --- | --- | --- |
| L1 (Original)  | 30.40 | 0.9018 |
| L1-$a$RGB | 30.85 (+0.45 dB) | 0.9096 (+0.0078) |

Reviewer comment (Writing):
Equation 6, L_{pair} should be L_{pixel}.

Author response:
We thank for your correction. We have revised the equation in the updated pdf.

Reviewer comment (Theory):
The space aRGB is originally designed to encode structure information on a pixel basis. Why it can exhibit suppression of artifacts in SR tasks?

Author response:
- We believe that there are at least two reasons the artifacts are well suppressed by our $a$RGB encoder.
    
    1. **Local information propagation**: Through the $a$RGB encoder, each pixel receives gradients from the pixel difference of its neighboring pixels. Visual artifacts can be seen as a disruption in high-frequency variation among adjacent pixels [12*]. The information propagation through our $a$RGB-based loss suppresses not only each pixel’s regression error, but each local region’s structural variations. This is not like in the conventional RGB space-based loss, which only cares about each pixel’s regression error. This analysis is further justified by our discussion in Appendix A. Here, we have shown that the amount of information propagation, i.e., the scale of gradients from adjacent pixels, is significant in our $a$RGB encoder.
        
    2. **Edge encoding**: We can also consider this phenomenon from the representation perspective. As visually demonstrated in Figures 5a, 17, and 18, the $a$RGB encoder appends local edge information to each pixel. This means that the $a$RGB embedding space effectively has channels responsible for edge information. This structural part is directly compared by a per-pixel $a$RGB loss, suppressing undesired **structure** presented in the restoration output.

- In short, we believe that the cause of artifact suppression you have mentioned is due to $a$RGB encoder’s **stronger guidance to minimize errors in high-frequency local structures**.

Reviewer comment (Theory):
The training process of aRGB auto-encoder does not involve any loss regarding local structure. Is it possible that the encoder also learns other information, e.g., texture, style?

Author response:
- As you may have noticed, our $a$RGB autoencoder embeds simple local neighborhood structures such as edges at each pixel as visualized in Figures 5a, 17, and 18. We have also tried to clarify the sources of this structural information by decomposing the $a$RGB space in Section 4.1.

- To our understanding, your concern is beyond this simple neighboring structures. It depends on how people define textures and styles. Nevertheless, we doubt that our $a$RGB space encodes **semantic** level of textures, for example, a checkerboard or long hairy textures. We have two reasons to believe so.

    1. **Small receptive fields**: Our $a$RGB encoder has a receptive field size of 9x9. This size is sufficient to suppress high-frequency artifacts generated by other auxiliary losses as demonstrated in Table 1, and to enhance the edge structures in deblurring and denoising as suggested in results in Appendix D. However, this is not sufficient for the encoder to perceive semantic textures. Thus, we believe that our $a$RGB encoder encodes local structure by appending local edge information to each pixel within neighborhood of small area corresponding to its receptive field.
    
    2. **Empirical reasons**: Although we have not included in our manuscript, we have conducted a simple experiment on training a single 1x1 convolutional layer to return VGG features from the $a$RGB embedding. If this experiment succeeds, we can conclude that the textual information up to VGG features’ level of abstract is indeed embedded in our $a$RGB space. However, this quickly has failed when we try to mimic the features from after a second downscaling block of the VGGNet.

- To sum up, **the advantage of our $a$RGB space is not based on the semantic embedding**. In other words, the $a$RGB space-based loss functions and the ones derived from deeper networks, e.g., perceptual losses and adversarial losses, are **complements rather than substitutes**. We believe that experimental results in Table 1 of our manuscript supports this claim.

Reviewer comment (Evaluation):
The additional architecture for aRGB representation transmission may introduce more computation consumption during the training phase.

Author response:
- We clearly note that our method only applies to the training phase of a restoration model, and therefore **does not cause any additional computational overhead in the test time**.
    
- For training time, our model does introduce additional computation for the embeddings. **The amount of this overhead is not significant** compared to the image restoration model itself. For example, training RRDB [5*], a widely used super-resolution model, with our $a$RGB loss increases training time about 9%.
    
- Moreover, the model itself is very lightweight, compared to the massive volume of the current state-of-the-art restoration models. The table below compares the computational burden with other loss functions and restoration models.

|  | # params |
| --- | --- |
| $a$RGB encoder* | 5.3M |
| VGG loss | 20.2M |
| Adversarial loss (SRGAN discriminator) | 80.2M |
| ESRGAN | 16.7M |
    
\*Our $a$RGB decoder has only 0.01M parameters. It is a linear layer. Furthermore, the decoder is not used in the training of image restoration models and therefore its computational cost does not affect the overall performance.

Reviewer comment (Evaluation):
The improved performance on image motion deblurring seems to be minimal.

Author response:
- To further justify our approach, we apply our method on another deblurring method [4*], as presented in the table below. Consistent with Table 2 in the main manuscript, our method demonstrates superior performance. The numbers are calculated from the author’s official code repository.

| Loss | GoPro PSNR | GoPro SSIM |
| --- | --- | --- |
| L1 (Original) | 30.40 | 0.9018 |
| L1-$a$RGB | 30.85 **(+0.45 dB)** | 0.9096 **(+0.0078)** |

Reviewer comment (Experiments):
The authors design a nonlinear mixture-of-experts encoder and a linear decoder for aRGB representation. Can this design principle be applied to guide the architecture design of image restoration networks?

Author response:
- Indeed exploiting experts in architecture may help, as suggested in [3*]. However, we would like to emphasize that our aRGB is not an additional branch for restoration models. Rather, it is similar to how perceptual loss does not count as an architectural augmentation. Thus, our aRGB space does not incur additional computational overhead during the test time, as described in our answer below.

Reviewer comment (Experiments):
Is the additional en/decoder equivalent to adding an additional branch for learning? I doubt whether the improved performance is yielded by the additional computation overhead.

Author response:
- Indeed exploiting experts in architecture may help, as suggested in [3*]. However, we would like to emphasize that our aRGB is not an additional branch for restoration models. Rather, it is similar to how perceptual loss does not count as an architectural augmentation. Thus, our aRGB space does not incur additional computational overhead during the test time, as described in our answer below.

Reviewer comment (Experiments):
The widely used dual-domain loss in models, such as MIMOUNet (Cho et al, ICCV'21) and SFNet (Cui et al, ICLR'23), can introduce global information refinement. How does aRGB compare to this loss function? This function does not lead to much computation overhead.

Author response:
- We clearly note that our method only applies to the training phase of a restoration model, and therefore **does not cause any additional computational overhead in the test time**.
    
- For training time, our model does introduce additional computation for the embeddings. **The amount of this overhead is not significant** compared to the image restoration model itself. For example, training RRDB [5*], a widely used super-resolution model, with our $a$RGB loss increases training time about 9%.
    
- Moreover, the model itself is very lightweight, compared to the massive volume of the current state-of-the-art restoration models. The table below compares the computational burden with other loss functions and restoration models.

|  | # params |
| --- | --- |
| $a$RGB encoder* | 5.3M |
| VGG loss | 20.2M |
| Adversarial loss (SRGAN discriminator) | 80.2M |
| ESRGAN | 16.7M |
    
\*Our $a$RGB decoder has only 0.01M parameters. It is a linear layer. Furthermore, the decoder is not used in the training of image restoration models and therefore its computational cost does not affect the overall performance.

Reviewer comment (Evaluation):
Does aRGB lead to extra computation overhead during training and inference?

Author response:
- In contrast to the Fourier space-based loss used in MIMO-UNet [6*] and SFNet[7*], our $a$RGB space-based loss has the following three unique characteristics.
    
    1. Whereas Fourier space loss is used in combination with the RGB space-based counterparts, our $a$RGB loss is designed **as a new alternative for the RGB space**.
    2. The dual representation in the Fourier space loss is obtained by applying the FFT algorithm to images. This is a constant linear operation, and hence the correlations it models only depend on the coordinate differences, and not the contents of the signal. In contrast, our **$a$RGB encoder is** **nonlinear and adaptive to the input signal**. The metrics in the $a$RGB space expands and contracts based on the images’ local structures as demonstrated in Figure 5d of our manuscript.
    3. Fourier space loss applies globally within the patch extent, yet its correlation model is fixed. In contrast, our $a$RGB encoder **behaves based on the local image structure**, yet it can **encode more complex structures adaptively**. Figures 5b, 5c and 20 in our manuscript demonstrates its adaptability on the local image structure.

- We design an experiment to further compare the two types of space modeling. Here, a deblurring model [4*] is trained using two types of losses: (1) the Fourier space loss with L1-RGB loss [6*, 7*] and (2) the $a$RGB loss only. Please understand that the time and resources permitted to us is limited, so we are temporarily reporting the intermediate scores. Still, the table clearly exhibits the advantage of our $a$RGB loss over the Fourier loss.

| DeepDeblur / GoPro (100 epochs) | PSNR (dB) | SSIM |
| --- | --- | --- |
| Fourier space loss + L1-RGB | 27.0007 | 0.8241 |
| L1-$a$RGB | 27.7348 **(+0.73 dB)** | 0.8416 **(+0.0175)** |