Paper: Prompt-Guided Dynamic Network for Image Super Resolution (ICLR-2024)

Reviewer comment (Experiments):
The only concern is unfair comparison. Since most of the baselines are trained on the DIV2K which has fewer images than the training data used in this work.

Author response:
We acknowledge that the baseline models referenced in our paper were originally trained on the DIV2K dataset, which indeed contains fewer images compared to the dataset used in our work. This discrepancy was an oversight on our part and was not intended to skew the comparisons in favor of our method.
Besides, it is worth mentioning that the primary intention behind using these baseline models was to demonstrate the effectiveness of our dynamic correlation module in a general super-resolution context. However, we recognize that for a fair and convincing comparison, the training conditions for all models need to be as similar as possible.

For a fair comparison, we provide results of some baseline methods trained on COCO and DIV2K datasets in Table 2.

**Table 2. Quantitative results of baseline methods trained on DIV2K and COCO datasets.**


|Method          | Set5          | Set14         | Urban100       |
| -----------    | :-----------: | :-----------: | :-----------:  | 
|EDSR(**COCO**)  | 37.25/0.935   |  33.16/0.898  | 31.01/0.902    |
|EDSR(**DIV2K**) | 37.89/0.961   | 33.47/0.917   | 31.84/0.926    |
|RCAN(**COCO**)  | 37.33/0.938   |  33.26/0.899  | 31.19/0.907    |
|RCAN(**DIV2K**) | 37.85/0.960   | 33.45/0.917   | 31.88/0.928    |

It is clear that when trained on the DIV2K dataset, EDSR and RCAN pose better quantitative results  (with slightly higher metrics) than those trained on the COCO dataset.
This mainly can be attributed to the high-resolution and high-quality trait of the DIV2K dataset, which enables SR methods to learn more representative and discriminative restoration capability.
Nevertheless, it will not be a major bottleneck for SR methods to be trained on the COCO dataset.

Reviewer comment (Experiments):
I cannot find the claim that the baseline models are trained on the same dataset from scratch. This may make the paper less convincing.

Author response:
We acknowledge that the baseline models referenced in our paper were originally trained on the DIV2K dataset, which indeed contains fewer images compared to the dataset used in our work. This discrepancy was an oversight on our part and was not intended to skew the comparisons in favor of our method.
Besides, it is worth mentioning that the primary intention behind using these baseline models was to demonstrate the effectiveness of our dynamic correlation module in a general super-resolution context. However, we recognize that for a fair and convincing comparison, the training conditions for all models need to be as similar as possible.

For a fair comparison, we provide results of some baseline methods trained on COCO and DIV2K datasets in Table 2.

**Table 2. Quantitative results of baseline methods trained on DIV2K and COCO datasets.**


|Method          | Set5          | Set14         | Urban100       |
| -----------    | :-----------: | :-----------: | :-----------:  | 
|EDSR(**COCO**)  | 37.25/0.935   |  33.16/0.898  | 31.01/0.902    |
|EDSR(**DIV2K**) | 37.89/0.961   | 33.47/0.917   | 31.84/0.926    |
|RCAN(**COCO**)  | 37.33/0.938   |  33.26/0.899  | 31.19/0.907    |
|RCAN(**DIV2K**) | 37.85/0.960   | 33.45/0.917   | 31.88/0.928    |

It is clear that when trained on the DIV2K dataset, EDSR and RCAN pose better quantitative results  (with slightly higher metrics) than those trained on the COCO dataset.
This mainly can be attributed to the high-resolution and high-quality trait of the DIV2K dataset, which enables SR methods to learn more representative and discriminative restoration capability.
Nevertheless, it will not be a major bottleneck for SR methods to be trained on the COCO dataset.

Reviewer comment (Experiments):
How does it perform if the prompt is fixed as "high detailed image"? For most cases of SR, it is hard to describe the image using text.

Author response:
Performance with a Fixed Prompt: The use of a general prompt like "high detailed image" in our multi-modal super-resolution process still offers significant benefits in most cases. Our dynamic correlation module, designed to leverage multi-modal prompts, is not solely dependent on specific or descriptive text input. Even with a generic prompt, the module can effectively guide the feature extraction process toward enhancing image details and clarity. The phrase "high detailed image" prompts the system to prioritize the refinement of textures and edges, ensuring that the resulting super-resolution image is sharp and detailed. We have conducted experiments with such fixed, general prompts and found that the system still outperforms traditional single-image super-resolution techniques, especially in terms of detail preservation and enhancement.

Challenge of Describing Images Using Text: We acknowledge that for many super-resolution scenarios, it might be challenging to describe the desired output precisely in text. However, the strength of our approach lies in its versatility. The system can work effectively with varying levels of prompt specificity - from detailed descriptions to general quality-oriented prompts. This flexibility ensures that our method is practical and applicable in a wide range of super-resolution tasks, even when detailed textual descriptions are not feasible. Furthermore, our approach encourages opening new pathways in image processing research.

To better demonstrate these points, we provide some failure cases of PDN in the supplementary material.
It can be observed that these examples contain unclear main objects or rather small objects.
This will lead to indiscriminative representations for subsequent multi-modal attention mask generation.
Besides, these images tend to generate image representations with disordered and indivisible features, which will deteriorate the precision of attention masks.
In such a scenario, the proposed PDN mistakes the areas with the highest intensity as the most similar areas instead of the most caption-relevant ones.
The aforementioned phenomena are the major drawbacks of PDN, i.e., irrelevant or unclear prompts cause misleading attention masks.

Reviewer comment (Experiments):
The CNN backbone is a little out of date. I would like to suggest the authors apply the similar idea to the SOTA SISR methods with both Transformers and diffusion models.

Author response:
It is worth pointing out that our initial decision to use a CNN backbone was driven by its proven **effectiveness** in various SISR tasks and its **suitability** for integrating with our newly introduced dynamic correlation module. This choice allowed us to clearly demonstrate the potential of multi-modal prompts in super-resolution processes.
Acknowledging your suggestion, we are enthusiastic about exploring the integration of our dynamic correlation module with more contemporary architectures such as Transformers. These models offer distinct advantages in terms of adaptability and learning complex data representations, which could potentially enhance the effectiveness of our approach.
For a comprehensive comparison, we further incorporate the proposed DCM into SwinIR.
Quantitative results are shown in Table 1.
It can be observed that the proposed DCM still shows superiority in boosting the performance of transformer-based SR methods.

Additionally, we are committed to furthering our research in this domain. Adapting our approach to include recent SOTA methods like Transformers and diffusion models will be a key focus of our ongoing work.


**Table 1. Quantitative results of SwinIR and SwinIR+ at the scale of 2.**


|Method          | Set5          | Set14         | Urban100       |
| -----------    | :-----------: | :-----------: | :-----------:  | 
|SwinIR          | 37.54/0.941   |  33.36/0.901  | 32.16/0.914    |
|SwinIR+         | 37.61/0.943   | 33.42/0.907   | 32.21/0.917    |

Reviewer comment (Novelty):
The high-level idea of using multi-modal prompts to improve super-resolution generalization is not entirely novel, as the TGSR paper proposed this first.

Author response:
The major differences between the proposed PDN and TGSR are as follows. **Dynamic Correlation Module**: Unlike TGSR, our method introduces a dynamic correlation module. This module actively leverages multi-modal prompts (textual and visual cues) to guide the image feature extraction process in a more contextually adaptive manner. This dynamic correlation enables our model to better interpret the nuances of the prompts, leading to more accurate super-resolution results.
**Multi-modal Prompt Integration**: While TGSR primarily focuses on text-guided image super-resolution, our approach extends the concept to incorporate a broader range of multi-modal prompts. This integration allows for a more versatile and comprehensive understanding of the image context, thereby enhancing the super-resolution process beyond the capabilities of TGSR.
The contribution of this work is two-fold. **Enhanced Feature Extraction**: Our dynamic correlation module contributes to a significant improvement in feature extraction. By interpreting multi-modal prompts in tandem, it ensures that the super-resolution process is not just guided by text but is also influenced by other relevant modalities, leading to a more holistic image enhancement.
**Improved Versatility and Adaptability**: The ability to process and integrate various types of prompts makes our method more versatile and adaptable to different scenarios and requirements. This is particularly beneficial in complex super-resolution tasks where the context and content vary greatly.
Thus, the proposed PDN shows clear superiority over TGSR.

Reviewer comment (Novelty):
While the specific techniques in this paper differ from TGSR, the overall framework is quite similar conceptually (leveraging prompts to aid super-resolution). Compared to TGSR, the innovations here seem more incremental—attention modules, dynamic convolutions, etc.—rather than proposing an entirely new overall architecture. The extent to which these specific contributions generalize may be limited.

Author response:
The differences between PDN and TGSR are three-fold.
**First**, TGSR utilizes a bi-directional LSTM network proposed by AttnGAN to extract textual features while PDN utilizes a more powerful and image-correlated textual encoder, i.e., CLIP\'s textual encoder.
**Second**, TGSR contains a well-designed dual-branch SR sub-network to generate a coarse SR result and enrich image details whereas PDN does not additional modification on existing SR networks rather than inserting the proposed DCM into existing SR networks.
**Third**, TGSR utilizes textual guidance by simply applying the text attention module (TAM) which may be sensitive to irrelevant or wrong prompts.
On the contrary, based on DCM, PDN can derive appropriate convolutional kernels from the kernel bank for feature transformation according to the prompts.

Reviewer comment (Novelty):
More analysis could be provided on how the approach differs from and improves upon TGSR technically. The conceptual similarity should be addressed more directly.

Author response:
The major differences between the proposed PDN and TGSR are as follows. **Dynamic Correlation Module**: Unlike TGSR, our method introduces a dynamic correlation module. This module actively leverages multi-modal prompts (textual and visual cues) to guide the image feature extraction process in a more contextually adaptive manner. This dynamic correlation enables our model to better interpret the nuances of the prompts, leading to more accurate super-resolution results.
**Multi-modal Prompt Integration**: While TGSR primarily focuses on text-guided image super-resolution, our approach extends the concept to incorporate a broader range of multi-modal prompts. This integration allows for a more versatile and comprehensive understanding of the image context, thereby enhancing the super-resolution process beyond the capabilities of TGSR.
The contribution of this work is two-fold. **Enhanced Feature Extraction**: Our dynamic correlation module contributes to a significant improvement in feature extraction. By interpreting multi-modal prompts in tandem, it ensures that the super-resolution process is not just guided by text but is also influenced by other relevant modalities, leading to a more holistic image enhancement.
**Improved Versatility and Adaptability**: The ability to process and integrate various types of prompts makes our method more versatile and adaptable to different scenarios and requirements. This is particularly beneficial in complex super-resolution tasks where the context and content vary greatly.
Thus, the proposed PDN shows clear superiority over TGSR.

Reviewer comment (Experiments):
Like most learning-based SR methods, the reliance on synthetic training data means real-world robustness is unclear. Evaluation of realistic degraded images could better validate effectiveness.

Author response:
**Dataset Generation**: We follow the common practice of organizing the training data [1].
For the COCO dataset, first, we generate low-resolution images via bicubic degradation.
Besides, we filter out images that have the largest edge smaller than 512.
Then we crop the raw images to $240\times240$ sub-images.
Additionally, dataset preparation scripts will further randomly crop the sub-images to $96\times96$ patches for training.

**Robustness to real-world degradations**:
We would like to clarify that our proposed method, as it currently stands, is indeed primarily tailored for and tested on synthetic datasets. This focus is a deliberate choice, stemming from our research objective, which is to explore and advance the theoretical and technical aspects of integrating multi-modal prompts into single-image super-resolution processes.
Here are key points addressing your concern:
Specific Research Scope: Our method is designed to explore the potential of multi-modal prompts in enhancing image super-resolution under well-defined and controlled conditions. The use of synthetic datasets allows us to precisely gauge the impact of these prompts and refine the underlying algorithms without the confounding variables present in real-world degradations.
Contribution to Theoretical Foundations: By focusing on synthetic data, our research contributes to the theoretical understanding and development of prompt-based super-resolution techniques. It lays the groundwork for future studies that may extend these techniques to more complex, real-world scenarios.
Future Research Directions: We acknowledge the importance of real-world applicability and see it as an essential direction for future research. Building upon the foundations laid by this study, subsequent research can adapt and extend our methodology to tackle real-world image degradations. This progression would be a natural next step, bringing the benefits of our approach to more practical and diverse applications.

[1]: Wang, Xintao, Liangbin Xie, Chao Dong, and Ying Shan. "Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1905-1914. 2021.
[2]: https://github.com/XPixelGroup/BasicSR/blob/master/docs/DatasetPreparation.md