Paper: PostRainBench: A Comprehensive Benchmark and A New Model for Precipitation Forecasting (ICLR-2024)

Reviewer comment (Novelty):
The authors claim in the paper that they are the first to surpass NWP using a deep learning approach. However, to my knowledge, several studies have already ventured into this area, as evidenced by references [1-3]. I suggest the authors review the accuracy of this statement and make appropriate amendments.

Author response:
We respectfully point out the key misunderstanding. To clarify, our assertion is that our model is the first to surpass NWP in the specific context of precipitation forecasting, particularly in extreme cases, within the realm of NWP post-processing tasks. This claim is based on three key criteria:

1. Focus on Precipitation Forecasting: Our model specifically addresses precipitation forecasting, which distinguishes it from other studies. While recent models like Pangu [2] and FengWu [3] forecast various upper-air and surface weather variables, they do not include precipitation in their scope. Our work fills this gap by providing insights specifically for precipitation forecasting, contributing significant advancements in this area.
2. NWP Post-processing Task: Our approach uniquely combines physics-based NWP results with AI methods in a post-processing task, a methodology that diverges from the purely data-driven deep learning methods employed in studies like Pangu and FengWu. This hybrid approach is in line with the future direction suggested in the Pangu study, which anticipates the combination of AI-based and NWP methods for improved performance.

3. Excellence in Extreme Cases: The specific area where our model excels is in forecasting heavy rain, which we define as 'extreme cases' in our paper. This is a crucial distinction, as none of the deep learning models in the related work, including KoMet, have surpassed NWP in this particular aspect of heavy rain forecasting.

Regarding NowcastNet [1], while it represents a significant advancement in using radar observations for weather prediction, its methodological framework differs from ours. NowcastNet relies on a data-driven approach without integrating physics-informed predictions (NWP Method), whereas our model is designed specifically for NWP post-processing, leveraging both AI and physics-based insights.

In summary, our claim of being the first to surpass NWP in the specific context of heavy rain forecasting in NWP post-processing tasks is based on these distinct aspects of our work. We believe this clarification underscores the unique contribution of our study in advancing the field of precipitation forecasting.

Reviewer comment (Experiments):
The selected baselines in the paper don't appear comprehensive. For instance, recent significant works in the relevant field, such as OpenSTL[4] and Fourcastnet[5], are not considered. I would recommend the authors to include these for comparison.

Author response:
We appreciate the reviewer's suggestion regarding the inclusion of comprehensive baselines in our paper. While OpenSTL [4] represents a comprehensive benchmark for spatiotemporal forecasting, our work focuses on NWP post-processing tasks, which differ from the typical spatiotemporal prediction task of inputting N frames to output N frames.

In our benchmark, we have defined two specific tasks:

1. Sequence to Image Prediction (Korea Dataset): Here, we input N frames to predict 1 frame under fixed lead times varying from 6 to 87 hours.

2. Image-to-Image Translation Prediction (Germany and China Datasets): This involves inputting 1 frame to predict the future frame under a fixed 3-hour lead time. Notably, these experiments, especially on the latter two datasets, do not involve temporal information.
As such, our NWP post-processing tasks are distinct from the typical input-output frame prediction models used in OpenSTL [4]. This difference in task nature and requirements is why we have not included a comparison with the spatiotemporal forecasting methods in OpenSTL in our study.

In selecting our baselines, we choose models like ConvLSTM, which is recurrent-based, and U-Net, which is recurrent-free, along with the MetNet network, specifically designed for precipitation forecasting. Our proposed SwinUnet model is also a recurrent-free method.
Acknowledging the reviewer's valuable input, we have included additional experiments with the latest FourCastNet as a baseline. We maintained consistent settings with baseline models for the respective dataset and averaged results over three random seeds. 

We have supplemented this result in Section A.5 in the appendix.

Korea Dataset

|             | CSI_Rain(Mean/Std) | CSI_Rain(Best) | HSS_Rain(Mean/Std) | HSS_Rain(Best) | CSI_Heavy(Mean/Std) | CSI_Heavy(Best) | HSS_Heavy(Mean/Std) | HSS_Heavy(Best) |
| :---------- | :----------------- | :------------- | :----------------- | :------------- | :------------------ | :-------------- | :------------------ | :-------------- |
| FourCastNet | 0.314(0.016)       | 0.325          | **0.391(0.023)**   | **0.409**      | 0.011(0.008)        | 0.017           | 0.020(0.014)        | 0.029           |
| Ours        | **0.321(0.005)**   | **0.326**      | 0.384(0.007)       | 0.389          | **0.052(0.010)**    | **0.058**       | **0.089(0.017)**    | **0.097**       |

Germany Dataset

|             | CSI_Rain(Mean/Std) | CSI_Rain(Best) | HSS_Rain(Mean/Std) | HSS_Rain(Best) | CSI_Heavy(Mean/Std) | CSI_Heavy(Best) | HSS_Heavy(Mean/Std) | HSS_Heavy(Best) |
| :---------- | :----------------- | :------------- | :----------------- | :------------- | :------------------ | :-------------- | :------------------ | :-------------- |
| FourCastNet | 0.494(0.009)       | 0.504          | 0.595(0.009)       | 0.601          | 0.157(0.034)        | 0.185           | 0.265(0.051)        | 0.306           |
| Ours        | **0.514(0.003)**   | **0.518**      | **0.609(0.006)**   | **0.616**      | **0.209(0.014)**    | **0.224**       | **0.339(0.020)**    | **0.359**       |

China Dataset

|             | CSI_Rain(Mean/Std) | CSI_Rain(Best) | HSS_Rain(Mean/Std) | HSS_Rain(Best) | CSI_Heavy(Mean/Std) | CSI_Heavy(Best) | HSS_Heavy(Mean/Std) | HSS_Heavy(Best) |
| :---------- | :----------------- | :------------- | :----------------- | :------------- | :------------------ | :-------------- | :------------------ | :-------------- |
| FourCastNet | 0.163(0.006)       | 0.167          | 0.219(0.010)       | 0.230          | 0.114(0.013)        | 0.129           | 0.166(0.023)        | 0.192           |
| Ours        | **0.208(0.007)**   | **0.216**      | **0.274(0.014)**   | **0.289**      | **0.145(0.015)**    | **0.163**       | **0.225(0.019)**    | **0.246**       |

For the Korea dataset, our model exhibits superior performance to FourCastNet in both rain CSI and heavy rain CSI metrics, with a marginal shortfall in rain HSS, where it trails by 1.8% behind FourCastNet. It is important to highlight that FourCastNet's predictive capability does not surpass that of NWP algorithms for heavy rain scenarios.

Regarding the Germany dataset, our model demonstrates an advancement over FourCastNet in all metrics for both rain and heavy rain, whereas FourCastNet does not demonstrate an advantage over NWP algorithms in heavy rain predictions.

For the China dataset, our model demonstrates comprehensive outperformance across all metrics when compared to FourCastNet. While FourCastNet posts a modest 3.6% gain over NWP methods in heavy rain forecasting, our approach achieves a substantial 31.8% improvement, marking a significant enhancement in predictive accuracy

Reviewer comment (Novelty):
The proposed algorithm is not particularly surprising.

Author response:
Please also refer to A2.1 for Reviewer Lop7.

We would like to emphasize our significant contribution to an under-explored task in NWP post-processing and highlight the novel contributions of our work, particularly in the context of multi-task learning and the Channel Attention Module (CAM).

Our algorithm stands out by providing a straightforward yet efficient solution to this specific challenge. It navigates the intricacies of imbalanced data scenarios, offering novel insights into optimizing models under such conditions. This approach is particularly relevant given the nature of precipitation data, which inherently exhibits significant class imbalances.

In summary, the value of our algorithm lies in its application-oriented focus, offering a practical and effective solution to a well-known challenge in precipitation forecasting. This contribution provides valuable insights for the field and can serve as a foundation for future advancements in this area.

Reviewer comment (Reproducibility):
Is there a part of the datasets newly provided by the authors?

Author response:
In our study, we introduce the China Dataset, which is a novel addition provided by us. This dataset offers unique insights specific to the Chinese region's precipitation patterns, contributing to the diversity and comprehensiveness of precipitation forecasting research.

While the Korea and Germany datasets used in our study have been utilized in previous research, we bring a new perspective to these existing datasets. Our comprehensive benchmarking approach includes a detailed comparison of different settings in temporal information and variable selection. This analysis not only enhances the understanding of these datasets but also extends their applicability.

The significance of our contribution lies in the novel application and analysis of these datasets, particularly in how we compare and contrast different modeling approaches. This benchmarking approach is valuable for wider downstream tasks, offering insights that can be adapted and applied in various contexts beyond the specific datasets we used.

In summary, our study not only introduces a new dataset with the China Dataset but also enriches the research landscape by providing a deeper and more nuanced analysis of existing datasets, broadening their potential applications in the field of precipitation forecasting.

We have provided the download link of all datasets we used in the supplementary material.

Reviewer comment (Novelty):
The model lacks substantial innovation; both the Channel Attention Model (CAM) and the multi-task learning loss function appear to be straightforward engineering optimizations rather than novel contributions.

Author response:
Firstly, We would like to highlight the novel contributions of our work in task formulation.

 In addressing the reviewer's comment on the innovation aspect of our model, we would like to emphasize our significant contribution to an under-explored task in NWP post-processing. Our comprehensive benchmark introduces a novel approach to this task, encompassing two distinct settings: image-to-image translation with fixed lead time, and sequence-to-image under the same fixed lead time conditions.

The innovative essence of our work lies in its problem-solving orientation. Based on our prior observations, despite deep learning models surpassing NWP in general rainfall prediction, a gap remained in their ability to outperform NWP models in forecasting heavy precipitation. Our proposed solution specifically addresses this challenge. We have supplemented our research with experimental results from the state-of-the-art network, FourCastNet, which also could not achieve better forecasting performance than NWP in heavy rain. This inability of existing advanced models to surpass NWP in heavy rain forecasting highlights the novelty and significance of our contribution.

Our work provides key insights into resolving this critical issue in precipitation forecasting, offering a pragmatic and effective approach to enhance predictive performance in extreme weather conditions. This not only demonstrates our model's innovative application but also its potential impact and value in real-world meteorological scenarios.

Secondly, we would like to highlight the novel contributions of our work, particularly in the context of multi-task learning and the Channel Attention Module (CAM):

1. Innovative Application of Multi-task Learning in Precipitation Forecasting: While multi-task learning is indeed a known optimization method, our research takes a unique approach by being the first to utilize regression to aid classification in data-imbalanced precipitation forecasting. Drawing inspiration from recent research [1] which suggests adding a classification loss in imbalanced data scenarios for regression tasks, our study extends this concept in a novel direction. We propose that for classification tasks with imbalanced data, integrating a regression loss can be beneficial. This approach deviates from traditional radar-based nowcasting tasks that predominantly rely on weighted loss for regression. Our contribution lies in introducing a new perspective to the optimization problem, potentially benefiting a broader spectrum of research within this domain.

2. Pioneering Use of CAM in Weather Variable Relation Modeling: Regarding the Channel Attention Model (CAM), our work stands out as the first to apply this mechanism to the complex task of modeling relationships between weather variables. Unlike general computer vision tasks, which often assume channel independence, our research bridges the gap between computer vision and weather forecasting by providing an innovative application of CAM. This integration is a significant step forward in enhancing the interpretability and efficacy of weather prediction models.

Reference:

[1] Pintea, S. L., Lin, Y., Dijkstra, J., & van Gemert, J. C. (2023). A step towards understanding why classification helps regression. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 19972-19981).

Reviewer comment (Evaluation):
The experimental section could benefit from an in-depth discussion analyzing how the model's performance varies with different lead times.

Author response:
We have supplemented this in Section A.6 with additional results for lead times from 6 to 87 hours on the test set to provide a more detailed view of our model's performance over varying lead times.

For the Korea dataset, we follow the setup of previous studies, selecting samples with various lead times ranging from 6 to 87 hours for our training, validation, and testing sets. The experimental results reported in our paper represent the average performance across different lead times within this 6-87 hour range. 

As shown in Figure 4 in the appendix of the revised PDF, within the lead time interval of 6 to 20, we observe that the CSI for rain reaches a peak at a lead time of 10 before exhibiting a declining trend, whereas the CSI for heavy rain peaks at a lead time of 9, subsequently showing a fluctuating trajectory. Expanding the analysis to a lead time range of 6 to 87, both rain and heavy rain CSI exhibit parallel trends, with heavy rain demonstrating superior performance over extended lead times, likely reflective of inherent data characteristics. Across all evaluated lead times from 6 to 87, our model's mean performance is enhanced, underscoring the comprehensive superiority of our modeling approach.

For the Germany and China datasets, the lead time is set at a fixed 3 hours, corresponding to the paired format in which NWP predictions and precipitation observations are provided.

In summary, while our approach adapts to the fixed 3-hour lead time in the Germany and China datasets, our comprehensive analysis and additional experiments on the Korea dataset underscore the versatility and robustness of our method across a range of lead times, affirming its applicability in varied precipitation forecasting contexts.

Reviewer comment (Evaluation):
Could the authors elaborate on the weaknesses in the model, particularly in relation to handling imbalanced data distributions?

Author response:
Discussion of Weakness:

One limitation of our model is the empirical selection of weights for the hyperparameters. Future research could explore an adaptive loss mechanism within a multi-task learning framework. 

●Empirical Selection of Weights: A key limitation lies in our approach to the empirical selection of weights for hyperparameters. This process is closely tied to the dataset's distribution, which can vary significantly in real-world scenarios. Due to this reliance, the model's performance might be overly dependent on the specific characteristics of the dataset used for training, potentially reducing its effectiveness when applied to datasets with different distributions.

●Potential for Adaptive Loss Mechanisms: Recognizing this limitation, we suggest that future research could explore the development of adaptive loss mechanisms within a multi-task learning framework. An adaptive approach would allow for more dynamic adjustment of weights based on the data distribution, potentially improving the model's handling of imbalanced datasets. This could lead to more nuanced and effective learning, especially in scenarios where data imbalances are pronounced and complex.

In summary, while our current model presents a solid foundation, its approach to weight selection and handling of imbalanced data could benefit from further refinement. The exploration of adaptive loss mechanisms, particularly in conjunction with multi-task learning strategies, represents a promising avenue for enhancing the model's robustness and versatility in diverse data scenarios.

Reviewer comment (Experiments):
The lead time in this paper is fixed as 3h. In practice, the length of lead time has significant influence on precipitation results. It would be better if the authors could exploit the lead time in their method, to show the robustness of their method.

Author response:
We have supplemented this in Section A.6 with additional results for lead times from 6 to 87 hours on the test set to provide a more detailed view of our model's performance over varying lead times.

For the Korea dataset, we follow the setup of previous studies, selecting samples with various lead times ranging from 6 to 87 hours for our training, validation, and testing sets. The experimental results reported in our paper represent the average performance across different lead times within this 6-87 hour range. 

As shown in Figure 4 in the appendix of the revised PDF, within the lead time interval of 6 to 20, we observe that the CSI for rain reaches a peak at a lead time of 10 before exhibiting a declining trend, whereas the CSI for heavy rain peaks at a lead time of 9, subsequently showing a fluctuating trajectory. Expanding the analysis to a lead time range of 6 to 87, both rain and heavy rain CSI exhibit parallel trends, with heavy rain demonstrating superior performance over extended lead times, likely reflective of inherent data characteristics. Across all evaluated lead times from 6 to 87, our model's mean performance is enhanced, underscoring the comprehensive superiority of our modeling approach.

For the Germany and China datasets, the lead time is set at a fixed 3 hours, corresponding to the paired format in which NWP predictions and precipitation observations are provided.

In summary, while our approach adapts to the fixed 3-hour lead time in the Germany and China datasets, our comprehensive analysis and additional experiments on the Korea dataset underscore the versatility and robustness of our method across a range of lead times, affirming its applicability in varied precipitation forecasting contexts.