RainNet: A Large-Scale Imagery Dataset and Benchmark for Spatial Precipitation Downscaling
Abstract
AI-for-science approaches have been applied to solve scientiﬁc problems (e.g., nuclear fusion, ecology, genomics, meteorology) and have achieved highly promising results.
Spatial precipitation downscaling is one of the most important meteorological problem and urgently requires the participation of AI.
However, the lack of a well-organized and annotated large-scale dataset hinders the training and veriﬁcation of more effective and advancing deep-learning models for precipitation downscaling.
To alleviate these obstacles, we present the ﬁrst large-scale spatial precipitation downscaling dataset named RainNet , which contains more than 62 , 400 pairs of high-quality low/high-resolution precipitation maps for over 17 years, ready to help the evolution of deep learning models in precipitation downscaling.
Speciﬁcally, the precipitation maps carefully collected in RainNet cover various meteorological phenomena (e.g., hurricane, squall), which is of great help to improve the model generalization ability.
In addition, the map pairs in RainNet are organized in the form of image sequences ( 720 maps per month ormap/hour), showing complex physical properties, e.g., temporal misalignment, temporal sparse, and ﬂuid properties.
Furthermore, two deep-learning-oriented metrics are speciﬁcally introduced to evaluate or verify the comprehensive performance of the trained model (e.g., prediction maps reconstruction accuracy).
To illustrate the applications of RainNet, 14 state-of-the-art models, including deep models and traditional approaches, are evaluated.
To fully explore potential downscaling solutions, we propose an implicit physical estimation benchmark framework to learn the above characteristics.
Extensive experiments demonstrate the value of
RainNet in training and evaluating downscaling models.
Our dataset is available at https://neuralchen.github.io/RainNet/ .
Introduction
Deep learning has made enormous breakthroughs in the natural sciences (e.g., nuclear fusion [11], ecology [52], genomics [28], meteorology [3]), which is extremely good at extracting valuable knowledge from numerous amounts of data.
In recent years, with computer science development, a deluge of earth system data is continuously being obtained, coming from sensors all over the earth and even in space.
These ever-increasing massive amounts of data with different sources and structures challenge the geoscience community, which lacks practical approaches to understand and further utilize the raw data [44].
Speciﬁcally, several preliminary works [16, 59, 21, 43, 1, 55] try to introduce deep-learning frameworks to solve meteorological problems, e [Figure] g., spatial precipitation downscaling. [Figure]
In this paper, we focus on the spatial precipitation downscaling task [60].
Spatial precipitation downscaling is a procedure to infer highresolution meteorological information from lowresolution variables, which is one of the most important upstream components for meteorological task [3].
The precision of weather and climate prediction is highly dependent on the resolution and reliability of the initial environmental input variables, and spatial precipitation downscaling is the most promising solution.
The improvement of the weather/climate forecast and geo-data quality saves tremendous money and lives; with the ﬁscal year 2021 budget of over $1 billion, NSF funds thousands of colleges in the U.S. to research on these topics [38].
Unfortunately, there are looming issues that hinder the research of spatial precipitation downscaling in the deep learning community: 1) Lack of "deep-learning ready" datasets.
The existing deep-learning based downscaling methods are only applied to ideal retrospective problems and veriﬁed on simulated datasets (e.g., mapping bicubic of precipitation generated by weather forecast model to original data [4]), which signiﬁcantly weakens the credibility of the feasibility, practicability, and effectiveness of the methods.
It is worth mentioning that the data obtained by the simulated degradation methods (e.g., bicubic) is completely different from the real data usually collected by two measurement systems (e.g., satellite and radar) with different precision.
The lack of a well-organized and annotated large-scale dataset hinders the training and veriﬁcation of more effective and complex deep-learning models for precipitation downscaling.
2) Lack of tailored metrics to evaluate deep-learning based frameworks.
Unlike deep learning (DL) communities, scientists in meteorology usually employ maps/charts to assess downscaling models case by case based on domain knowledge [21, 56], which hinders the application of RainNet in DL communities.
For example, He et.
al.
[21] uses log-semivariance (spatial metrics for local precipitation) and quantile-quantile maps to analyze the maps.
The efﬁcient deep-learning framework for downscaling should be established.
Contrary to image data, the proposed real precipitation dataset covers various types of real meteorological phenomena (e.g., hurricane, squall), and shows the physical characters (e.g., temporal misalignment , temporal sparse and ﬂuid properties ) that challenge the downscaling algorithms.
Traditional computationally dense physics-driven downscaling methods are powerless to handle the increasing meteorological data size and are ﬂexible to multiple data sources.
To alleviate these obstacles, we propose the ﬁrst large-scale spatial precipitation downscaling dataset named RainNet , which contains more than 62 , 400 pairs of high-quality low/high-resolution precipitation maps for over 17 years, ready to help the evolution of deep models in spatial precipitation downscaling.
The proposed dataset covers more than 9 million square kilometers of land area, which contains both wet and dry seasons and diverse meteorological phenomena.
To facilitate DL/ML and other researchers to use RainNet, we introduce 6 most concerning indices to evaluate downscaling models: mesoscale peak precipitation error (MPPE), heavy rain region error (HRRE), cumulative precipitation mean square error (CPMSE), cluster mean distance (CMD), heavy rain transition speed (HRTS) and average miss moving degree (AMMD).
In order to further simplify the application of indices, we abstract them into two weighted and summed metrics: Precipitation Error Measure (PEM) and Precipitation Dynamics Error Measure (PDEM).
Unlike video super-resolution, the motion of [Figure] the precipitation region is non-rigid (i.e., ﬂuid), while video super-resolution mainly concerns rigid body motion estimation.
To fully explore how to alleviate the mentioned predicament, we propose an implicit dynamics estimation driven downscaling benchmark model.
Our model hierarchically aligns adjacent precipitation maps, that is, implicit motion estimation, which is very simple but exhibits highly competitive performance.
Based on meteorological science, we also proved that the dataset we constructed contained the full information people may need to recover the higher resolution observations from lower resolution ones.
We believe RainNet provides an imagenet-like track for natural science research in the deep-learning community.
The main contributions of this paper are:
• To the best of our knowledge, we present the ﬁrst REAL (non-simulated) large-scale spatial precipitation downscaling dataset for deep learning; • We introduce 2 easy-to-use metrics to evaluate the performance of downscaling models; • We propose a benchmark downscaling model with strong competitiveness.
We evaluate 14 competitive potential solutions on the proposed dataset and analyze the feasibility and effectiveness of these solutions.
Background
At the beginning of the 19 th century, geoscientists recognized that predicting the state of the atmosphere could be treated as an initial value problem of mathematical physics, wherein future weather is determined by integrating the governing partial differential equations, starting from the observed current weather.
Today, this paradigm translates into solving a system of nonlinear differential equations at about half a billion points per time step and accounting for dynamic, thermodynamic, radiative, and chemical processes working on scales from hundreds of meters to thousands of kilometers and from seconds to weeks [3].
The Navier–Stokes and mass continuity equations (including the effect of the Earth’s rotation), together with the ﬁrst law of thermodynamics and the ideal gas law, represent the full set of prognostic equations in the atmosphere, describing the change in space and time of wind, pressure, density and temperature are described (formulas given in supplementary) [3].
These equations have to be solved numerically using spatial and temporal discretization because of the mathematical intractability of obtaining analytical solutions, and this approximation creates a distinction between so-called resolved and unresolved scales of motion.
2.1 Spatial Downscaling of Precipitation
The global weather forecast model is treated as a computational problem, and relying on high-quality initial data input.
The error of weather forecast would increase exponentially over time from this initial error of the input dataset.
Downscaling is one of the most important approaches to improving the initial input quality.
Precipitation is one of the essential atmospheric variables that are related to daily life.
It could easily be observed, by all means, e.g., gauge station, radar, and satellites.
Applying downscaling methods to precipitation and creating high-resolution rainfall is far more meaningful than deriving other variables, while it is the most proper initial task to test deep learning’s power in geo-science.
The traditional downscaling methods can be separated into dynamic and statistical downscaling.
Dynamic downscaling treats the downscaling as an optimization problem constraint on the physical laws.
The dynamic downscaling methods ﬁnd the most likely precipitation over space and time under the pre-deﬁned physical law.
It usually takes over 6 hours to downscale a 6-hour precipitation scenario globally on supercomputers [10].
As the dynamic downscaling relies on pre-deﬁned known macroscopic physics, a more ﬂexible weather downscaling framework that could easily blend different sources of observations and show the ability to describe more complex physical phenomena on different scales is desperately in need.
Statistical downscaling is trying to speed up the dynamic downscaling process.
The input of statistical downscaling is usually dynamic model results or two different observation datasets on different scales.
However, due to the quality of statistical downscaling results, people rarely apply statistical downscaling to weather forecasts.
These methods are currently applied in the tasks not requiring high data quality but more qualitative understanding, e.g., climate projection, which forecasts the weather for hundreds of years on coarse grids and uses statistical downscaling to get detailed knowledge of medium-scale future climate system.
RainNet: Spatial Precipitation Downscaling Imagery Dataset 3.1 Data Collection and Processing
To build up a standard realistic (non-simulated) downscaling dataset for computer vision, we selected the eastern coast of the United States, which covers a large region (7 million km 2 ; 105 ◦ ∼ 65 ◦ W , 25 ◦ ∼ 50 ◦ N , GNU Free Documentation License 1.2) and has a 20-year high-quality precipitation observations.
It is worth mentioning that this region is selected for two reasons: on the one hand, this region has both LR and HR instrument observations, and on the other hand, it covers the various meteorological phenomena.
We collected two precipitation data sources from National Stage IV QPE
Product (StageIV [37]; high resolution at 0 .
04 ◦ (approximately 4 km ), GNU Free Documentation
License 1.2) and North American Land Data Assimilation System (NLDAS [62]; low resolution at 0 .
125 ◦ (approximately 13 km )).
StageIV is mosaicked into a national product at National Centers for
Environmental Prediction (NCEP), from the regional hourly/6-hourly multi-sensor (radar+gauges) precipitation analyses (MPEs) produced by the 12 River Forecast Centers over the continental United
States with some manual quality control done at the River Forecast Centers (RFCs).
NLDAS is constructed quality-controlled, spatially-and-temporally consistent datasets from the gauges and remote sensors to support modeling activities.
Both products are hourly updated and both available from 2002 to the current age.
In our dataset, we further selected the eastern coast region for rain season ( July ∼ November , covering hurricane season; hurricanes pour over 10% annual rainfall in less than 10 days).
We matched the coordinate system to the lat-lon system for both products and further labeled all the hurricane periods happening in the last 17 years.
These heavy rain events are the largest challenge for weather forecasting and downscaling products.
As heavy rain could stimulate a wide-spreading ﬂood, which threatens local lives and arouses public evacuation.
If people underestimate the rainfall, a potential ﬂood would be underrated; while over-estimating the rainfall would lead to unnecessary evacuation orders and ﬂood protection, which is also costly.
3.2 Dataset Statistics
At the time of this work, we collected and processed precipitation data for the rainy season for 17 years from 2002 to 2018.
One precipitation map pair per hour, 24 precipitation map pairs per day.
In detail, we have collected 85 months or 62424 hours, totaling 62424 pairs of highresolution and low-resolution precipitation maps.
The size of the high-resolution precipitation map is 624 × 999 , and the size of the low-resolution is 208 × 333 .
Various meteorological phenomena and precipitation conditions (e.g., hurricanes, squall lines.) are covered in these data.
The precipitation map pairs in RainNet are stored in HDF5 ﬁles that make up 360 GB of disk space.
We select 2 typical meteorological phenomena and visualize them in Fig.
Our data is collected from satellites, radars, gauge stations, etc., which covers the inherent working characteristics of different meteorological measurement systems.
Compared with traditional methods that generate data with different resolutions through physical model simulation, our dataset is of great help for deep models to learn real meteorological laws.
The proposed dataset can be used under the Creative Common
License (Attribution CC BY).
3.3 Dataset Analysis
In order to help design a more appropriate and effective precipitation downscaling model, we have explored the property of the dataset in depth.
As mentioned above, our dataset is collected from multiple sensor sources (e.g., satellite, weather radar), which makes the data show a certain extent of misalignment .
Our efforts here are not able to vanquish the misalignment.
This is an intrinsic problem brought about by the fusion of multi-sensor meteorological data.
Limited by observation methods (e.g., satellites can only collect data when they ﬂy over the observation area), meteorological data is usually temporal sparse , e.g., in our dataset, the sampling interval between two precipitation maps is one hour.
The temporal sparse leads to serious difﬁculties in the utilization of precipitation sequences.
Additionally, the movement of the precipitation position is directly related to the cloud.
It is a ﬂuid movement process that is completely different from the rigid body movement concerned in image/video super-resolution.
At the same time, the cloud will grow or dissipate in the process of ﬂowing and even form new clouds, which further complicates the process.
In the nutshell, although existed SR is a potential solution for downscaling, there is a big difference between the two.
Especially, the three characteristics of downscaling mentioned above: temporal misalignment , temporal sparse , ﬂuid properties , which make the dynamic estimation of precipitation more challenging.
Evaluation Metrics
Due to the difference between downscaling and traditional image/video super-resolution, the metrics that work well under SR tasks may not be sufﬁcient for precipitation downscaling.
By gathering the metrics from the meteorologic literature (the literature includes are [66, 35, 14, 21, 40, 61], we select 6 most common metrics (a metric may have multiple names in different literature) to reﬂect the downscaling quality: mesoscale peak precipitation error (MPPE), cumulative precipitation mean square error (CPMSE), heavy rain region error (HRRE) , cluster mean distance (CMD), heavy rain transition speed (HRTS) and average miss moving degree (AMMD).
These 6 metrics can be separated as reconstruction metrics: MPPE, HRRE, CPMSE, AMMD, and dynamic metrics: HRTS and CMD.
The MPPE ( mm/hour ) is calculated as the difference of top quantile between the generated/real rainfall dataset which considers both spatial and temporal properties of mesoscale meteorological systems, e.g., hurricane, squall.
This metric is used in most of these papers (for example [66, 35, 14, 21, 40, 61] suggest the quantile analysis to evaluate the downscaling quality).
The CPMSE ( mm 2 /hour 2 ) measures the cumulative rainfall difference on each pixel over the time-axis of the test set, which shows the spatial reconstruction property. Similar metrics are used RRDB C onv
U p S a m p li ng C onv RRDB RRDB …… [Figure]
Downscaling
[Figure]
Results
L1+Percetual Loss
Ground Truth
Losses
A
A 𝑁−
A 𝑁− መ𝐈 𝑇𝐻 𝐈 𝑇 𝐻 ……
𝐈 𝑇 − 𝑁 − 12 + 1 𝐿 𝐈 𝑇 + 𝑁 − 12 + 1 𝐿 𝐈 𝑇 + 𝑁 − 12 𝐿 in [66, 35, 61] calculated as the pixel level difference of monthly rainfall and used in [21] as a pixel level difference of cumulative rainfall with different lengths of the record.
The HRRE ( km 2 ) measures the difference in heavy rain coverage on each time slide between generated and labeled test set, which shows the temporal reconstruction ability of the models.
The
AMMD ( radian ) measures the average angle difference between main rainfall clusters.
Similar metrics are used in [66, 35, 61] as rainfall coverage of a indeﬁnite number precipitation level and used in [21, 40] as a continuous spatial analysis.
As a single variable dataset, it is hard to evaluate the ability of different models to capture the precipitation dynamics when temporal information is not included (a multi-variable dataset may have wind speed, a typical variable representing dynamics, included).
So here we introduce the ﬁrst-order temporal and spatial variables to evaluate the dynamical property of downscaling results.
Similar approaches are suggested in [35, 14, 40].
The CMD ( km ) physically compares the location difference of the main rainfall systems between the generated and labeled test set, which could be also understood as the RMSE of the ﬁrst-order derivative of precipitation data in spatial directions.
The
HRTS ( km/hour ) measures the difference between the main rainfall system moving speed between the generated and labeled test set which shows the ability of models to capture the dynamic property, which could be also understood as the RMSE of the ﬁrst-order derivative of precipitation data on the temporal direction.
Similar metrics are suggested in [35, 14, 40] as the auto-regression analysis and the differential analysis.
More details about the metrics and their equations are given in supplementary materials.
One metrics group (MPPE, HRRE, CPMSE, AMMD) mainly measures the rainfall deviation between the generated precipitation maps and GT.
The other group (HRTS and CMD) mainly measures the dynamic deviation of generated precipitation maps.
So many measures will create difﬁculties in application and understanding.
In order to further simplify the application of indices, we ﬁnally abstract them into two weighted and summed metrics: Precipitation Error Measure (PEM) and Precipitation Dynamics Error
Measure (PDEM).
We ﬁrst align the dimensions of these two groups of metrics respectively.
The ﬁrst group of metrics (MPPE, HRRE, CPMSE, AMMD) is normalized, weighted , and summed to get the precipitation error measure (PEM).
According to [17], all the metrics are transferred to Percent Bias (PBIAS) to be suitable for metrics weighting.
The original deﬁnition of PBIAS is the bias divided by observation, as PBIAS = | Q model − Q obs | / | Q obs | .
Here we rewrite the original metrics to PBIAS by dividing the metrics with annual mean observations of the original variables (AMO), as PBIAS PEMi = | Metrics PEMi | / | AMO PEMi | , Metrics PEMi = { MPPE, HRRE, CPMSE, AMMD } .
In our dataset, AMO PEMMPPE = 64 , AMO PEMHRRE = 533 , AMO PEMAMMD = 0 .
64 , AMO PEMCPMSE = 332 , AMO PEMHRTS = 15 , AMO PEMCMD = 26 .
The metrics then are ensembled to a single metric (PEM) with equal weight, as PEM =
(cid:80) i 0 .
25 · PBIAS PEMi .
Following the same procedure, we then ensemble the second group of dynamic metrics (HRTS and CMD) to a single metrics PDEM =
(cid:80) i 0 .
5 · PBIAS PDEMi .
We also include the most commonly used metric RMSE as one single metric in our metrics list.
RMSE could evaluate both the reconstruction and dynamic property of the downscaling results.
Benchmark of RainNet
As a potential solution, Super-Resolution (SR) frameworks are generally divided into the SingleImage Super-Resolution (SISR) and the Video Super-Resolution (VSR).
Video Super-Resolution is able to leverage multi-frame information to restore images, which better matches the nature of downscaling.
We will demonstrate this judgment in Sec. 6.1.
The VSR pipeline usually contains three components: deblurring, inter-frame alignment, and super-resolution.
Deblurring and inter-frame alignment are implemented by the motion estimation module.
There are four motion estimation frameworks: 1) RNN based [29, 49, 25, 19]; 2) Optical Flow [65]; 3) Deformable Convolution based [51, 63, 57]; 4) Temporal Concatenation [26, 6, 32].
In fact, there is another motion estimation scheme proposed for the ﬁrst time in the noise reduction task [50], which achieves an excellent video noise reduction performance.
Inspired by [50], we design an implicit dynamics estimation model for the spatial precipitation downscaling.
It is worth mentioning that our proposed model and the above four frameworks together form a relatively complete candidate set of dynamic estimation solutions.
Proposed Framework.
As shown in Fig.
3, our framework consists of two components: Implicit dynamic estimation module and downscaling Backbone .
These two parts are trained jointly.
Suppose there are N adjacent low-resolution precipitation maps { I LT − N − 1 2 , .., I LT , ..., I LT + N − 1 2 } .
Here, we employ superscript · H denotes the high resolution maps and · L for the low resolution counterpart.
The task is to reconstruct the high-resolution precipitation map I HT of I LT .
The implicit dynamic estimation module is composed of multiple vanilla networks A
=
{A 1 , ..., A N − 2 } ( N = 5 in this paper)
sharing weights.
Each vanilla network receives three adjacent frames as input, outputs, and intermediate results.
The intermediate result can be considered as a frame with implicit dynamic alignment.
We concatenate all the intermediate frames as the input of the next module.
The speciﬁc structure of the vanilla network can be found in the supplementary materials.
The main task of the downscaling backbone is to restore the high-resolution precipitation map
I HT based on the aligned intermediate frames.
In order to make full use of multi-scale information, we use multiple Residualin-Residual Dense Blocks [58] in the network.
We employ the interpolation+convolution [39] as the up-sampling operator to reduce the checkerboard artifacts.
After processing by downscaling backbone we get the ﬁnal estimated HR map ˆI H T .
Model objective.
The downscaling task is essentially to restore high-resolution precipitation maps.
We learn from the super-resolution task and also apply L 1 and perceptual loss [27] as the training loss of our model.
The model objective is shown below:
[Equation]
where φ denotes the pre-trained VGG19 network [47], we select the Relu 5 − 4 (without the activator [58]) as the output layer.
λ is the coefﬁcient to balance the loss terms.
In our framework, we set λ = 20 to balance the order of magnitude between losses.
Experimental Evaluation
We conduct spatial precipitation downscaling experiments to illustrate the application of our proposed RainNet and evaluate the effectiveness of the benchmark downscaling frameworks.
Following the mainstream evaluation protocol of DL/ML communities, cross-validation is employed.
In detail, we divide the dataset into 17 parts (2002.7 ∼ 2002.11, 2003.7 ∼ 2003.11, 2004.7 ∼ 2004.11, 2005.7 ∼ 2005.11, 2006.7 ∼ 2006.11, 2007.7 ∼ 2007.11, 2008.7 ∼ 2008.11, 2009.7 ∼ 2009.11, 2010.7 ∼ 2010.11, 2011.7 ∼ 2011.11, 2012.7 ∼ 2012.11, 2013.7 ∼ 2013.11, 2014.7 ∼ 2014.11, 2015.7 ∼ 2015.11, 2016.7 ∼ 2016.11, 2017.7 ∼ 2017.11, 2018.7 ∼ 2018.11) by year, and sequentially employ each year as the test set and the remaining 16 years as the training set, that is, 17-fold cross-validation.
All models maintain the same training settings and hyperparameters during the training phase.
These data cover various complicated precipitation situations such as hurricanes, squall lines, different levels of rain, and sunny days.
It is sufﬁcient to select the rainy season of the year as the test set from the perspective of meteorology, as the climate of one area is normally stable.
6.1 Baselines
The SISR/VSR and the spatial precipitation downscaling are similar to some extent, so we argue that the SR models can be applied to the task as the benchmark models.
The input of SISR is a single image, and the model infers a high-resolution image from it.
Its main focus is to generate high-quality texture details to achieve pleasing visual effects.
In contrast, VSR models input multiple frames of [Table] images (e.g., 3 frames, 5 frames.).
In our experiments, we employ 5 frames.
The core idea of VSR models is to increase the resolution by complementing texture information between different frames.
It is worth mentioning that VSR models generally are equipped with a motion estimation module to alleviate the challenge of object motion to inter-frame information registration.
We evaluated 7 state-of-the-art SISR frameworks (i.e., Bicubic [29], SRCNN [12], SRGAN [30], EDSR [33], ESRGAN [58], DBPN [18], RCAN [67] and 5 VSR frameworks (i.e., SRGANV, EDSR-V, ESRGAN-V, RBPN [19],
EDVR [57], of which 3 VSR methods (i.e., SRGAN-V, EDSR-V, ESRGAN-V) are modiﬁed from SISR.
In order to verify the impact of increasing the number of input frames on performance, we build SRGAN-V, EDSR-V and ESRGAN-V by concatenating multiple frames of precipitation maps as the input of the model.
In addition, we also evaluate Kriging [48], which is the de-facto standard method in the meteorological community.
The mentioned 9 metrics are used to quantitatively evaluate the performance of these SR models and our method.
Further, we select some disastrous weather as samples for qualitative analysis to test the model’s ability to learn the dynamic properties of the weather system.
And we employ the implementation of Pytorch for Bicubic.
We use 4 NVIDIA [-
-]2080 Ti GPUs for training.
We train all models with following setting.
The batch size is set as 24.
Precipitation maps are random crop into 64 × 64 .
We employ the Adam optimizer, beta1 is 0.9, and beta2 is 0.99.
The initial learning rate is 0.001, which is reduced to 1/10 every 50 epochs, and a total of 200 epochs are trained.
We evaluate benchmark frameworks with 17-fold cross-validation.
The downscaling performances are shown in Tab.
We divide the indicators mentioned above into two groups.
PDEM measures the model’s ability to learn the dynamics of precipitation.
PEM illustrates the model’s ability to reconstruct precipitation. [Figure]
H R [Figure] From Tab. 1, we can learn that the overall performance of the VSR methods are better than SISR models, which shows that the dynamic properties mentioned above are extremely important for the downscaling model.
Furthermore, it can be seen from Fig.
4 that the SISR method is clustered in the upper right corner of the scatter plot, and the VSR method is concentrated in the lower-left corner, which further shows that the dynamic properties of the VSR methods are overall better than the SISR methods.
In addition, our method achieves the 1 st best performance in RMSE,
PDEM, and achieve the second-best performance on PEM, which demonstrates that the proposed implicit dynamic estimation framework used is feasible and effective.
It is worth mentioning that the traditional downscaling method Kriging performs better than many deep learning models (e.g.,
SRGAN, ESRGAN).
6.1.1 Qualitative analysis
We visualized the tropical cyclone precipitation map of the 166 th hour (6th) in September 2010 and the high-resolution precipitation map generated by different methods.
As shown in Fig.
5, the best perceptual effects are generated by EDVR and Our framework.
Zooming in the result image, the precipitation maps generated by SRGAN and EDSR present obvious checkerboard artifacts.
The reason for the checkerboard artifacts should be the relatively simple and sparse texture pattern in precipitation maps.
The results generated by Bicubic, RCAN, Kriging, and SRCNN are over-smooth.
DBPN even cannot reconstruct the eye of the hurricane.
Especially, the result generated by Kriging is as fuzzy as the input LR precipitation map.
In conclusion, the visual effects generated by the
VSR methods are generally better than the SISR methods and the traditional method.
From the perspective of quantitative and qualitative analysis, the dynamics estimation framework is very critical for downscaling.
Related Works
Downscaling in Geoscience.
Downscaling is a fundamental task in geoscience and meteorology, which researchers have been interested in for a long time [60].
Most statistical downscaling methods regard this problem as point-wise regression [36, 53, 46] or direct maximum likelihood estimation of the high-resolution data from the low-resolution data [2].
Bürger et.
al .
[5] compared bias-correction spatial disaggregation (BCSD), quantile regression networks, and expanded downscaling (XSD) for climate downscaling.
Four fundamental statistical methods and three more advanced machine learning methods to downscale daily precipitation in the Northeast United States were compared in [53].
Recently, deep learning methods have been applied to solve the climate downscaling problem.
DeepSD [54] tried to employ SRCNN [12] on precipitation downscaling.
White et.
al.
[59] tested the performance of Generative Adversarial Networks (GANs) on downscaling weather models.
ClimAlign [16] proposed a novel deep learning method for statistical downscaling, treating it as an unsupervised domain alignment problem without using paired low/high-resolution training data.
Single Image Super-Resolution(SISR).
For deep learning based method, Dong et.
al.
ﬁrst proposed an end-to-end model SRCNN [12] using deep convolutional network.
SRGAN [30] proposed by Ledig et.
al. uses perceptual loss for ﬁner texture.
The enhanced version ESRGAN [58] introduces RRDB as basic network unit, using relative discriminator and perceptual loss of features before activation.
Video Super-Resolution(VSR).
VSR utilizes the temporal information of image sequences.
Wang et.
al. proposed a framework EDVR [57], which devises PCD module to handle large motion alignment, and the TSA fusion module with temporal and spatial attention.
Xie et.
al. proposed the tempoGAN [64] for ﬂuid ﬂow Super-Resolution, synthesizing 4-D physics ﬁelds with a temporal discriminator.
Future Work and Limitations
The recent success of vision based transformer (i.e., ViT) models [13, 34, 31, 7, 8, 42, 15] in image generation tasks has sparked a performance revolution. With its larger effective receptive ﬁeld and high-order interaction, ViT greatly surpasses the previous dominant convolutional neural networks (i.e., CNNs) [20, 23, 24, 12, 30, 58, 57, 67, 18] in both video and image processing. Witnessing this situation, we believe that the introduction of the ViT model will be of great help to the real downscaling task presented by RainNet.
At the same time, the diffusion based models [22, 9, 45, 41, 45] also shows unparalleled performance potential in generating images/videos (e.g., text to image), despite its notorious high computational complexity.
In future work, we will deeply explore the tailored application of ViT model and diffusion-based model in RainNet to better solve this important meteorological problem.
Conclusion
In this paper, we built the ﬁrst large-scale real precipitation downscaling dataset for the deep learning community.
This dataset has 62424 pairs of HR and LR precipitation maps in total.
We believe this dataset will further accelerate the research on precipitation downscaling.
Furthermore, we analyze the problem in-depth and put forward three key challenges: temporal misalignment, temporal sparse and ﬂuid properties.
In addition, we propose an implicit dynamic estimation model to alleviate the above challenges.
At the same time, we evaluated the mainstream SISR and VSR models and found that none of these models can perfectly solve RainNet’s problems.
Therefore, the downscaling task on this dataset is still very challenging.
This work still remains several open problems.
Currently, the data domain of this research is limited to the eastern U.S.
In future research, we would enlarge the dataset to a larger domain.
The dataset is only a single variable now.
In future research, we may include more variables, e.g. temperature and wind speed.
Acknowledgements
This work was supported by National Science Foundation of China (U20B2072, 61976137).
We also appreciate the supporting of computing resources by the student innovation center of Shanghai Jiao
Tong University.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 3.2.
• Did you include the license to the code and datasets?
[Yes] See Section 3.2.
• Did you include the license to the code and datasets?
[Yes]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Discussed in Conclusion section (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec.4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Sec.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec.2.
(b) Did you mention the license of the assets?
[Yes] See Sec.2.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]