Spatiotemporal data fusion for precipitation nowcasting
Abstract
Precipitation nowcasting using neural networks and ground-based radars has become one of the key components of modern weather prediction services, but it is limited to the regions covered by ground-based radars.
Truly global precipitation nowcasting requires fusion of radar and satellite observations.
We propose the data fusion pipeline based on computer vision techniques, including novel inpainting algorithm with soft masking.
Introduction
Precipitation nowcasting is a problem of precipitation prediction on a very short term (up to two hours).
In this time range, extrapolation techniques working with detailed observational data can compete with and even win over meteorological models.
The sources of data for this problem include ground-based radars and geostationary satellites.
Radars provide more detailed and accurate observations, so using radars is more desirable when they are available.
Nowcasting based on radar data was investigated in detail in literature [1, 2].
However, radar coverage is not complete across the globe.
There are huge areas where only the satellite observations available, so any global solution for precipitation nowcasting have to incorporate satellite data and through the following steps
Detect precipitation on satellite observations 2.
Perform fusion of radar and satellite data 3.
Train model and perform prediction
The detection is described in [3, 4] and approaches proposed for prediction step include extrapolation with optical ﬂow [5] and convolutional LSTM networks [1].
In this paper, we focus on the fusion step of this pipeline that was not explored before.
The fusion of radar and satellite observations is a challenging task, as two data domains differ both spatially (each radar detects precipitation in the range of 200 km around its location) and temporally (radars and satellites provide observations in different moments of time, with different cadence).
We address temporal fusion by simple interpolation algorithm using optical ﬂow, and spacial fusion (image blending) by alpha-blending and special modiﬁcation of inpainting algorithm with a soft mask to correct artifacts.
In the simplest case of spatial fusion, the satellite data is replaced by radar data when the latter is available.
As both radar and satellite have errors, this procedure produces sharp circular seams.
In case of early fusion (fusion before prediction), the ﬁxed in space seams will confuse the nowcasting model, which is supposed to estimate and predict the movement of precipitation ﬁelds.
In case of late fusion, the seams will be left in the output map and confuse the users.
Inpainting networks allow setting mask explicitly.
Starting from neural network with partial convolutions [6], we improve inpaint quality by developing a new mechanism of mask updates.
In summary, we make the following contributions:
• We develop a global nowcasting using radar and satellite data.
• We develop a soft mask inpainting which outperform binary mask inpainting.
• We show that it will better variant for certain types of tasks like large area recovery and image blending.
Related Work
Image blending is a well-developed topic of classical computer vision.
Smooth transition between composited images can be guided by the pyramid image representation [7].
Optimal seam methods [8, 9] enhance the visual quality of the composited image by cutting at optimal places, and Poisson image editing [10] corrects different exposure or tint of the composed images.
Recently, GANs were applied to the problem [11], allowing to make non-local photorealistic changes.
While our task essentially is the one of image blending, several factors stop us from adopting these methods: in our case, seam position is predeﬁned by the radar range, there is no need for exposure correction, and it is desirable to make only the local changes.
Image inpainting, the process of reconstructing lost or corrupted regions of the image, can also be used for blending: compose two images together, declare the region near the seam to be corrupted, then inpaint.
Multiple approaches for image inpainting were developed in the pre-CNN era of computer vision (i.e., reconstruction of isophotes [12] or Patch-Match algorithm [13]), but the modern approaches rely on neural networks, their ability to learn the shapes and textures of the objects in the training set and reproduce them to cover corrupted parts of the image.
These approaches commonly use GANs with rectangular binary masks to cover the ’corrupted’ parts of the image [14, 15].
Alternatively, [6] propose a CNN architecture (inspired by UNet [16] model) with the partial convolution layers and combination of losses which outperform [14, 15].
The main idea behind partial convolution from [6] is the following.
Let W be the convolutional weights and b the corresponding bias.
X are the pixels values for the current convolution window and
M is the corresponding binary mask.
The partial convolution at every location is expressed as:
[Equation]
where (cid:12) is an element-wise multiplication.
After each layer with the partial convolution, the mask values of the next layer m (cid:48) are updated:
[Equation]
Method 3.1 Temporal fusion
The ﬁrst problem we encounter during mixing satellite and radar data is a different time resolution.
While most radars provide images once every 10 minutes, the time resolution for the satellite imagery (Meteosat-8) is 15 minutes.
Most of the temporal change in the data is associated with precipitation ﬁelds moving in the wind, so the optical ﬂow based algorithm comes up as a natural solution to produce intermediate frames required to link radar and satellite data.
We use a simpliﬁed dense optical-ﬂow based frame rate conversion algorithm [17] to generate synthetic satellite data at 10-minute intervals. [Equation]
3.2 Soft-mask inpainting
We extend partial convolutions [6] by allowing non-binary values in the mask, interpreted as the conﬁdence levels of the corresponding pixels in a source image.
With this modiﬁcation, the inpainting algorithm incorporates some information from the corrupted part of the image and successfully ﬁlls larger regions.
In case of smooth masking, Equation 2 changes the following way:
[Equation]
where m is a correspondent element of the mask from the previous layer.
3.3 Soft-mask preprocessing for blending
Since our inpainting model uses information from the area of blending, we preprocess the data to allow the network using both sources.
During inference, we use simple alpha-blending to compose two images I
and I 2 with smooth transition between them
[Equation]
where values of the map α deﬁne this transition, linearly changing from 0 to 1 in the blending area.
Note that ground truth can’t be obtained for this scheme, so we need another approach for training.
During training, we use the satellite data I as the ground truth, and apply noise N to the simulated blending area to generate the training data I training
[Equation]
In this case, α is equal to zero in the middle of blending area, equal to one outside the blending area, and changes linearly between these two points.
Experiments 4.1 Comparison with binary mask inpainting
In this section, we perform experiments on a different dataset to demonstrate the validity of our approach with soft masking.
We use CelebA-HQ [18, 19] dataset for training and testing.
For CelebA-HQ, we randomly partition into 27K images for training and 3K images for testing.
All images scaled and cropped to 256x pixels.
For the training process, we add noise to the images in these datasets using Equation 5 and try to predict source images.
We use quick-draw [20] as a shape of the mask.
We compare binary inpainting network and two soft-mask networks: one with a constant mask ﬁlled with value 0 .
5 (semi-transparent mask) and another with mask M = α , where α is taken directly from Equation 5.
We compare these networks using PSNR and SSIM metrics and present the results in Table 1, demonstrating improvement of both variants of soft masking compared to binary masking.
[Table]
The best performance is achieved in the case of M = α , but in context of the task this is data leak.
However, the network with semi-transparent mask have almost the same metrics, demonstrating that the main improvement is caused by utilization of information from the masked region of the original image, and not from the leak of parameters of data synthesis through α .
Thus, we can use soft-mask inpainting with any reasonable α instead of exact values and still have an improvement in quality compared to binary masking.
4.2 Radars and satellite data fusion
Figure 1 shows results of the same model on our target task, satellite and radar data fusion.
It can be seen that our approach successfully hides the border between two types of data.
[Figure]
Conclusion
We presented a pipeline for spatiotemporal fusion of precipitation data.
Our pipeline uses simple techniques, such as optical ﬂow based interpolation and alpha-blending, as well as a novel approach of image inpainting with soft masking.
We have demonstrated an advantage of soft masking in the case then useful information can be recovered from the corrupted region.
The proposed pipeline allows expanding precipitation nowcasting from the list of selected locations covered with radars (USA, Hong Kong, Europe), to the rest of the world.