Spatiotemporal data fusion for precipitation nowcasting
Anonymous Author s)
Abstract
Precipitation nowcasting using neural networks and ground-based radars has become one of the key components of modern weather prediction services, but it islimited to the regions covered by ground-based radars.
Truly global precipitationnowcasting requires fusion of radar and satellite observations.
We propose the datafusion pipeline based on computer vision techniques, including novel inpaintingalgorithm with soft masking.
Introduction
Precipitation nowcasting is a problem of precipitation prediction on a very short term (up to twohours).
In this time range, extrapolation techniques working with detailed observational data cancompete with and even win over meteorological models.
The sources of data for this probleminclude ground-based radars and geostationary satellites.
Radars provide more detailed and accurateobservations, so using radars is more desirable when they are available.
Nowcasting based on radardata was investigated in detail in literature [1, 2].
However, radar coverage is not complete across the globe. There are huge areas where only thesatellite observations available, so any global solution for precipitation nowcasting have to incorporatesatellite data and through the following steps
Detect precipitation on satellite observations
Perform fusion of radar and satellite data3.
Train model and perform prediction
The detection is described in [3, 4] and approaches proposed for prediction step include extrapolationwith optical ﬂow [5] and convolutional LSTM networks [1].
In this paper, we focus on the fusionstep of this pipeline that was not explored before.
The fusion of radar and satellite observations is a challenging task, as two data domains differ bothspatially (each radar detects precipitation in the range of 200 km around its location) and temporally(radars and satellites provide observations in different moments of time, with different cadence).
We address temporal fusion by simple interpolation algorithm using optical ﬂow, and spacial fusion(image blending) by alpha-blending and special modiﬁcation of inpainting algorithm with a soft maskto correct artifacts.
In the simplest case of spatial fusion, the satellite data is replaced by radar data when the latter isavailable.
As both radar and satellite have errors, this procedure produces sharp circular seams.
Incase of early fusion (fusion before prediction), the ﬁxed in space seams will confuse the nowcastingmodel, which is supposed to estimate and predict the movement of precipitation ﬁelds.
In case of latefusion, the seams will be left in the output map and confuse the users.
Inpainting networks allow setting mask explicitly.
Starting from neural network with partial convolutions [6], we improve inpaint quality by developing a new mechanism of mask updates.
In summary, we make the following contributions:
• We develop a global nowcasting using radar and satellite data.
• We develop a soft mask inpainting which outperform binary mask inpainting.•
We show that it will better variant for certain types of tasks like large area recovery andimage blending.
Related Work
Image blending is a well-developed topic of classical computer vision.
Smooth transition betweencomposited images can be guided by the pyramid image representation [7].
Optimal seam methods [8,9] enhance the visual quality of the composited image by cutting at optimal places, and Poissonimage editing [10] corrects different exposure or tint of the composed images.
Recently, GANs wereapplied to the problem [11], allowing to make non-local photorealistic changes.
While our task essentially is the one of image blending, several factors stop us from adopting thesemethods: in our case, seam position is predeﬁned by the radar range, there is no need for exposurecorrection, and it is desirable to make only the local changes.
Image inpainting, the process of reconstructing lost or corrupted regions of the image, can also beused for blending: compose two images together, declare the region near the seam to be corrupted,then inpaint.
Multiple approaches for image inpainting were developed in the pre-CNN era ofcomputer vision (i.e., reconstruction of isophotes [12] or Patch-Match algorithm [13]), but themodern approaches rely on neural networks, their ability to learn the shapes and textures of theobjects in the training set and reproduce them to cover corrupted parts of the image.
These approachescommonly use GANs with rectangular binary masks to cover the ’corrupted’ parts of the image [14,15].
Alternatively, [6] propose a CNN architecture (inspired by UNet [16] model) with the partialconvolution layers and combination of losses which outperform [14, 15].
The main idea behind partial convolution from [6] is the following.
Let W be the convolutionalweights and b the corresponding bias.
X are the pixels values for the current convolution window and M is the corresponding binary mask.
The partial convolution at every location is expressed as:
[Equation]
where (cid:12) is an element-wise multiplication.
After each layer with the partial convolution, the maskvalues of the next layer m (cid:48) are updated:
[Equation]
Method3.1 Temporal fusion
The ﬁrst problem we encounter during mixing satellite and radar data is a different time resolution.
While most radars provide images once every 10 minutes, the time resolution for the satellite imagery(Meteosat-8) is 15 minutes.
Most of the temporal change in the data is associated with precipitation ﬁelds moving in the wind,so the optical ﬂow based algorithm comes up as a natural solution to produce intermediate framesrequired to link radar and satellite data.
We use a simpliﬁed dense optical-ﬂow based frame rateconversion algorithm [17] to generate synthetic satellite data at 10-minute intervals.
3.2 Soft-mask inpainting
We extend partial convolutions [6] by allowing non-binary values in the mask, interpreted as theconﬁdence levels of the corresponding pixels in a source image.
With this modiﬁcation, the inpaintingalgorithm incorporates some information from the corrupted part of the image and successfully ﬁllslarger regions.
In case of smooth masking, Equation 2 changes the following way:
[Equation]
where m is a correspondent element of the mask from the previous layer.
3.3 Soft-mask preprocessing for blending
Since our inpainting model uses information from the area of blending, we preprocess the data toallow the network using both sources.
During inference, we use simple alpha-blending to composetwo images I
and I 2 with smooth transition between them
[Equation]
where values of the map α deﬁne this transition, linearly changing from 0 to 1 in the blending area.
Note that ground truth can’t be obtained for this scheme, so we need another approach for training.
During training, we use the satellite data I as the ground truth, and apply noise N to the simulatedblending area to generate the training data I training
[Equation]
In this case, α is equal to zero in the middle of blending area, equal to one outside the blending area,and changes linearly between these two points.
Experiments
4.1 Comparison with binary mask inpainting
In this section, we perform experiments on a different dataset to demonstrate the validity of ourapproach with soft masking.
We use CelebA-HQ [18, 19] dataset for training and testing.
For CelebA-HQ, we randomly partitioninto 27K images for training and 3K images for testing. All images scaled and cropped to 256x256pixels.
For the training process, we add noise to the images in these datasets using Equation 5 and tryto predict source images.
We use quick-draw [20] as a shape of the mask.
We compare binary inpainting network and two soft-mask networks: one with a constant mask ﬁlledwith value 0 .
5 (semi-transparent mask) and another with mask M = α , where α is taken directlyfrom Equation 5.
We compare these networks using PSNR and SSIM metrics and present the resultsin Table 1, demonstrating improvement of both variants of soft masking compared to binary masking.
[Table] The best performance is achieved in the case of M = α , but in context of the task this is data leak.
However, the network with semi-transparent mask have almost the same metrics, demonstrating that the main improvement is caused by utilization of information from the masked region of the original [Figure] image, and not from the leak of parameters of data synthesis through α .
Thus, we can use soft-mask [Figure] inpainting with any reasonable α instead of exact values and still have an improvement in qualitycompared to binary masking.
[Figure]
[Figure]
Figure 1 shows results of the same model on our target task, satellite and radar data fusion. It can beseen that our approach successfully hides the border between two types of data. [Figure]  Conclusion
We presented a pipeline for spatiotemporal fusion of precipitation data.
Our pipeline uses simpletechniques, such as optical ﬂow based interpolation and alpha-blending, as well as a novel approachof image inpainting with soft masking.
We have demonstrated an advantage of soft masking in thecase then useful information can be recovered from the corrupted region.
The proposed pipelineallows expanding precipitation nowcasting from the list of selected locations covered with radars(USA, Hong Kong, Europe), to the rest of the world.