Convolutional LSTMs for Cloud-Robust Segmentation of Remote Sensing Imagery
Abstract
Clouds frequently cover the Earth’s surface and pose an omnipresent challenge to optical Earth observation methods. The vast majority of remote sensing approaches either selectively choose single cloud-free observations or employ a pre-classiﬁcation strategy to identify and mask cloudy pixels. We follow a different strategy and treat cloud coverage as noise that is inherent to the observed satellite data.
In prior work, we directly employed a straightforward convolutional long short-term memory network for vegetation classiﬁcation without explicit cloud ﬁltering and achieved state-of-the-art classiﬁcation accuracies. In this work, we investigate this cloud-robustness further by visualizing internal cell activations and performing an ablation experiment on datasets of different cloud coverage.
In the visualizations of network states, we identiﬁed some cells in which modulation and input gates closed on cloudy pixels.
This indicates that the network has internalized a cloud-ﬁltering mechanism without being speciﬁcally trained on cloud labels.
Overall, our results question the necessity of sophisticated pre-processing pipelines for multi-temporal deep learning approaches.
We have published the source code of your work on GitHub 1 .
Introduction
An increasing number of satellites monitor dynamic spatiotemporal processes on the Earth’s surface. Optical satellites measure spectral reﬂectances at multiple short electromagnetic wavelengths in regular time intervals of a few days.
The surface, however, is often completely or partially covered by clouds. This poses an omnipresent challenge for the majority of remote sensing approaches are designed with cloud-free imagery in mind. In prior published work [Rußwurm and Körner, 2018, 2017], we focused on the identiﬁcation of crop classes with convolutional recurrent neural networks [Xingjian et al., 2015] to approximate life cycle events of vegetation.
These phenological cycles and other land cover dynamics can be monitored at weekly intervals at spatial resolutions of several meters which allows distinguishing large single objects. We observed that state-of-the-art accuracies could be achieved without dedicated pre-processing mechanisms. In this work, we investigate this cloud-robustness further and treat clouds as data-inherent noise.
Related Work
Clouds distinguish themselves from ground pixels by their high reﬂectance.
Rule-based models [Hollstein et al., 2016, Zhu and Woodcock, 2012, Hagolle et al., 2010] are used for many remote f t i t j t o t (cid:12) (cid:12) + (cid:12) h t −c t c t −h t x t classA classB classC predictions c T sensing applications and rely on expert-designed features [Li et al., 2018].
The fmask algorithm [Zhu and Woodcock, 2012] and improved versions [Zhu et al., 2015, Frantz et al., 2015] implement a projection of the detected cloud on the surface to additionally predict the shadow cast by the cloud.
Other approaches extract multi-temporal features and utilize the sudden temporal increase in reﬂectance [Hagolle et al., 2010].
Convolutional neural networks (CNNs) have also shown to compare well on cloud and cloud-shadow classiﬁcation [Li et al., 2018] indicating that these features can be learned from the data by deep neural networks.
However, masking single pixels by a pre-classiﬁcation introduces an additional layer of complexity and raises the question of how to treat these masked pixels accordingly.
For instance, one recently published approach [Interdonato et al., 2018] replaces cloudy pixels by temporal interpolation of cloud-free observations, while we introduced pre-classiﬁed cloud labels as additional prediction class in early work on this topic [Rußwurm and Körner, 2017].
Overall, cloud-ﬁltering remains a pre-processing necessity for most remote sensing approaches that are prone to fail in the presence of data noise.
Similar to our work, only a few approaches have tried to design robust methods that do not require this additional pre-classiﬁcation step.
For instance, Man et al.
[2018] used ensemble-based methods of supervised classiﬁers which have shown a certain robustness to the presence of clouds.
Method
In this section, we outline the theoretical basis of convolutional long short-term memory (convLSTM) networks utilized in this work and provide detail on the employed network topology.
3.1 Convolutional Long Short-term Memory
Long short-term memory (LSTM) networks [Hochreiter and Schmidhuber, 1997] implement internal gates to control the gradient-ﬂow through time and an additional container for long-term memory c t .
This yields the LSTM update ( h t ,
c t − 1 ) ← ( x t , h t − 1 , c t − 1 ) that maps an input x t and short-term context h t − 1 to a hidden representation h t .
Additionally, a long-term cell state c t − 1 is updated to c t at each iteration and can store information for a theoretically unlimited number of iteration.
Three gates control the update of the cell state c t ← c t − 1 (cid:12) f t + i t (cid:12) j t by element-wise multiplication (cid:12) .
The forget gate f t = σ ( x t ∗ θ fx + h t − 1 ∗ θ fh + 1 ) evaluates the inﬂuence of the previous cell state c t − 1 with a sigmoidal σ ( · ) ∈ ] 0, 1 [ activation function.
The input and modulation gates i t
=
σ ( x t ∗ θ ix + h t − 1 ∗ θ ih ) , and
[Equation]
(1) are element-wise multiplied for the cell state update.
The output gate o t = tanh ( x t ∗ θ ox + h t − 1 ∗ θ oh ) determines together with the cell state the current cell output x i j c t. . . tt. . ttttttttt  -  -  h t ← o t (cid:12) c t .
Convolutional recurrent networks implement a convolution, denoted by ∗ , instead of a matrix multiplication.
Each respective gate activation, referred by subscripts f , i , j , o, is controlled by trainable weights for input θ fx , θ ix , θ jx , θ ox ∈ R k × k × d × r and hidden representation θ fh , θ ih , θ jh , θ oh ∈
R k × k × r × r where d represents the number of image channels, k the convolutional kernel size, and r a hyper-parameter that determins the number of the hidden states in the recurrent layer.
With this change, image data of certain width, height and depth can be processed with convolutions partially connecting the local pixel neighborhoods between layers.
3.2 Network Topology
We utilize this single-layer convolutional LSTM neural network to encode a sequence of T satellite images to the ﬁxed length representation c T , as illustrated in Fig.
In our initial work [Rußwurm and
Körner, 2018], we found that 256 recurrent cells performed best and used this hyper-parameter for the number of hidden states within the ConvLSTM network.
To balance the inﬂuence of the sequence order, we also encode the reversed sequence and append the ﬁnal cell states.
Applying softmax normalization produces activations that can be interpreted as network-conﬁdences per class.
We used convolutional kernels of 3 × 3 px throughout the network.
To train, we evaluate the cross-entropy between the last layer and a one-hot representation of the ground truth labels.
The inﬂuence of each weight on the evaluated loss is determined by back-propagated gradients which are used to iteratively adjustment the respective weight using the Adam optimizer [Kingma and Ba, 2014] with a learning rate of λ = 0.001 .
Results
The primary objective for this network was to identify the type of cultivated crops in an area of interest of 100 km × 40 km, as described at greater detail in Rußwurm and Körner [2018].
Hence, we trained our network end-to-end on label data describing the crop-type on distinct ﬁeld parcels and provided no additional label information about cloud coverages.
We used a sequence of 46 S ENTINEL 2 satellite images from the year 2016 for this objective.
This satellite measures the reﬂectances of 13 spectral bands at 10 m, 20 m, and 60 m resolution.
To harmonize the data sources, we bi-linearly interpolated these to 10 m resolution and rasterized the crop labels accordingly.
Additional information about area of interest and data partitioning strategy is given in Appendix A.
In the remainder of this section, we evaluate the robustness of the proposed network to cloud coverage.
4.1 Exploration of Hidden LSTM States
We trained the network on ﬁeld crop labels for thirty epochs using raw sequences of cloudy and noncloudy observations.
The top row of Fig.
2 shows an particular example sequence with 48 × 48 px images.
The following rows illustrate activations of the internal convolutional LSTM for input gate
samples seen ov .
acc .
samples seen ov .
acc .
< 1% (4 obs.)
< 10% (10 obs.)
< 25% (17 obs.)
< 50% (23 obs.)
all images (46 obs.)
i , modulation gate j , and c t .
While all of the 256 recurrent cells likely contribute to the classiﬁcation decision, only a few were visually interpretable similar to the shown example, as can be compared in
Appendix B.
The activations in Fig.
2 of these gates in the second and third row show that the input gate i approaches zero at pixels that are covered by clouds.
This effect can be observed at time steps t = { 13, 26, 31, 33 } .
At time t = 32 the input gate seems unchanged, however, the modulation gate j changes sign.
Overall, these results indicate that the convolutional recurrent network has internalized a mechanism for cloud-ﬁltering.
4.2 Ablation Experiment on Cloud Coverage
In this experiment, we trained the network on datasets with different degrees of cloud coverage.
To determine the actual coverage of clouds per observation, all satellite images have been pre-processed using the fmask algorithm implemented in the Sen2Cor software, as is common practice in remote sensing [Conrad et al., 2014, Foerster et al., 2012].
With this, a cloud coverage pixel ratio per observation can be calculated.
Based on this, several sub-datasets have been created with either all 46 observations, the 26 images covered with less than 50%, 17 images with less than 25%, 10 with less than 10%, and 4 completely cloud-free images.
We trained the network on these pre-ﬁltered datasets and show the overall accuracy recorded during training on validation data in Fig.
Even though the ratio of cloudy images varies greatly between these datasets, the classiﬁcation accuracy remains similar for all of the sub-sampled datasets.
The right graph shows a zoomed view and reveals some differences between the dataset performances.
Datasets containing all observations and the four completely cloud-free observations have been slightly worse classiﬁed than the intermediate ones of 10%, 25%, and 50% coverage.
It seems that the manual removal of the completely cloud-covered observations had a minor beneﬁt on the classiﬁcation accuracy.
Also, the four cloud-free observations were classiﬁed slightly worse which indicates that these may have missed some characteristic vegetation-related events.
Intuitively, these results show a trade-off between restrictions on cloud coverage and sequence length and demonstrate that cherry-picking single cloud-free observations may lead to inferior classiﬁcation accuracy.
Overall, these results demonstrate the robustness of the convolutional long short-term memory network to handle data containing temporal noise induced by cloud coverage.
Conclusion
Noise in temporal data is a common challenge for a variety of disciplines.
In this work, we focused on noise induced by cloud coverage in multi-temporal remote sensing imagery.
Most Earth observation approaches either select few completely cloud-free observations or use a pre-classiﬁcation to mask cloudy pixels.
The experiments of this work showed that this cloud-induced temporal noise can be suppressed by training on an other objective in an end-to-end fashion with an appropriate model design.
Our results demonstrate that convolutional recurrent networks are able to consistently extract the classiﬁcation-relevant features from observations between clouds without dedicated cloud labels. Overall, this work questions the necessity of sophisticated, partly hand-crafted pre-processing pipelines for remote sensing imagery. Appendices
This addendum provides further background information on the crop classiﬁcation task out of which the presented evaluation on cloud-robustness has emerged. A Dataset and Area of Interest
The area of interest (AOI) north of Munich, Germany, was divided into partitions for the training, validation [Figure]
and evaluation, as shown in Fig. This ensures independence between the optimization of network parameters, hyper-parameter tuning, and ﬁnal evaluation. Further, we took special care to avoid introducing dependence between these partitions by spatial proximity by subdividing the AOI into rectangular blocks of 3.84 km × 3.84 km (multiples of the chosen tile sizes 240 m and 480 m). Margins between blocks of 480m avoid that tiles of two distinct partitions are located in immediate proximity.
These blocks were randomly assigned to the partitions in a 4:1:1 train/valid/eval ratio.
All tiles (240m and 480m) that were fully within the respective blocks have been assigned to their respective partition.
The overall cloud coverage over the area of interest is shown in Fig.
[Figure]
[Figure]
Further [Table] x fijcfijcfijcfijcfijct. . .
[Table]
[Table]
[Table]
B.2 Tile B x fijcfijcfijcfijcfijct. . .
tttttt. . .
tttttttt
[Table]
[Table]