[Re] Satellite Image Time Series Classiﬁcation with Pixel-Set Encoders and Temporal Self-Attention
Anonymous Author(s)
AfﬁliationAddress email
The presented study evaluates “Satellite Image Time Series Classiﬁcation with Pixel-Set Encoders and Temporal
Self-Attention” by Garnot et al.
(2020) within the scope of the ML Reproducibility Challenge 2020.
Our work focuseson both aspects constituting the paper: the method itself and the validity of the stated results.
We show that, despitesome unforeseen design choices, the investigated method is coherent in itself and performs the expected way.
Scope
of
Reproducibility
The evaluated paper presents a method to classify crop types from multispectral satellite image time series with anewly developed pixel-set encoder and an adaption of the Transformer (Vaswani et al ., 2017), called temporal attentionencoder .
Methodology
In order to assess both the architecture and the performance of the approach, we ﬁrst attempted to implement the methodfrom scratch, followed by a study of the authors’ openly provided code.
Additionally, we also compiled an alternativedataset similar to the one presented in the paper and evaluated the methodology on it.
Results
During the study, we were not able to reproduce the method due to a conceptual misinterpretation of ours regarding theauthors’ adaption of the Transformer (Vaswani et al ., 2017).
However, the publicly available implementation helpedus answering our questions and proved its validity during our experiments on different datasets.
Additionally, wecompared the papers’ temporal attention encoder to our adaption of it, which we came across while we were trying toreimplement and grasp the authors’ ideas.
What was easy
Running the provided code and obtaining the presented dataset turned out to be easily possible.
Even adapting themethod to our own ideas did not cause issues, due to a well documented and clear implementation.
What was difﬁcult
Reimplementing the approach from scratch turned out to be harder than expected, especially because we had a certaintype of architecture in mind that did not ﬁt the dimensions of the layers mentioned in the paper.
Furthermore, knowinghow the dataset was exactly assembled would have been beneﬁcial for us, as we tried to retrace these steps, and thereforewould have made the results on our dataset easier to compare to the ones from the paper.
Communication with original authors
While working on the challenge, we stood in E-mail contact with the ﬁrst and second author, had two online meetingsand got feedback to our implementation on G IT H UB .
Additionally, one of the authors of the Transformer paper( et al ., 2017) provided us with further answers regarding their models’ architecture.
Introduction
The machine learning community showcases impressively and with great success how to design systems for analysinglarge data inventories, with the goal of identifying relevant patterns within them.
At the same time, the remote sensingand Earth observation sector found itself confronted with the availability of novel dedicated sensor platforms.
Theseare now capable of continuously acquiring new data at high temporal, spatial, and spectral frequencies, requiring thedevelopment of innovative and efﬁcient ways to process these data stocks.
In light of that, several machine learningmethods have found wide application in the ﬁeld of remote sensing and Earth observation.
As the availability ofobservation data increased massively during the last decades (Lary et al ., 2018), various retrieval, detection, andprediction problems can be addressed this way.
Nevertheless, Earth observation data is mostly of a very inhomogeneousnature, which is due to the different designs and layouts of the receiving sensor platforms.
In addition, geophysicalprocesses on the Earth’s surface are complex and manifest themselves in observable changes with different dynamicpatterns.
Therefore, the goal of gaining deeper understanding of such processes requires the use of interpretable models(Maxwell, Warner, and Fang, 2018).
In their recent CVPR publication “Satellite Image Time Series Classiﬁcation with Pixel-Set Encoders and Temporal
Self-Attention”, Garnot et al .
(2020) propose a new method to address these issues.
Motivated by the practical problemof crop type classiﬁcation from sequences of optical satellite imagery—that we consider a proxy for the entirety ofvegetative processes on the Earth’ surface—, the authors made use of attention mechanisms .
In particular, they claimedadapting the Transformer architecture (Vaswani et al ., 2017) that has gained considerable popularity in the recent past,enabling it to digest such speciﬁc Earth observation data modalities.
Additionally, they introduce a pixel-set encoder asa new option to deal with medium-resolution satellite images instead of the known convolutional neural networks inimage processing.
Due to their aforementioned properties, the handling of Earth observation data in practical applications requires specialcare.
Most prevalently, they exhibit a considerable amount of spatial autocorrelation (Spiker and Warner, 2007),reinforcing the already known issues of underspeciﬁcation inherent to data-driven machine learning models (D’Amouret al ., 2020).
This generally leads to overﬁtting effects and poor generalisation performance.
Hence, we carefullyreproduced the proposed methods, ﬁrst by starting from scratch just following the descriptions given in the paper underinvestigation, and subsequently by adapting the reference implementation openly provided by the original authors.
Tosanity-check both implementations and to further assess the transferability and generalisation properties of the studiedmodel, we carried out further experiments relying on an alternative but comparable dataset.
Following the idea ofthis reproducibility challenge, we will make our implementation and data public, allowing the community to likewiseevaluate our ﬁndings.
1.1 Reproducibility questions
As a foundation of our reproduction study, we identiﬁed the following key questions, each one examining one particularaspect or claim of the original paper:
i) Is it possible to reproduce the presented methods and their performance with and without referring to the
authors’ publicly available code?
ii)
To which extent do the author’s implicit claim of adapting the transformer architecture affect the model and
its
performance?
iii) Does the model perform comparably well when being only tested or both trained and tested on a differentdataset?
iv) How is the outcome inﬂuenced by the choice of the test set assembly, namely by splitting the data randomly orregionally?
1.2 Contributions beyond the original paper
In addition to the results in the original paper, we also report the performance of the method evaluated on an alternativedataset and on a test set that does not show regionally overlap with the training set.
Methodology and experimental setup of the reproduction study
In order to study the reproducibility of the original publication, we followed different approaches to answer thequestions raised in Section 1.1.
Each of the following subsections will address one of these questions by introducing the methodology behind the chosen experiments and present our obtained results as replies to the author’s claims.
Generally, we traversed the following three different experimental stages that we will also publish on G IT H UB 1 forbetter traceability: We ﬁrst started by developing the entire approach ourselves in P YTHON and P Y T ORCH (Paszkeet al ., 2019), as described in Section 2.1.1, followed by a short study on the original implementation in Section 2.1.2 anda comparison of one particular technical aspect in Section 2.2.
Eventually, we conducted experiments on the inﬂuenceof input data in the remaining sections.
2.1 Reproduction and accuracy of the satellite time series classiﬁcation
The proposed architecture for satellite time series classiﬁcation consists of two components which the authors introduceas the pixel set encoder (PSE) and the temporal attention encoder (TAE) .
While the former takes care of a randomlysampled pixel set from a crop parcel and produces an embedding of the input, the latter, an adapted variant of the
Transformer architecture (Vaswani et al ., 2017), produces an output by applying self-attention to these multi-temporalembeddings.
Unlike practices familiar from natural language processing , PSE and TAE get both optimised during thetraining phase.
A detailed summary of the composition and number of parameters in the networks can be obtained from
Table 1 in the studied paper.
All experiments were conducted on an U BUNTU 20.10 workstation equipped with 64 GB of RAM, an I NTEL I 7-
CPU and an NV IDIA G E F ORCE RTX 2060 GPU.
2.1.1 Full replication study of the approach
We reimplemented the proposed architecture described in Section 3: Methods of the investigated paper almost literallyin P YTHON .
As the section is subdivided into the three parts describing the spatial encoder , also referred to as thepixel-set encoder , the temporal attention encoder, and the spatio-temporal classiﬁer , these three modules likewise buildthe core of this reproduction study.
More precisely, Table 1 of Garnot et al .
(2020) allowed us to inherit all modelhyper-parameters straightforwardly.
For one of the four used multi-layer perceptrons (MLPs) , it was stated that itconsisted of fully-connected layers FC , batch normalisation , and ReLU activations .
We, thus, inductively assumedthese components to be part of the other three MLPs as well.
While we managed to develop an inefﬁcient yet workingversion of the spatial encoder, some aspects of the temporal attention encoder appeared unintuitive at ﬁrst sight.
Unlikethe original Transformer model (Vaswani et al ., 2017), on which this module is based, the values v are not calculatedby a fully-connected layer.
Instead, the sum of the spatial encoder outputs and the positional encoding is multipliedwith the attention mask a , which is visualised in Figure 1a.
The authors motivated this change with the claim thatit “removes needless computations, and avoids a potential information bottleneck” ( cf.
Garnot et al ., 2020, Section3.2.).
Having the original Transformer implementation in mind, we misinterpreted this design choice.
Thus, in ourimplementation, we divided the multi-head attention input into four equal tracks, as we were not able to think of anotherway to end up having 512 nodes to be passed over to the MLP 3 ( cf.
Table 1 of Garnot et al ., 2020).
This misconceptionwas partially reinforced by the superscript ( h ) in formula (5) in Garnot et al ., 2020, i.e. ,
[Equation]
suggesting that, for each head h , an entirely independent fully-connected layer FC ( h ) was used.
This way, our networkreimplementation became incredibly blown up and we were not able to spot the correct approach.
All hyper-parameters and training details were directly taken from the Section 3.4: Implementation details from Garnotet al .
After completing the ﬁrst presented stages of the implementation, we were able to achieve an accuracy ofabout 60 %.
Subsequently, we had a ﬁrst online meeting with the ﬁrst and second author of the investigated paper, wherewe identiﬁed some misconception concerning the used labels: Instead of using all 20 classes from the label_19classdictionary in the provided lables.json ﬁle, only the top-20 classes from the label_44class dictionary of the sameﬁle were utilised by the authors, i.e. , the classes with more than 100 occurrences in the dataset.
It also got to our attentionthat, in lieu of the proposed batch normalisation, the multi-layer perceptrons should perform layer normalisation .
Unfortunately, despite the ﬁrst author providing helpful feedback on our implementation via G IT H UB , we were stillnot able to achieve a relevant increase in accuracy compared to the previously stated 60 %.
After comparing ourimplementation to the reference implementation provided by the authors, it became apparent that we struggled to graspthe authors’ ideas about the data organisation and the spatio-temporal classiﬁer.
Therefore, we will investigate what ledus to misinterpret the Transformer’s adaption in Section 2.2 and the data organisation in Section 2.3.2.
encoding
[Equation]
attention
[Equation]
input
[Equation]
[Figure]
encoding
[Equation]
input
[Equation]
[Figure]
2.1.2 Evaluation of the original implementation
Obtaining the authors’ code and running it locally proved to be easy thanks to a well-documented G IT H UB repository 2 .
In general, their reference implementation is modular, clearly structured, and sufﬁciently commented which makes theentire architecture easy to adapt to one’s own needs.
The model can either be trained from scratch or, together with aprovided checkpoint, used to solely run the inference.
We present a comparison of the test results from the checkpointand when training from scratch in Table 1a.
Using the system speciﬁed in Section 2.1, training the model for one singleepoch took approximately 27 s, while running the inference on one sample batch containing 128 parcels completedwithin about 0.04 s. For the following experiments, we exclusively used the ofﬁcial reference implementation to ensurecomparability.
2.2 Experiments on the transformer architecture
As stated in Section 2.1.1, we previously relied on some faulty assumptions related to the architecture of the adapted
Transformer multi-head attention.
Fortunately, the original code provided by the authors helped us to reconstruct theirinitial idea.
Nevertheless, having the vanilla off-the-shelf Transformer implementation in mind, we were inquisitiveabout the impact of the particular architectural change proposed by Garnot et al .
Hence, we took the referenceimplementation and changed two lines of it in a way that the temporal attention encoder then employed a third fullyconnected layer FC v , just like the vanilla Transformer attention model does, as described by Vaswani et al .
Figure 1b illustrates this subtle change in comparison to the architecture realized by the authors of the paper underinvestigation, shown in Figure 1a.
We provide the code for this version as a fork of the original repository.
[Table] [Table]
While this adaption of the proposed architecture reduces the number of trainable model parameters to 80 % we wereable to reproduce the reported performance or even experienced a slight increase in accuracy, as summarised in Table 1a.
2.3 Generalisation and transferability
Until that stage, we mainly considered the theoretical aspects of this reproduction study.
Conversely, this section focuseson the application-oriented side.
In light of our observations described before we wanted to investigate whether theincreased number of parameters in the original model can become beneﬁcial when scaling the underlying classiﬁcationproblem by confronting it with an alternative dataset.
2.3.1 Datasets
Although a tremendous amount of satellite data, especially from the S ENTINEL -2 platforms, is publicly available,the computer vision and machine learning community still lacks labels or annotations for addressing most relevantresearch questions.
Therefore, Garnot et al .
(2020) did not only publish a new method, but also complemented it with adataset containing crop type labels of more than 190 000 agricultural parcels within the area of a particular tile of the
S ENTINEL -2 tiling grid T31TFM , located in France.
Additionally and to evaluate the reproducibility of the presentedmethods’ results on a different input, an analogous dataset with parcels located in Slovenia was constructed in thecourse of this study.
The following two sections gives insight into the background and properties of these two datasets.
FR- T31TFM : Dataset from the paper Unlike CNN-based methods, the approach by Garnot et al .
does not requirethe observation data to be stored as images with deﬁned neighborhood relations, but rather as an unordered set of pixelsfor each parcel.
From their G IT H
UB page, a toy dataset containing 500 parcels, each saved as N UM P Y data ﬁles, canbe obtained.
To get access to the entire dataset, an inquiry needs to be sent to the authors, which they reply to withinno time.
This dataset includes 192 056 N UM P Y data ﬁles of dimension T × C × N , with T , C = 10 , and N beingthe number of observations dates, spectral bands, and pixels for each particular parcel, respectively.
It additionallycomes with several metaﬁles, i.e. , the dates of the observations, the labels of the parcels, the geometric features of theparcels—which are stated to be necessary for the pixel set encoder—, and pre-computed normalisation values.
Bydesign, the dataset is randomly partitioned into test, validation, and training parcels, following a split ratio of 3:1:1.
This dataset also faces one of the biggest challenges in crop type classiﬁcation from satellite data, namely the unevendistributed data foundation, as visualised in Figure 3a.
In their original paper, Garnot et al .
refer to several data preprocessing steps, such as reducing the number of spectralbands delivered by the S ENTINEL -2 satellite from 13 to 10, linearly interpolating ground pixels that are affected bycloud cover, normalising the reﬂectance data, and adding Gaussian noise.
SI- T33TWM : Additional dataset As part of another project at our lab, a pan-European reference dataset for croptype classiﬁcation is currently under development and will be made publicly available early next year.
This way, wehad the chance to use some of the data obtained from Slovenia to construct a pixel set similar to the one presentedby Garnot et al .
In order to keep it as close as possible to the original, we selected similar time steps and its appeared not be
Considering the
, we two
One with the previously stated top- classes from the dataset, called top-20- and one containing the classes from ouralternatively chosen region, analogously named top-20-S and illustrated in Figure 3b.
Since the crop cultivation in
Slovenia differs from France, several classes of the original top-20-F were not represented in the new dataset, as shownin Figure 3c.
This appears to render the classiﬁcation on top-20-F an easier problem than on top-20-S. Section 2.4 willprovide more background regarding the split of the dataset and to one of the particular difﬁculties inherent to geospatialdata, which is why we put one region aside in advance.
This way, we were able to evaluate the performance of themethod also on regionally unseen data.
A visual explanation of the train and test split is shown in
2.3.2 Cross-dataset evaluation
The ﬁrst experiment on the generalisation abilities of the described method was performed with the model being trainedon the FR- T31TFM dataset.
After the descriptions of the chosen classes were obtained from the original authors, wepicked the same classes from our SI- T33TWM dataset and ran the inference on our prepared test pixel set.
Table 1bsummarises
the overall results, showing that the performance dropped signiﬁcantly compared to the ones reportedpreviously ( cf.
Table 1a).
2.3.3 Method application to a different region
Taking advantage of the relatively short training time, it was easily possible to train the entire model on our own data.
This way, we evaluated whether the outcomes reported by Garnot et al .
could be reproduced, even without havingaccess to data at exactly the same preprocessing level.
For this purpose, we ran the experiment twice on the new
SI- T33TWM dataset, i.e. , ﬁrst with the top-20-F labels and then with the top-20-S labels.
In Table 2, the columns randomsplit show the results of these experiments.
These can directly be compared to the ones from the paper.
2.4 Experiments on the choice of the test set
Beside the issue of limited geometric resolution in S ENTINEL -2 data, the inﬂuence of spatial autocorrelation has alwaysto be taken into account when dealing with Earth observation satellite imagery (Spiker and Warner, 2007).
Due tothe coarse sampling of the Earth’s surface, adjacent pixels might share relevant amounts of information with eachother.
Therefore, using contiguous ﬁeld parcels for training as well as for testing can lead to non-representative resultsover-estimating the true performance of the classiﬁcation model.
We tried to account for these effects by reserving an
M ea do w
W i n t e r C e r ea l
S u
mm
e
r
C e r ea l
G r a p e v i n e
N on - a g r i c u lt u r a l S u rf ace
W i n t e r R a p e s ee d S oy
L e gu m i nou s F odd e r
F r u it s / V e g e t a b
l e s / F l o w e r s C e r ea l
S un ﬂ o w e r
S p
r
i ng
C e r ea l
L i gn e ou s
P r o t e i n C r op P o t a t o
S o r ghu m / M ill e t/
M oh a
O r c h a r d s
W i n t e r D u r u m W h ea t W ood
O t h e r F
[Table]
e r 0 .
[Table]
r e l a ti v e o cc u rr
France
[Table]
V
[Table]
s
F r e
[Table]
h V e
[Table]
a
[Table]
s /
M
[Table]
l on
[Table] [Table]
r a w
[Table]
rr
[Table]
e s T r
[Table]
l e
W i n t e r R a p e s ee d / T u r n i p R a p e s ee
d O a t s R y e
L e gu m i nou s P l a n t s
O t h e r C e r ea l N u t
s H op
R a p e s ee d / T u r n i p
R a p e s ee d M ill e t 0 .0 .0 .
Slovenia
M ea do w
W i n t e r C e r ea l
S u
mm
e
r
C e r ea l
G r a p e v i n e
N on - a g r i c u lt u r a l S u rf ace
W i n t e r R a p e s ee d S oy
L e gu m i nou s F odd e r
F r u it s / V e g e t a b
l e s / F l o w e r s C e r ea l
S un ﬂ o w e r
S p
r
i ng
C e r ea l
L i gn e ou s
P r o t e i n C r op P o t a t o
S o r ghu m / M ill e t/
M oh a
O r c h a r d s
W i n t e r D u r u m W h ea t W ood
O t h e r F odd e r 0 .0 .0 .
France
Slovenia test set from the training set.
The labels either represented the top 20 crop type classes found in the FR- T31TFM dataset (top-20-F) or the top 20 classes from the region of SI- T33TWM (top-20-S).
As we set the parcels from one region of Slovenia aside, we were able to evaluate the methods not only on the proposed randomly drawn, but also on a regionally separated test set.
It is necessary to recollect that with the stated top-20-F classes the entire dataset has less crop types to classify, which is illustrated in Figure 3c.
[Table]
entirely separate region, shown in Figure 2, as an additional test set.
This region was selected to approximate one-ﬁfthof the parcels and a class distribution representative for the entire SI- T33TWM dataset.
Results of this experiment areincluded into Table 2, where we compare the original random split to the new regional split.
Discussion
This section focuses on the analysis of reproducibility in general and will not justify the authors claim that their methodis the current state-of-the-art approach to solve satellite time series classiﬁcation.
We therefore split the ﬁndings ofthe study into two aspects that we tried to evaluate: On the one hand, we will discuss the insights we have gained byreproducing the methodological process itself, and on the other hand, we will elaborate our approach to reproduce thedesired results produced by the method.
3.1 Reproducibility of the method
When reimplementing the full architecture, we found that, despite having all parameters at hand, it would have beenhelpful to have access to more information concerning the data preprocessing and organisation, as well as to the adaptionof the original Transformer model, to achieve performance similar to that reported by the authors.
From our perspective, we cannot tell whether or not better P YTHON programming skills would have been beneﬁcial and if someone with moreexperience with the multi-head attention would have been able to understand and implement the method right away.
Inany case, the authors certainly developed a coherent methodology and, by providing the corresponding code alongsidewith the paper, have ensured that all interested parties can clearly follow their ideas.
As a result of our misinterpretation of the Transformer adaptation proposed by Garnot et al .
, we came across analternative approach that performed comparably well as the one from the paper, but requiring only 80 % of itsparameters.
When we asked the authors about this observation, we concluded that we all had different opinions onthe most obvious derivative of the method developed by Vaswani et al .
applied to the problem of satellite time seriesclassiﬁcation.
Upon request, the authors of the Transformer paper conﬁrmed that keeping the fully-connected layer
FC v has proven to be helpful and acknowledged the validity of our approach.
Under these circumstances, it is notstraightforwardly answerable whether the implicit claim of having the Transformer adapted in the papers’ way can besupported.
However, it can be said that the well-documented code and clean GitHub repository contributed strongly to ourunderstanding of the method and helped us answer most of our comprehension question.
Since this all is publiclyavailable, a reproduction of the presented method based on that implementation is possible.
3.2 Reproducibility of the outcomes
Besides the possibility to reproduce the methodological aspects of the original paper, we were also interested in whetherwe could achieve the results stated in the investigated paper.
By training the entire model by ourselves with the originaldataset, we faced no considerable difﬁculties using the data.
We were, in fact, able to achieve a slightly higher overallaccuracy on the original test set compared to using the model pre-trained by the authors.
However, a slight drop in performance became observable when we used an alternative yet similarly preprocesseddataset.
While we still reached an overall accuracy of over 87 % for the random split on our new and potentiallymore challenging dataset, we were not able to reach 84 % when running the inference on a regionally separated testset.
This result highlights the importance of the right choice of a representative test set, especially when using Earthobservation imagery, while still acknowledging that the presented method is potentially able to generalise to unseen andnew data.
Nevertheless, our reproduction experiments conﬁrmed the claimed validity of the approach and it is to be leftto the authors and the entire research community to investigate whether the presented model is able to compete withstate-of-the-art methods given broader and more diverse datasets.
The only issue with the presented method arose when we used one dataset for training and another one for testing.
Although we tried to preprocess our new dataset exactly the way the authors did, we were not able to obtain convincingresults.
There might be several reasons causing this issue, like an incorrect harmonisation of the dataset, other parametersfor the interpolation of the cloud-covered pixels, or assumptions about the data we unknowingly and implicitly madedifferent than the authors.
Hence, it remains interesting for us to know how exactly the data was processed.
To summarise, we support the claim that the method can successfully classify crop parcels when it was trained on datathat was acquired under the same conditions, as the data it eventually gets tested on.
Conclusion
During the proposed study of “Satellite Image Time Series Classiﬁcation with Pixel-Set Encoders and Temporal
Self-Attention” by Garnot et al .
(2020), we assessed several questions regarding different aspects of the reproducibilityof the paper.
Therefore, we ﬁrst attempted to reimplement the methodology from scratch based on the descriptionsgiven in that paper.
As this proved to be more challenging than expected and prone to misunderstandings, we proceededto evaluate the provided clean implementation in terms of an adaption of the Transformer architecture (Vaswani et al .,2017).
There, we came across a discrepancy between our understanding of the vanilla multi-head attention concept andthe one used in the paper.
Our obtained results show that, by changing the proposed adaptation in a subtle way towardsthe more basic multi-head attention, the model uses considerably fewer parameters, while still performing equally well.
When employing the authors’ implementation and dataset, we were able to reproduce the presented results straightforwardly and even on a new dataset that we speciﬁcally developed for this survey, the approach delivered meaningfulresults.
The only issues arose when the training and the test dataset did not share exactly the same properties lifting theaccurate preprocessing of the data to a crucial component of the proposed method.
In conclusion, we can state that the examined method is conclusive in itself and valid.
Our experiments speak in favourof the approach and our ﬁndings might highlight a path in which further works should proceed.
This direction ofresearch could take advantage of the dataset which we will make publicly available within the next few months.