Transfer Learning Approaches for Knowledge Discovery in Grid-based Geo-Spatiotemporal Data
Abstract
Extracting and meticulously analyzing geo-spatiotemporal features is crucial torecognize intricate underlying causes of natural events, such as ﬂoods.
Limitedevidence about hidden factors leading to climate change makes it challenging topredict regional water discharge accurately.
In addition, the explosive growth incomplex geo-spatiotemporal environment data that requires repeated learning bythe state-of-the-art neural networks for every new region emphasizes the needfor new computationally efﬁcient methods, advanced computational resources,and extensive training on a massive amount of available monitored data.
We,therefore, propose HydroDeep, an effectively reusable pretrained model to addressthis problem of transferring knowledge from one region to another by effectivelycapturing their intrinsic geo-spatiotemporal variance.
Further, we present fourtransfer learning approaches on HydroDeep for spatiotemporal interpretability thatimprove
Nash–Sutcliffe efﬁciency by 9% to 108% in new regions with a 95%reduction in time.
Introduction
Spatiotemporal variance inﬂuencing local water discharge is inevitable due to varying soil property,climate, and land usage among different regions.
Training individual hydrological models to captureregional spatiotemporal features requires extensive training and computational resources.
Theseregions, although varying in local characteristics, still follow fundamental hydrological dependencies.
Transfer learning is a well-known solution to reuse a trained model to reduce training duration for anew dataset.
Although most commonly applied in computer vision [1, 2, 3] and time-series prediction[4, 5, 6], recent works [7, 8] in other application domains show that these techniques have also beeneffective in knowledge guided neural networks that are powered by process-based (PB) mechanisms.
In hydrology, PB models that rely on domain-speciﬁc principles and mathematical formulations,although often criticized as overly complex are elegant in describing large-scale patterns [9, 10] whereknowledge of distributed state variables and physical constraints is essential.
However, understandinga system’s general organization does not provide insights into how the principal variables interactover space and time.
[11] discuss all the prevalent challenges of distributed hydrological PB models.
On the other hand, several works show an increase in deep learning (DL) applications in hydrology.
A recent survey of 129 publications ﬁnds the use of various state-of-the-art DL network architectures,namely, convolutional neural networks (CNNs), long short-term memory networks (LSTMs), andgated recurrent units (GRUs) [12].
However, limited knowledge of underlying PB mechanismsin DL algorithms makes PB models irreplaceable in studying environmental systems.
As a result,knowledge-guided deep neural networks are an active research area in hydrology [13, 14, 15] as wellas other domains [16, 17, 18].
In this paper, we present HydroDeep that couples a PB hydro-ecological model with a combinationof one-dimensional CNN and LSTM, to capture regional grid-based geo-spatiotemporal features of a watershed contributing to water discharge that inﬂuence an event of ﬂood.
The combination of
CNN and LSTM that our PB-DL based network uses showed promising results in computer vision[19], speech recognition and natural language processing [20], and other time-series analysis [21].
Inour experiments, HydroDeep outperformed the Nash–Sutcliffe efﬁciency [22] of standalone CNNsand LSTMs by 1.6% and 10.5% respectively.
We further propose a new application area of transferlearning to analyze similarities and dissimilarities in geo-spatiotemporal characteristics of watershedswhile reducing the extensive training time required in training local DL models.
Recent research inhydrology shows transfer learning has been applied in predicting water temperature in unmonitoredlakes [23] and water quality prediction system [24].
In [25] transfer learning was applied for ﬂoodprediction, similar to us.
However, they converted their time-series data to images to use transferlearning in a purely CNN architecture for time-series prediction with a reduction in computationalcost by 18% .
In this paper, we explore how transferrable geo-spatiotemporal features are when keptin their original spatiotemporal form.
Approach2.1 HydroDeep
If g i ∈ { g 1 , g 2 , . .
. , g L } is the spatial grid vector where L is the total number of grids coveringa region having spatial coordinates c g i ∈ { ( x g 1 , y g 1 ) , ( x g 2 , y g 2 ) , . . .
, ( x g L , y g L ) } , the distance tothese grids from the nearest river or water source is d g
i ∈
{ d g 1 , d g 2 , . . .
, d g L } .
On a certain day t ,the grids have precipitation measurements p g i where p g i ∈ { p g 1 , p g 2 , . .
, p g L } which are mapped totheir corresponding grid-based PB simulated runoff, r g i
where r g i ∈
{ r g 1 , r g 1 , . . .
, r g L } .
The extentto which each grid’s precipitation contributes to the river discharge depends extensively on the grid’sdistance to the nearest water source.
The distance vector is thus transformed to a distance weightvector (cid:101)
d g i such that higher distance weights are applied to p g i if the said grid is closer to a local riverthus contributing more to the regional river discharge (Appendix Section 1).
We denote weightedprecipitation vector as (cid:101) p g i
= (cid:101)
d i (cid:12) p g i .
On day t, the input vectors (cid:101) p g i ,t and r g i ,t are mapped with 62 respective daily river discharge observations D t ∈ { D 1 , D 2 , . .
, D T } where T is the total numberof daily river discharge observations.
Input vector x t can be shown as x t = f
[( (cid:101) p g i ,t ) , ( r g i ,t )] where 64 i ∈ { 1 , 2 , . . .
, L } and t ∈ { 1 , 2 , . . .
, T } .
From a multivariate time-series point of view, we denoteour inputs as X =
( x 1 , x 2 , . . .
, x T ) ∈ R n × T where x t ∈ R n and n denotes the total number of inputvariables per day which in our case is 2 L .
We want to predict the corresponding target outputs D t .
The aim is to obtain a non-linear mapping between X and D .
The motive behind the integration of
CNN and LSTM in HydroDeep lies in capturing both the spatial and temporal dependencies of awatershed.
The CNN layers help extract local geospatial features between the input variables andpass them to LSTMs to support temporal sequence prediction.
HydroDeep has an initial input layercustomizable to different input shapes (number of grids) to make our model easily transferable toother watersheds.
If t p denotes the prediction day, HydroDeep trains on inputs from ( t p − 7) to( t p − 1) , a weekly time window validated by empirical studies, to predict the output of day t p .
Acontinuous drought for days can be followed by a hurricane, leading to a ﬂood overnight, making itcrucial to include the target day t p ’s inputs as the second input to our network, which is concatenatedwith the last LSTM layer’s output, and are processed collectively by a fully connected layer.
We usedata from Jan 1, 2000, to Dec 31, 2011, for training and Jan 1, 2012, to Dec 31, 2016, for evaluation.
We optimized the hyperparameters of HydroDeep by random search. (Appendix Section 2)
2.2 Transfer Learning
An event of ﬂood or drought is heavily dependent on environmental drivers.
Likewise, each of thesedrivers’ inherent local spatiotemporal patterns is bound to be unique based on their geographicallocation.
Consequently, one model that has learned the local spatiotemporal patterns of a regionwill fail to perform accurately for a geographically distant region with different characteristics.
Asa result, the model should be retrained perpetually for every new unique region.
Alternatively, aglobal model can be trained on a larger area covering many watersheds to address this problem, butit will fail to capture the local patterns.
Besides, both these methods are expensive as they requiremore training time and computational power to train such a model of global extent.
Therefore, weuse transfer learning to reuse HydroDeep’s knowledge from one region to another.
More formally,transfer learning consists of a domain (cid:101) D and a task (cid:101) T where the domain (cid:101)
D is the marginal probabilitydistribution P ( X ) over an input feature space X
= { x 1 , x 2 , . . .
, x N } where N is the total numberof input features.
Given a domain (cid:101) D
= { X, P ( X ) } , a task (cid:101) T consists of a conditional probabilitydistribution P ( Y | X ) over a label space Y .
The conditional probability distribution is usually learned from the pairs { x i , y i } in the training samples where x i ∈ X and y i ∈ Y .
Suppose there is asource domain (cid:101)
D source with a source task (cid:101) T source and a target domain (cid:101) D target with a target task(cid:101)
T target , through transfer learning we try to learn the target conditional probability distribution
P ( Y target | X target ) in (cid:101) D target , from the knowledge learned from (cid:101) D source and (cid:101) T target [26].
Experimental Design
3.1 Dataset and the Process-based Model - DLEM
For experiments, we use a PB hydro-ecological model named the Dynamic Land Ecosystem Model(DLEM) that mimics the plant physiological, biogeochemical, and hydrological processes in theplant-soil-water-river continuum [27, 28].
In DLEM, the design of grid-to-grid connection trackssigniﬁcant features of a region, including within-grid heterogeneity, grid-to-grid ﬂow, and landaquatic linkage.
DLEM models the water movement from land to aquatic systems at a daily time stepwith each grid cell comprising multiple land cover types, rivers, and lakes with their area percentageprescribed by land-use history data [29].
Research shows that DLEM has been extensively validatedagainst measurements from LTER, NEON, Ameri Flux, USDA crop yield survey, and USGS gaugemonitoring and are widely used to quantify the spatiotemporal variations in the pool and ﬂuxes ofwater, carbon, and nitrogen coupling (water-C-N) at the site, and regional scales [27, 28, 30, 31, 32].
The preliminary results from DLEM at the outlet of the Mississippi and Atchafalaya river basin(MARB) show that the variations of daily river discharge are very close to the USGS observed riverdischarge over the years [29].
We pulled dataset from the U.S. Geological Survey’s (USGS) daily discharge measurements for
Iowa Streams having daily discharge measurements of 23 watersheds covering the state of Iowa[33].
Alongside, we use daily precipitation at a 5-arc-min resolution generated from high-resolutiongridded meteorological data products from station observations by the Climatic Research Unit (CRU)of the University of East Anglia [34], and North America Regional Reanalysis (NARR) datasetfrom a combination of modeled and observed data [35].
We also use DLEM-simulated surface andsubsurface runoff to guide our DL network [29].
Climate, land management, and environmentaldrivers steer DLEM simulation and are used to represent our “best estimate” of land-to-aquaticsurface and subsurface runoff across the watershed.
For our experiment, we chose the Thompson
Fork Grand River basin at David City (watershed 13) as our source domain due to its smaller size andtransferred its knowledge to 5 target domains - West Nodaway River near Shambaugh (watershed14), East Nishnabotna River near Shenandoah (watershed 15), Turkey River near Garber (watershed4), South Skunk River near Oskaloosa (watershed 10), and Rock River near Hawarden (watershed23) [33].
Watershed 13 (w13) has 29 grids with each grid having two input time-series signalscorresponding to precipitation and DLEM runoff.
Daily river discharge observations are given aslabels to all 29 grids.
3.2 Regional Knowledge Transfer for Spatiotemporal Analysis
We have experimented with four transfer learning approaches to reuse HydroDeep’s knowledge inpredicting river discharge in distant watersheds (Figure 1).
The goal was to ﬁnd the best approach totransfer HydroDeep’s grid-based spatiotemporal knowledge from one watershed to another to reducethe required training iterations on a new region and interpret the geo-spatiotemporal similaritiesand dissimilarities between the source and the target.
In the ﬁrst approach, we preserve the original
HydroDeep’s spatiotemporal knowledge from the source w13 and use it solely in T-HydroDeep1 (T-HD-1) to test the new target watersheds.
Secondly, we transfer the original HydroDeep’sspatiotemporal knowledge in T-HydroDeep-2 (T-HD-2) and allow it to ﬁnetune on the target.
In thethird and fourth approaches, we take turns in ﬁnetuning just the temporal features in T-HydroDeep-3(T-HD-3) and the spatial features in T-HydroDeep-4 (T-HD-4).
The CNN layers and the LSTMlayers are responsible for learning spatial and temporal features, respectively.
When we ﬁnetune onekind of layer, we freeze the other to keep the originally learned features intact.
Table 1 shows theobservations from our ablation study.
Results
The watersheds w14 and w15, although being adjacent to the source w13 [33], are observed to havedistinct spatial features as T-HD-4 proved to be the best approach for both the watersheds.
Wealso observed that the second-best approach for w14 is T-HD-1, which means w13 and w14 havesimilar temporal features, but since they vary in spatial features, only the spatial layers needed to be ﬁnetuned (T-HD-4) to improve prediction accuracy.
Similarly, the result shows that w15 hasmore distinct spatial features than temporal features from w13, as T-HD-2 shows the second-bestperformance, just behind T-HD-4.
However, w4 and w10, both being far from w13, show distinctspatiotemporal features as T-HD-2 shows the best performance.
Note that w13 has only 29 spatialgrids, whereas both w4 and w10 have 61 and 65 grids, respectively.
HydroDeep’s knowledge istransferred to targets almost double the area of the source and still achieved the best performancesamong all the watersheds included in our experiment.
This supports our arguments that more dataavailability will increase HydroDeep’s performance; transfer learning approaches work in targetslarger in area than the source and can be used to analyze watersheds’ spatiotemporal characteristics.
The original HydroDeep was pretrained for 300 iterations on w13 in ∼ 3138 seconds to achieve 0.
NSE outperforming the baseline neural network architectures - CNN by 1 . 6% , LSTM by 10 .
GRU by 26% , and Bi-directional LSTM by 53 . 6% that are commonly used for hydrological modeling(Appendix Section 3).
In contrast, the transferred models on the new targets (w14, w15, w4, w10,and w23) achieved signiﬁcant performance in just 20 training iterations in ∼ 155 seconds, a ∼ 95%reduction in time.
In Table 1, training HydroDeep (HD) from scratch in each target watershed remainslow when run for 20 iterations.
Thereby, using transfer learning, the prediction performance on theindividual watersheds increased by 44% in w14, 108% in w15, 15% in w4, 9% in w10, and 39% inw23 in terms of NSE. where HydroDeep was pretrained on w13 with 29 grids before being transferred to other watersheds of varying distance and size. (Appendix Section 4)
[Table]
Conclusion and Future Work
This paper illustrates a new application of transfer learning techniques in interpreting geospatiotemporal characteristics of watersheds with limited computational resources and a 95% reduction in time.
We believe that a smaller grid-scale resolution will help HydroDeep to capture localfeatures on a ﬁner scale.
In the future, we will run our experiments on more watersheds to betterquantify the performance of HydroDeep and its variants.
Further, we plan to perform extensiveresearch on how to select the source watershed(s) more effectively.
Evaluation of HydroDeep’s Performance
(Related to Section 2.1, 4)
(Related to Section 4)
Hardware Specification
1. Transformation of distance into spatial weights for precipitation
The distance between each grid and its nearest local river is calculated by the Euclidean distance tool of ArcMap 10.5.
These distances add weights to the grid based precipitation measurements such as the farthest grid contributes the lowest weight to the local river and finally into the river outlet where river drainage discharge has been recorded.
The raw distances (Supplementary Fig 1a.) were first transformed, so that the closest grids to the local rivers having the shortest distances is at 0, and the farthest distances lie at negative (Supplementary Fig 1b.).
The transformed distances are then scaled between 0 and 1, which means the shortest distance is brought to 1 having the highest contributing weight, whereas the highest distance at 0 will have the lowest contributing weight to the concerned local river grid (Supplementary Fig 1c.).
Supplementary
Watershed 13 has 29 grids.
The distance is measured in geographic degree.
(Related to Section 2.1)
Supplementary
2. Hyperparameter Tuning for HydroDeep 2.1. Learning Rate
Choosing an optimal learning rate is challenging since a higher learning rate can make the network stuck in a sub-optimal set of weights, whereas a lower learning rate may result in a longer training process.
We did an extensive random search in finding the optimal learning rate for HydroDeep.
[Table]
2.2. Time-window
Out of all the hyperparameters, HydroDeep was most susceptible to the lookback (time-window) parameter.
In the first iteration, the lookback window sets itself from the start day of the training sample-set and then slides over with a step value of 1 day per iteration for the following iterations.
We experimented with lookback values varying from 3 to 11 days and observed that increasing the number of days did not consistently improve the performance.
The value was set manually before each run to observe the NSE performance (Supplementary Figure 2a).
From our analysis, a lookback period of 7 days in HydroDeep has achieved the best performance in NSE throughout.
3. Evaluation of HydroDeep’s performance
After the training is complete, Hydrodeep's performance is evaluated on the test dataset (Supplementary Figure 3a).
Supplementary Figure 3b shows an arbitrarily chosen window ofmonths from 2013 in w13.
We compare the prediction power of the three approaches, DL which represents a similar neural network architecture as HydroDeep but without DLEM simulated runoff as input, standalone DLEM, and our proposed HydroDeep (with DLEM), against the observations to evaluate where they stand in comparison.
According to our results, Hydrodeep performs better than its counterparts.
Supplementary Figure 3b .
We present a closer look at a period from February 20, 2013 to May 20,in watershed 13 where HydroDeep, with the use of DLEM, outperforms the other models.
DL (without
DLEM) achieves an NSE of 0.49 compared to HydroDeep's 0.63.
[Table]
HydroDeep on w13 achieves an NSE of 0.63 outperforming the baseline neural network architecturesCNN by 1.6%, LSTM by 10.5%, GRU by 26%, and Bi-directional LSTM by 53.6% that are commonly used for hydrological modeling (Supplementary Table 2).
We use DLEM (PB) simulated runoff and similar preprocessing of all the grid-based input vectors for a fair assessment of the benchmark architectures.
T-HydroDeep-1,
T-HydroDeep-2, T-HydroDeep-3, T-HydroDeep-4, with HydroDeep (with no pretraining on source watershed 13).
(Related to Section 4)
[Table]
[Table]
[Table]
[Table]
[Table]
HydroDeep-
[Table]
[Table]
[Table]
5. Hardware Specification
The extensive search to optimize hyperparameters for HydroDeep on the source watershed was done on the following hardware specification.
No. of
Nodes
Processors/N ode
Cores/N ode
Memory/
Node
Interconn ect
Local $TMPDI
R Disk
Accelerator
Card
CPU-H
our
Cost
Factor
Two 18-Core
Intel Skylake  192 GB 100G IB 1.5 TB 1 NVIDIA
Tesla
V100-32GB 2.
The final training time for optimized HydroDeep and its variants T-HydroDeep-1,
T-HydroDeep-2, T-HydroDeep-3, T-HydroDeep-4 were measured on Intel® Core™ i5-7200U
CPU @
2.50GHz × 4 with Intel® HD Graphics 620 (KBL GT2) and 12 GB RAM