Transfer Learning Approaches for Knowledge Discovery in Grid-based Geo-Spatiotemporal Data
Abstract
Extracting and meticulously analyzing geo-spatiotemporal features is crucial to recognize intricate underlying causes of natural events, such as ﬂoods.
Limited evidence about hidden factors leading to climate change makes it challenging to predict regional water discharge accurately.
In addition, the explosive growth in complex geo-spatiotemporal environment data that requires repeated learning by the state-of-the-art neural networks for every new region emphasizes the need for new computationally efﬁcient methods, advanced computational resources, and extensive training on a massive amount of available monitored data.
We, therefore, propose HydroDeep, an effectively reusable pretrained model to address this problem of transferring knowledge from one region to another by effectively capturing their intrinsic geo-spatiotemporal variance.
Further, we present four transfer learning approaches on HydroDeep for spatiotemporal interpretability that improve Nash–Sutcliffe efﬁciency by 9% to 108% in new regions with a 95% reduction in time.
Introduction
Spatiotemporal variance inﬂuencing local water discharge is inevitable due to varying soil property, climate, and land usage among different regions.
Training individual hydrological models to capture regional spatiotemporal features requires extensive training and computational resources.
These regions, although varying in local characteristics, still follow fundamental hydrological dependencies.
Transfer learning is a well-known solution to reuse a trained model to reduce training duration for a new dataset.
Although most commonly applied in computer vision [1, 2, 3] and time-series prediction [4, 5, 6], recent works [7, 8] in other application domains show that these techniques have also been effective in knowledge guided neural networks that are powered by process-based (PB) mechanisms.
In hydrology, PB models that rely on domain-speciﬁc principles and mathematical formulations, although often criticized as overly complex are elegant in describing large-scale patterns [9, 10] where knowledge of distributed state variables and physical constraints is essential.
However, understanding a system’s general organization does not provide insights into how the principal variables interact over space and time.
[11] discuss all the prevalent challenges of distributed hydrological PB models.
On the other hand, several works show an increase in deep learning (DL) applications in hydrology.
A recent survey of 129 publications ﬁnds the use of various state-of-the-art DL network architectures, namely, convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and gated recurrent units (GRUs) [12].
However, limited knowledge of underlying PB mechanisms in DL algorithms makes PB models irreplaceable in studying environmental systems.
As a result, knowledge-guided deep neural networks are an active research area in hydrology [13, 14, 15] as well as other domains [16, 17, 18].
In this paper, we present HydroDeep that couples a PB hydro-ecological model with a combination of one-dimensional CNN and LSTM, to capture regional grid-based geo-spatiotemporal features of a watershed contributing to water discharge that inﬂuence an event of ﬂood.
The combination of
CNN and LSTM that our PB-DL based network uses showed promising results in computer vision [19], speech recognition and natural language processing [20], and other time-series analysis [21].
In our experiments, HydroDeep outperformed the Nash–Sutcliffe efﬁciency [22] of standalone CNNs and LSTMs by 1.6% and 10.5% respectively.
We further propose a new application area of transfer learning to analyze similarities and dissimilarities in geo-spatiotemporal characteristics of watersheds while reducing the extensive training time required in training local DL models.
Recent research in hydrology shows transfer learning has been applied in predicting water temperature in unmonitored lakes [23] and water quality prediction system [24].
In [25] transfer learning was applied for ﬂood prediction, similar to us.
However, they converted their time-series data to images to use transfer learning in a purely CNN architecture for time-series prediction with a reduction in computational cost by 18% .
In this paper, we explore how transferrable geo-spatiotemporal features are when kept in their original spatiotemporal form.
Approach 2.1 HydroDeep
If g i ∈ { g 1 , g 2 , . .
. , g L } is the spatial grid vector where L is the total number of grids covering a region having spatial coordinates c g i ∈ { ( x g 1 , y g 1 ) , ( x g 2 , y g 2 ) , . . .
, ( x g L , y g L ) } , the distance to these grids from the nearest river or water source is d g
i ∈
{ d g 1 , d g 2 , . . .
, d g L } .
On a certain day t , the grids have precipitation measurements p g i where p g i ∈ { p g 1 , p g 2 , . .
, p g L } which are mapped to their corresponding grid-based PB simulated runoff, r g i
where r g i ∈
{ r g 1 , r g 1 , . . .
, r g L } .
The extent to which each grid’s precipitation contributes to the river discharge depends extensively on the grid’s distance to the nearest water source.
The distance vector is thus transformed to a distance weight vector (cid:101)
d g i such that higher distance weights are applied to p g i if the said grid is closer to a local river thus contributing more to the regional river discharge (Appendix Section 1).
We denote weighted precipitation vector as (cid:101) p g i
= (cid:101)
d i (cid:12) p g i .
On day t, the input vectors (cid:101) p g i ,t and r g i ,t are mapped with respective daily river discharge observations D t ∈ { D 1 , D 2 , . .
, D T } where T is the total number of daily river discharge observations.
Input vector x t can be shown as x t = f
[( (cid:101) p g i ,t ) , ( r g i ,t )] where i ∈ { 1 , 2 , . . .
, L } and t ∈ { 1 , 2 , . . .
, T } .
From a multivariate time-series point of view, we denote our inputs as X =
( x 1 , x 2 , . . .
, x T ) ∈ R n × T where x t ∈ R n and n denotes the total number of input variables per day which in our case is 2 L .
We want to predict the corresponding target outputs D t .
The aim is to obtain a non-linear mapping between X and D .
The motive behind the integration of
CNN and LSTM in HydroDeep lies in capturing both the spatial and temporal dependencies of a watershed.
The CNN layers help extract local geospatial features between the input variables and pass them to LSTMs to support temporal sequence prediction.
HydroDeep has an initial input layer customizable to different input shapes (number of grids) to make our model easily transferable to other watersheds.
If t p denotes the prediction day, HydroDeep trains on inputs from ( t p − 7) to ( t p − 1) , a weekly time window validated by empirical studies, to predict the output of day t p .
A continuous drought for days can be followed by a hurricane, leading to a ﬂood overnight, making it crucial to include the target day t p ’s inputs as the second input to our network, which is concatenated with the last LSTM layer’s output, and are processed collectively by a fully connected layer.
We use data from Jan 1, 2000, to Dec 31, 2011, for training and Jan 1, 2012, to Dec 31, 2016, for evaluation.
We optimized the hyperparameters of HydroDeep by random search.
(Appendix Section 2)
2.2 Transfer Learning
An event of ﬂood or drought is heavily dependent on environmental drivers.
Likewise, each of these drivers’ inherent local spatiotemporal patterns is bound to be unique based on their geographical location.
Consequently, one model that has learned the local spatiotemporal patterns of a region will fail to perform accurately for a geographically distant region with different characteristics.
As a result, the model should be retrained perpetually for every new unique region.
Alternatively, a global model can be trained on a larger area covering many watersheds to address this problem, but it will fail to capture the local patterns.
Besides, both these methods are expensive as they require more training time and computational power to train such a model of global extent.
Therefore, we use transfer learning to reuse HydroDeep’s knowledge from one region to another.
More formally, transfer learning consists of a domain (cid:101) D and a task (cid:101) T where the domain (cid:101)
D is the marginal probability distribution P ( X ) over an input feature space X
= { x 1 , x 2 , . . .
, x N } where N is the total number of input features.
Given a domain (cid:101) D
= { X, P ( X ) } , a task (cid:101) T consists of a conditional probability distribution P ( Y | X ) over a label space Y .
The conditional probability distribution is usually learned from the pairs { x i , y i } in the training samples where x i ∈ X and
y i ∈ Y .
Suppose there is a source domain (cid:101) D source with a source task (cid:101) T source and a target domain (cid:101) D target with a target task (cid:101) T target , through transfer learning we try to learn the target conditional probability distribution
P ( Y target | X target ) in (cid:101) D target , from the knowledge learned from (cid:101) D source and (cid:101) T target [26].
Experimental Design 3.1 Dataset and the Process-based Model - DLEM
For experiments, we use a PB hydro-ecological model named the Dynamic Land Ecosystem Model (DLEM) that mimics the plant physiological, biogeochemical, and hydrological processes in the plant-soil-water-river continuum [27, 28].
In DLEM, the design of grid-to-grid connection tracks signiﬁcant features of a region, including within-grid heterogeneity, grid-to-grid ﬂow, and landaquatic linkage.
DLEM models the water movement from land to aquatic systems at a daily time step with each grid cell comprising multiple land cover types, rivers, and lakes with their area percentage prescribed by land-use history data [29].
Research shows that DLEM has been extensively validated against measurements from LTER, NEON, Ameri Flux, USDA crop yield survey, and USGS gauge monitoring and are widely used to quantify the spatiotemporal variations in the pool and ﬂuxes of water, carbon, and nitrogen coupling (water-C-N) at the site, and regional scales [27, 28, 30, 31, 32].
The preliminary results from DLEM at the outlet of the Mississippi and Atchafalaya river basin (MARB) show that the variations of daily river discharge are very close to the USGS observed river discharge over the years [29].
We pulled dataset from the U.S. Geological Survey’s (USGS) daily discharge measurements for
Iowa Streams having daily discharge measurements of 23 watersheds covering the state of Iowa [33].
Alongside, we use daily precipitation at a 5-arc-min resolution generated from high-resolution gridded meteorological data products from station observations by the Climatic Research Unit (CRU) of the University of East Anglia [34], and North America Regional Reanalysis (NARR) dataset from a combination of modeled and observed data [35].
We also use DLEM-simulated surface and subsurface runoff to guide our DL network [29].
Climate, land management, and environmental drivers steer DLEM simulation and are used to represent our “best estimate” of land-to-aquatic surface and subsurface runoff across the watershed.
For our experiment, we chose the Thompson
Fork Grand River basin at David City (watershed 13) as our source domain due to its smaller size and transferred its knowledge to 5 target domains - West Nodaway River near Shambaugh (watershed 14), East Nishnabotna River near Shenandoah (watershed 15), Turkey River near Garber (watershed 4), South Skunk River near Oskaloosa (watershed 10), and Rock River near Hawarden (watershed 23) [33].
Watershed 13 (w13) has 29 grids with each grid having two input time-series signals corresponding to precipitation and DLEM runoff.
Daily river discharge observations are given as labels to all 29 grids.
3.2 Regional Knowledge Transfer for Spatiotemporal Analysis
We have experimented with four transfer learning approaches to reuse HydroDeep’s knowledge in predicting river discharge in distant watersheds (Figure 1).
The goal was to ﬁnd the best approach to transfer HydroDeep’s grid-based spatiotemporal knowledge from one watershed to another to reduce the required training iterations on a new region and interpret the geo-spatiotemporal similarities and dissimilarities between the source and the target.
In the ﬁrst approach, we preserve the original
HydroDeep’s spatiotemporal knowledge from the source w13 and use it solely in T-HydroDeep1 (T-HD-1) to test the new target watersheds.
Secondly, we transfer the original HydroDeep’s spatiotemporal knowledge in T-HydroDeep-2 (T-HD-2) and allow it to ﬁnetune on the target.
In the third and fourth approaches, we take turns in ﬁnetuning just the temporal features in T-HydroDeep- (T-HD-3) and the spatial features in T-HydroDeep-4 (T-HD-4).
The CNN layers and the LSTM layers are responsible for learning spatial and temporal features, respectively.
When we ﬁnetune one kind of layer, we freeze the other to keep the originally learned features intact.
Table 1 shows the observations from our ablation study.
Results
The watersheds w14 and w15, although being adjacent to the source w13 [33], are observed to have distinct spatial features as T-HD-4 proved to be the best approach for both the watersheds.
We also observed that the second-best approach for w14 is T-HD-1, which means w13 and w14 have similar temporal features, but since they vary in spatial features, only the spatial layers needed to be ﬁnetuned (T-HD-4) to improve prediction accuracy.
Similarly, the result shows that w15 has more distinct spatial features than temporal features from w13, as T-HD-2 shows the second-best performance, just behind T-HD-4.
However, w4 and w10, both being far from w13, show distinct spatiotemporal features as T-HD-2 shows the best performance.
Note that w13 has only 29 spatial grids, whereas both w4 and w10 have 61 and 65 grids, respectively.
HydroDeep’s knowledge is transferred to targets almost double the area of the source and still achieved the best performances among all the watersheds included in our experiment.
This supports our arguments that more data availability will increase HydroDeep’s performance; transfer learning approaches work in targets larger in area than the source and can be used to analyze watersheds’ spatiotemporal characteristics.
The original HydroDeep was pretrained for 300 iterations on w13 in ∼ 3138 seconds to achieve [-0.
-]0.
NSE outperforming the baseline neural network architectures - CNN by 1 . 6% , LSTM by 10 .
GRU by 26% , and Bi-directional LSTM by 53 . 6% that are commonly used for hydrological modeling (Appendix Section 3).
In contrast, the transferred models on the new targets (w14, w15, w4, w10, and w23) achieved signiﬁcant performance in just 20 training iterations in ∼ 155 seconds, a ∼ 95% reduction in time.
In Table 1, training HydroDeep (HD) from scratch in each target watershed remains low when run for 20 iterations.
Thereby, using transfer learning, the prediction performance on the individual watersheds increased by 44% in w14, 108% in w15, 15% in w4, 9% in w10, and 39% in w23 in terms of NSE.
[Table]
Conclusion and Future Work
This paper illustrates a new application of transfer learning techniques in interpreting geospatiotemporal characteristics of watersheds with limited computational resources and a 95% reduction in time.
We believe that a smaller grid-scale resolution will help HydroDeep to capture local features on a ﬁner scale.
In the future, we will run our experiments on more watersheds to better quantify the performance of HydroDeep and its variants.
Further, we plan to perform extensive research on how to select the source watershed(s) more effectively.