A UTOMATICALLY I NFERRING D ATA Q UALITY FOR S PATIOTEMPORAL F ORECASTING
Anonymous authors
Paper under double-blind review
A BSTRACT
The importance of learning from spatiotemporal data has been growing with the increasing number of data sources distributed over space.
While numerous studies have been done for analyzing the spatiotemporal signals, existing models have assumed that heterogeneous data sources in spatial domain are equally reliable over time.
In this paper, we propose the novel method that estimates the time-varying data quality levels based on the local variations of spatiotemporal signals without explicitly assigned labels.
Furthermore, we integrate the formulation of data quality level with graph convolutional networks to exploit their efﬁcient architecture.
Finally, we evaluate the proposed method by simulating a forecasting task with real-world climate data.
I NTRODUCTION
Recent advances with GPS-equipped technology have facilitated the collection of large spatiotemporal datasets.
As the number of spatiotemporal data increases, many have proposed representing this data as time-varying graph signals in various domains, such as sensor networks (Shi et al., 2015; Zhu & Rabbat, 2012), climate analysis (Chen et al., 2014; Mei & Moura, 2015), and trafﬁc control systems (Li et al., 2017; Yu et al., 2017).
Furthermore, the study of time-varying graph signals has been applied to biology (Mutlu et al., 2012; Yu et al., 2015).
While many works have been proposed to exploit the spatial structures and temporal signals , most models have implicitly assumed that each signal source node in a structure is equally reliable over time, and without considering the data quality of different sources.
However, a large amount of data is from heterogeneous sources that have different levels of noise (Song et al., 2015; Zhang & Chaudhuri, 2015).
Moreover, the noises of each source can be time-varying due to abrupt malfunction of sensors.
In order to fully consider the varying credibility of every source over time, it is required to give a different estimate for data quality level.
However, relying on domain experts to manually assign such labels is laborious and expensive. Furthermore, it is extremely challenging to build a model that can differentiate between high and low-quality sources in an unsupervised setting.
In this paper, we consider the learning problem of spatiotemporal signals which can be represented by time-varying graph signals with different data qualities.
Given a graph G =
( V , E , W ) and observations X ∈ R N × M × T where N, M, T are the number of vertices, the types of signals, and the length of time-varying signals, respectively, we propose a model that is able to exploit structural information (spatial features) as well as the temporal behaviors (time series).
On top of the timevarying graph signals, we implement graph-based neural networks which encode the level of data quality at each vertex into a scalar by looking at neighbor signals observed at the set of K -hop neighbor vertices.
Once the quality level for each vertex is extracted from the data, we use the variable as a weight of a loss function that is dependent on a type of task, classiﬁcation or regression, on the vertex.
A summary of the key contributions of the paper are the following.
First, we deﬁne the concept of a data quality level at each vertex in a graph using the local variation of the vertex.
The local variation at each vertex depends on local structure as well as neighboring signals.
We assume that the data quality level at a vertex is a function of the local variation at the vertex, and the parameters of the function can be trained in a way similar to the training of layer-wise graph convolutional networks (GCNs) (Kipf & Welling, 2017).
Secondly, we combine the quality level with an objective function deﬁned by a given task.
In this setting, the quality level behaves as a regularizer constant in the objective function.
Finally, we demonstrate that the data quality is an essential factor which can improve the performance of given tasks, such as time-series forecasting, on a number of datasets.
Related work The ﬁrst set of work related to our study is about learning and processing graph signals or features.
Spectral graph theory (Chung, 1997; Von Luxburg, 2007; Shi et al., 2015) has been developed as a main study to understand two aspects of graph signals: structures and signals.
Under this theory many works have been introduced to exploit convolutional neural networks (CNNs) which provide an efﬁcient architecture to extract localized features or patterns from regular grids, such as images (Krizhevsky et al., 2012).
Bruna et al.
(2014) introduce how to learn convolutional parameters based on the spectrum of the graph Laplacian.
Later, Henaff et al.
(2015) extend the spectral aspect of CNNs on graphs into large-scale learning problems.
From the pioneer works about graph convolutional networks (GCNs), Defferrard et al.
(2016) propose a spectral formulation for fast localized ﬁltering with efﬁcient pooling.
Furthermore, Kipf & Welling (2017) re-formularize existing ideas into layer-wise neural networks that can be tuned through backpropagation rule with a ﬁrst-order approximation of spectral ﬁlters introduced in Hammond et al.
On top of these studies, we propose a graph convolutional layer that maps spatiotemporal features into a data quality level.
There are a number of works about considering data qualities and handling heterogeneous data sources.
Urner et al.
(2012) is the ﬁrst theoretical work that proposes a mixture of two types of labels for supervised learning.
One type of the labels is considered as high quality labels from an expensive source (domain experts) while another type is from error-prone crowdsourcing.
Since the reliability or quality of the labels is different, it is not desired to consider them equally.
They propose a learning algorithm that is able to utilize the error-prone labels to reduce the cost required for the expert labeling.
Zhang & Chaudhuri (2015) address issues from strong and weak labelers by developing an active learning algorithm minimizing the number of label requests.
Song et al.
(2015) focus on the data of variable quality resulting from heterogeneous sources.
They deﬁne the concept of heterogeneity of data and develop a method of adjusting the learning rate based on the heterogeneity.
Unlike these existing works, we provide a method to differentiate heterogeneous sources based on neighborhood signals without any explicit labels.
Outline We review graph signal processing to deﬁne the local variation and a data quality level (DQL) with graph convolutional networks in Section 2.
In Section 3, we provide how the data quality levels are exploited with recurrent neural networks to differentiate reliability of observations on vertices.
Also, we construct a forecasting model, DQ-LSTM.
Our main result is presented in
Section 4 with other baselines.
In Section 5 we discuss its properties and interpret the data reliability inferred from our model.
We then conclude with Section 6.
P RELIMINARIES
We ﬁrst show how to deﬁne the local variation at a vertex based on graph signals.
Then, we explain how the variational features at each vertex can be used to generate a data quality level.
2. L OCAL V ARIATION
We focus on the graph signals deﬁned on an undirected, weighted graph G =
( V , E , W ) , where V is a set of vertices with |V| = N and E is a set of edges.
W ∈ R N × N is a random-walk normalized weighted adjacency matrix which provides how two vertices are relatively close.
When the elements
W ij are not be expliticly provided by dataset, the graph connectivity can be constructed by various distance metrics, such as Euclidean distance, cosine similarity, and a Gaussian kernel (Belkin &
Niyogi, 2002), on the vertex features V ∈ R N × F where F is the number of the features.
Once all the structural connectivity is provided, the local variation can be deﬁned by the edge derivative of a given graph signal x ∈ R N deﬁned on every vertex (Zhou & Sch¨olkopf, 2004). ∂
[Equation]
[Figure]
where e =
( i, j ) is deﬁned as a direction of the derivative and x ( i ) is a signal value on the vertex i .
The graph gradient of x at vertex i can be deﬁned by Eq.
1 over all edges joining the vertex i .
[Equation]
where N i is a set of neighbor vertices of the vertex i .
While the dimesion of the graph gradient is different due to the different number of neighbors of each vertex, the local variation at vertex i can be deﬁned by a norm of the graph gradient:
[Equation]
Eq.
3 provides a measure of local variation of x at vertex i .
As it indicates, if all neighboring signals of i are close to the signal at i , the local variation at i should be small and it means less ﬂuctuated signals around the vertex.
As Eq.
3 shows, the local variation is a function of a structural connectivity W and graph signals x .
Figure 1 illustrates how the two factors affect the local variation at the same vertex.
The concept of the local variation is easily generalized to multivariate graph signals of M different measures by repeatedly computing Eq.
3 over all measures.
[Equation]
where x m ∈ R N corresponds the m th signals from multiple sensors.
As Eq.
4 indicates, L i is a
M dimensional vector describing local variations at the vertex i with respect to the M different measures.
Finally, it is desired to represent Eq.
4 in a matrix form to be combined with graph convolutional networks later.
[Equation]
where D is a degree matrix deﬁned as D ii = (cid:80) j W ij and (cid:12) is an element-wise product operator.
X is a N × M matrix describing multivariate graph signals on N vertices and x m is a m th column of
X .
L ∈ R N × M is a local variation matrix and L im is the local variation at the vertex i with respect to the m th signal.
2. D ATA Q UALITY L EVEL
While the term of data quality has been used in various ways, it generally means “ ﬁtness of use ” for intended purposes (Juran & Godfrey, 1999).
In this section, we will deﬁne the term under the data property we are interested in and propose how to exploit the data quality level into a general framework.
Given a multivariate graph signal X ∈ R N × M on vertices represented by a feature matrix V ∈ R N × F , we assume that a signal value at a certain vertex i is desired not be signiﬁcantly different with signals of neighboring vertices j ∈ N i .
This is a valid assumption if the value at the vertex i is dependent on (or function of) features of the vertex i when an edge weight is deﬁned by a distance in the feature vector space between two vertices.
In other word, if two vertices have similar features, they are connected to form the graph structure and the signal values observed at the vertices are highly likely similar.
There are a lot of domains which follow the assumption, for instance, geographical features (vertex features) and meteorological observations (graph signal) or sensory nervous features and received signals.
Under the assumption, we deﬁne the data quality at a vertex i as a function of local variations of i :
[Equation]
It is ﬂexible to choose the function q .
For example, w i can be deﬁned as an average of L i .
If so, all measures are equally considered to compute the data quality at vertex i .
In more general sense, we can introduce parameterized function q ( L i ; Φ ) and learn the parameters through data.
Kipf & Welling (2017) propose a method that learns parameters for graph-based features by the layer-wise graph convolutional networks (GCNs) with arbitrary activation functions.
For a single layer GCN on a graph, the latent representation of each node can be represented as:
[Equation]
where ˆ A provides structural connectivities of a graph and Θ is a trainable parameter matrix.
σ is an activation function.
By stacking σ ( ˆ AX Θ ) , it is able to achieve larger receptive ﬁelds with multi-layer GCNs.
See details in Kipf & Welling (2017).
We can replace ˆ AX with L which is also a function of the weighted adjacency W and the graph signal X .
Note that values in row i of ˆ AX and L are a function of values at i as well as neighbors of i .
Although L only exploits nearest neighbors (i.e., 1-hop neighbors), it is possible to consider
K -hop neighbors to compute the local variations by stacking GCN before applying Eq.
The generalized formula for the data quality level can be represented as:
[Equation]
where K is the number of GCN layers and w = ( w 1 , w 2 , · · · , w N ) is the data quality level of each vertex incorporating K -hop neighbors.
We propose some constraints to ensure that a higher w i corresponds to less ﬂuctuated around i .
First, we constrain Φ to be positive to guarantee that larger elements in L cause larger L Φ that are inputs of σ L .
Next, we use an activation function that is inversely proportional to an input, e.g., σ L ( x ) = 11+ x , to meet the relation between the data quality w i and the local variations L i .
Once w is obtained, it will be combined with an objective function to assign a penalty for each vertex loss function.
M ODEL
In this section, we give the details of the proposed model, which is able to exploit the data quality deﬁned in Section 2.2 for practical tasks.
First, it will be demonstrated how the data quality network
DQN is combined with recurrent neural networks (LSTM) to handle time-varying signals.
We, then, provide how this model is trained over all graph signals from all vertices.
Data quality network In Section 2.2, we ﬁnd that the local variations around all vertices can be computed once graph signals X are given.
Using the local variations matrix L , the data quality level at each vertex w i can be represented as a function of L i with parameters Φ (See Eq. 6).
While the function q is not explicitly provided, we can parameterize the function and learn the parameters Φ ∈ R M through a given dataset.
One of straightforward parameterizations is based on fully connected neural networks.
Given a set of features, neural networks are efﬁcient to ﬁnd nonlinear relations among the features.
Furthermore, the parameters in the neural networks can be easily learned through optimizing a loss function deﬁned for own purpose.
Thus, we use a single layer neural networks Φ followed by an activation function σ L to transfer the local variations to the data quality level.
Note that multi-layer GCNs can be used between graph signals and DQN to extract convolved signals as well as increase the size of convolution
[Figure]
Long short term memory Recurrent neural networks (RNNs) are especially powerful to extract latent patterns in time series which has inherent dependencies between not only adjacent data points but also distant points.
Among existing various RNNs, we use Long short term memory (LSTM) (Hochreiter & Schmidhuber, 1997) to handle the temporal signals on vertices for a regression task.
We feed ﬁnite lengths k of sequential signals into LSTM as an input series to predict the observation at next time step.
The predicted value is going to be compared to a true value and all parameters in LSTM will be updated via backpropagation through time.
DQ-LSTM Figure 2 illustrates how the data quality networks with LSTM (DQ-LSTM) consists of submodules.
Time-varying graph signals on N vertices can be represented as a tensor form, X ∈ R N × M × T where the total length of signals is T .
First, the time-varying graph signals for each vertex are segmentized to be fed into LSTM.
For example, X ( i, : , h : h + k − 1) is one of segmentized signals on vertex i starting at t
= h . Second, the graph signals for all vertices at the last time stamp X (: , : , h + k − 1) are used as an input of GCNs followed by DQN.
Hence, we consider the data quality level by looking at the local variations of the last signals and the estimated quality level w i is used to assign a weight on the loss function deﬁned on the vertex i .
We use the mean squared error loss function.
For each vertex i , DQ-LSTM repeatedly reads inputs, predicts outputs, and updates parameters as many as a number of segmentized length k time series.
[Equation]
where n i is the number of segmentized series on vertex i and ˆ X ( i, : , k + j − 1) is a predicted value from a fully connected layer (FC) which reduces a dimension of an output vector from LSTM.
Lregularization is used to prevent overﬁtting.
Then, the total loss function over all vertices is as
L = 1 N (cid:80) Ni = 1 L i .
E XPERIMENTS
In this section, we evaluate DQ-LSTM on real-world climate datasets.
In the main set of experiments, we evaluate the mean absolute error (MAE) of the predictions produced by DQ-LSTM over entire weather stations.
In addition, we analyze the data quality levels estimated by DQ-LSTM.
4. D ATASETS
In order to have a fair comparison, we use real-world meteorological measurements from two commercial weather service providing real-time weather information, Weather Underground(WU)
Both observations from automated personal weather (PWS).
In the dataset, all stations are distributed around Los Angeles County and geographical characteristics where the station is located at are provided.
These characteristics would be used as a set of input features V to build a graph structure.
The list of the static 11 characteristics is, Latitude,
Longitude, Elevation, Tree fraction, Vegetation fraction, Albedo, Distance from coast, Impervious fraction, Canopy width, Building height, and Canopy direction .
At each station, a number of meteorological signals are observed through the installed instruments.
The types of the measurements are Temperature, Pressure, Relative humidity, Solar radiation, Precipitation, Wind speed , and Wind direction .
Since each weather station has observed measurements under its own frequecy (e.g., every 5 minutes or every 30 minutes), we ﬁx the temporal granularity as 1 hour and aggregate observations in each hour by averaging them.
We want to ensure that the model can be veriﬁed and explained physically with in one meteorological regime before applying it to the entire year with many other regimes.
Especially, it is more challenging to predict temperatures in the summer season of Los Angeles since the daytime temperatures can vary as much as 36 ◦ F (19 ◦ C) between inland areas and the coastal Los Angeles Basin.
On the other hand, it varies only 6 ◦ F (3.3 ◦ C) for December and January in the same regions.
We use 2 months observations from each service, July/2015 and August/2015, for our experiments.
The dataset description is provided in Appendix A.
4. G RAPH GENERATION
Since any structural information between a pair of stations is not given, we need to construct a graph of the weather stations.
In general graphs, two nodes can be interpreted as similar nodes if they are connected.
Thus, as mentioned in Section 2.1, we can compute a distance between two nodes in the feature space.
A naive approach to deﬁning the distance is using only the geolocation features, Latitude and Longitude .
However, it might be inappropriate because other features can be signiﬁcantly different even if two stations are fairly close.
For example, the distance between stations in the Elysian park and Downtown LA is less than 2 miles, however, the territorial characteristics are signiﬁcantly different.
Furthermore, the different characteristics (e.g., Tree fraction or Impervious fraction) can affect weather observations (especially, temperature due to urban heat island effect).
Thus, considering only the physical distance can mislead to compute the meteorological similarity between two nodes.
To alleviate the issue, we assume that all static features are equally important.
This is a reasonable assumption because we do not know which feature is more important and each feature can affect weather measurements.
Thus, we normalize the spatial features not to make some features dominate the distance.
In this experiment, we use Gaussian kernel e ( − γ (cid:107) V
i − V j (cid:107) 2 ) with γ = 0 .
2 and 0 .
6 for WU and WB, respectively, and make weights less than 0 .
9 zero (i.e., disconnected) to build similar topologies (the average number of neighbors is around 10) from two datasets.
4. B ASELINES
Since we are focusing on a forecasting task, there are a lot of models that have been used to predict future values.
First, we compare against a stochastic process, autoregressive (AR), which estimates future values based on past values.
Second, we further compare against a simple LSTM.
This model is expected to infer mixed-dependencies among the input multivariate signals and provide a reference error of neural networks based model.
Lastly, we use graph convolutional networks (Kipf & Welling, 2017) which are also able to infer the data quality level from a given dataset.
We test a single layer GCN ( K = 1 ) and two-layer GCNs ( K = 2 ).
4. E XPERIMENTAL SETING
Since all the baselines and DQ-LSTM are dependent on the previous observations, we set the same lag length k = 10 .
For the deep recurrent models, the k -steps previous observations are sequentially fed into them to predict next values.
We have the same setting for the deep recurrent module, i.e. the 50 units of the hidden states.
We use Adam optimizer (Kingma & Ba, 2015) with a learning rate of [Table]
001 and the objective is given by a mean squared error function.
For GCN-LSTM and DQ-LSTM, we evaluate different number of layers ( K ) of GCN.
The ﬁnal layer always provides a set of scalars for every vertices and we set β = 0 .
for the L 2 regularization of the ﬁnal layer.
For each month dataset, we split it into three subsets, training, validation, and testing sets.
The ﬁrst 60% observations are used to train the models and next 20% validation set is used to tune hyperparameters.
Finally, the left 20% testing set is used to report error results.
Among given measurements, Temperature is used as a target measurement, i.e., output of LSTM, and others including
Temperature are used as input signals.
We repeat 20 times with random initializations.
R ESULTS AND D ISCUSSION 5. F ORECASTING EXPERIMENT
Experimental results are summarized in Table 1.
We report the forecasting mean absolute error (MAE) with standard deviations.
The two months observations, July and August, are denoted as 7 and 8 , and K indicates the number of GCN layers.
Overall, models considering a given graph structure outperform AR and LSTM.
While the node connectivities and weights are dependent on the distance function, it cleary shows that knowing neighbor signals of a given node can help predict next value of the node.
Although GCN is able to transfer a given signal of a vertex to a latent representation that is more compact and expressive, it is difﬁcult to learn a mapping function to the data quality level from neighbor signals directly unlike DQ-LSTM which pre-transfers the signals to local variations explicitly.
In other word, given signal X , what GCN learns is w = f
( X ) where w is the data quality we want to infer from data, however, DQ-LSTM learns w =
g
( Y = h ( X )) where Y is a local variation matrix given by X in a closed form, h .
Thus, lower MAEs of DQ-LSTM verify that our assumption in Section 2.2 is valid and the local variations are a useful metric to measure the data quality level.
It is also noticeable that DQ-LSTM with a GCN reports the lowest MAE among other models.
This is because the additionally trainable parameters in GCN increase the number of nodes to compute better local variations.
5. N ODE EMBEDDING AND ANOMALY DETECTION
As DQ-LSTM can be combined with GCN, it is also possible to represent each node as an embedding obtained from an output of GCN.
Embeddings from deep neural networks are especially interesting since they can capture distances between nodes which are not explicitly provided but implicitly inherent in data.
Once the embeddings are extracted, they can be used for further tasks, such as classiﬁcation and clustering (Grover & Leskovec, 2016; Kipf & Welling, 2017).
Moreover, since the embeddings have low dimensional representations, it is more efﬁcient to visualize the nodes by manifold learning methods, such as t-SNE (Maaten & Hinton, 2008).
Visualization with spatiotemporal signals is especially effective to show how a similarity between nodes changes.
Figure 3 provides embedding distributions over time.
Green dots are neighbors of a red dot and they are closely distributed to form a cluster.
There are two factors that affect embeddings: temporal signals and structure.
Ideally, connected nodes have observed similar signals and thus, they are 20 21 23 25(a) t = 20 21 29 (b) t = 2025 29(c) t = mapped closely in the embedding space.
However, if one node v i measures a fairly different value, the node’s embedding will also be far from its neighbors.
On the other hand, if one node v i is connected with a group of nodes { g } and an additional node v j which is not a member of g , the v i node is affected by the v j node as well.
If the signals observed at v i are similar with the signals at { g } , the embedding of v i is still close to those of { g } .
However, if the signals of v i is close to that of v j or the weight of e ( i, j ) is signiﬁcantly high, v i will be far away from { g } in the embedding space.
Figure 3b provides an example that a node ( i = 25 ) is affected by another node ( j = 4 ) which is not included in the cluster (green dots).
Since we can detect potentially anomalous nodes by looking at the distribution of nodes, the latent representations can be applicable for anomaly detection tasks.
5. D ATA QUALITY ANALYSIS
Since we do not have explicit labels for the data quality levels, it is not straightforward to evaluate the inferred data quality from DQ-LSTM.
Instead, we can verify the validity of the inferred data quality by looking at high and low quality examples.
Table 4 provides observations in Figure 3b.
The values x 25 at v 25 same as x 20 , x 21 , and x 23 , however, x 25 is quite different with x 19 , x 22 , x 29 .
Moreover, the edge weights between v 25 and other green nodes are not as large as weights between the green nodes.
( v 25 is much closer to the coast than other green nodes.)
As a result, it is not easy for v 25 to be close to the green nodes because of the opposite behaviors ( v 20 , v 21 , and v 23 pull vand v 19 , v 22 , and v 29 push v 25 ).
On the other hand, since x 25 is similar with x 4 and W 25 , 4 is large, v 25 is more likely close to v 4 as in Figure 3b.
[Table]
Note that v 4 has far different geological features compared to features of the green nodes and thus, v 4 is not connected to v 22 or other green nodes excluding v 25 .
Consequently, v 25 is the bridge node between v 4 and the cluster of v 22 .
Under our assumption (See Section 2.2) it is natural to have high local variations at a bridge node since connected clusters on the other side have different features and therefore, the bridge node highly likely has a lower data quality level due to the heterogeneity.
As Table 4 provides, w 25 is the lowest number among the green nodes and the data quality levels are correctly inferred.
C ONCLUSION
In this work we study a model that enables the automatic inference of data quality for spatiotemporal data.
While existing work has typically addressed the problem by assuming that all the signals are equally reliable over time we argue that it is important to differentiate the data quality because the signals come from heterogeneous sources.
To address the issue we proposed our formulation of data quality levels that are assigned to different sources and inferred from a given dataset.
We evaluate the concept of the data quality by comparing other baselines in the weather forecasting task.
A D ATASET
In order to have a fair comparison
[Table]
we use real-world meteorological measurements from two commercial weather service providing real-time weather information, Weather Underground (WU) and
WeatherBug (WB).
Both services use observations from automated personal weather stations (PWS).
The PWS are illustrated in Figure 4.
[Figure]
In the dataset, each station is distributed around Los Angeles County and land characteristics where the station is located at are provided.
These characteristics would be used as a set of input features,
V i .
The list of the static characteristics is:
[Table]
At each station, a number of weather data are observed through the installed instruments and recorded.
The types of measurements are Temperature, Pressure, Relative humidity, Solar radiation, Precipitation, Wind speed , and Wind direction .
[Table]