A UTOMATICALLY I NFERRING D ATA Q UALITY FOR S PATIOTEMPORAL F ORECASTING
A BSTRACT
The importance of learning from spatiotemporal data has been growing with increasing amounts of data sources distributed over space.
While numerous studies have been done for analyzing spatiotemporal signals, existing models have assumed that heterogeneous data sources in the spatial domain are equally reliable over time.
In this paper, we propose a novel method that estimates time-varying data quality levels based on local variations of spatiotemporal signals without explicit labels that designate data quality.
Furthermore, we integrate our estimate of data quality level with graph convolutional networks to exploit their efﬁcient structure, because spatially distributed data sources form natural graphs.
Finally, we evaluate the proposed method by simulating a forecasting task with real-world climate data.
I NTRODUCTION
Recent advances with GPS-equipped technology have facilitated the collection of large spatiotemporal datasets.
As the amount of spatiotemporal data increases, many have proposed representing this data as time-varying graph signals in various domains, such as sensor networks (Shi et al., 2015;
Zhu & Rabbat, 2012), climate analysis (Chen et al., 2014; Mei & Moura, 2015), trafﬁc control systems (Li et al., 2017; Yu et al., 2017), and biology (Mutlu et al., 2012; Yu et al., 2015).
While many works have been proposed to exploit both spatial structures and temporal signals , most models have implicitly assumed that each signal source node in a spatial structure is equally reliable over time, without considering the data quality of different sources.
However, a large amount of data comes from heterogeneous sources that have different levels of noise (Song et al., 2015; Zhang &
Chaudhuri, 2015).
Moreover, the noises of each source can vary over time due to abrupt malfunctions of sensors.
The concept of the data quality should be considered to train and use deep neural network that is called a black box because even a small perturbation of a data can extremely deceive the model and turn out unexpected behaviors (Goodfellow et al., 2014; Koh & Liang, 2017).
Thus, it is naturally desired to differentiate the data quality level of each source.
An important technical problem arises when developing an estimate of data quality level.
How can such data quality for many different sources be estimated without labels, which are laborious and expensive to obtain?
By answering this question, we can develop a practical model that automatically differentiates between high and low quality sources.
In this paper, we consider a learning problem of spatiotemporal signals that are represented by timevarying graph signals for different data qualities.
Given a graph G =
( V , E , W ) and observations
X ∈ R N × M × T where N, M, T are the number of vertices, the types of signals, and the length of time-varying signals, respectively, we propose a model that is able to exploit structural information (spatial features) as well as temporal behaviors (time series).
On top of the time-varying graph signals, we implement graph-based neural networks that encode the level of data quality at each vertex into a scalar by looking at neighboring signals observed at a set of K -hop neighboring vertices.
There are multiple deﬁnitions of data quality but high quality data is generally considered as it is ﬁt for intended uses in operations, decision making and planning (Redman, 2008). Here, we narrow down the deﬁnition of the term as a penalizing quantity for high local variations .
The contributions of our paper are the following.
First, we deﬁne the concept of a data quality level at each vertex in a graph using a local variation of the vertex.
The local variation at each vertex depends on the local spatial structure and neighboring signals.
We propose a function of the local variation at the vertex to infer the data quality level in an unsupervised manner without any explicitly given quality labels. It is worth to note that it is ﬂexible to deﬁne the data quality function depending on tasks and characteristics of datasets.
Secondly, we combine the quality level with an objective function deﬁned by the task at hand.
Speciﬁcally, the quality level is used as a regularizer constant in the objective function.
Finally, we demonstrate that the data quality is an essential factor which can improve the predictive performance of common tasks, such as time-series forecasting, on real-world climate datasets.
Related work The ﬁrst set of work related to our study is about learning and processing graph signals or features.
Spectral graph theory (Chung, 1997; Von Luxburg, 2007; Shi et al., 2015) has been developed as a main study to understand two aspects of graph signals: structures and signals.
Under this theory many works have been introduced to exploit convolutional neural networks (CNNs) which provide an efﬁcient architecture to extract localized patterns from regular grids, such as images (Krizhevsky et al., 2012).
Bruna et al.
(2014) introduce how to learn convolutional parameters based on the spectrum of the graph Laplacian.
Later, Henaff et al.
(2015) extend the spectral aspect of CNNs on graphs into large-scale learning problems.
From the pioneer works about graph convolutional networks (GCN), Defferrard et al.
(2016) propose a spectral formulation for fast localized ﬁltering with efﬁcient pooling.
Furthermore, Kipf & Welling (2017) re-formularize existing ideas into layer-wise neural networks that can be tuned through backpropagation rule with a ﬁrst-order approximation of spectral ﬁlters introduced in Hammond et al.
On top of these studies, we propose a graph convolutional layer that maps spatiotemporal features into a data quality level.
There are a number of works about considering data qualities and handling heterogeneous data sources.
Urner et al.
(2012) is the ﬁrst theoretical work that proposes a mixture of two types of labels for supervised learning.
One type of the labels is considered as high quality labels from an expensive source (domain experts) while another type is from error-prone crowdsourcing.
Since the reliability or quality of the labels is different, it is not desired to consider them equally.
They propose a learning algorithm that is able to utilize the error-prone labels to reduce the cost required for the expert labeling.
Zhang & Chaudhuri (2015) address issues from strong and weak labelers by developing an active learning algorithm minimizing the number of label requests.
Song et al.
(2015) focus on the data of variable quality resulting from heterogeneous sources.
They deﬁne the concept of heterogeneity of data and develop a method of adjusting the learning rate based on the heterogeneity.
Unlike these existing works, we provide a method to differentiate heterogeneous sources based on neighborhood signals without any explicit labels.
Outline We review graph signal processing to deﬁne the local variation and a data quality level (DQL) with graph convolutional networks in Section 2.
In Section 3, we provide how the data quality levels are exploited with recurrent neural networks to differentiate reliability of observations on vertices.
Also, we construct a forecasting model, DQ-LSTM.
Our main result is presented in
Section 4 with other baselines.
In Section 5 we discuss its properties and interpret the data reliability inferred from our model.
We then conclude with Section 6.
P RELIMINARIES
We ﬁrst show how to deﬁne the local variation at a vertex based on graph signals.
Then, we explain how the variational features at each vertex can be used to generate a data quality level.
2. L OCAL V ARIATION
We focus on the graph signals deﬁned on an undirected, weighted graph G =
( V , E , W ) , where V is a set of vertices with |V| = N and E is a set of edges.
W ∈ R N × N is a random-walk normalized weighted adjacency matrix which provides how two vertices are relatively close.
When the elements
W ij are not be expliticly provided by dataset, the graph connectivity can be constructed by various distance metrics, such as Euclidean distance, cosine similarity, and a Gaussian kernel (Belkin &
Niyogi, 2002), on the vertex features V ∈ R N × F where F is the number of the features.
[Figure] Once all the structural connectivity is provided, the local variation can be deﬁned by the edge derivative of a given graph signal x ∈ R N deﬁned on every vertex (Zhou & Sch¨olkopf, 2004).
[Equation]
where e =
( i, j ) is deﬁned as a direction of the derivative and x ( i ) is a signal value on the vertex i .
The graph gradient of x at vertex i can be deﬁned by Eq.
1 over all edges joining the vertex i .
[Equation]
where N i is a set of neighbor vertices of the vertex i .
While the dimesion of the graph gradient is different due to the different number of neighbors of each vertex, the local variation at vertex i can be deﬁned by a norm of the graph gradient:
[Equation]
Eq.
3 provides a measure of local variation of x at vertex i .
As it indicates, if all neighboring signals of i are close to the signal at i , the local variation at i should be small and it means less ﬂuctuated signals around the vertex.
As Eq.
3 shows, the local variation is a function of a structural connectivity W and graph signals x .
Figure 1 illustrates how the two factors affect the local variation at the same vertex.
The concept of the local variation is easily generalized to multivariate graph signals of M different measures by repeatedly computing Eq.
3 over all measures.
[Equation]
where x m ∈ R N corresponds the m th signals from multiple sensors.
As Eq.
4 indicates, L i is a
M dimensional vector describing local variations at the vertex i with respect to the M different measures.
Finally, it is desired to represent Eq.
4 in a matrix form to be combined with graph convolutional networks later.
[Equation]
where D is a degree matrix deﬁned as D ii = (cid:80) j W ij and (cid:12) is an element-wise product operator.
X is a N × M matrix describing multivariate graph signals on N vertices and x m is a m th column of
X .
L ∈ R N × M is a local variation matrix and L im is the local variation at the vertex i with respect to the m th signal.
2. D ATA Q UALITY L EVEL
While the term of data quality has been used in various ways, it generally means “ ﬁtness of use ” for intended purposes (Juran & Godfrey, 1999).
In this section, we will deﬁne the term under the data property we are interested in and propose how to exploit the data quality level into a general framework.
Given a multivariate graph signal X ∈ R N × M on vertices represented by a feature matrix V ∈ R N × F , we assume that a signal value at a certain vertex i is desired not be signiﬁcantly different with signals of neighboring vertices j ∈ N i .
This is a valid assumption if the signal value at the vertex i is dependent on (or function of) features of the vertex i when an edge weight is deﬁned by a distance in the feature vector space between two vertices.
In other words, if two vertices have similar features, they are connected to form the graph structure and the signal values observed at the vertices are highly likely similar.
There are a lot of domains which follow the assumption, for instance, geographical features (vertex features) and meteorological observations (graph signal) or sensory nervous features and received signals.
Under the assumption, we deﬁne the data quality level (score) at a vertex i as a function of local variations of i :
[Equation]
[Equation]
It is ﬂexible to choose the function q .
For example, s i can be deﬁned as an average of L i .
If so, all measures are equally considered to compute the data quality at vertex i .
In more general sense, we can introduce parameterized function q ( L i ; Φ ) and learn the parameters through data.
Kipf &
Welling (2017) propose a method that learns parameters for graph-based features by the layer-wise graph convolutional networks (GCN) with arbitrary activation functions.
For a single layer GCN on a graph, the latent representation of each node can be represented as:
[Equation]
where ˆ A =
( D + I N ) − 12 ( A + I N )
( D + I N ) − 12 provides structural connectivities of a graph and Θ is a trainable parameter matrix.
σ is an activation function.
By stacking σ (ˆ AX Θ ) , it is able to achieve larger receptive ﬁelds with multi-layer GCN.
See details in Kipf & Welling (2017).
We can replace ˆ AX with L which is also a function of the weighted adjacency W and the graph signal X .
Note that values in row i of ˆ AX and L are a function of values at i as well as neighbors of i .
Although L only exploits nearest neighbors (i.e., 1-hop neighbors), it is possible to consider
K -hop neighbors to compute the local variations by stacking GCN before applying Eq.
The generalized formula for the data quality level can be represented as:
[Equation]
where K is the number of GCN layers and s = ( s 1 , s 2 , · · · , s N ) is the data quality level of each vertex incorporating K -hop neighbors.
We propose some constraints to ensure that a higher s i corresponds to less ﬂuctuated around i .
First, we constrain Φ to be positive to guarantee that larger elements in L cause larger L Φ that are inputs of σ L .
Next, we use an activation function that is inversely proportional to an input, e.g., σ L ( x ) = 11+ x , to meet the relation between the data quality s i and the local variations L i .
Once s is obtained, it will be combined with an objective function to assign a penalty for each vertex loss function.
M ODEL
In this section, we give the details of the proposed model, which is able to exploit the data quality deﬁned in Section 2.2 for practical tasks.
First, it will be demonstrated how the data quality network
DQN is combined with recurrent neural networks (LSTM) to handle time-varying signals.
We, then, provide how this model is trained over all graph signals from all vertices.
Data quality network In Section 2.2, we ﬁnd that the local variations around all vertices can be computed once graph signals X are given.
Using the local variation matrix L , the data quality level at each vertex s i can be represented as a function of L i with parameters
Φ (See Eq. 6).
While the function q is not explicitly provided, we can parameterize the function and learn the parameters Φ ∈ R M through a given dataset. [Figure]
One of straightforward parameterizations is based on fully connected neural networks.
Given a set of features, neural networks are efﬁcient to ﬁnd nonlinear relations among the features.
Furthermore, the parameters in the neural networks can be easily learned through optimizing a loss function deﬁned for own purpose.
Thus, we use a single layer neural networks Φ followed by an activation function σ L ( · ) to transfer the local variations to the data quality level.
Note that multi-layer GCN can be used between graph signals and DQN to extract convolved signals as well as increase the size of convolution ﬁlters.
(See Eq.
8 and 9).
This multi-layer neural networks are efﬁcient to learn nonlinear interactions of K -hop nodes which are not easily learnable by graph polynomial ﬁlters in Sandryhaila & Moura (2013).
Long short term memory Recurrent neural networks (RNNs) are especially powerful to extract latent patterns in time series which has inherent dependencies between not only adjacent data points but also distant points.
Among existing various RNNs, we use Long short term memory (LSTM) (Hochreiter & Schmidhuber, 1997) to handle the temporal signals on vertices for a regression task.
We feed ﬁnite lengths k of sequential signals into LSTM as an input series to predict the observation at next time step.
The predicted value is going to be compared to a true value and all parameters in LSTM will be updated via backpropagation through time.
DQ- LSTM Figure 2 illustrates how the data quality networks with LSTM (DQ-LSTM) consists of submodules.
Time-varying graph signals on N vertices can be represented as a tensor form,
X ∈ R N × M × T where the total length of signals is T .
First, the time-varying graph signals for each vertex are segmentized to be fed into LSTM.
For example, X ( i, : , h : h + k − 1) is one of segmentized signals on vertex i starting at t
= h .
Second, the graph signals for all vertices at the last time stamp X (: , : , h + k − 1) are used as an input of GCN followed by DQN.
Hence, we consider the data quality level by looking at the local variations of the last signals and
the estimated quality level s i is used to assign a weight on the loss function deﬁned on the vertex i .
We use the mean squared error loss function.
For each vertex i , DQ-LSTM repeatedly reads inputs, predicts outputs, and updates parameters as many as a number of segmentized length k time series.
[Equation]
[Equation] where n i is the number of segmentized series on vertex i and ˆ X ( i, : , k + j − 1) is a predicted value from a fully connected layer (FC) which reduces a dimension of an output vector from LSTM.
Lregularization is used to prevent overﬁtting.
Then, the total loss function over all vertices is as
L = 1 N (cid:80) Ni =
1 L i .
E XPERIMENTS
In this section, we evaluate DQ-LSTM on real-world climate datasets.
In the main set of experiments, we evaluate the mean absolute error (MAE) of the predictions produced by DQ-LSTM over entire weather stations.
In addition, we analyze the data quality levels estimated by DQ-LSTM.
4. D ATASETS
We use real-world datasets on meteorological measurements from two commercial weather services,
Weather Underground(WU) and WeatherBug(WB).
Both datasets provide real-time weather information from personal weather stations.
In the datasets, all stations are distributed around Los Angeles County, and geographical characteristics of each station are also provided.
These characteristics would be used as a set of input features V to build a graph structure.
The list of the static 11 characteristics is, Latitude, Longitude, Elevation, Tree fraction, Vegetation fraction, Albedo, Distance from coast, Impervious fraction, Canopy width, Building height, and Canopy direction .
Meteorological signals at each station are observed through the installed instruments.
The types of the measurements are Temperature, Pressure, Relative humidity, Solar radiation, Precipitation,
Wind speed , and Wind direction .
Since each weather station has observed the measurements under its own frequency (e.g., every 5 minutes or every 30 minutes), we ﬁx the temporal granularity athour and aggregate observations in each hour by averaging them.
We want to ensure that the model can be veriﬁed and explained physically within one meteorological regime before applying it to the entire year with many other regimes.
Since it is more challenging to predict temperatures in the summer season of Los Angeles due to the large ﬂuctuation of daytime temperatures (summer: 36 ◦ F / 19 ◦ C and winter: 6 ◦ F / 3.3 ◦ C between inland areas and the coastal Los Angeles Basin), we use 2 months observations from each service, July/2015 and August/2015, for our experiments.
The dataset description is provided in Appendix A.
4. G RAPH GENERATION
Since structural information between pairs of stations is not directly known, we need to construct a graph of the weather stations.
In general graphs, two nodes can be interpreted as similar nodes if they are connected.
Thus, as mentioned in Section 2.1, we can compute a distance between two nodes in the feature space.
A naive approach to deﬁning the distance is using only the geolocation features, Latitude and Longitude .
However, it might be inappropriate because other features can be signiﬁcantly different even if two stations are fairly close.
For example, the distance between stations in the Elysian Park and Downtown LA is less than 2 miles, however, the territorial characteristics are signiﬁcantly different.
Furthermore, the different characteristics (e.g., Tree fraction or Impervious fraction) can affect weather observations (especially, temperature due to urban heat island effect).
Thus, considering only physical distance may improperly approximate the meteorological similarity between two nodes.
To alleviate this issue, we assume that all static features are equally important.
This is a reasonable assumption because we do not know which feature is more important since each feature can affect weather measurements.
Thus, we normalize all spatial features.
In this experiment, we use the
Gaussian kernel e ( − γ (cid:107) V
i − V j (cid:107) 2 ) with γ = 0 .
2 and 0 .
6 for WU and WB, respectively, and make weights less than 0 .
9 zero (i.e., disconnected) such that the average number of node neighbors is around 10.
4. B ASELINES
We compare our approach to well-studied baselines for time-series forecasting.
First, we compare against a stochastic process, autoregressive (AR), which estimates future values based on past values.
Second, we further compare against a simple LSTM.
This model is expected to infer mixeddependencies among the input multivariate signals and provide a reference error of the neural networks based model.
Lastly, we use graph convolutional networks (Kipf & Welling, 2017) which are also able to infer the data quality level from a given dataset.
We test a single layer GCN ( K = 1 ) and two-layer GCN ( K = 2 ). [Table]
4. E XPERIMENTAL SETING
Since DQ-LSTM and our baselines are dependent on previous observations
[Table]
we set a common lag length of k = 10 . For the deep recurrent models, the k -steps previous observations are sequentially inputted to predict next values.
All deep recurrent models have the same 50 hidden units and one fully connected layer ( R 50 × 1 ) that provides the target output.
For GCN-LSTM and DQ-LSTM, we evaluate with different numbers of layers ( K ) of GCN.
We set the dimension of the ﬁrst ( K = 1 ) and second ( K = 2 ) hidden layer of GCN as 10 and 5 , respectively, based on the cross validation.
The ﬁnal layer always provides a set of scalars for every vertex, and we set β = 0 .
for the Lregularization of the ﬁnal layer.
We use the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0 .
001 and a mean squared error objective.
We split each dataset into three subsets: training, validation, and testing sets.
The ﬁrst 60% observations are used for training, the next 20% is used to tune hyperparameters (validation), and the remaining 20% is used to report error results (test).
Among the measurements provided, Temperature is used as the target measurement, i.e., output of LSTM, and previous time-step observations, including Temperature, are used as input signals.
We report average scores over 20 trials of random initializations.
R ESULTS AND D ISCUSSION 5. F ORECASTING EXPERIMENT
Experimental results are summarized in Table 1.
We report the temperature forecasting mean absolute error (MAE) of our DQ-LSTM model with standard deviations.
Meteorological measurements for July and August are denoted by 7 and 8 , and K indicates the number of GCN layers.
Overall, the models that account for graph structures outperform AR and LSTM.
While the node connectivities and weights are dependent on our distance function (Section 4.2), Table 1 clearly shows that knowing the neighboring signals of a given node can help predict next value of the node.
Although GCN are able to transfer a given signal of a vertex to a latent representation that is more compact and expressive, GCN have difﬁculty learning a mapping from neighboring signals to data quality level directly, unlike DQ-LSTM which pre-transfers the signals to local variations explicitly.
In other words, given signal X , what GCN learns is s = f ( X ) where s is the data quality we want to infer from data; however, DQ-LSTM learns s =
g ( Y = h ( X ))
where Y is a local variation matrix given by X in a closed form, h .
Thus, lower MAEs of DQ-LSTM verify that our assumption in Section 2.2 is valid and the local variations are a useful metric to measure data quality level.
It is also noteworthy that DQ-LSTM with a GCN reports the lowest MAE among all models.
This is because the additional trainable parameters in GCN increase the number of neighboring nodes that are accounted for to compute better local variations.
5. N ODE EMBEDDING AND LOW - QUALITY DETECTION
As DQ-LSTM can be combined with GCN, it is possible to represent each node as an embedding obtained from an output of GCN.
Embeddings from deep neural networks are especially interesting 20 21 23 25(a) t = 20 21 29 (b) t = 2025 29(c) t = since they can capture distances between nodes.
These distances are not explicitly provided but inherently present in data.
Once the embeddings are extracted, they can be used for further tasks, such as classiﬁcation and clustering (Grover & Leskovec, 2016; Kipf & Welling, 2017).
Moreover, since the embeddings have low dimensional representations, it is more efﬁcient to visualize the nodes by manifold learning methods, such as t-SNE (Maaten & Hinton, 2008).
Visualization with spatiotemporal signals is especially effective to show how similarities between nodes change.
Figure 3 shows varying embedding distributions over time.
The green dots are neighbors of a red dot, and they are closely distributed to form a cluster.
There are two factors that affect embeddings: temporal signals and spatial structure .
Ideally, connected nodes have observed similar signals and thus, they are mapped closely in the embedding space.
However, if one node v i measures a fairly different value compared to other connected values, the node’s embedding will also be far from its neighbors.
Furthermore, if one node v i is connected to a subset of a group of nodes { g } as well as an additional node v j / ∈ { g } , v i would be affected by the subset and v j / ∈ { g } simultaneously.
For example, if the signals observed at v i are similar to the signals at { g } , the embedding of v i is still close to those of { g } .
However, if the signals of v i are close to that of v j , or the weight of e ( i, j ) is signiﬁcantly high, v i will be far away from { g } in the embedding space.
Such intuition of the embedding distribution can be used to ﬁnd potentially low-quality nodes, which we analyze next.
Figure 3b shows that a node v 25 is affected by its neighboring green nodes and v 4 that is not included in the cluster (green dots).
The red dot v 22 is connected with the green dots ( v 19 , v 20 , v 21 , v 23 , v 25 , v 29 ).
Since these nodes have similar spatial features and are connected, the nodes are expected to have similar observations.
At t = 0 , the distribution of the nodes seems like a cluster.
However, v 25 is far away from other green nodes and the red node at t = 4 .
There are two possible reasons.
First, observations of v 25 at t = 4 may be too different with those of other green nodes and the red node.
Second, observations at v 4 , which is only connected to v 25 (not to other green nodes and the red node), might be too noisy.
The ﬁrst case violates our assumption (See Section 2.2, such that the observation at v 25 should be similar to those of other green nodes.); therefore, the observations of v 25 at t = 4 might be noisy or not reliable.
In the second case, the observations of v 4 at t = 4 might be noisy.
Thus, v 25 and v 4 are candidates of low-quality nodes.
5. D ATA QUALITY ANALYSIS
Since we do not have explicit labels for the data quality levels, it is not straightforward to directly evaluate the data quality inferred from DQ-LSTM.
Instead, we can verify the inferred data quality by studying high and low quality examples from embedding distributions and associated meterological observations.
Table 2 shows meterological observations associated with the previously discussed embedding distribution at t = 4 (Figure 3b).
The values x 25 at v 25 are the same as x 20 , x 21 , and x 23 ; however, x 25 is quite different than x 19 , x 22 , and x 29 .
Moreover, the edge weights between vand other green nodes are not as large as weights between other green nodes.
( v 25 is much closer to the ocean than other green nodes.)
As a result, it is not easy for v 25 to be close to the green nodes ( v 20 , v 21 , and v 23 pull v 25 and v 19 , v 22 , and v 29 push v 25 ).
On the other hand, since x 25 is similar to x 4 and W 25 , 4 is large, v 25 is more likely to be close to v 4 as in Figure 3b.
[Table]
Note that v 4 has very different geological features compared to the features of the green nodes and thus, v 4 is not connected to v 22 or other green nodes except v 25 .
Consequently, v 25 is the bridge node between v 4 and the cluster of v 22 .
Since a bridge node is affected by two (or more) different groups of nodes simultaneously, the quality level at the bridge node is more susceptible than those of other nodes.
However, this does not directly mean that a bridge node must have a lower data quality level.
As Table 2 shows, s 4 has the lowest data quality level, which comes from the discrepancy between its neighboring signals and x 4 .
Since v 4 is connected to v 25 , v 4 pulls v 25 and s 4 , lowering srelative to data quality levels of the other green nodes that are correctly inferred.
C ONCLUSION
In this work, we proposed our model, DQ-LSTM, which enables the automatic inference of data quality for spatiotemporal data.
While existing works have modeled spatiotemporal data by assuming that all signals are equally reliable over time, we argue that it is important to differentiate data quality because the signals come from heterogeneous sources.
To address the issue, we proposed our formulation of data quality levels that are assigned to different sources and inferred from a given dataset.
We evaluate our concept of data quality by comparing DQ-LSTM to baselines in real-world climate forecasting tasks
A CKNOWLEDGMENTS
This work is supported in part by NSF Research Grant IIS-1254206 and IIS-1539608.
The views and conclusions are those of the authors and should not be interpreted as representing the ofﬁcial policies of the funding agency, or the U.S. Government.
We thank Michael Tsang for discussion and his comments that greatly improved the manuscript
A D ATASET
In order to have a fair comparison, we use real-world meteorological measurements from two commercial weather service providing real-time weather information, Weather Underground (WU) 1 and WeatherBug (WB) 2 .
Both services use observations from automated personal weather stations (PWS).
The PWS are illustrated in Figure 4.
[Figure]
In the dataset, each station is distributed around Los Angeles County and land characteristics where the station is located at are provided.
These characteristics would be used as a set of input features, V i .
The list of the static characteristics is:
[Table]
An angle which ranges from 0 at the Equator to 90 at the poles
[Table]
angule which ranges from 0 at the Prime Meridian to
[Table]
180 eastward and 180 westward
[Table] sea [Table] At each station, a number of weather data are observed through the installed instruments and recorded.
The types of measurements are Temperature, Pressure, Relative humidity, Solar radiation, Precipitation, Wind speed , and Wind direction .
[Table]