Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations
Anonymous Author(s)
AffiliationAddress email
Abstract
Modeling multivariate time series as temporal signals over a (possibly dynamic)graph is an effective representational framework that allows for developing modelsfor time series analysis.
In fact, discrete sequences of graphs can be processed byautoregressive graph neural networks to recursively learn representations at eachdiscrete point in time and space.
Spatiotemporal graphs are often highly sparse,with time series characterized by multiple, concurrent, and even long sequences ofmissing data, e.g., due to the unreliable underlying sensor network.
In this context,autoregressive models can be brittle and exhibit unstable learning dynamics.
Theobjective of this paper is, then, to tackle the problem of learning effective models toreconstruct, i.e., impute, missing data points by conditioning the reconstruction onlyon the available observations.
In particular, we propose a novel class of attentionbased architectures that, given a set of highly sparse discrete observations, learn arepresentation for points in time and space by exploiting a spatiotemporal diffusionarchitecture aligned with the imputation task.
Representations are trained end-toend to reconstruct observations w.r.t.
the corresponding sensor and its neighboringnodes.
Compared to the state of the art, our model handles sparse data withoutpropagating prediction errors or requiring a bidirectional model to encode forwardand backward time dependencies.
Empirical results on representative benchmarksshow the effectiveness of the proposed method.
Introduction
Exploiting structure – both temporal and spatial – is arguably the key ingredient for the success ofmodern deep learning architectures and models.
Structures and invariances allow imposing inductivebiases to learning systems that act as strong regularizations, thus limiting the space of possible modelsto the most plausible ones and, consequently, greatly reducing sample complexity.
This is the casewith spatiotemporal graph neural networks (STGNNs) [1–3] which learn to process multivariatetime series while taking into account underlying space and time dependencies by encoding structuralspatiotemporal inductive biases in their architectures.
In particular, graphs are used to model the presence of spatial relationships which can be thought of as soft functional dependencies existingamong the sensors generating the time series [4].
However, even when spatiotemporal relationshipsare present, available data are almost always incomplete and irregularly sampled both spatially and temporally.
This is definitely true for data coming from real sensor networks (SNs), where the presence of missing data is a phenomenon inherent to any data acquisition or communication process.
In this case, a common approach is to reconstruct missing time series observations with simple interpolation strategies before proceeding with the downstream task.
However, arguably, this induces a bias in the inference procedure, possibly made even worse by the strong regularization imposed onthe model.
More advanced methods deal with missing data by autoregressively replacing missingobservations with predicted ones, eventually using bidirectional architectures [5, 6] to exploit bothforward and backward temporal dependencies.
To account also for spatial dependencies, Cini et al.[4] introduced a method, named GRIN, combining a bidirectional autoregressive architecture withmessage passing graph neural networks [7–10] for spatiotemporal imputation.
Despite being the stateof the art in imputation, GRIN suffers from the error propagation typical of autoregressive modelsthat bootstrap future inferences from their own predictions.
In fact, we argue that the propagation ofimputed (biased) values through space and time combined with noisy observations might exacerbateerror accumulation in highly sparse data and drive the hidden state of GRIN-like models to possiblydrift away.
In this paper, we aim at tackling this problem by designing an architecture based on a novelattention mechanism that takes spatiotemporal sparsity into account while learning representationsand imputing missing values.
Attention-based models allow distributed representations to emerge by letting each discrete element– representing an entity or point within a structured environment – interact with each other.
Infact, these mechanisms for propagating information through structures have been recently linkedto collective intelligence [11].
In our context, each token, represented as a node in a sequence ofgraphs, represents a point in space and time.
A trivial way to account for sparsity in this setting is toconstrain the attention coefficients of time steps corresponding to missing data to be zero or simply toadd an auxiliary token to indicate missing observations; basically, the resulting reconstruction modelwould behave similarly to a denoising autoencoder, in the same spirit of BERT-like encoders popularin natural language processing [12].
Conversely, we propose a novel architecture exploiting aninter-node sparse spatiotemporal attention mechanism within the neural message-passing framework.
In particular, we seek to design an architecture where each processing stage is aligned with thetask of reconstructing missing spatiotemporal observations.
Our model can be seen as a learned,self-organizing, spatiotemporal propagation process that, as we will motivate throughout the paper, ismore apt to the purpose of missing data reconstruction than standard encoder-decoder architectures.
In fact, compared with the alternatives discussed so far, our method exploits the aforementionedpropagation process to learn a predictive representation for each missing observation by relyingonly on observed values propagated through the spatiotemporal structure.
This approach achievesthe twofold objective of avoiding propagating biased representation – typical in the autoregressiveframework – and reconstructing observations at arbitrary nodes in the sensor network.
In summary,our main contributions are:
1. introduction of a sparse spatiotemporal attention mechanism to learn, from sparse data,
representations localized in time and space;
design of a novel graph neural network architecture based on the aforementioned spatiotem-
poral attention mechanism and equipped with inductive biases that make the model tailored
for the multivariate time series imputation task;
an empirical assessment of the proposed method showing how it overcomes the limits of
existing approaches, particularly in settings with highly sparse data.
The paper is structured as follows.
In Section 2 we discuss related works and contrast them with ours.
We formulate the problem of multivariate time series imputation in the context of spatiotemporalgraphs in Section 3, and present our approach in Section 4, providing an in-depth discussion ofmotivations, design choices, as well as methodological and technical issues.
Finally, in Section 5,we report the empirical results on several relevant benchmarks.
Conclusions and future works arediscussed in Section 6.
Related works
Multivariate time series imputation is a core task in time series analysis.
Besides standard statisticalapproaches based on linear autoregressive models [13–15] or interpolation [16], methods based on matrix factorization are widely popular [17] which can also incorporate temporal [18] and graph-sideinformation [19].
Deep learning methods are also commonly used in this regard.
In particular, deepautoregressive models based on recurrent neural networkss (RNNs) are currently among the mostwidely adopted methods [20, 5, 6, 21].
In this direction, BRITS [6] is archetypal of several relatedworks which exploit bidirectional RNNs to perform imputation.
Notably, differently from a denoisingautoencoder, BRITS uses one-step-ahead forecasting as a surrogate task to learn an imputation modelwhile using a simple linear regression layer to incorporate spatial information.
Several approaches inthe literature, then, rely on generative adversarial neural networks [22] – often paired with RNNs – togenerate imputed subsequences by matching the underlying data distribution [23, 24, 21].
Recently,several attention-based imputation techniques have also been proposed [25–27], however, none ofthese explicitly account for spatial dependencies within the graph processing framework and overlookthe spatial dimension of the problem. Other works, instead, address this problem in the contextof continuous-time models [28].
Huang et al.
[29], in particular, exploit graph representations tomodel spatial dynamics. The limits of deep autoregressive approaches in data reconstruction havealso been tackled by using hierarchical imputation methods [30, 31].
More related to our work, GRIN [4] uses a bidirectional graph RNN, paired with a message passing spatial decoder, to imputetime series based on spatiotemporal dependencies.
Other graph-based architectures have been usedin application-specific settings, such as traffic data [32, 33] and load profiles from smart grids [34].
While GRIN achieves remarkable performance, we argue that the spatial regularization might not beenough to prevent errors to accumulate in the states of the recurrent graph network.
The attention mechanism has been exploited in several contexts within the graph deep learningliterature, in particular in anisotropic graph convolutional filters [35–38].
Among STGNNs, attentionbased architectures have been exploited in time series forecasting [37, 39–41].
In particular, TraverseNet [41] is specially related to our work, since it relies on spatiotemporal autoregressive attentionto compute messages exchanged between nodes.
One striking example of attention being used successfully to process incomplete data is in pretraining routines for representation learning in naturallanguage processing [12, 42].
Finally, graph neural networks are also popular for reconstructingmissing features in static graphs [43–45].
Preliminaries
We model multivariate spatiotemporal time series as observations coming from a SNs.
In a SN everyi -th node, i.e., sensor , acquires a d -dimensional x
it ∈ R
d observation at each t -th time step.
Wedenote by X t ∈ R N t ×
d the matrix collecting the measurements of N t sensors at time step t , with
X t : t + T being the sequence of T measurements collected in the time interval [ t, t + T ) .
We modelfunctional relationships among the sensors as graph edges.
Relationships can be often inferred fromavailable side information, e.g., one can extract a graph from the position of each sensor and theirreciprocal physical proximity or the structure of the physical system where sensors are placed.
In othercases instead, functional dependencies can be inferred directly from data by exploiting some affinityscore (e.g., Pearson correlation, Granger causality [46], correntropy [47], etc.), or more advancedtechniques (e.g., graph learning methods [48]).
We model the extracted relational information with aweighted, possibly asymmetric adjacency matrix A t ∈ R N t × N t , in which each nonzero entry a i,jtindicates the weight of the edge going from the i -th node to the j -th.
While our framework canaccount for dynamic relationships, we mostly focus on settings where the topology is static, i.e.,
A t = A and N t = N .
Finally, we assume to have available sensor-level covariates Q t ∈ R N × d qthat act as spatiotemporal coordinates to localize a point in time and space (e.g., date/time featuresand geographic location).
Note that coordinates q it are assumed available for each node at each timestep; in section 4 we discuss how Q t can be learned as a spatiotemporal positional encoding.
We thenmodel observations as a discrete sequence of spatiotemporal graphs, where each graph is a triplet
G t = ⟨ X t , Q t , A ⟩ .
Note that, in general, we indicate with capital letters network-level attributes,while we use the lowercase for the sensor level.
As already mentioned, observations might be incomplete due to faults, partial observability, and costsof the data acquisition process.
We model data availability with a binary mask m it ∈ { 0 , 1 } which is 1 if the measurements associated with the i -th sensor are valid at time step t .
Conversely, if m it = 0 ,we consider the measurements x it to be completely missing, with the exogenous variables q it beinginstead available.
Notice that usually each m it are realizations of random variables with non-trivialdistributions exhibiting complex spatiotemporal dynamics and correlations.
For instance, a sensorfault may last for more than one time step, or spatially correlated errors may result in missing data inan entire sub-region of the network.
Multivariate Time Series Imputation Given G t : t + T with mask M t : t + T , we denote by (cid:102)
X t : t + Tthe unknown corresponding complete sequence, i.e., the sequence where no observation is missing.
Formally, the goal of multivariate time series imputation (MTSI) is to find an estimate (cid:99)
X t : t
+ Tminimizing the reconstruction error
[Equation]
where ℓ ( · , · ) is an element-wise error function and m iτ is the logical binary complement of m iτ .
Notice that, since (cid:102) X t : t + T is not available, one should find a surrogate objective or simulate thepresence of missing data, for which the reconstruction error can be computed.
Learning representations from sparse spatiotemporal data
The autoregressive approach to reconstruction consists in directly modeling distributions p (cid:0) x it | X <t (cid:1)and using one-step-ahead forecasting as a surrogate objective to learn how to recover missingobservations.
To exploit available data subsequent to the target time step, it is common to use abidirectional architecture, i.e., to mirror time w.r.t.
the time step of interest and have a second modelfor estimating p (cid:0) x it | X >t (cid:1) [49, 6].
Moreover, a third component p (cid:0) x it | { x j ̸ =
i t } (cid:1) must be introducedto account for spatial information at each step.
Architectures like GRIN, follow exactly this schemewith different components dedicated to model each of these three aspects.
While being effective inpractice, these approaches can have multiple drawbacks.
Besides the computational overhead ofhaving three separate components and the compounding error in the autoregressive models, they canfall short in capturing global context, as the processing of the structural information is decomposed.
Furthermore, merging the information coming from the different modules is also problematic,yielding to further compounding of errors.
Finally, in the case of highly sparse observations, thespatial processing should be dealt with special care as propagating information through partiallyobserved spatiotemporal graphs adds another layer of complexity.
For instance, simply masking outfaulty sensors would compromise the flow of information through message passing layers [9].
Model overview To address the limitation of existing methods, we act on the problem moredirectly by aligning the structure of our proposed architecture closely with the reconstructiontask.
We denote as observed set X t : t + T = (cid:8)(cid:10) x iτ , q iτ (cid:11) | m iτ =
∧ τ ∈ [ t, t + T ) (cid:9) the set ofall observations, paired with their spatiotempotal coordinates.
Conversely, we name target set
Y t : t + T = (cid:8) q iτ | m iτ =
∧ τ ∈ [ t, t + T ) (cid:9) the complement set collecting the coordinates of thediscrete spatiotemporal points for which we want to reconstruct an observation.
We refer to theset of observed and target points of the i -th node as X it : t + T and Y it : t + T , respectively.
Then, for allq iτ ∈ Y t : t + T , we aim at learning a structured model for
[Equation]
Our approach, named Spatiotemporal Point Inference Network (SPIN), is a graph attention networkfor MTSI, designed to learn representations of discrete points associated with nodes of a sequence ofspatiotemporal graphs.
Given disjoint observed and target sets X t : t + T and Y t : t + T , SPIN is trained tolearn a model
[Equation]
for all discrete points x iτ ∈ Y t : t + T .
To this end, SPIN learns a parameterized propagation processwhere each representation, corresponding to a specific node and time step, is updated by aggregating
[Figure]
V a li d
O b s e r v a t i on
[Figure]
L ×
M L P information from all the available observations acquired at neighboring nodes weighted by inputdependent attention coefficients.
Figure 1 shows an overview of the architecture.
In the nextparagraphs we will go through each component explaining in details and providing motivations foreach design choice.
We start by describing how we set up the propagation process.
Sparse spatiotemporal attention The core component of SPIN is a novel sparse spatiotemporalattention layer used to propagate information at the level of single observations.
Indeed, leveragingon the attention mechanism, we learn representations for each i -th node at each τ -th time step bysimultaneously aggregating information from (1) the observed set of i -th node X it : t + T (2) the observed
[Figure]
set X jt : t + T of its neighbors j ∈ N ( i ) .
Figure 2 shows a schematic representation of this procedure.
[Figure]
Let h i, ( l )
τ ∈ R d h be the learned representation for the i -th node and time step τ at the l -th layer.
Theencoding is initialized as
[Equation]
) where q iτ indicates a positional encoding, as previously anticipated, and MLP is a generic multilayer perceptron.
The representation is then updated by joint temporal and spatiotemporal attentionoperations.
To describe inner working of the proposed attention mechanism we adopt the terminologyof [50] and indicate as query the token for which we want to compute an updated representation, keya representation of the source tokens, and value the content representation of each token.
The nextsteps involve computations of spatiotemporal messages, i.e., representations computed to propagateinformation from one discrete space-time point to another.
We indicate the propagation along thetemporal dimension from time step s to time step τ as subscripts s → τ .
Conversely, superscriptsj → i indicate messages sent from the j -th node to the i -th.
To avoid overloading the notation, weomit the layer superscript in the following.
The message r j → i s → τ ∈ R d h from the j -th node at time steps to the i -th node at time step τ is computed as
[Equation]
where, following the adopted terminology, target point representation h iτ acts as query, source pointrepresentation h js as key and the computed message r j → i t → τ is the value.
Notice that, differently fromthe dot-product attention mechanism by Vaswani et al.
[50], we take into account both source andtarget representations to compute the value, similarly to the Bahdanau attention [51].
To account forspatial information, this mechanism is used to perform an inter-node temporal cross-attention.
Moreprecisely, for every neighbor j ∈ N ( i ) , we use h iτ to query – separately – every encoding in h jt : t + Tassociated with a valid observation, and collect the messages in set
[Equation]
Then, we linearly transform messages in R j → i τ (using trainable weights w α ∈ R d h × 1 ) and obtainmessage scores α j → i s → τ with a softmax layer, i.e.,
[Equation]
Then, we aggregate temporal messages coming from each j -th node separately, and obtain anedge-level context vector e j → i τ
[Equation]
which encodes the observed sequence at each j -th node w.r.t
the i -th node and time step τ .
Analogously, to account for the observed sequence of the i -th node itself, we exploit an intra-node temporalself-attention mechanism to query the encodings h
it : t + T corresponding to valid observations.
Then,resulting representations are weighted and aggregated to obtain a temporal context vector c iτ as
[Equation]
where message score α is → τ is obtained similarly as Eq.
Note that parameters are not sharedbetween the cross-attention and self-attention blocks.
After having obtained context vectors c iτ, ( l )and e j → i, ( l ) τ , the encoding h
i, ( l ) τ is updated with a final aggregation step as
[Equation]
After L layers, we obtain imputations for all spatiotemporal points in Y t : t + T with a nonlinear readout
[Equation]
Two-phase propagation Masking out tokens in the target set allows SPIN to propagate only validinformation.
As a downside, this results in blocking the flow of information on paths through points inthe target set.
This can be problematic when the input observations are extremely sparse.
Nonetheless,it is reasonable to assume that, after only a few propagation steps, the available information hasalready been partially diffused to locations with missing observations.
At this point, interrupted pathscan be unlocked, allowing for reaching higher-order neighborhoods.
In practice, we introduce ahyperparameter η to control the number of layers with masked connections and effectively split thepropagation process into two phases.
It is important to notice that what is being propagated in thesecond phase are learned representations, not observations (unavailable for masked tokens).
Graph subsampling and hierarchical attention Roughly speaking, the proposed spatiotemporalattention mechanism can be viewed as performing attention over the spatiotemporal graph S , obtainedby considering the product graph between space and time dimensions – with some connections prunedw.r.t.
unavailable data.
Let N max , E max be the largest number of nodes and edges, respectively,among graphs in G t : t + T .
Performing graph attention on the surrogate graph S has time and memory complexities that scale with O (( N max + E max ) T 2 ) (sparse implementations can alleviate thiscomplexity by replacing T 2 with the number of valid time step pairs).
To reduce this computationalburden – which undermines the application of the proposed method to large graphs and long timehorizons – we propose two different approaches.
The straightforward approach consists in trainingthe model by exploiting graph subsampling, using one of the many possible subsampling strategiesfrom the literature
(e.g., [52]).
In practice, at training time, we sample a k -hop subgraph centered onn target nodes and then compute the loss only w.r.t.
these n nodes.
In this way, we can reduce theamount of computation required by acting on n and k .
This type of subsampling can also be seen asa form of regularization [53].
A more interesting and orthogonal approach to reduce complexity is to rewire the attention mechanismto be hierarchical [54].
We do this by adding K dummy nodes that act as hubs for propagatinginformation.
Let Z i ∈
R K ×
d z be the the hub nodes’ representations for central node i , and then, forhub k :
1. update z ik by querying { h iτ | (cid:10) x iτ , q iτ (cid:11)
∈
X t : t + T } , i.e., node encodings associated with valid
observations, obtaining ˜ z
ik ;
2. update node encoding h iτ by querying updated (cid:101)
Z i and (cid:101)
Z j of every j -th neighbor in N ( i ) .
The spatiotemporal attention is effectively split in two phases.
At first, we update each hub noderepresentation similarly as Eq.
[Equation]
[Equation]
Then, we obtain context vectors from the updated hub representations as:
[Equation]
and update node representation h iτ as in Eq.
In this way, we can reduce the spatiotemporal attention complexity to O (( N max + E max ) KT ) with K ≪ T , at the cost of introducing an informationbottleneck.
We initialize the hub representations at layer l = 0 with random trainable parameters.
Spatiotemporal positional encoding We now come back to the problem of learning the spatiotemporal positional encodings Q t : t + T .
The encoding has to capture both spatial and temporal structureand be decoupled from the observations x it to allow inference for points in the target set.
Theseencodings can be directly extracted from available exogenous information (e.g., sensor location anddate/time features) or learned end-to-end jointly with the other model parameters.
We propose aspatiotemporal positional encoding q i t
= ρ ( u t , v i ) obtained by combining a temporal encoding
U ∈ R T × d u and a spatial encoding V ∈ R N × d v with a non linear transformation ρ , e.g.,
[Equation]
For the temporal encoding u t , we use sine and cosine transforms of the time step t w.r.t.
a period ofinterest (e.g., day and/or week), to account for seasonalities.
For the spatial encoding v i , we resortto a vector of learnable parameters different for each i -th node.
More advanced methods could beconsidered [55, 56].
Empirical evaluation
In this section, we evaluate our method on three real-world datasets and compare the performanceagainst state-of-the-art methods and standard approaches for MTSI.
As the objective of our approachis to address the imputation problem in highly sparse settings, in a second experiment we assess howperformance changes as the percentage of missing values increases.
[Table]
5.1 Experimental setting
In following experiments, we consider both SPIN and the hierarchical version SPIN-H (Sec. 4).
The figure of merit used is the mean absolute error (MAE), averaged across imputation windows.
We consider only the out-of-sample scenario [4], in which every parametric model is trained andtested on disjoint sets.
All the baselines have been implemented in PyTorch [57] using, wheneveravailable, public code provided by the authors.
Please refer to the appendix for more details about theexperimental setup.
Datasets We consider three openly available datasets coming from real-world SNs.
The firsttwo, namely PEMS-BAY and METR-LA [2], are two widely used benchmarks in spatiotemporalforecasting literature.
Each of them records traffic measurements every 5 minutes from 325 speedsensors in San Francisco Bay Area and 207 in Los Angeles County Highway, respectively.
Sincethe original datasets have a low number of missing values, we use the same setup of [4] to injectmissing data with two different policies: 1) Block missing , in which we randomly mask out 5% ofthe available data and, in addition, we simulate a failure lasting for S ∼ U (12 , 48) steps with 0 .
15%probability; 2) Point missing , in which we randomly drop 25% of the available data.
As a thirddataset, we consider AQI [58], which collects one year of hourly measurements of air pollutantsfrom 437 air quality monitoring stations over 43 cities in China.
We consider also a smaller versionof this dataset ( AQI-36 ) with only the 36 sensors scattered over the city of Beijing.
This dataset is apopular benchmark for imputation for the high number of missing values ( 25 .
67% in the completedataset)
and provides a mask for evaluation that simulates the true missing data distribution [15].
Fora given month, such a mask replicates the missing values patterns of the previous month, making thisscenario more similar to the block missing setting, for a total of ≈ 36% missing data.
In all settings,all the valid observations masked out are used as targets for evaluation.
We obtain an adjacencymatrix from the pairwise distance of sensors following previous works [2–4].
Baselines As the target of our approach are spatiotemporal graphs with missing observations, wecompare our methods against GRIN [4], a graph-based bidirectional RNN for MTSI with state-ofthe-art performance.
We then consider a spatiotemporal Transformer, where we alternate temporaland spatial Transformer encoder layers from [50] and replace missing values with a [MASK] token(as in [12]).
We consider also other deep imputation methods: 1) SAITS [25], a recent attentionbased architecture based on diagonally-masked self-attention; 2) BRITS [6], which leverages on abidirectional RNN; 3) rGAIN , an adversarial approach which shares similarities with GAIN [23]and SSGAN [21].
Finally, we report results of simpler methods that imputes missing values using4) node-level sequence mean ( MEAN ) or 5) neighbors mean ( KNN ); 6) Matrix Factorization ( MF ); 7)
[Table]
5.2 Results
[Figure]
reports the experiments results.
Both SPIN methods outperform the baselines in almost allscenarios.
Not surprisingly, improvements are more evident when entire blocks of data are missing,as in AQI datasets and block missing settings.
Differently from autoregressive methods, the sparsespatiotemporal attention mechanism of SPIN, in fact, is not autoregressive and allows an unboundedmemory capacity.
Conversely, in the point missing setting, SPIN methods perform on par withthe state of the art.
With respect to the spatiotemporal Transformer, SPIN performs better in allsettings except for AQI-36, which can be attributed to the ineffectiveness of spatial attention alone indetermining the dependencies among the different spatial locations.
Notice also that in almost allcases SPIN-H performs on par with SPIN (even better in the smaller dataset AQI-36), making it avalid lightweight alternative to SPIN.
To further show the advantages of SPIN in sparsely observed structured sequences, we compare ourmethods against the most relevant baselines on the same datasets by progressively increasing thesparseness in the input data distribution.
In practice, we test the models on the same test horizon ofthe previous experiment but, instead of using the previous evaluation masks, we reduce the numberof available observations by associating a probability p of being removed to each observation.
Thehigher is p , the more likely it is to have blocks of consecutive missing values.
For the traffic datasets,we use the weights of the models trained on the point missing setting.
Table 2 shows results (interms of MAE) averaged over 5 different evaluation masks for each missing probability p .
In all theconsidered scenarios, SPIN-based models rank as best performing methods for any sparsity level.
Moreover, they are more robust to changes in missing data distribution.
In fact, compared to thebaselines, the performance of both SPIN and SPIN-H deteriorates at a slower rate as the sparsity dataincreases.
Notably, GRIN would require much more data to match the performance that SPIN is ableto achieve with only 5 % of valid observations.
Improvements over attention-based methods SAITSand Transformer are also striking.
Conclusion and Future Works
We introduced a graph-based architecture, named SPIN, to reconstruct missing observations in sparsespatiotemporal time series.
To overcome the major limitations of autoregressive methods, we designeda novel sparse spatiotemporal attention mechanism to propagate valid observations through discretepoints in time and space, jointly.
Furthermore, we showed how the time and space complexities of theapproach can be drastically reduced considering a novel hierarchical attention mechanism.
Empiricalanalysis shows that the proposed method outperforms by a wide margin state-of-the-art methodsfor imputation in highly sparse settings.
We noticed that in some extremely sparse training settings,
SPIN might suffer from a lack of supervision that may slow down learning; future works might try toaddress this problem by introducing an additional auxiliary learning task.
Other possible extensionscould investigate spatiotemporal positional encoding methods and their application in SNs.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 4 and 5.
(b) Did you describe the limitations of your work?
[Yes] Assumptions discussed in Section 3, limits in Sections 4 and 6.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We do notsee any potential harm to society in the proposed method.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide the
experimental setup, as well as all datasets used, as supplementary material. (b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] As supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] As supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] We provide links to datasets and code
that contain also information about the license
c ) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
AppendixA Detailed experimental setup
In this appendix, we discuss in details the experimental settings.
We use the same setup of Cini et al.[4] 12 .
Indeed, Table 1 reports results from [4] whenever possible.
We refer to [4] for details on thesebaselines.
For SPIN, we use the same hyperparameters in all datasets: L = 4 layers, with first η = 3 layerswith masked connections; hidden size d h = 32 ; 2 layers with hidden size 32 for every MLP; ReLUactivation functions.
For SPIN-H, we use similar hyperparameters, but 5 layers, with η = 3 ; K = 4hubs per node with d z = 128 units each.
These hyperparameters have been selected among a smallsubset of options on the validation set; we expect far better performance to be achievable with furtherhyperparameter tuning.
Depending on the dataset, the number of parameters ranges from ≈ 55 Kto ≈ 95 K for SPIN and ≈ 540 K to ≈ 800 K for SPIN-H.
We use Adam optimizer [60], learningrate lr = 0 . 0008 and a cosine scheduler with a warm-up of 12 steps and (partial) restarts every100 epochs.
We train our models with 300 minibatches of 8 random samples per epoch, fixing themaximum number of epochs to 300 and using early stopping on the validation set with patience of40 epochs.
Due to constraints on memory capacity on some of the GPUs (see the description ofthe hardware resources below), for SPIN-H we set the batch size to 6 and 16 in AQI and AQI-36,respectively.
To train SPIN-based models, we minimize the following loss function:
[Equation]
where ℓ ( · , · ) is the absolute error and ˆ x i, ( l ) τ is l -th layer imputation for the i -th node at time step τ .
Note that, to provide more supervision to the architecture, the loss is computed and backpropagatedw.r.t.
representations learned at each layer, not only at the last one.
The error is computed only ondata not seen by the model at each forward pass.
For this reason, we randomly remove p ratio of theinput data for each minibatch sample, with p sampled uniformly from [0 . 2 , 0 . 5 , 0 . 8] , and use them tocompute the loss.
We never use data masked for evaluation to train any model.
For the spatiotemporal Transformer baseline, we use the same training strategy and a similar hyperparameters configuration of SPIN-H:
L = 5 layers; 4 attention heads; hidden size and feed-forwardsize of 64 and 128 units, respectively.
For SAITS, we use the code provided by the authors 3 .
Hyperparameters for SAITS have been selected on the validation set with a random search by usinghyperparameter ranges from the original paper.
We recall that the time and memory complexities of SPIN and SPIN-H scale with O (( N max +
E max ) T 2 ) and O (( N max + E max ) KT ) , respectively.
For the sake of comparison, here we alsoreport the asymptotic complexities of the spatiotemporal Transformer and GRIN.
The Transformeralternates temporal attention (i.e., O ( N max T 2 ) ) and spatial attention (i.e., O ( TN 2max ) ), with aresulting O (( N max + T ) N max T ) complexity.
Let R be the spatial receptive field (i.e., number ofgraph convolution layers) of the inner MPGRU cell, the time complexity required to process a singledirection in GRIN scales with O ( TRE max ) .
Note also that while most of the operations in theattention-based models can be executed in parallel, GRIN would need to process the entire sequencerecurrently, with a consequent performance slowdown at execution time.
All the models were developed in Python [61] using PyTorch [57], PyG [62] and Torch Spatiotemporal [63].
We use Neptune 4 [64] for experiments tracking.
The code to reproduce the experiments of the paper is available as supplementary material.
All the experiments have been run in a cluster using
GPU-enabled nodes with different hardware setups.
Running times of SPIN-H training on a nodeequipped with a 12GB NVIDIA Titan V GPU range from 4 to 14 hours (depending on the dataset).
For SPIN we used a node with 40GB NVIDIA A100 GPU, with running times ranging from 4 to 26hours.
B Datasets
In this appendix, we provide details on datasets and preprocessing used for the experiments.
Weuse temporal windows of T = 24 steps for all datasets except AQI-36, for which we set T = 36 .
For traffic datasets, we split the data sequentially as 70% for training, 10% for validation, and 20%for test.
For air quality datasets, following Yi et al.
[15], we consider as the test set the months of
March, June, September, and December and we use valid observation x iτ as ground-truth if the valueis missing at the same hour and day in the following month.
For data preprocessing we use thesame approach of Cini et al.
[4], by normalizing data across the feature dimension (graph-wise forgraph-based models) to zero mean and unit variance.
In line with [3, 4], we obtain the adjacency matrix from the node pairwise geographical distancesusing a thresholded Gaussian kernel [65]
[Equation]
otherwise
[Equation]
[Equation]
where dist ( · , · ) is the geographical distance operator, γ is a shape parameter and δ is the threshold.
C Virtual sensing
[Figure]
no . no .
The spatiotemporal cross-attention mechanism allows SPIN variants to exploit validobservations at neighboring nodes to impute missing values at spatial locationswhere no sensor is available, thus performing virtual sensing (also referred to as kriging [66]).
We assess the performance of
SPIN-H in the virtual sensing task by completely masking out observations at twonodes during training and then using thetrained model to infer the entire sequencesat the masked nodes.
For this experiment,we keep the same architecture hyperparameters and consider the dataset AQI-36,masking out the two nodes with the highest(station no. 1015) and lowest (no.
1032)number of neighbors, reproducing the experimental settings of [4].
Figure 3 showsthe performance of SPIN-H in reconstructing a temporal window of 36 time steps for the two virtualsensors at test time.
Results qualitatively show that our method is able to reconstruct observations forsensors with no available information other than their location in the network.
D Ablation study
Table 3 shows the results of an ablation study on METR-LA (Point missing) and AQI-36.
Here, weevaluate the performance in terms of mean absolute error (MAE) and mean relative error (MRE).
Weconsider two different versions of SPIN-H in which we remove the spatiotemporal cross-attention
[Table]
and the attention components, the performance
SPIN,
SPIN-H and the for reference.
Results clearly show both components contribute positively to imputation accuracy.
We are masked out
E Performance on data with simulated failures
In Table 2, we show the reconstruction error of the different methods as the number of validobservations in input sequences decreases.
In detail, for every considered dataset, we removeobservations uniformly at random and then compute the reconstruction error for the masked-outsamples.
To further assess the performance of our model, we show results for a different setting,where data are removed according to the Block missing policy (see section 5).
In particular, wesimulate the case in which, at each time step, every sensor has a constant probability ¯ p of going offlinefor a random number S ∼ U (12 , 36) of future (consecutive) time steps.
Table 4 shows results for alldatasets with ¯ p = 5% , ¯ p = 10% , and ¯ p = 15% .
Note that, depending on the dataset, the portion ofvalid observations in each of these cases amounts to ≈ 25 - 30% , ≈ 8 - 10% , and ≈ 3 - 4% , respectively.
[Table]
SPIN models, differently from the baselines, can handle all the considered scenarios.
In particular,improvements in performance w.r.t.
the best performing baseline (GRIN) are more evident as thenumber of available observations decreases.