Paper: Graph-based Virtual Sensing from Sparse and Partial Multivariate Observations (ICLR-2024)

Reviewer comment (Theory):
This paper has a lot of assumptions, for example, it assumes the covariates are always available, and the mutual dependencies between target and covariates are invariant and can be leveraged at all locations. Here are some concerns: we cannot guarantee that those assumptions hold for all cases (at least, there should be more examples or references to support them), and if they do not hold, does the proposed method still work?

Author response:
**W1. Paper's assumptions**\
Please allow us to clarify a possible misunderstanding. Our assumptions are only those dictated by the task which our method has been specifically designed to solve. In particular, for what concerns the availability of covariates, at least some of these are needed for the task to make sense, otherwise, no reconstruction would be possible in sparse settings. 
Furthermore, note that we do *not* require all covariates to be available at all locations. On the contrary, as shown in Fig.1, covariates can be partially missing at any location.
Clearly, the quality of the reconstruction will depend on the amount of relevant information available to condition the virtual sensing model.
We would like to point out that the covariate availability assumption is not a weakness of our proposed approach, but a hard requirement of the addressed sparse virtual sensing task. In general, this corresponds to the necessity of all ML models to condition inference on observations.

Then, similarly to the previous point, assuming a mutual dependency between the target and covariates is a requirement imposed by the task. If this cannot be leveraged at the target location, it is not possible to perform any inference in a sparse setting.

Finally, assuming synchronous observations and overlapping time frames among sensors is necessary to exploit the spatial dimension. 
Although the synchronicity assumption could be alleviated to some extent, we consider this aspect orthogonal to those we deal with in the present paper and mention this direction as a possible future work.

Reviewer comment (Theory):
It seems the method requires the latent representations of locations where the target values need to be reconstructed are close, so is it possible to deal with the cases where the latent representations of locations are remote?

Author response:
**W2. Far-apart latent representation of locations**\
In our framework, far-apart representations in the latent space can, potentially, translate into widely different observed dynamics. In the transductive setting, the masking strategy still brings supervision for training locations with isolated representations. However, in general, if no available observed location is at all similar to the target one, there may be no guarantee of the accuracy of the reconstruction. This is due to the intrinsic limits of any data-driven method used to extrapolate beyond the training conditions.

Reviewer comment (Novelty):
The model somewhat lacks novelty, it seems the final model just combines several existing models.

Author response:
**W3. Discussion on the paper's novelty**\
The focus of the paper is not on introducing new layers or new neural operators. 
Instead, we are the first to explore (multivariate) virtual sensing for sparse settings, which is relevant in many practical applications.
Most importantly, we introduce a conceptualization and propose a methodology to address the general problem, which goes far beyond the possible specific implementations. Nevertheless, we also propose a possible implementation (GgNet) of such a methodology, which utilizes an original nested graph representation for effectively modeling dependencies in such a context.
We finally provide experimental evidence of how the proposed model can overcome the limitations of previous approaches. 
We believe these claims are significant and contribute to advancing the field in a meaningful way.
We refer the reviewer to the introduction of the paper where our core novel contributions are explicitly stated. After the revision, we have added references to the section supporting each claim.
We hope this clarifies any doubts regarding the novelty aspect of the paper; if not, we would deeply appreciate any further comment.

Reviewer comment (Writing):
The paper, while generally well-structured, could benefit from more explicit and detailed explanations in certain technical aspects. For instance, the paper could provide a more comprehensive explanation of the nested graph structure used in GgNet, which is a key component of the proposed method.

Author response:
**W1. More detailed explanations in certain technical aspects**\
We understand that there are several important technical details that were deferred to the appendix (in particular, in Appendix B). However, due to space constraints, we preferred to allocate more space to the conceptualization of the problem and to the methodological aspects that are the main paper contributions.

Reviewer comment (Experiments):
The paper mentions the computational time for GgNet and other baselines but does not discuss scalability. How does GgNet's performance scale with more extensive sensor networks or larger datasets?

Author response:
**Q1. Discussion on scalability**\
Scalability is discussed in Appendix E.1. The computational complexity of GgNet originates from the sum of the complexities of its components.  
The time and space complexity are both $\mathcal O(N^2TD) + \mathcal O(NTD^2)$.  
As it is often safe to assume that $D^2 \ll N^2$, the overall complexity reduces to $\mathcal O(N^2TD)$.  
If scalability to large N is found to be a concern, existing sparse graph learning alternative methods, e.g., [1], can be integrated into the approach.  
We thank the reviewer for the question and have expanded the discussion on scalability in Appendix E.1 of the revision.  

We hope to have clarified all of the reviewer’s concerns and are happy to provide further details.  

**References:**\
[1] Niculae et al., Discrete latent structure in neural networks. arXiv preprint arXiv:2301.07473, 2023.

Reviewer comment (Evaluation):
This article lacks a thorough analysis of the experimental results and a comprehensive interpretation of the data. It primarily presents the experimental outcomes in tables without providing detailed explanations.

Author response:
**W1/Q3. Discussion of the experimental results**\
In Sec. 5 of our paper, several paragraphs are dedicated to the discussion of the experimental results.
Our discussion focuses on the limitations of the existing methods compared to GgNet, which we investigate across different temporal and spatial problem settings. 
We believe that discussing limitations is the most important aspect when interpreting experimental results, as it directly relates to the observed differences in performance. 
From a qualitative angle, Fig. 5 shows results w.r.t. the quality of the prediction and the structure of the learned embeddings. 
However, we agree that further comments might be needed, so we have extended our discussion in Sec. 5.1 of the revised paper. Furthermore, we have performed $4$ additional analyses (see highlighted content in the revised paper Appendix) that significantly contribute to the discussion:
- in Appendix E.3, we now deepen the disadvantages of graph-based approaches in contrast to GgNet by means of an illustrative example in a geographically remote location; 
- in Appendix E.4, we now show and discuss the GgNet's learned dependencies between variables (intra-location graph $g$); 
- in Appendix E.5, we now investigate location-wise performance and show that the improvement of GgNet with respect to SAITS is spread across locations rather than concentrated around a few locations;
- in Appendix E.6, we now qualitatively compare uncertainty estimates of GgNet across different scenarios.

Any comment on these revisions would be appreciated.

Reviewer comment (Presentation):
The article did not include an overview diagram of the overall model structure, which made it difficult for readers to comprehend the architecture of the model.

Author response:
**W2. Overview diagram of the overall model structure**\
Please, refer to Fig. 3, which provides an overview of the model's architecture and highlights the different components.
In the revised paper, we bring modifications to the figure's caption.
We would appreciate further feedback if, in the reviewer's opinion, any aspect of the figure could be improved.
Further details about the architecture are provided in Appendix B.

Reviewer comment (Evaluation):
The experiment evaluated accuracy-related metrics such as MAE and MRE but did not provide information on the computation time and performance on large datasets. As a result, readers cannot assess the efficiency of this model.

Author response:
**W3. Computation time and performance on large datasets.**\
Appendix E.1 reports the total computation time of GgNet and relevant baselines, on the largest adopted dataset, i.e., the daily climatic dataset.
In the revised paper, we extend the pertinent discussion.
The temporal modeling in GgNet is based on temporal convolutions, which makes it faster than the popular graph-based representative GRIN, which uses a recurrent model instead.
As for BRITS and SAITS, these are faster than GgNet as they do not account for spatial dependencies.
The main bottleneck in GgNet is due to the inter-locations graph which makes message-passing operations scale quadratically w.r.t. the number of locations. If scalability to large $N$ is a concern, existing sparse graph learning methods can be considered, e.g., [1].
However, note that scenarios that involve significantly more sensors than the selected datasets would not likely be sparse and a different set of techniques should be used.
We thank the reviewer for highlighting this and we have integrated the above considerations into Appendix E.1 of the revision.

Reviewer comment (Experiments):
The experimental training methods have the potential to enhance the robustness of the network, but in the end, no robustness test results for the model were provided.

Author response:
**W4/Q2. Robustness test results.**\
As mentioned in Sec. 4.4, in the first place, we mask out a fraction of the available training channels (entirely for all timestamps in a batch) and train the models to reconstruct such masked portions of data.
This is a core aspect, as it pushes the training toward learning the virtual-sensing task. Secondly, we also mask out randomly a portion of the input to provide additional supervision to the training routine.
We recognize that the paper would benefit from a sensitivity analysis of these aspects. As such, we have carried out additional experiments by varying the number of channels being masked out at training time. 
Results are reported in Appendix E.7 of the revised paper.

Reviewer comment (Experiments):
The experiment did not conduct tests of this framework with different data types of other domains, which makes it difficult to assess the comprehensiveness of its application in various fields.

Author response:
**W5. Different data types of other domains.**\
The focus of the paper is addressing virtual sensing for sensor networks with poor spatial coverage by proposing a graph-based reconstruction framework. 
For this purpose, we select $3$ datasets from very relevant application domains, which we believe are sufficient to validate the relevance of the proposed methodology and the potential effectiveness of GgNet. 
The application to more general domains and the design of appropriate benchmarks are extremely relevant, yet they go beyond this paper's scope and will constitute interesting future research.

Reviewer comment (Theory):
Why is GgNet only applicable to transductive learning settings in its current form? What are the limitations? How to extend GgNet to support inductive learning?

Author response:
**Q1. Why is GgNet only applicable to transductive learning settings in its current form?**\
GgNet is intrinsically transductive as the node embeddings, which are key to building the inter-location graph and adding local components to different layers, need to be learned for every location from the available data.
Despite that, for these types of models, the fine-tuning of new node embeddings can be done very efficiently, as shown in [2].
Different strategies can be considered to make GgNet inductive, e.g., by relying on training-free approaches to obtain node embeddings. We thank the reviewer for this question and clarify this in the revision.

We hope to have clarified all of the reviewer’s concerns and are happy to provide further details if needed.

**References:**\
[1] Niculae et al., Discrete latent structure in neural networks. arXiv preprint arXiv:2301.07473, 2023.\
[2] Cini et al., Taming Local Effects in Graph-based Spatiotemporal Forecasting, arXiv preprint arXiv:2302.04071, 2023.

Reviewer comment (Experiments):
The claim that Kriging or similar approach won't work due to sparse sensors lacks sufficient justification. The proposed solution can be solved by multivariate Kriging. The authors should at least show the results on such a well-known spatial solution and compare it with their solution.

Author response:
**W1/Q1 (1). Limitations of Kriging in sparse settings**\
Kriging, in general, relies on geographical coordinates, which are used to model spatial dependencies (physical proximity). 
With a dense sensor coverage and high, local, spatial correlation, general kriging methods will perform competitively w.r.t. deep learning methods [1]. However, without the possibility of relying on such pre-defined spatial dependencies, kriging methods might fail. Our method, instead, explicitly learns (end-to-end from data) latent non-linear dependencies among the observed variables both intra and inter-location. This property becomes more important when poor sensor coverage prevents from performing geographical interpolation. 
Lastly, note that OKriging is unsuitable for applications on irregular spatial domains [1], e.g., transportation networks.


**W1/Q1 (2). Kriging baseline**\
While we recognize that adding more standard Kriging baselines would improve completeness, we do argue that it is not a critical element to support our claims. In our study, we benchmark against GRIN, a representative of spatio-temporal Graph Neural Networks (ST-GNN) and a state-of-the-art model for coordinate-based reconstruction. The established literature consistently shows the superior performance of these modern virtual sensing methodologies over standard Kriging in similar settings, e.g., [1,2,3,4].

In particular, note that:
- Most of the standard kriging methods cannot handle temporal data (fundamental in our setting) while the selected (state-of-the-art) baselines explicitly target these scenarios; 
- Learning a proper spatiotemporal variogram is very computationally demanding, especially for long and multivariate time series;
- Kriging consists of a *linear interpolation* of the observed data and has a strong Gaussian assumption. More recent models (ours included), instead, are designed to capture *non-linear* dependencies between locations in a learned latent space. 
    
Given the above, we argue that our choice of baselines is coherent with the problem settings and the current state of the art.

Reviewer comment (Theory):
The graph of sensors is built by measuring similarities between node embeddings. However, it is unclear how to learn such embeddings and why it is not affected by sensor sparsity. If two sensors are too far away their values have no correlation. The edge defined in under such a case would be meaningless.

Author response:
**W2/Q3. Embedding learning**\
There might be a critical misunderstanding here. As stated in Sec. 4.2, node embeddings are a table of learnable parameters [5,6] and, as such, are end-to-end learned jointly with the model weights given the task at hand. In our case, node embeddings are learned together with the reconstruction model by minimizing the reconstruction loss. Note that the computations carried out to extract the inter-location graph from such embeddings (Eq. 3) are seamlessly integrated within the end-to-end learning architectures. In other words, the graph is learned end-to-end as well.

Once learned, the embeddings identify each location in a common latent space, of which we show an example in Fig. 5 (right). This placement is not directly influenced by the geographical coordinates of each sensor but is learned exclusively from the observed data (targets and covariates).
Similarity in such space, then, is not limited to model physical proximity but can capture non-linear dependencies beyond that.

Reviewer comment (Experiments):
There is no demonstration of a successful prediction of missing values on a spatial map.

Author response:
**W3. Visualization on spatial map**\
Generating such visualizations poses challenges in our context. The extensive area that is covered by sparse sensors makes creating spatial maps computationally very demanding. Additionally, our data is multivariate, adding complexity to visualizing all dimensions effectively in a single spatial map. Moreover, as already mentioned, dependencies in our framework extend beyond physical proximity. 

Nevertheless, we argue that both the main and the appendix provide an in-depth analysis of the empirical results highlighting several aspects of the proposed method such as metrics for each channel and location, robustness, uncertainty estimation, and visualizations of the learned embeddings.

Reviewer comment (Theory):
Why Kriging would not work in this case is not adequately justified. The semi-variogram function can be selected from a variety of options depending on the assumptions. As long as the two locations are not farther than a threshold (the range) apart, they can be assumed to have correlations.

Author response:
**W1/Q1 (1). Limitations of Kriging in sparse settings**\
Kriging, in general, relies on geographical coordinates, which are used to model spatial dependencies (physical proximity). 
With a dense sensor coverage and high, local, spatial correlation, general kriging methods will perform competitively w.r.t. deep learning methods [1]. However, without the possibility of relying on such pre-defined spatial dependencies, kriging methods might fail. Our method, instead, explicitly learns (end-to-end from data) latent non-linear dependencies among the observed variables both intra and inter-location. This property becomes more important when poor sensor coverage prevents from performing geographical interpolation. 
Lastly, note that OKriging is unsuitable for applications on irregular spatial domains [1], e.g., transportation networks.


**W1/Q1 (2). Kriging baseline**\
While we recognize that adding more standard Kriging baselines would improve completeness, we do argue that it is not a critical element to support our claims. In our study, we benchmark against GRIN, a representative of spatio-temporal Graph Neural Networks (ST-GNN) and a state-of-the-art model for coordinate-based reconstruction. The established literature consistently shows the superior performance of these modern virtual sensing methodologies over standard Kriging in similar settings, e.g., [1,2,3,4].

In particular, note that:
- Most of the standard kriging methods cannot handle temporal data (fundamental in our setting) while the selected (state-of-the-art) baselines explicitly target these scenarios; 
- Learning a proper spatiotemporal variogram is very computationally demanding, especially for long and multivariate time series;
- Kriging consists of a *linear interpolation* of the observed data and has a strong Gaussian assumption. More recent models (ours included), instead, are designed to capture *non-linear* dependencies between locations in a learned latent space. 
    
Given the above, we argue that our choice of baselines is coherent with the problem settings and the current state of the art.

Reviewer comment (Experiments):
Another traditional model that might solve this problem is Markov random field. Why not considering it in the baseline?

Author response:
**Q2. Why not consider Markov random field**\
The extension of Markov random fields to our problem settings is not trivial, as discussed above with respect to standard kriging methods. 
While we have discussed why we believe that our choice of baselines is sufficient for the purpose of this paper, if the reviewer has a particular method in mind, that would be suitable to our settings, we will be happy to consider including it in the discussion.