Paper: Exploiting River Network Topology for Flood Forecasting with Graph Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
The key issue with the current paper is the message is mostly a negative result, that incorporating the network structure does not help with the forecast of flood. Although there are some experimental support to this, it is difficult to draw this conclusion because the authors have only tried a limited set of models. As the claim is counter-intuitive, more analysis, especially analysis of the raw data, is required for supporting the claim.

Author response:
> The key issue with the current paper is the message is mostly a negative result, that incorporating the network structure does not help with the forecast of flood.

We believe that especially unintuitive negative results should be published to highlight the finding to the community. This may motivate other researchers to investigate the issue from their different viewpoints and complete the picture.


---

> Although there are some experimental support to this, it is difficult to draw this conclusion because the authors have only tried a limited set of models.

The set of models we employed is actually quite large: 3 major architectures times 18 topology combinations and each cross-validated on 6 different folds of a big dataset, which amounts to 324 models. This does not include the additional 240 models for the depth experiment. While we agree there are more architectures and choices that can be explored, we had to restrict our search space to retain computational feasibility. This issue would be mitigated by publishing our results so the community can address the problem with its accumulated computational resources.

Reviewer comment (Evaluation):
Are there correlations between the water level in two gauge stations upstream and downstream? And are there time lags between a spike in an upstream gauge station and a downstream gauge station? If so, the spike in the upstream gauge station should be useful for predicting the spike in the downstream gauge station. Why is this not reflected in the experiment results?

Author response:
> Are there correlations between the water level in two gauge stations upstream and downstream?

Physically, they are clearly correlated since they share the same water stream. Statistically, we also find linear cross-correlations, but the corresponding shift naturally varies.

---

> How long does it take on average for the flow from one upstream gauge station to reach a neighboring downstream gauge station? I think these values should be taken into account for the history window size (24 hours currently) and the prediction horizon (6 hours currently). 

> And are there time lags between between a spike in an upstream gauge station and a downstream gauge station?

We argue that the time lags are not relevant for prediction since the flow fluctuation and spikes will ultimately be reflected in the downstream gauge station's own discharge history.

Reviewer comment (Evaluation):
I have reservations about the way the problem is modeled. Since we are interested in predicting flood, which is a rare event, fitting the average MSE loss to the time series data might not be the best approach. Assuming the floods are rare large spikes in the data, a conservative model will do best by trying NOT to predict a large spike, as getting the timing of the spike wrong can incur a huge MSE loss.

Author response:
> I have reservations about the way the problem is modeled. Since we are interested in predicting flood, which is a rare event, fitting the average MSE loss to the time series data might not be the best approach. Assuming the floods are rare large spikes in the data, a conservative model will do best by trying NOT to predict a large spike, as getting the timing of the spike wrong can incur a huge MSE loss. It could, for example, be modeled alternatively as a time series prediction problem, where we try to predict if there is a flood event within the next 6 hours given the water level in the past 24 hours. There could be multiple ways to model this but MSE on the water level does not seem the right fit for modeling rare spike events.

We agree that a rare event detection setup would be strictly preferable. However, such an approach requires ground-truth annotations of all flooding events, which the dataset unfortunately does not provide. Hence, the only option with the given data is regression of river discharge. On the other hand, the datasets that do have flood annotations are lacking the graph structure. We apologize for not having made this clear in the draft and updated Section 3.1 accordingly.

Reviewer comment (Evaluation):
It could, for example, be modeled alternatively as a time series prediction problem, where we try to predict if there is a flood event within the next 6 hours given the water level in the past 24 hours. There could be multiple ways to model this but MSE on the water level does not seem the right fit for modeling rare spike events.

Author response:
> I have reservations about the way the problem is modeled. Since we are interested in predicting flood, which is a rare event, fitting the average MSE loss to the time series data might not be the best approach. Assuming the floods are rare large spikes in the data, a conservative model will do best by trying NOT to predict a large spike, as getting the timing of the spike wrong can incur a huge MSE loss. It could, for example, be modeled alternatively as a time series prediction problem, where we try to predict if there is a flood event within the next 6 hours given the water level in the past 24 hours. There could be multiple ways to model this but MSE on the water level does not seem the right fit for modeling rare spike events.

We agree that a rare event detection setup would be strictly preferable. However, such an approach requires ground-truth annotations of all flooding events, which the dataset unfortunately does not provide. Hence, the only option with the given data is regression of river discharge. On the other hand, the datasets that do have flood annotations are lacking the graph structure. We apologize for not having made this clear in the draft and updated Section 3.1 accordingly.

Reviewer comment (Experiments):
How long does it take on average for the flow from one upstream gauge station to reach a neighboring downstream gauge station? I think these values should be taken into account for the history window size (24 hours currently) and the prediction horizon (6 hours currently).

Author response:
> How long does it take on average for the flow from one upstream gauge station to reach a neighboring downstream gauge station? I think these values should be taken into account for the history window size (24 hours currently) and the prediction horizon (6 hours currently). 

> And are there time lags between between a spike in an upstream gauge station and a downstream gauge station?

We argue that the time lags are not relevant for prediction since the flow fluctuation and spikes will ultimately be reflected in the downstream gauge station's own discharge history.

Reviewer comment (Experiments):
What is the sample size for each gauge station? Or more importantly, what is the average number of flood events that each gauge station experience in the data? If the sample size is small, it could be beneficial to just pool all the data and train one model for the time series prediction of rare events, than to spread the rare event samples across multiple stations and try to model their correlations. Could that be a reason why the incorporation of network structure is not helping here?

Author response:
> What is the sample size for each gauge station? Or more importantly, what is the average number of flood events that each gauge station experience in the data? If the sample size is small, it could be beneficial to just pool all the data and train one model for the time series prediction of rare events, than to spread the rare event samples across multiple stations and try to model their correlations. Could that be a reason why the incorporation of network structure is not helping here?

Pooling across all gauges, as suggested by the reviewer, would contract the river network graph into a singular point, thus making any topology considerations meaningless. There is a body of research on improving discharge prediction by training on pooled data from multiple gauges, but that caters to an entirely different research question than the one we are addressing.

Reviewer comment (Evaluation):
The authors claim that the methods perform similarly. From Table 2 it seems to be true for NSE, but MSE has a lot more variations across the different methods. Why is this the case?

Author response:
> The authors claim that the methods perform similarly. From Table 2 it seems to be true for NSE, but MSE has a lot more variations across the different methods. Why is this the case?

Discharge scale across gauges varies massively. The MSE we show in Table 2 is not scale-normalized, while NSE is (cp. Section 3.2, Metrics). Hence, MSE is highly sensitive to even minor errors on gauges in broad waterbeds. However, we want the model to do equally well on all gauges of the river network. For this reason (among others), hydrology papers typically look at scale-normalized metrics such as NSE. We only included the non-scale-normalized MSE for completeness. Note that during training, the data is normalized, and thus, the optimization objective an implicitly scale-normalized MSE. We updated the caption of Table 2 to prevent any potential confusion in this regard.