Paper: DASFormer: Self-supervised Pretraining for Earthquake Monitoring (ICLR-2024)

Reviewer comment (Evaluation):
Ignoring the DASFormer approach for a second, the results table shows that the "traditional" aggregation baseline "aggregation-0" is actually really strong, on par with or better than all the deep learning models, which could illustrate that there might be some other, much simpler modelling approach to achieve good results.

Author response:
Thanks for pointing out this. Aggregation-0 in Table 1 is a variant of well-known traditional method "STA/LTA" for DAS data. 

The traditional time-series methods like Random Forest and ARIMA-family are not applicable to our task, as they are designed for stationary time series. However, DAS data is characterized by dynamic and evolving patterns, making it challenging to model using traditional stationary time series methods

Reviewer comment (Experiments):
There is literature, for example [Makridakis et al. 2020] and [Nixtla], showing that simple models outperform deep learning methods on time-series tasks, so it would be interesting to see what results could be achieved with a Random Forest, which could also offer more insights into explaining a model prediction.

Author response:
Thanks for pointing out this. Aggregation-0 in Table 1 is a variant of well-known traditional method "STA/LTA" for DAS data. 

The traditional time-series methods like Random Forest and ARIMA-family are not applicable to our task, as they are designed for stationary time series. However, DAS data is characterized by dynamic and evolving patterns, making it challenging to model using traditional stationary time series methods

Reviewer comment (Experiments):
The authors point out several times that it is not appropriate to train models with the standard point-wise loss. However, this point-wise loss is applied in all the baseline models that DASFormer is compared to, so there should be a comparison to a baseline that does not have this weakness.

Author response:
Thanks for pointing out this. To alleviate the drawback of point-wise loss, DASFormer utilizes an extra pretrained VGG-based on intermediate features (it's a features-to-features autoencoder, not on the final predictions) in DASFormer to extract the high-level features, and use the feature-wise loss as the metric. This feature extractor is not universal for other methods, so we have to design feature extractors/encoders to suit the specific architecture and characteristics of each baseline model. Therefore, we only use the point-wise loss for other baselines.

Reviewer comment (Evaluation):
Given the crucial task of earthquake monitoring and the acknowledgment that "DAS data includes various stochastic noise, such as environmental and instrumental noise," it is surprising that there is no mention or discussion of "uncertainty" in the paper. Even though uncertainty is not trivial to handle, especially with overparameterized deep learning models, there should be at least an effort to discuss its influences and impacts.

Author response:
Thanks for your suggestion. We show the confidence intervals of the results in Table 4 in Appendix, where we can observe that the results are stable and reliable with small variance. Due to the limitations of time and computation, we will explore and add more uncertainty analysis in the future work. 

We also conduct comparison experiments for different magnitude (different noise levels) of earthquakes. The results are shown in the following tables:

Distribution of the magnitude of the 45 earthquakes in the training set:

| Magnitude | Numbers |
|-----------|---------|
| M0 ~ M1   | 0       |
| M1 ~ M2   | 0       |
| M2 ~ M3   | 31      |
| M3 ~ M4   | 12      |
| M4 ~ M5   | 2       |

Distribution of the magnitude of the 45 earthquakes in the evaluation set and the performance:

|DASFormer          |         | AE   |      | EMD  |      | sliced EMD |      |
|-----------|---------|------|------|------|------|------------|------|
| Magnitude | Numbers | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| M0 ~ M1   | 0       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M1 ~ M2   | 1       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M2 ~ M3   | 27      | 0.84 | 0.4  | 0.88 | 0.46 | 0.86       | 0.42 |
| M3 ~ M4   | 14      | 0.89 | 0.59 | 0.92 | 0.58 | 0.93       | 0.6  |
| M4 ~ M5   | 3       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |

|Aggregation-0 |         | AE   |      | EMD  |      | sliced EMD |      |
|-----------|---------|------|------|------|------|------------|------|
| Magnitude | Numbers | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| M0 ~ M1   | 0       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M1 ~ M2   | 1       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M2 ~ M3   | 27      | 0.74 | 0.24 | 0.7  | 0.28 | 0.7        | 0.25 |
| M3 ~ M4   | 14      | 0.73 | 0.24 | 0.69 | 0.26 | 0.7        | 0.25 |
| M4 ~ M5   | 3       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |


For the earthquakes in the range of M2 ~ M3 and M3 ~ M4, we can see that the performance is becoming better for larger magnitude earthquakes. In each level, the performance of DASFormer is better than Aggregation-0. For the earthquakes in the range of M0 ~ M2 and M4 ~ M5, the number of samples is too small to draw a conclusion.

Reviewer comment (Experiments):
Under "5.1 Implementation Details," it is stated that the anomaly score is defined between the predicted model values and the actual values. It is unclear how the approach works in a "real" setting when forecasting into regions without any data, or whether it is sufficient, due to the time-scale of seconds, to detect an anomaly "in hindsight."

Author response:
In a real setting, our approach works by predicting the DAS sginals for a future time window, given the current time window. Specifically, at time $t$, our model predicts the DAS waveforms for the subsequent time window $t+ /delta t$. Once the true DAS waveforms for the $/delta t$ time period are received, we calculate the difference between the predicted waveforms and the actual waveforms as an anomaly score. Typically, the time window $/delta t$ is set to be around 1 second, which is sufficient for detecting anomalies in real-time.

Reviewer comment (Evaluation):
Are there advantages or disadvantages of the selected anomaly score distance metrics that could be discussed?

Author response:
The main advantage of Earth Moving Distance (EMD) over Absolute Error (AE) is that EMD is more robust to temporal and spatial dynamics. EMD considers the shift or displacement of the predicted earthquake waveforms compared to the actual waveforms, while AE is a point-to-point error. However, EMD with a large window size can be more sensitive to noise and small local variations in the waveforms. To trade-off between EMD and AE, we propose to use sliced EMD, which breaks down the waveforms into smaller segments and aggregates EMDs of all segments, enabling a more localized evaluation.

Reviewer comment (Novelty):
The first concern is the goal of the paper. Indeed, DAS earthquake detectors exist (one of them was cited by the authors, PhaseNet-Das, Zhu et al. 2023, there might be others), and no comparison was made, nor a justification on the benefit of your method against theirs. If the claim is to say that this is a foundation model, and the test on this task is only as a proof of concept, it should be clearer, and then show or justify a future useful application.

Author response:
To the best of our knowledge, the existing DAS earthquake detectors like PhaseNet-DAS are for P/S phase pickup tasks, which are different from seismic detection tasks we did. Please refer to the Answer to A6.

P/S phase pickup refers to the identification and classification of seismic wave phases, specifically the arrival times of the P (primary) and S (secondary) waves, which focuses on accurately detecting and labeling these specific phases point-wise in DAS data.  On the other hand, seismic detection task (in Table 1) involves identifying and classifying anomalies or abnormal patterns in DAS data. The main differences between P/S phase pickup and seismic detection tasks can be summarized in the following aspects: 

(1) **Wave identification**: In P/S phase pickup, the method needs to accurately distinguish between P and S waves in seismic data. But seismic detection tasks do not require this specific wave identification, only with earthquake events. Most of the existing P/S phase pickup methods like PhaseNet-DAS learn on the context of P/S phases, while our method is a forecasting-based method which is easier and more suitable for real-time systems.

(2) **Data representation**: P/S phase pickup typically provides point-wise results, indicating the exact arrival times of P and S waves. In contrast, seismic detection tasks focus on determining whether there is an event or anomaly within a specific time span, without the need for precise point-wise results. Moreover, our training/valid/testing splits are different from PhaseNet/PhaseNet-DAS.

(3) **Learning setting**: P/S phase pickup methods are typically implemented in a supervised learning setting, where labeled data with ground-truth P/S wave information is required for training. However, our seismic detection task is addressed in an unsupervised learning setting like anomaly detection, as obtaining labeled data for anomalies can be challenging.

As a self-supervised learning model, DASFomer can be adapted to the P/S phase pickup tasks and we've done some preliminary experiments shown in the paper. However, considering the above points and the limited ground-truth annotations of P/S phases of our data, we hardly find a fair evaluation metric to compare with PhaseNet-DAS. We will explore more in the future work.

Reviewer comment (Evaluation):
The whole idea of this method is that earthquakes are unpredictable. It is clever indeed, but I see two major limitations: 1) this foundation model is thus harder to use for other tasks (which could be predictable) and 2) in a series of aftershocks (which could maybe be seen as more predictable), how does your measure perform?

Author response:
The effectiveness of our model for unpredictable earthquakes is derived from its good capabilities for other predictable events.

In this work, we primarily showcase the usage of DASFormer as a forecasting-based earthquake monitor. For tasks that are predictable, we can remove the masking and directly employ DASFormer as a feature extractor. We demonstrate its capability as a feature extractor in detecting malfunctioning sensors in Figure 4. Additionally, Figure 3 also illustrates its ability to detect predictable events such as traffic signals.

Regarding your concern about the assumption that "earthquakes are unpredictable" may potentially confuse the model, it is important to note that our look-back window is only around 13 seconds, with a prediction window of approximately 1 second. At this time scale, the level of predictability is nearly indistinguishable for relatively predictable events like aftershocks. Thus, we can treat each aftershock as a separate earthquake event, considering them as individual instances rather than relying on their predictability.

Reviewer comment (Experiments):
The comparison with other multi-variate time series is somehow misleading. Indeed, in multi-variate time-series, we suppose that the different time series (or sensors) are not ordered and not equally-spaced: DAS is a very particular type of 'multi-variate time-series'. I don't think it is worth presenting all of these methods (maybe only one), and it should be clearly stated in the paper. Yet, a comparison with image 2D foundation models, or by modifying a video framework from a 2D+t to a 1D+t, would be more relevant.

Author response:
Thanks for your valuable suggestion. Actually we did conduct baseline experiments using image-related methods, such as CNN-GAN or CNN-VAE, shown in Table 1. We have indeed considered representing DAS data as images, but drawing an analogy between DAS data and images can also be misleading. Image data does not inherently consider the causal information along the time axis that is present in DAS data. Adapting image-based methods to a causal-related forecasting-based approach would require additional design considerations. We will explore the possibilities like adapting ViT-based image methods in the future.

Reviewer comment (Novelty):
I don't understand the dynamic masking, and I don't see why it is useful. Does it come from other studies?

Author response:
The purpose of dynamic masking in our method serves two main functions.

Firstly, dynamic masking ensures that only unmasked tokens are attended to within the self-attention blocks, while disregarding the masked (invalid, useless) tokens. In this way, the model can more effectively capture the temporal/spatial dependencies.

Secondly, the dynamic masking approach updates the masked tokens around the unmasked tokens in a step-by-step manner, similar to a time series auto-regression rather than a one-step regression. This helps the model to progressively refine its predictions and incorporate the newly revealed information from the previously masked tokens.

Reviewer comment (Experiments):
The ablation study should first be 'ablation': i.e. what if we only use the fine generator, or the fine coarse? What if we only use a 'generator' without the need of the discriminator? Also, why is the fine generator a convolutional U-net, and the coarse a Swin-based Unet?

Author response:
Yes, we can have coarse-only, coarse-and-fine, coarse-and-fine w/ discriminator, these three components. The results are shown in the following table:

|                       | AE   |      | EMD  |      | sliced EMD |      |
|-----------------------|------|------|------|------|------------|------|
|                       | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| Coarse-only           | 0.79 | 0.41 | 0.81 | 0.44 | 0.83       | 0.47 |
| Coarse-to-Fine        | 0.85 | 0.52 | 0.87 | 0.53 | 0.87       | 0.50 |
| Coarse-to-Fine w/ GAN | 0.88 | 0.55 | 0.89 | 0.52 | 0.90       | 0.56 |

The reason to use different networks for two stages is that Swin Transformer is patch-level model, which can more effciently update the masked regions (see dynamic masking), while U-Net is a point-level model and more fine-grained.

Reviewer comment (Experiments):
Comparison with PhaseNet is rather unclear. First, it is stated that table 1 shows comparisons with time series forecasting and anomaly detection, but PhaseNet is neither of those, and is only single-station. In the table, it is unclear the use of 'traditional method', and the 'aggregation'. It would have been best to compare with PhaseNet-DAS, no?

Author response:
Please refer to A1.

Reviewer comment (Evaluation):
Detection becomes more important and challenging for smaller earthquakes. It is unclear how the method performs for smaller earthquakes and whether there are many false positives.

Author response:
**Q1:**  It is unclear how the method performs for smaller earthquakes and whether there are many false positives.

**A1:** For performance on smaller earthquakes, we visualized several cases ranging from M2.0 to M3.66 in Figure 9 in Appendix. And we show the performance for different magnitudes in answer **A3**. The false positive rate (FPR) and true positive rate (TPR) for DASFormer are shown as follows:

|     | AE   | EMD  | sliced EMD |
|-----|------|------|------------|
| FPR | 0.22 | 0.24 | 0.19       |
| TPR | 0.79 | 0.82 | 0.81       |

Reviewer comment (Evaluation):
What is the distribution (histogram) of the metric as a function of earthquake magnitude? How does it compare to the baseline? How about traditional signal processing techniques?

Author response:
**Q3:** What is the distribution (histogram) of the metric as a function of earthquake magnitude? How does it compare to the baseline? How about traditional signal processing techniques?

**A3:** We provide the magnitude distribution of our datasets with performance here:

Distribution of the magnitude of the 45 earthquakes in the training set:

| Magnitude | Numbers |
|-----------|---------|
| M0 ~ M1   | 0       |
| M1 ~ M2   | 0       |
| M2 ~ M3   | 31      |
| M3 ~ M4   | 12      |
| M4 ~ M5   | 2       |

Distribution of the magnitude of the 45 earthquakes in the evaluation set and the performance on DASFormer and Aggregation-0(STA/LTA baseline):

|DASFormer          |         | AE   |      | EMD  |      | sliced EMD |      |
|-----------|---------|------|------|------|------|------------|------|
| Magnitude | Numbers | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| M0 ~ M1   | 0       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M1 ~ M2   | 1       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M2 ~ M3   | 27      | 0.84 | 0.4  | 0.88 | 0.46 | 0.86       | 0.42 |
| M3 ~ M4   | 14      | 0.89 | 0.59 | 0.92 | 0.58 | 0.93       | 0.6  |
| M4 ~ M5   | 3       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |

|Aggregation-0 |         | AE   |      | EMD  |      | sliced EMD |      |
|-----------|---------|------|------|------|------|------------|------|
| Magnitude | Numbers | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| M0 ~ M1   | 0       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M1 ~ M2   | 1       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M2 ~ M3   | 27      | 0.74 | 0.24 | 0.7  | 0.28 | 0.7        | 0.25 |
| M3 ~ M4   | 14      | 0.73 | 0.24 | 0.69 | 0.26 | 0.7        | 0.25 |
| M4 ~ M5   | 3       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |


For the earthquakes in the range of M2 ~ M3 and M3 ~ M4, we can see that the performance is becoming better for larger magnitude earthquakes. In each level, the performance of DASFormer is better than Aggregation-0 (STA/LTA baseline). For the earthquakes in the range of M0 ~ M2 and M4 ~ M5, the number of samples is too small to draw a conclusion.

Reviewer comment (Experiments):
The resampling of the dataset from 2000-4000Hz to 10Hz for pre-training lacks a comparative verification of its impact on downstream tasks. Notably, the usual data sampling frequency for seismometer-based seismic detection tasks is within the range of 100-250Hz.

Author response:
**A4:** As you mentioned, the original sampling of our DAS data is 250Hz. Resampling the data from 250Hz to 10Hz allows us to retain the important features and characteristics of the seismic signals while reducing the data size and computational complexity. ****For seismic detection tasks, the lasting time of P waves typically spans a few seconds, while S waves can last even longer, up to several minutes. Given this context, a sampling frequency of 10Hz is sufficient to capture the necessary temporal information required for accurate seismic detection.

Reviewer comment (Experiments):
The paper does not adequately address the low signal-to-noise ratio (SNR) characteristic of DAS data, failing to provide clear information about the magnitude distribution and SNR of the training data. This omission leaves unanswered questions regarding the model's ability to detect earthquakes below a certain magnitude or signal-to-noise ratio.

Author response:
**A5:** Thanks for your suggestion. We provide magnitude distribution of our datasets here:

We provide magnitude distribution of our datasets here:

Distribution of the magnitude of the 45 earthquakes in the training set:

| Magnitude | Numbers |
|-----------|---------|
| M0 ~ M1   | 0       |
| M1 ~ M2   | 0       |
| M2 ~ M3   | 31      |
| M3 ~ M4   | 12      |
| M4 ~ M5   | 2       |

Distribution of the magnitude of the 45 earthquakes in the evaluation set and the performance:

|DASFormer          |         | AE   |      | EMD  |      | sliced EMD |      |
|-----------|---------|------|------|------|------|------------|------|
| Magnitude | Numbers | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| M0 ~ M1   | 0       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M1 ~ M2   | 1       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M2 ~ M3   | 27      | 0.84 | 0.4  | 0.88 | 0.46 | 0.86       | 0.42 |
| M3 ~ M4   | 14      | 0.89 | 0.59 | 0.92 | 0.58 | 0.93       | 0.6  |
| M4 ~ M5   | 3       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |

|Aggregation-0 |         | AE   |      | EMD  |      | sliced EMD |      |
|-----------|---------|------|------|------|------|------------|------|
| Magnitude | Numbers | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| M0 ~ M1   | 0       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M1 ~ M2   | 1       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |
| M2 ~ M3   | 27      | 0.74 | 0.24 | 0.7  | 0.28 | 0.7        | 0.25 |
| M3 ~ M4   | 14      | 0.73 | 0.24 | 0.69 | 0.26 | 0.7        | 0.25 |
| M4 ~ M5   | 3       | N/A  | N/A  | N/A  | N/A  | N/A        | N/A  |


For the earthquakes in the range of M2 ~ M3 and M3 ~ M4, we can see that the performance is becoming better for larger magnitude earthquakes. In each level, the performance of DASFormer is better than Aggregation-0. For the earthquakes in the range of M0 ~ M2 and M4 ~ M5, the number of samples is too small to draw a conclusion.

Reviewer comment (Experiments):
The paper lacks ablation experiments to elucidate the quantity of training data necessary for effective seismic monitoring.

Author response:
**A8:** We agree that ablation studies on the quantity of training data can investigate the performance especially the scalability of DASFormer. The results of this ablation study is shown in following table:

|                      | AE   |      | EMD  |      | sliced EMD |      |
|----------------------|------|------|------|------|------------|------|
|                      | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| 5 earthquake events  | 0.71 | 0.34 | 0.74 | 0.37 | 0.72       | 0.37 |
| 10 earthquake events | 0.78 | 0.37 | 0.80 | 0.44 | 0.80       | 0.42 |
| 20 earthquake events | 0.86 | 0.52 | 0.88 | 0.51 | 0.88       | 0.54 |
| 45 earthquake events (Full data) | 0.88 | 0.55 | 0.89 | 0.52 | 0.90       | 0.56 |

Reviewer comment (Experiments):
In the 'Time Series Modeling' section, the paper emphasizes the limitations of existing time series models in incorporating spatial information. However, there are network structures that can effectively integrate both spatial and temporal analyses. Including and comparing these structures with the work presented in this section would enhance the paper's precision.

Author response:
**A10:** Thanks for the suggestion. There are many existing spatio-temporal time series methods aim to capture and integrate spatial information for multi-variate time series data. We select three representative methods (Autoformer, GDN, and Crossformer) and compare them with DASFormer. The results are shown in Table 1 and Table 6. Autoformer and Crossformer are two Transformer-based methods utilizing spatial dependency in multi-variate time series. GDN is a graph-based method that uses a graph neural network to capture the spatial dependency of the multi-variate time series.

There are many existing spatio-temporal time series methods aim to capture and integrate spatial information for multi-variate time series data. However, to the best of our knowledge, all of them are order-agnostic models (with GNNs or Transformers) to capture spatial dependency, which typically do not consider the order or sequence of the variates in the data. This can be problematic in the context of DAS data, where the order of variates is crucial as it represents the location of stations along the fiber optic cable.

Reviewer comment (Experiments):
The paper emphasizes the role of the coarse and fine steps in its approach, but it does not include ablation experiments to demonstrate the individual contributions and effectiveness of these two components. Such experiments would provide valuable insights into the significance of each step.

Author response:
**A11:** Thanks for the suggestion. We conduct ablation studies on the three steps (Coarse, Coarse-to-Fine, Coarse-to-Fine with Discriminator). The results are shown in the following table:

|                       | AE   |      | EMD  |      | sliced EMD |      |
|-----------------------|------|------|------|------|------------|------|
|                       | AUC  | F1   | AUC  | F1   | AUC        | F1   |
| Coarse-only           | 0.79 | 0.41 | 0.81 | 0.44 | 0.83       | 0.47 |
| Coarse-to-Fine        | 0.85 | 0.52 | 0.87 | 0.53 | 0.87       | 0.50 |
| Coarse-to-Fine w/ GAN | 0.88 | 0.55 | 0.89 | 0.52 | 0.90       | 0.56 |

The results demonstrate the effectiveness of the coarse-to-fine framework and the discriminator. We will add these results to the revision.