Paper: Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting (ICLR-2024)

Reviewer comment (Experiments):
The current time series forecasting datasets are pretty small, and performance may be satuated or over-fitting. Could the method be used for larger datasets?

Author response:
**Q1:** Results on larger datasets.

**A1:** We seek larger datasets from two perspectives: data volume and the number of variables. We add two datasets, the Wind Power dataset and the PEMS07 dataset, to evaluate the performance of Pathformer on larger datasets. The Wind Power dataset comprises 7397147 timestamps, reaching a sample size in the millions, and the PEMS07 dataset includes 883 variables. If the reviewer may mention other larger datasets, we are also willing to test on them. Pathformer also demonstrates superior predictive performance on these larger datasets compared with some state-of-the-art methods such as PatchTST, DLinear, and Scaleformer. We add experiments on these larger datasets in $\underline{\text{Section A.6 of the revised supplementary}}$.

| Methods |     | Pathformer        | PatchTST |     DLinear |     Scaleformer |     
|:---------:|:----------------:|:-------------:|:-------------------:|:-------------:|:-----:|
| Metrics |     | MSE    / MAE   | MSE   /  MAE   | MSE   / MAE   | MSE   / MAE |
| PEMS07    | 96  | **0.135**    / **0.243** | 0.146  / 0.259 | 0.564  / 0.536 |       0.152      /  0.268  |
|         | 192 | **0.177**    / **0.271** | 0.185   / 0.286 | 0.596  / 0.555 |        0.195     /   0.302  |
|         | 336 | **0.188**     / **0.278** | 0.205   / 0.289 | 0.475  / 0.482 |      0.276       /  0.394  |
|         | 720 | **0.208**     / **0.296** | 0.235    / 0.325 | 0.543  / 0.523 |        0.305     /   0.410  |
| Wind Power   | 96  | **0.062**     / **0.146** | 0.070    / 0.158 | 0.078   / 0.184 |      0.089       /  0.167   |
|         | 192 | **0.123**     / **0.214** | 0.131    / 0.237 | 0.133  / 0.252 |      0.163       /  0.246   |
|         | 336 | **0.200**       / **0.283** | 0.215   / 0.307 | 0.205  / 0.325 |      0.225       /    0.352 |
|         | 720 | **0.388**     / **0.414** | 0.404   / 0.429 | 0.407  / 0.457 | 0.414 / 0.426 |

Reviewer comment (Novelty):
Scalformer [1] also uses the multi-scale nature of time series data, this paper didn't mention and compare the similarities and differences with Scalformer.

Author response:
**Q2:** Compare with Scaleformer.

**A2:** We mentioned the method of Scaleformer in $\underline{\text{the related work of the original submission}}$. In the revision, we add Scaleformer results in $\underline{\text{Tabel 1 of the revised paper}}$ as an important baseline of multi-scale models. We also provide a more detailed comparison between Scaleformer and our model in $\underline{\text{the related work of the revised paper and Section A.5.3 of the revised supplementary}}$, as follows:

Scaleformer also utilizes the modeling of multi-scale features for time series forecasting. It differs from our proposed Pathformer in the following aspects:

- Scaleformer employs fixed sampling rates, while Pathformer has the capability for adaptive multi-scale modeling based on the differences in time series samples.
- Scaleformer obtains multi-scale features with different temporal resolutions through downsampling. In contrast, Pathformer not only considers time series features of different resolutions but also models from the perspective of temporal distance, taking into account global correlations and local details. This provides a more comprehensive approach to multi-scale modeling through both temporal resolutions and temporal distances.
- Scaleformer requires the allocation of a predictive model at different temporal resolutions, resulting in higher model complexity than Pathformer.

Reviewer comment (Experiments):
The current time series forecasting datasets are pretty small, and performance may be satuated or over-fitting. Could the method be used for larger datasets?

Author response:
**Q1:** Results on larger datasets.

**A1:** We seek larger datasets from two perspectives: data volume and the number of variables. We add two datasets, the Wind Power dataset and the PEMS07 dataset, to evaluate the performance of Pathformer on larger datasets. The Wind Power dataset comprises 7397147 timestamps, reaching a sample size in the millions, and the PEMS07 dataset includes 883 variables. If the reviewer may mention other larger datasets, we are also willing to test on them. Pathformer also demonstrates superior predictive performance on these larger datasets compared with some state-of-the-art methods such as PatchTST, DLinear, and Scaleformer. We add experiments on these larger datasets in $\underline{\text{Section A.6 of the revised supplementary}}$.

| Methods |     | Pathformer        | PatchTST |     DLinear |     Scaleformer |     
|:---------:|:----------------:|:-------------:|:-------------------:|:-------------:|:-----:|
| Metrics |     | MSE    / MAE   | MSE   /  MAE   | MSE   / MAE   | MSE   / MAE |
| PEMS07    | 96  | **0.135**    / **0.243** | 0.146  / 0.259 | 0.564  / 0.536 |       0.152      /  0.268  |
|         | 192 | **0.177**    / **0.271** | 0.185   / 0.286 | 0.596  / 0.555 |        0.195     /   0.302  |
|         | 336 | **0.188**     / **0.278** | 0.205   / 0.289 | 0.475  / 0.482 |      0.276       /  0.394  |
|         | 720 | **0.208**     / **0.296** | 0.235    / 0.325 | 0.543  / 0.523 |        0.305     /   0.410  |
| Wind Power   | 96  | **0.062**     / **0.146** | 0.070    / 0.158 | 0.078   / 0.184 |      0.089       /  0.167   |
|         | 192 | **0.123**     / **0.214** | 0.131    / 0.237 | 0.133  / 0.252 |      0.163       /  0.246   |
|         | 336 | **0.200**       / **0.283** | 0.215   / 0.307 | 0.205  / 0.325 |      0.225       /    0.352 |
|         | 720 | **0.388**     / **0.414** | 0.404   / 0.429 | 0.407  / 0.457 | 0.414 / 0.426 |

Reviewer comment (Novelty):
Scalformer [1] also uses hierarchical design and the scales of time series data, Could this paper compare the similarities and differences with Scalformer.

Author response:
**Q2:** Compare with Scaleformer.

**A2:** We mentioned the method of Scaleformer in $\underline{\text{the related work of the original submission}}$. In the revision, we add Scaleformer results in $\underline{\text{Tabel 1 of the revised paper}}$ as an important baseline of multi-scale models. We also provide a more detailed comparison between Scaleformer and our model in $\underline{\text{the related work of the revised paper and Section A.5.3 of the revised supplementary}}$, as follows:

Scaleformer also utilizes the modeling of multi-scale features for time series forecasting. It differs from our proposed Pathformer in the following aspects:

- Scaleformer employs fixed sampling rates, while Pathformer has the capability for adaptive multi-scale modeling based on the differences in time series samples.
- Scaleformer obtains multi-scale features with different temporal resolutions through downsampling. In contrast, Pathformer not only considers time series features of different resolutions but also models from the perspective of temporal distance, taking into account global correlations and local details. This provides a more comprehensive approach to multi-scale modeling through both temporal resolutions and temporal distances.
- Scaleformer requires the allocation of a predictive model at different temporal resolutions, resulting in higher model complexity than Pathformer.

Reviewer comment (Experiments):
Time-series Dense Encoder (TiDE) is also a popular long-term time-series forecasting benchmark. But the paper does not include experiments comparing the proposed method to TiDE.

Author response:
**Q1:** Compare with Time-series Dense Encoder (TiDE).

**A1:** The results of TiDE presented in the original paper are the best ones selected from different input sequence lengths of (48,96,192,336,720). Considering the urgency and fairness, we conduct a comparison under the condition of a fixed input sequence length of 96. We show results of some datasets here, and the complete results of other datasets and methods are available in $\underline{\text{Table 1 of the revised paper}}$.


| Methods |  | Pathformer | TIDE |
| :-------:|:-------:|:----------:|:---------:|
|Metrics|  |MSE / MAE| MSE / MAE | 
| ETTm2      | 96|  0.170 / 0.248 | 0.182 / 0.264 |
|            | 192|    0.238 / 0.295 |0.256 / 0.323 |
|            | 336|  0.293 / 0.331 | 0.313 / 0.354 |
|            | 720|  0.390 / 0.389 | 0.419 / 0.410|
| Electricity| 96|  0.145 / 0.236   |    0.194/ 0.277  |
|            | 192|  0.167 / 0.258  |   0.193 / 0.280   |
|            | 336|  0.186 / 0.275  |    0.206 / 0.296  |
|            | 720|   0.231 / 0.309 |    0.242 / 0.328  |
| Weather    | 96 |  0.156 / 0.192  |      0.202 / 0.261             |
|            | 192|  0.206 / 0.240  |      0.242 / 0.298 |
|            | 336|  0.254 / 0.282  |      0.287 / 0.335 |
|            | 720|  0.340 / 0.336  |      0.351 / 0.386 |

Reviewer comment (Experiments):
Not all datasets are present in the study. Please include additional results on ILI and Traffic from PatchTST.

Author response:
**Q2:** Include additional results on ILI and Traffic from PatchTST

**A2:** We add the ILI and Traffic datasets in $\underline{\text{Table 1 in the revised paper}}$. Here, we show some models with superior performance: PatchTST, DLinear, FEDformer, and Autoformer (bold indicates the best). Results of other compared methods are also included in the revised paper.



| Methods      |     | PathFormer |     PatchTST  | DLinear |      Fedformer   |   Autoformer |
|:-------------:|:----------:|:-------------------:|:----------------:|:-----------------:|:------------:|:-------------:|
| Metrics     |     | MSE / MAE   | MSE / MAE   | MSE  / MAE   | MSE / MAE|   MSE / MAE |
| ILI       | 96  | **1.587** / **0.758** | 1.724 / 0.843 | 2.573 / 1.073 | 2.624 / 1.095 | 2.906 / 1.182 |
|             | 192 | **1.429** / **0.711** | 1.536 / 0.752 | 2.673 / 1.085 | 2.516 / 1.021 | 2.585 / 1.038 |
|             | 336 | **1.505** / **0.742** | 1.821 / 0.832 | 2.773 / 1.126 | 2.505 / 1.041 | 3.024 / 1.145 |
|             | 720 | **1.731** / **0.799** | 1.923 /  0.842 | 2.827 / 1.152 | 2.742 / 1.122 | 2.761 / 1.114 |
| Traffic       | 96  | **0.479** / **0.283** | 0.492  / 0.324 | 0.648 / 0.396 | 0.576 / 0.359 | 0.597 / 0.371 |
|             | 192 | **0.484** / **0.292** | 0.487  / 0.303 | 0.613 / 0.614 | 0.610 / 0.380 | 0.607 / 0.382 |
|             | 336 | **0.503** / **0.299** | 0.505 / 0.317 | 0.614 / 0.383 | 0.608 / 0.375 | 0.623 / 0.387 |
|             | 720 | **0.537** / **0.322** | 0.542  / 0.337 | 0.655 / 0.405 | 0.621 / 0.375 | 0.639 / 0.395 |

Reviewer comment (Experiments):
Please include results from Zeng et al. Are transformers effective for time series forecasting? AAAI'23 in your table and you will see that your results are not state of the art. This makes the results unconvincing, because basically a very complex model is not able to use the same inputs as very simple models in an effective way. Transformer based papers have consistently failed to include appropriate baselines in the studies creating a large gap in methodology and undermining the ultimate reliability of these studies. The work can be interesting if authors show that with the proposed modifications a transformer based model can be more effective than much simpler and faster models presented in Zeng et al. and Li et al.

Author response:
**Q3:** Include and compare with the results from Zeng et al.

**A3:** Some results presented in the paper of Zeng et al. (DLinear) are better than ours in Table 1, which is because that DLinear uses a **larger input length** (H=336) than ours (H=96). To ensure a fair comparison, we conduct separate experiments for input sequence lengths (H) of 96 and 336. Here, we show results of some models and datasets, with the complete results of other models and on other datasets available in the $\underline{\text{Table 1 in the revised paper and Table 9 in supplementary}}$.

**The results for the input sequence length H=96:**

| Methods |     | Pathformer |    DLinear     | NLinear | 
|:---------:|:-----:|:------------------:|:---------------:|:----:|
| Metrics |     | MSE       /MAE   | MSE     / MAE   |MSE / MAE| 
| ETTm1   | 96  | **0.316**      / **0.346** | 0.342   / 0.370 |0.339 / 0.369| 
|         | 192 | **0.366**      / **0.370** | 0.383   / 0.394 |0.379 / 0.386|
|         | 336 | **0.386**     / **0.394** | 0.413   / 0.414 |0.411 / 0.407| 
|         | 720 | **0.460**    / **0.432** | 0.472   / 0.452 |0.478 / 0.442| 
| Weather | 96  | **0.156**      / **0.192** | 0.195   / 0.253 |0.168 / 0.208|
|         | 192 | **0.206**     / **0.240** | 0.239   / 0.299 |0.217 / 0.255| 
|         | 336 | **0.254**     / **0.282**| 0.282   / 0.333 |0.267 / 0.292| 
|         | 720 | **0.340**    / **0.336** | 0.352   / 0.390 |0.351 / 0.346| 
| ILI     | 24  | **1.587**     / **0.758** | 2.573   / 1.073 |2.725 / 1.069|
|         | 36  | **1.429**     / **0.711** | 2.673   / 1.085 |2.530 / 1.032|
|         | 48  | **1.505**     / **0.742**| 2.773   / 1.126 |2.510 / 1.031| 
|         | 60  | **1.731**     / **0.799** | 2.827   / 1.152 |2.492 / 1.026|

**The results for the input sequence length H=336:**

| Methods  |     | Pathformer        | DLinear     | NLinear    |
|:---------:|:-----:|:------------------:|:----------------:|:----------------:|
| Metrics  |     | MSE       / MAE    | MSE     / MAE    | MSE     / MAE    | MSE   / MAE    |
| ETTm1   | 96  | **0.285**    / **0.336**  | 0.299   / 0.353  | 0.306   / 0.348  | 
|         | 192 | **0.331**    / **0.361**  | 0.335   / 0.365  | 0.349   / 0.375  | 
|         | 336 | **0.362**     / **0.382**  | 0.369  / 0.386  | 0.375   / 0.388  | 
|         | 720 | **0.412**    / **0.414**  | 0.425   / 0.421  | 0.433   / 0.422  | 
| Weather | 96  | **0.144**   / **0.184**  | 0.176  / 0.237  | 0.182   / 0.232  | 
|         | 192 | **0.191**   / **0.229**  | 0.220 / 0.282  | 0.225  / 0.269  |
|         | 336 | **0.234**   / **0.268**  | 0.265  / 0.319  | 0.271  / 0.301  | 
|         | 720 | **0.316**    / **0.323**  | 0.323   / 0.362  | 0.338    / 0.348  | 
| ILI     | 24  | **1.411**      / **0.705**  | 2.215   / 1.081  | 1.683    / 0.868  | 
|         | 36  | **1.365**      / **0.727**  | 1.963   / 0.963  | 1.703    / 0.859  | 
|         | 48  | **1.537**      / **0.764**  | 2.130   / 1.024  | 1.719    / 0.884  | 
|         | 60  | **1.418**      / **0.772**  | 2.368   / 1.096  | 1.819    / 0.917  | 

The results above reveal that our proposed model Pathformer outperforms DLinear with both input sequence lengths of 96 and 336. Zeng et al. point out that previous Transformer cannot extract temporal relations well from longer input sequences, but our proposed Pathformer performs better with a longer input length, indicating that considering adaptive multi-scale modeling can be an effective way to enhance such a relation extraction ability of Transformers.

Reviewer comment (Novelty):
The model seems to borrow conceptually very heavily from the PatchTST model without explicitly recognizing the source of inspiration. Without a detailed explanation of the actual difference between the two architectures the proposed architecture appears to be a minor perturbation of the original PatchTST.

Author response:
**Q4:** Compare with PatchTST

**A4:** We mention the source of inspiration of patching in $\underline{\text{Section 3 of our revised paper}}$. We also want to clarify that Pathformer extends from patch division to realize adaptive multi-scale modeling, which is a novel design. It is not a perturbation of PatchTST, as how the series are divided, and how the correlations are modeled are both designed differently. The main differences with PatchTST are as follows:

- **Adaptive Multi-scale Modeling:** 
PatchTST employs a fixed patch size for all data, hindering the grasp of critical patterns in different time series. We are the first to propose adaptive pathways that dynamically select varying patch sizes tailored to the dynamic features of individual samples, enabling adaptive multi-scale modeling.

- **Partitioning with Multiple Patch Sizes:**
PatchTST employs a single patch size to partition time series, obtaining features with a singular resolution. Pathformer utilizes multiple different patch sizes for partitioning, which captures multi-scale features from various temporal resolutions.

- **Global correlations between patches and local details in each patch:**
PatchTST performs attention between divided patches, overlooking the internal details in each patch. Pathformer not only considers the correlations between patches but also the detailed information within each patch. It introduces dual attention (inter-patch attention and intra-patch attention) to integrate global correlations and local details, capturing multi-scale features from various temporal distances.

We also add the above discussion in $\underline{\text{Section A.5.1 of the revised supplementary}}$.