Paper: PatchMixer: A Patch-Mixing Architecture for Long-Term Time Series Forecasting (ICLR-2024)

Reviewer comment (Novelty):
The novelty of work is limited, and its contribution to the field of time series forecasting is incremental. The idea of channel independence and separately modeling inter-patches and intra-patches correlations have already been proposed in PatchTST [1]. In my view, PatchMixer simply replaces self-attention with convolutions. The essence of dual forecasting heads is skip connection, which has also been seen in previous works about time series forecasting such as TiDE [2].

Author response:
Q4.1: Novelty clarification.

A4.1: We point out our contributions below:
1. Our work is inspired by PatchTST's patch processing technique, which is proposed for use in the Transformer architecture. We hypothesize that the strong predictive performance observed in time series forecasting is attributed more to the patch embedding methods rather than the inherent predictive capabilities of the Transformers. Therefore, we design PatchMixer based on CNN architectures, which have shown to be faster at a similar scale to Transformers.
2. Our proposed approach, PatchMixer, has achieved state-of-the-art results, particularly enhancing the performance of convolutional networks in the field of time series forecasting. Quantitatively, PatchMixer outperforms the best-performing Transformer (PatchTST) by 3.9% on MSE and surpasses the leading CNN (TimesNet) by 21.2% on MSE.
3. PatchMixer significantly improves the efficiency of long-term time series forecasting with linear computational complexity. It is three times faster for inference and two times faster during training compared to the current state-of-the-art model.

Q4.2: Comparison to DLinear, PatchTST and TiDE:

A4.2: The method of channel independence was introduced by DLinear, while PatchTST proposed a patch processing technique to demonstrate the effectiveness of the original Transformer architecture. Based on these previous works, our paper hypothesizes that the efficacy in forecasting is largely due to the patch embedding approach rather than the Transformer's innate predictive capabilities. Thus, the replacement of self-attention with convolutions is not merely a simplification but a strategic choice to enhance forecasting performance. 
We also clarify the distinction from TiDE in our forecasting heads' implementation.  TiDE employs global residual connections at the start, before the embedding. The whole process operates at a point-wise level. In contrast, our use of the linear connection is not global but rather post-embedding, with dimensional transformations involving both time steps and patches.

Reviewer comment (Evaluation):
The improvement in prediction accuracy is negligible. From the results in Table 1, it can be seen that PatchMixer has very little or no improvement compared to PatchTST on datasets other than ETTh1 and ETTh2. In the ablation experiments in Table 2, the performance of Standard Convolution, Separable Convolution, and Attention Mechanism are comparable, which does not prove the superiority of Separable Convolution.

Author response:
Q4.3: Model performance clarification and significant test.

A4.3: To ensure a fair comparison of the performance improvements, we have conducted extensive T-tests on both PatchTST and PatchMixer across three datasets: weather, ETTm2, and ETTh2. These tests are performed over all forecast lengths using 10 sets of random seeds to maintain statistical robustness. The results yield a T-statistic of -2.33 and a P-value of 0.02. This statistically significant outcome suggests that the improvements observed with PatchMixer are not due to random chance. The P-value, being less than the conventional threshold of 0.05, indicates that there is only a 2% probability that the observed improvements could occur due to variance in the data alone. Hence, we can assert that PatchMixer provides a consistent and significant enhancement in forecasting accuracy across datasets.

Reviewer comment (Experiments):
The paper mentions two types of patch representation from PatchTST and TimesNet [3] respectively, but the experiments do not compare these two.

Author response:
Q4.6: Different patch representations.

A4.6: Thank you for your valuable feedback. Our primary focus is on investigating the impact of patch embedding methods on time series forecasting performance. We hypothesize that the strong predictive performance observed in PatchTST is attributed more to the patch embedding methods rather than the inherent predictive capabilities of the Transformers. Therefore in this paper, we use the same patch representation as PatchTST. As you imply, the discussion of TimesNet's patch method strays away from our motivation and focus. So we move this description from the Methods section to the Related Work section and briefly mention it.

Reviewer comment (Writing):
The overall content of the paper is not substantial and there are some redundant paragraphs. For example, in the Method section, there are many introductions to Patch Representation and Embedding, but these parts can even be described in just a single paragraph.

Author response:
Q4.5: Typos and redundant paragraphs.

A4.5: Thank you for pointing out the redundancy and typographical error.
We have carefully modified the content of our paper, by moving the patch operation of TimesNet to the Related Work section and simplifying the descriptions of the Method section. Besides, the reference to "equation 8" has been corrected to "equations 10" in the Pointwise Convolution section.

Reviewer comment (Experiments):
Are the slight performance improvements in PatchMixer solely due to the use of dual forecasting heads and MSE+MAE loss, rather than the architecture? Because other models in Table 1 can also be equipped with dual forecasting heads and MSE+MAE loss.

Author response:
Q4.4: The effectiveness of the proposed components and different loss functions.

A4.4: Thanks for pointing these out. We argue that the performance improvements in PatchMixer are primarily attributed to the architecture itself. Our model's design and the combination of these components contribute to its superior performance. We show the additional experiments below to support our claim, which are included in the revised draft.

We have updated Table 2 and provided supplementary information to clarify this in the revised draft. We bold the best results, and supplement the prediction performance using only Dual Heads. The comparison of convolution is placed in a separate section in the appendix.

| Methods                       | Metrics | Traffic 96 | Traffic 192 | Traffic 336 | Traffic 720 | ETTm1 96 | ETTm1 192 | ETTm1 336 | ETTm1 720 | ETTh1 96 | ETTh1 192 | ETTh1 336 | ETTh1 720 |
|--------------------------------|-------------------|------------|-------------|-------------|-------------|-----------|------------|------------|------------|-----------|------------|------------|------------|
| **Dual Heads**                 | MSE               | 0.377      | 0.391       | 0.399       | 0.429       | 0.290     | 0.327      | 0.356      | 0.420      | **0.355** | **0.373**  | 0.392      | **0.440**  |
| **Dual Heads**                 | MAE               | 0.251      | 0.256       | 0.258       | 0.283       | 0.340     | 0.362      | 0.381      | 0.416      | 0.384     | 0.394      | 0.412      | **0.455**  |
| **Attention Mechanism + Dual Head** | MSE         | 0.368      | 0.388       | 0.401       | 0.447       | 0.294     | 0.331      | 0.360      | 0.422      | **0.355** | 0.378      | 0.393      | 0.451      |
| **Attention Mechanism + Dual Head** | MAE         | **0.240**  | **0.249**   | **0.256**   | **0.273**   | 0.340     | 0.365      | 0.386      | 0.417      | **0.382** | 0.397      | 0.411      | 0.460      |
| **PatchMixer Layer + Dual Head**    | MSE         | **0.362**  | **0.382**   | **0.392**   | **0.428**   | **0.290** | **0.325**  | **0.353**  | **0.413**  | **0.355** | **0.373**  | **0.391**  | 0.446      |
| **PatchMixer Layer + Dual Head**    | MAE         | 0.242      | 0.252       | 0.257       | 0.282       | 0.340     | **0.361**  | **0.382**  | **0.413**  | 0.383     | **0.394**  | **0.410**  | 0.463      |

Our model PatchMixer can be represented by PatchMixer Layer + Dual Heads. As shown in the revised Table 2, it shows significant improvements compared to only Dual Heads and replacing PatchMixer Layer with Attention Mechanism. Note that all experiments adhere to the same hypermeter settings and the best results are bolded. For the small dataset (ETTh1), both our model and the utilization of only the Dual Heads component outperform the use of Attention Mechanism + Dual Heads. For medium (ETTm1) and large-scale (Traffic) datasets, PatchMixer Layer with Dual Heads demonstrates noticeable improvements over the use of Dual Heads alone. This underscores the indispensable predictive capacity of our convolutional module. 

Besides, we have also conducted additional experiments to evaluate the impact of four different loss functions. The revised Table 4 now presents a comprehensive comparison between PatchMixer and PatchTST using these loss functions, with the superior results highlighted in bold.
Here we provide the following key observations, which are included in the revised paper. 
First, PatchMixer consistently outperforms PatchTST across the board, affirming our model's superior performance. 
Secondly, SmoothL1loss is clearly inferior to the other two losses. 
Thirdly, the model trained with combined MAE+MSE loss shows a better performance on the MAE/MSE metrics than the model trained with MSE loss only.

Reviewer comment (Writing):
Typo. The second paragraph in Pointwise Convolution section, "The above equations 8 demonstrate the process ..." should be "The above equations 10 demonstrate the process ...".

Author response:
Q4.5: Typos and redundant paragraphs.

A4.5: Thank you for pointing out the redundancy and typographical error.
We have carefully modified the content of our paper, by moving the patch operation of TimesNet to the Related Work section and simplifying the descriptions of the Method section. Besides, the reference to "equation 8" has been corrected to "equations 10" in the Pointwise Convolution section.

Reviewer comment (Evaluation):
The improvement is marginal, especially over PatchTST. Also, there is no substantial benefit of the proposed model using depthwise separable convolution or dual head in the ablation study (Table 2 and Figure 4).

Author response:
Q3.2: Table 2 clarification.

A3.2: Thanks for pointing this out. Table 2 has been modified following your suggestions in the updated draft. We have provided supplementary information on the prediction performance using only dual forecasting heads and relocated the convolution comparison to a separate appendix section.

Our model is represented by PatchMixer Layer + Dual Heads. As shown in the revised Table 2, it shows significant improvements compared to only Dual Heads and replacing PatchMixer Layer with Attention Mechanism. Note that all experiments adhere to the same hypermeter settings and the best results are bolded. For the small dataset (ETTh1), both our model and the utilization of only the Dual Heads component outperform the use of Attention Mechanism + Dual Heads. For medium (ETTm1) and large-scale (Traffic) datasets, PatchMixer Layer with Dual Heads demonstrates noticeable improvements over the use of Dual Heads alone. This underscores the indispensable predictive capacity of the convolutional module.

| Methods                       | Metrics | Traffic 96 | Traffic 192 | Traffic 336 | Traffic 720 | ETTm1 96 | ETTm1 192 | ETTm1 336 | ETTm1 720 | ETTh1 96 | ETTh1 192 | ETTh1 336 | ETTh1 720 |
|--------------------------------|-------------------|------------|-------------|-------------|-------------|-----------|------------|------------|------------|-----------|------------|------------|------------|
| **Dual Heads**                 | MSE               | 0.377      | 0.391       | 0.399       | 0.429       | 0.290     | 0.327      | 0.356      | 0.420      | **0.355** | **0.373**  | 0.392      | **0.440**  |
| **Dual Heads**                 | MAE               | 0.251      | 0.256       | 0.258       | 0.283       | 0.340     | 0.362      | 0.381      | 0.416      | 0.384     | 0.394      | 0.412      | **0.455**  |
| **Attention Mechanism + Dual Head** | MSE         | 0.368      | 0.388       | 0.401       | 0.447       | 0.294     | 0.331      | 0.360      | 0.422      | **0.355** | 0.378      | 0.393      | 0.451      |
| **Attention Mechanism + Dual Head** | MAE         | **0.240**  | **0.249**   | **0.256**   | **0.273**   | 0.340     | 0.365      | 0.386      | 0.417      | **0.382** | 0.397      | 0.411      | 0.460      |
| **PatchMixer Layer + Dual Head**    | MSE         | **0.362**  | **0.382**   | **0.392**   | **0.428**   | **0.290** | **0.325**  | **0.353**  | **0.413**  | **0.355** | **0.373**  | **0.391**  | 0.446      |
| **PatchMixer Layer + Dual Head**    | MAE         | 0.242      | 0.252       | 0.257       | 0.282       | 0.340     | **0.361**  | **0.382**  | **0.413**  | 0.383     | **0.394**  | **0.410**  | 0.463      |

Reviewer comment (Presentation):
Table 2 is very confusing. It took significant effort to understand what the table means. For instance, the first column, which is PatchMixer "minus" another module, was not initially clear. Because PatchMixer and another module are in separate rows, it looks like the first row is PatchMixer, and the second column is a comparing model.

Author response:
Q3.2: Table 2 clarification.

A3.2: Thanks for pointing this out. Table 2 has been modified following your suggestions in the updated draft. We have provided supplementary information on the prediction performance using only dual forecasting heads and relocated the convolution comparison to a separate appendix section.

Our model is represented by PatchMixer Layer + Dual Heads. As shown in the revised Table 2, it shows significant improvements compared to only Dual Heads and replacing PatchMixer Layer with Attention Mechanism. Note that all experiments adhere to the same hypermeter settings and the best results are bolded. For the small dataset (ETTh1), both our model and the utilization of only the Dual Heads component outperform the use of Attention Mechanism + Dual Heads. For medium (ETTm1) and large-scale (Traffic) datasets, PatchMixer Layer with Dual Heads demonstrates noticeable improvements over the use of Dual Heads alone. This underscores the indispensable predictive capacity of the convolutional module.

| Methods                       | Metrics | Traffic 96 | Traffic 192 | Traffic 336 | Traffic 720 | ETTm1 96 | ETTm1 192 | ETTm1 336 | ETTm1 720 | ETTh1 96 | ETTh1 192 | ETTh1 336 | ETTh1 720 |
|--------------------------------|-------------------|------------|-------------|-------------|-------------|-----------|------------|------------|------------|-----------|------------|------------|------------|
| **Dual Heads**                 | MSE               | 0.377      | 0.391       | 0.399       | 0.429       | 0.290     | 0.327      | 0.356      | 0.420      | **0.355** | **0.373**  | 0.392      | **0.440**  |
| **Dual Heads**                 | MAE               | 0.251      | 0.256       | 0.258       | 0.283       | 0.340     | 0.362      | 0.381      | 0.416      | 0.384     | 0.394      | 0.412      | **0.455**  |
| **Attention Mechanism + Dual Head** | MSE         | 0.368      | 0.388       | 0.401       | 0.447       | 0.294     | 0.331      | 0.360      | 0.422      | **0.355** | 0.378      | 0.393      | 0.451      |
| **Attention Mechanism + Dual Head** | MAE         | **0.240**  | **0.249**   | **0.256**   | **0.273**   | 0.340     | 0.365      | 0.386      | 0.417      | **0.382** | 0.397      | 0.411      | 0.460      |
| **PatchMixer Layer + Dual Head**    | MSE         | **0.362**  | **0.382**   | **0.392**   | **0.428**   | **0.290** | **0.325**  | **0.353**  | **0.413**  | **0.355** | **0.373**  | **0.391**  | 0.446      |
| **PatchMixer Layer + Dual Head**    | MAE         | 0.242      | 0.252       | 0.257       | 0.282       | 0.340     | **0.361**  | **0.382**  | **0.413**  | 0.383     | **0.394**  | **0.410**  | 0.463      |

Reviewer comment (Evaluation):
Clarification is needed on whether all the methods in Table 1 are comparable. Questions include: How are the training setups? Are the number of parameters similar? How many runs were done for these experiments? Are the results consistent with different runs? There is interest in knowing whether this error gap is actually significant.

Author response:
Q3.3: Experiment setup clarification.

A3.3: The errors at different prediction lengths and comparison with transformer-based and conv-based methods are shown in Table 1 in Experiment chapter. The baselines and implementation details can be checked in Section A.1.2 in Baselines and Section A.1.3 Implementation Details of Appendix section. In short, the input length of ours follows 336 time steps, as DLinear and PatchTST. We directly use the numbers reported in their papers in our tables. We repeat our PatchMixer for 5 random seeds to verify its robustness. The patch hypermeters follow PatchTST for a fair comparison.

Q3.4: Model performance clarification and significant test.

A3.4: To ensure a fair comparison of the performance improvements, we have conducted extensive T-tests on both PatchTST and PatchMixer across three datasets: weather, ETTm2, and ETTh2. These tests are performed over all forecast lengths using 10 sets of random seeds to maintain statistical robustness. The results yield a T-statistic of -2.33 and a P-value of 0.02, indicating a significant improvement by PatchMixer is unlikely due to data variance alone. This affirms PatchMixer's accuracy enhancement.

Reviewer comment (Evaluation):
The contributions of the paper are unclear. Although there are three points at the bottom of the Introduction, the experimental results do not support these arguments. For instance, the reviewer requests an explanation of why the PatchMixer layer is particularly helpful for long-term time series forecasting.

Author response:
Q3.1: Clarification of contribution.

A3.1: We point out the key contributions of our paper below:
1. Our work is inspired by PatchTST's patch processing technique, which is proposed for use in the Transformer architecture. We hypothesize that the strong predictive performance observed in time series forecasting is attributed more to the patch embedding methods rather than the inherent predictive capabilities of the Transformers. Therefore, we design PatchMixer based on CNN architectures, which have shown to be faster at a similar scale to Transformers. The experimental results have verified our hypothesis.
2. Our proposed approach, PatchMixer, has achieved state-of-the-art results, particularly enhancing the performance of convolutional networks in the field of time series forecasting. Quantitatively, PatchMixer outperforms the best-performing Transformer (PatchTST) by 3.9% on MSE and surpasses the leading CNN (TimesNet) by 21.2% on MSE.
3. PatchMixer significantly improves the efficiency of long-term time series forecasting. It is three times faster for inference and two times faster during training compared to the current state-of-the-art model.

Reviewer comment (Evaluation):
Regarding efficiency compared to PatchTCT (Transformer-based), it is unclear where the speed-up comes from. Is it due to less computational cost but a similar model size compared to PatchTCT, or high parallelizability? It is assumed that both models have great parallelization capabilities. The reviewer is trying to understand where the efficiency comes from.

Author response:
Q3.4: Model Efficiency Analysis (where the speed-up comes from).

A3.4: Thanks for your reminder, we analyze the source of efficiency improvement from two aspects: model structure and computational cost. The details can also be found in Appendix 5 Computational Efficiency Analysis of our revised script.
1. PatchMixer leverages the strength of convolutions for local modeling while overcoming the limitation of their receptive fields. Through patch embedding, which transforms the 1D sequence into a 2D matrix, we address this limitation effectively. A moderately sized kernel (k=8) is sufficient to process stacked patches (p=16), as the number of patches is treated as the channel dimension. This novel approach allows for feature extraction within and across patches, enabling simple 1D convolutions to capture temporal patterns both locally and globally. 
2. Although both PatchMixer and PatchTST utilize parallelization, when it comes to computational cost, under default configurations, PatchTST requires more than double the flops of PatchMixer (e.g., for forecasting 720 future time points in the ETTm1 dataset, the flops are 154.29M for PatchTST and 66.32M for PatchMixer). This substantial reduction in computational load without sacrificing performance underscores PatchMixer's efficiency.

Reviewer comment (Writing):
There is an error in a citation in the depthwise separable convolution section of the related work section.

Author response:
Thank you for pointing out the citation error in the related work section. This has been corrected in the revised manuscript. We have also realigned all tables to the top of the pages, placing their captions accordingly.

Reviewer comment (Presentation):
I am not sure I understand the relationship between the 2D patches of CV methods and the method introduced here. Please correct me if I am wrong but, the patches used in this approach are 1D, right? I understand that other methods (Eqs. 1-3) are 2D, but I think that this is somewhat confusing. Perhaps it would be better to describe these methods in less detail here (and move details to the related work section)?

Author response:
We recognize the potential confusion between 2D patches in CV methods and our 1D patch approach in PatchMixer. The 2D patch is an image patch with C channels, in the shape of (H, W), obtained by the conv2d function. A 1D patch is a sequence patch of length P with N channels, where N is the number of patches, obtained using the unfold function.  
The revised manuscript now clearly differentiates between the two, with both the related work and method sections updated to enhance clarity and prevent any misunderstanding.

Reviewer comment (Presentation):
Following the style guidelines of ICLR, I would recommend the authors to align tables with the top of the pages and put their caption on the top.

Author response:
Thank you for pointing out the citation error in the related work section. This has been corrected in the revised manuscript. We have also realigned all tables to the top of the pages, placing their captions accordingly.

Reviewer comment (Presentation):
The results shown in Table 2 are very confusing. How are these divided? Basically all numbers here are either bold or underlined.

Author response:
We acknowledge that the presentation in Table 2 may not be clear and this has been modified following your suggestions. We bold the best results and supplement the prediction performance using only dual heads. We also put the analysis between ordinary convolution and depthwise separable convolution in a separate section in the appendix. The comparison of convolution is placed in a separate section in the appendix. 

As we see below, for small datasets like ETTh1, methods with only dual heads can achieve SOTA performance. 
For medium and large-scale datasets like Traffic and ETTm1, our PatchMixer layer is shown to be better.

| Methods                       | Metrics | Traffic 96 | Traffic 192 | Traffic 336 | Traffic 720 | ETTm1 96 | ETTm1 192 | ETTm1 336 | ETTm1 720 | ETTh1 96 | ETTh1 192 | ETTh1 336 | ETTh1 720 |
|--------------------------------|-------------------|------------|-------------|-------------|-------------|-----------|------------|------------|------------|-----------|------------|------------|------------|
| **Dual Heads**                 | MSE               | 0.377      | 0.391       | 0.399       | 0.429       | 0.290     | 0.327      | 0.356      | 0.420      | **0.355** | **0.373**  | 0.392      | **0.440**  |
| **Dual Heads**                 | MAE               | 0.251      | 0.256       | 0.258       | 0.283       | 0.340     | 0.362      | 0.381      | 0.416      | 0.384     | 0.394      | 0.412      | **0.455**  |
| **Attention Mechanism + Dual Head** | MSE         | 0.368      | 0.388       | 0.401       | 0.447       | 0.294     | 0.331      | 0.360      | 0.422      | **0.355** | 0.378      | 0.393      | 0.451      |
| **Attention Mechanism + Dual Head** | MAE         | **0.240**  | **0.249**   | **0.256**   | **0.273**   | 0.340     | 0.365      | 0.386      | 0.417      | **0.382** | 0.397      | 0.411      | 0.460      |
| **PatchMixer Layer + Dual Head**    | MSE         | **0.362**  | **0.382**   | **0.392**   | **0.428**   | **0.290** | **0.325**  | **0.353**  | **0.413**  | **0.355** | **0.373**  | **0.391**  | 0.446      |
| **PatchMixer Layer + Dual Head**    | MAE         | 0.242      | 0.252       | 0.257       | 0.282       | 0.340     | **0.361**  | **0.382**  | **0.413**  | 0.383     | **0.394**  | **0.410**  | 0.463      |

Reviewer comment (Experiments):
While the paper presents several ablation studies, it takes over several assumptions from previous papers. Although the current paper already presents several ablations, I believe that the impact of the paper could be further enhanced by doing ablations on these different assumptions to analyze if these also hold in the current setting.

Author response:
We appreciate the reviewer's valuable suggestions on adding more ablations. We have conducted the experiments accordingly and show the results below:

●**Effectiveness of Channel Independence vs. Channel Mixing:**
The comparative analysis of the experiments has been detailed in Appendix 7.1 of PatchTST, which highlights the superiority of channel-independent models in adaptability, training data efficiency, and overfitting prevention. To avoid duplication, we will not replicate these experiments but will reference them in the appendix of our revised manuscript.

●**Impact of Different Patch Sizes:**
We are grateful for your comment. The selection of patch length has been analyzed in Appendix 4.1 of PatchTST, demonstrating that MSE scores remain consistent across various P values. We also select one large dataset, Electricity, to verify in our model. The result table with a prediction length of 720 time steps is as follows, affirming our model's robustness to the patch length hyperparameter: 

| Patch Size | 2     | 4     | 8     | 12    | 16    | 24    | 32    | 40    |
|-----------|-------|-------|-------|-------|-------|-------|-------|-------|
| MSE      | 0.207 | 0.207 | 0.207 | 0.207 | 0.206 | 0.205 | 0.206 | 0.206 |
| MAE      | 0.292 | 0.293 | 0.292 | 0.293 | 0.292 | 0.291 | 0.292 | 0.292 |

●**The Impact of Different Overlaps:**
The patch overlap is related to the patch step size, and the relationship is $O=P-S$. We conduct this experiment on two large datasets, Weather and Electricity. One observation is that the MSE score oscillated in a small range (between 0.003 and 0.001) with different choices of S, indicating the robustness of our model to patch overlaps.  Please refer to Figure 6 in the Appendix 2 of the revised script.

●**The Influence of Varying Convolutional Kernel Sizes.**
We have conducted the experiments and put the results in Figure 7 in Appendix 2. There was no significant pattern in the MSE score with different choices of K, the ideal patch overlap and convolution kernel size may depend on the datasets.

Reviewer comment (Experiments):
The assumption that channel independence is better than channel mixing for forecasting tasks should be tested through ablation studies.

Author response:
We appreciate the reviewer's valuable suggestions on adding more ablations. We have conducted the experiments accordingly and show the results below:

●**Effectiveness of Channel Independence vs. Channel Mixing:**
The comparative analysis of the experiments has been detailed in Appendix 7.1 of PatchTST, which highlights the superiority of channel-independent models in adaptability, training data efficiency, and overfitting prevention. To avoid duplication, we will not replicate these experiments but will reference them in the appendix of our revised manuscript.

Reviewer comment (Experiments):
The size of the patches and their overlap should be tested through ablation studies.

Author response:
We appreciate the reviewer's valuable suggestions on adding more ablations. We have conducted the experiments accordingly and show the results below:

●**Impact of Different Patch Sizes:**
We are grateful for your comment. The selection of patch length has been analyzed in Appendix 4.1 of PatchTST, demonstrating that MSE scores remain consistent across various P values. We also select one large dataset, Electricity, to verify in our model. The result table with a prediction length of 720 time steps is as follows, affirming our model's robustness to the patch length hyperparameter: 

| Patch Size | 2     | 4     | 8     | 12    | 16    | 24    | 32    | 40    |
|-----------|-------|-------|-------|-------|-------|-------|-------|-------|
| MSE      | 0.207 | 0.207 | 0.207 | 0.207 | 0.206 | 0.205 | 0.206 | 0.206 |
| MAE      | 0.292 | 0.293 | 0.292 | 0.293 | 0.292 | 0.291 | 0.292 | 0.292 |

●**The Impact of Different Overlaps:**
The patch overlap is related to the patch step size, and the relationship is $O=P-S$. We conduct this experiment on two large datasets, Weather and Electricity. One observation is that the MSE score oscillated in a small range (between 0.003 and 0.001) with different choices of S, indicating the robustness of our model to patch overlaps.  Please refer to Figure 6 in the Appendix 2 of the revised script.

Reviewer comment (Experiments):
The size of the conv kernels should be tested through ablation studies.

Author response:
We appreciate the reviewer's valuable suggestions on adding more ablations. We have conducted the experiments accordingly and show the results below:

●**The Influence of Varying Convolutional Kernel Sizes.**
We have conducted the experiments and put the results in Figure 7 in Appendix 2. There was no significant pattern in the MSE score with different choices of K, the ideal patch overlap and convolution kernel size may depend on the datasets.

Reviewer comment (Experiments):
There has been a line of work on long convolutional models, which are able to model long term dependencies without patching, e.g., S4 [1], CKConv [2], Hyena [3], etc. Do you have any thoughts regarding the use of these models for forecasting?

Author response:
Thank you for pointing this out. In the Related Work section of our revised manuscript, we have included citations to these works to enrich the discussion around long convolutional models. Specifically, we acknowledge the following:

●**S4 (Structured State Spaces):** This method effectively processes long sequences through structured state spaces and has been applied in the field of large language models. Its theoretical explanation for long-term modeling provides valuable insights, particularly in understanding the underlying mechanisms of long-term dependency modeling.

●**CKConv (Continuous Kernel Convolution):** The approach constructs continuous convolutional kernels using a small neural network, aiming to develop a versatile CNN architecture. This innovative approach inspires our future research towards handling data of varying resolution, length, and dimension, broadening the scope of convolutional neural network applications.

●**Hyena (Hierarchical Convolutional Language Models):** It adeptly models long-term contexts by interweaving implicit parameterized long convolutions with data-controlled gating. This methodology presents a compelling approach that we believe could be adapted for temporal tasks, potentially enhancing the modeling of complex time series data.

Reviewer comment (Theory):
The motivation is unclear. The author only elaborates on how the module is designed, but ignores why it is designed this way.

Author response:
Our work is inspired by PatchTST's patch processing technique, which is proposed for use in the Transformer architecture. We hypothesize that the strong predictive performance observed in time series forecasting is attributed more to the patch embedding methods rather than the inherent predictive capabilities of the Transformers. Therefore, we design PatchMixer based on the CNN architecture, which has been shown to be faster at a similar scale to Transformers. Through theoretical analysis and practical experiments, we have demonstrated that our approach not only surpasses Transformers in predictive performance but also achieves 2-3 times faster training and inference speeds, thereby substantiating the efficacy of patch embedding in temporal prediction.

Reviewer comment (Writing):
The writting should be improved. It would be better if the motivation is highlighted.

Author response:
Our work is inspired by PatchTST's patch processing technique, which is proposed for use in the Transformer architecture. We hypothesize that the strong predictive performance observed in time series forecasting is attributed more to the patch embedding methods rather than the inherent predictive capabilities of the Transformers. Therefore, we design PatchMixer based on the CNN architecture, which has been shown to be faster at a similar scale to Transformers. Through theoretical analysis and practical experiments, we have demonstrated that our approach not only surpasses Transformers in predictive performance but also achieves 2-3 times faster training and inference speeds, thereby substantiating the efficacy of patch embedding in temporal prediction.

Reviewer comment (Experiments):
Some experimental results are not convincing. E.g. in Table 4, how PatchTST performs with MSE+MAE and SmoothL1Loss?

Author response:
Thanks for the suggestions. We have conducted additional experiments to evaluate the impact of four different loss functions on our model's performance including the mentioned losses. The revised Table 4 now presents a comprehensive comparison between PatchMixer and PatchTST using these loss functions, with the superior results highlighted in bold. 

Here we provide the following key observations, which are included in the revised paper. 

First, PatchMixer consistently outperforms PatchTST across the board, affirming our model's superior performance. 

Secondly, SmoothL1loss is inferior to the other two losses. 

Thirdly, the model trained with combined MAE+MSE loss shows a better performance on the MAE/MSE metrics than the model trained with MSE loss only.

Reviewer comment (Writing):
More elaboration about the motivation.

Author response:
Our work is inspired by PatchTST's patch processing technique, which is proposed for use in the Transformer architecture. We hypothesize that the strong predictive performance observed in time series forecasting is attributed more to the patch embedding methods rather than the inherent predictive capabilities of the Transformers. Therefore, we design PatchMixer based on the CNN architecture, which has been shown to be faster at a similar scale to Transformers. Through theoretical analysis and practical experiments, we have demonstrated that our approach not only surpasses Transformers in predictive performance but also achieves 2-3 times faster training and inference speeds, thereby substantiating the efficacy of patch embedding in temporal prediction.