Paper: Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators (ICLR-2024)

Reviewer comment (Evaluation):
While the motivation lies in the exploitation of dependence between variables, the experiment shows better performance and improvement for CI methods compared with CD methods, which is kind of contradicted. Could you provide more insight into the possible reasons?

Author response:
**Q1: The experiment shows better performance and improvement for CI methods compared with CD methods. Could you provide possible reasons?**

**A1:** Let "$>$" mean "outperforms". The following is the rationale for the experimental results.

$\text{Why CI backbones} > \text{CD backbones}?$

Crossformer focuses on variables that share similarities at the same time but ignores leading indicators that look different at the same time (as illustrated in Figure 1) due to a large leading step. MTGNN learns a fixed set of indicators and overlooks the varying leading steps. Thus MTGNN can easily overfit training data, of which the lead-lag relationships are different from test data (see Figure 7-8). Other CD backbones (*e.g.*, Autoformer and FEDformer) simply mix up all variables and suffer from noise.

$\text{Why CI+LIFT}>\text{CI} > \text{CD+LIFT}?$

This happens when $CI \gg CD$ and the performance gap is too large for LIFT to fill. The reason is that LIFT is affected by the error from the backbone. As defined in Eq. (5), the shifted window of leading indicators is padded by a part of the backbone's forecasts when the leading step is smaller than the forecast horizon. 

$\text{Why LIFT achieves higher relative improvement on CI than CD}?$

As both CD backbones and the LIFT module exploit dependence between variables, their leveraged dependence information can overlap. By contrast, CI backbones treat each variate independently and hence benefit more from the channel dependence exploited by LIFT.

Reviewer comment (Theory):
It is not clear how to mathematically interpret the term **locally stationary lead-lag relation** as "influence requires a certain time delay to propagate and take effect." Is it possible to provide a formal definition? Furthermore, in the context of cross-correlation used to quantify such relationships, does this indicate that a lead-lag relationship is inferred whenever we observe relatively larger cross-correlation in differing time slots between two variables? (In the causality field, correlation is not equivalent to influence and effect).

Author response:
**Q2: Provide a formal definition of locally stationary lead-lag relation. Is a lead-lag relationship inferred whenever we observe relatively larger cross-correlation?**

**A2:** Formally, there is a locally stationary lead-lag relationship between variate $i$ and $j$ at time $t$ if
\\begin{aligned}
\exists \delta >0,\\ &D(\mathcal X^{(j)} _{t+1:t+H}, \mathcal X^{(i)} _{t+1-\delta:t+H-\delta})\approx D(\mathcal X^{(j)} _{t-L+1:t}, \mathcal X^{(i)} _{t-L+1-\delta:t-\delta})\approx 0 \\\\
\wedge\exists t'> t,\\ &D(\mathcal X^{(j)} _{t'+1:t'+H}, \mathcal X^{(i)} _{t'+1-\delta:t'+H-\delta}) \not\approx 0,
\\end{aligned}
where $D(\mathcal W,\mathcal W')$ is the distance between two windows $\mathcal W$ and $\mathcal W'$, and $D(\cdot,\cdot)$ can be estimated by a neural network or a non-parametric metric (*e.g.*, $1-\text{cross-correlation coefficient}$). Furthermore, if there exists $i\neq j$ such that $D(\mathcal X^{(j)} _{t+1:t+H}, \mathcal X^{(i)} _{t+1:t+H})\approx 0$, we dismiss variate $i$ as it is not a leading indicator and offers no advance information. 

Since most time series are not extremely non-stationary, we assume the lead-lag relationship keeps stationary across the lookback and horizon windows, *i.e.*, $\mathcal X^{(j)} _{t-L+1:t}$ and $\mathcal X^{(j)} _{t+1:t+H}$ has the same leading indicator $i$ with the leading step close to $\delta$.

A lead-lag relationship is not inferred by the correlation-based estimation alone. The motivation of using cross-correlation is to estimate all variates *as fast as possible* and filter out most noisy variates. A large correlation between two lookback windows just suggests that the indicator can serve as a potential reference for the horizon window. We rely on the Lead-aware Refiner which is the key component to exploit the advance information from potential leading indicators. The Lead-aware Refiner is trained to learn how much we can trust the suggestion and how we leverage the potential leading indicators.

Reviewer comment (Experiments):
As a plug-and-play module that works on top of the baseline predictions, some fair comparison about the computation cost such as the total computation load should be considered. Otherwise, it is not surprised to see that after injecting some inductive bias, the performance is better than baselines.

Author response:
**A3:** Following LSTF-Linear [1], we compare the backbones and LIFT per sample by the number of parameters, the number of multiply-accumulate operations (MACs, $\approx\frac{1}{2}$FLOPs), the GPU memory consumption, and the inference time in the following table.

|||PatchTST|PatchTST+LIFT|relative additional cost|Crossformer|Crossformer+LIFT|relative additional cost|
|--|--|--|--|--|--|--|--|
|**Weather**|**Parameter** (M)|4.3|4.7|9.3%|11.7|12.5|7.1%|
||**MACs** (G)|0.5|0.5|1.8%|4.4|4.4|0.2%|
||**Memory** (MB)|43|46|7.7%|206|209|1.6%|
||**Time** (ms)|4.7|6.0|29.6%|37.7|40.6|7.6%|
|**Electricity**|**Parameter** (M)|4.3|4.7|9.3%|2.4|2.8|16.7%|
||**MACs** (G)|7.1|7.2|1.8%|8.6|8.7|1.4%|
||**Memory** (MB)|406|411|1.2%|873|878|0.6%|
||**Time** (ms)|5.1|6.0|17.4%|31.5|33.1|5.1%|
|**Traffic**|**Parameter** (M)|4.3|4.7|9.3%|3.2|3.6|12.4%|
||**MACs** (G)|19.0|19.4|1.8%|10.6|10.9|3.2%|
||**Memory** (MB)|1061|1092|2.9%|1558|1578|1.3%|
||**Time** (ms)|5.1|5.8|14.5%|32.7|36.4|11.3%|

Our empirical study shows that LIFT additionally requires an average of 10.7% parameters, 1.7% MACs, 2.5% memory, and 14.2% inference time more than its backbone. It is noteworthy that our Lead Estimator is non-parametric and can precompute estimation only once on the training data, reducing practical training time.

---
[1] Zeng & Chen et. al. Are Transformers Effective for Time Series Forecasting? AAAI 2023.

Reviewer comment (Theory):
Consider a setting in which the influence structure among the time series is a chain, i.e., $X^{(1)} -> X^{(2)} -> \cdots -> X^{(C)}$. In this case, there are scenarios in which the pairwise cross-correlations between $X^{(C)}$ and {$X^{(i)}: 1\leq i\leq C-2$} are all higher than the correlation between $X^{(C)}$ and $X^{(C-1)}$. This means that LIFT will not pick $X^{(C-1)}$ for forecasting while the only relevant time series for forecasting $X^{(C)}$ is $X^{(C-1)}$.

Author response:
**A2:** The chain structure $X^{(1)} \rightarrow X^{(2)} \rightarrow \cdots \rightarrow X^{(C)}$ indicates that the leading steps of the $C-1$ indicators *w.r.t.* $X^{(C)}$ are in a descending order, *i.e.*, $\delta _1^{(1)}>\cdots>\delta _{C-2}^{(C)}>\delta _{C-1}^{(C)}$. On the one hand, a leading indicator with a larger leading step offers more advance information. On the other hand, a higher cross-correlation coefficient indicates a more possible lead-lag relationship. Thus, it is better to pick $\{X^{(i)}\} _{i=1}^{C-2}$ which are more helpful and more reliable than $X^{(C-1)}$.

Reviewer comment (Theory):
Is it possible to discover causal relationships (e.g., Granger causality) among time series via modifications of LIFT?

Author response:
**A3:** A possible way to compute a time-dependent causal graph via the modifications of LIST is the following. Initially, we build a graph based on the lead estimation results at time $t$, where each node (a variate $j$) is connected to $K$ nodes (its potential indicators $\mathcal I _t^{(j)}$), *i.e.*, $\{X _{t-\delta _{i, t}^{(j)}}^{(i)} \rightarrow X _t^{(j)} \mid 1 \leq j \leq C, i \in \mathcal I _t^{(j)}\}$. Then, we prune the graph as follows.

1. Replacing the sum pooling in Eq. (11) by mean pooling, we rewrite our frequency mixing as

$$
\widetilde{V}^{(j)} = g\left({V}^{(j)}\parallel \frac{1}{|\mathcal K^{(j)}|}\sum _{k\in \mathcal K^{(j)}} {\widetilde U}^{(j)} _k \parallel \frac{1}{|\mathcal K^{(j)}|}\sum _{k\in \mathcal K^{(j)}} {\widetilde \Delta}^{(j)} _k \right),
$$

where $\mathcal K^{(j)}=\{1,2,\cdots,K\}$ denotes the indices of $\mathcal I _t^{(j)}$.

2. We train LIFT on the training data and calculate the forecast loss on the validation data. 

3. For each target node $j$ in $[1, C]$, we repeat removing the most useless indicator by the following iteration algorithm until the loss of forecasting $j$ cannot be decreased.

   - For each $k$ in $\mathcal K^{(j)}$, we specially dismiss the $k$-th indicator, re-conduct frequency mixing, and calculate a new forecast loss by using the remaining $|\mathcal K^{(j)}|-1$ indicators. A lower loss indicates that the $k$-th selected one is just a noise. 

   - Then, we update $\mathcal K^{(j)}$ as $\mathcal K^{(j)}-\{k^{\prime}\}$ if dismissing the $k^{\prime}$-th indicator leads to the lowest forecast loss.

Reviewer comment (Writing):
Why there is subscript $\Delta$ and $U$ in equation (9)?

Author response:
**A4:** To simplify notations in the following discussion, we leave out the superscript $(j)$ and the subscript $k$. Let $U$ denote the information of one leading indicator, $V$ denote the prediction of the target variate, and $\Delta=U-V$. Eq. (9) generates different filters $r _U$ and $r _{\Delta}$ for $U$ and $\Delta$, respectively.

The reason is that we consider two kinds of relationships: 

- $V$ is directly influenced by $U$, and the ground-truth $V$ contains a degree of $U$, *e.g.*, $V _{true} \approx V + r _U\odot U$; 

- $V$ is similar to $U$ when they are both influenced by a latent factor, and the ground-truth $V$ is the interpolation between $V$ and $U$, *e.g.*, $V _{true} \approx (1- r _{\Delta})\odot V + r _{\Delta}\odot U=V+ r _{\Delta}\odot\Delta$. 

Note that both situations may exist in practice. We thus use different layers to generate $r _U$ and $r _{\Delta}$. Our ablation study in Figure 5b demonstrates the effectiveness of modeling both kinds of influence.

Reviewer comment (Presentation):
In Figure 3, “all layers in the grey background” refer to which part in the figure?

Author response:
**A2:** The grey rectangles refer to normalization, denormalization, Lead Estimator, Fourier transform, and the inverse Fourier transform. We would like to highlight that we avoid repeating Lead Estimation every epoch but precompute it *only once* on training data since our Lead Estimator is non-parametric.