Paper: Feature-aligned N-BEATS with Sinkhorn divergence (ICLR-2024)

Reviewer comment (Presentation):
For ease of understanding, I think that either the Contributions paragraph or a different subsection before Section 2 could be used to more clearly signpost how the algorithm will be presented in Sections 2 and 3. Including something like equation (3.7) earlier to demonstrate the kind of training objective being constructed could be helpful for readers to identify the role played by feature alignment throughout.

Author response:
We are grateful for the insightful suggestions, especially those aimed at improving our presentation. For ease of understanding, before starting the main section (Sections 2 and 3), we have signposted how the training objective with the corresponding algorithm will be presented,

> The remainder of the article is organized as follows. In Section 2, we set the domain generalization problem in the context of time series forecasting, review the doubly residual stacking architecture of N-BEATS, and introduce the error analysis for domain generalization models. Section 3 is devoted to defining the marginal feature measures inspiring the stack-wise alignment, introducing the Sinkhorn divergence together with the corresponding representation learning bound, and presenting the training objective with the corresponding algorithm. In particular, Figure 1 therein illustrates the overall architecture of Feature-aligned N-BEATS. In Section 4, comprehensive experimental evaluations are provided. Section 5 concludes the paper. Other technical descriptions, visualized results, and ablation studies are given in Appendix.

In addition, we have contextualized $\alpha$, $\beta$, $\gamma$, $K$, $M$, and $L$ in 'Experimental details' in Section 4.

> The lookback horizon, forecast horizon, and the number of source domains are set to be $\alpha=50$, $\beta=10$, and $K=3$, respectively (noting that it depends on the characteristics of source domains' datasets). Furthermore, the number of stacks and blocks, and the dimension of feature space are set to be $M=3$, $L=4$, and $\gamma=512$, respectively (noting that it is consistent with N-BEATS [45]).

Reviewer comment (Writing):
A large number of parameters are introduced in these sections, such as $\alpha$, $\beta$, $\gamma$, $K$, $M$, and $L$; I think some of these could be better contextualized, so that the reader can know which are expected to be large and which small. (For instance, we should probably think of $L$ and $M$ as relatively small constants, due to their impact on Lipschitzness.)

Author response:
We are grateful for the insightful suggestions, especially those aimed at improving our presentation. For ease of understanding, before starting the main section (Sections 2 and 3), we have signposted how the training objective with the corresponding algorithm will be presented,

> The remainder of the article is organized as follows. In Section 2, we set the domain generalization problem in the context of time series forecasting, review the doubly residual stacking architecture of N-BEATS, and introduce the error analysis for domain generalization models. Section 3 is devoted to defining the marginal feature measures inspiring the stack-wise alignment, introducing the Sinkhorn divergence together with the corresponding representation learning bound, and presenting the training objective with the corresponding algorithm. In particular, Figure 1 therein illustrates the overall architecture of Feature-aligned N-BEATS. In Section 4, comprehensive experimental evaluations are provided. Section 5 concludes the paper. Other technical descriptions, visualized results, and ablation studies are given in Appendix.

In addition, we have contextualized $\alpha$, $\beta$, $\gamma$, $K$, $M$, and $L$ in 'Experimental details' in Section 4.

> The lookback horizon, forecast horizon, and the number of source domains are set to be $\alpha=50$, $\beta=10$, and $K=3$, respectively (noting that it depends on the characteristics of source domains' datasets). Furthermore, the number of stacks and blocks, and the dimension of feature space are set to be $M=3$, $L=4$, and $\gamma=512$, respectively (noting that it is consistent with N-BEATS [45]).

Reviewer comment (Experiments):
While Table 2 of the empirical results compares the runtime of computing different divergences, I'd be interested in seeing comparisons between the end-to-end runtimes of the methods in Table 1, in order to evaluate whether computing the regularization term for feature alignment adds a substantial computational cost.

Author response:
We appreciate the suggestion for evaluating the potential increase in computational cost due to the introduction of FA.
We have added a new row to Table 1, which presents the training time per iteration.

|               | N-HiTS | + FA (Ours) | N-BEATS-I | + FA (Ours) | N-BEATS-G | + FA (Ours) | NLinear | DLinear | Autoformer | Informer |
| ------------- | ------ | ----------- | --------- | ----------- | --------- | ----------- | ------- | ------- | ---------- | -------- |
| Runtime (sec) | 0.26   | 0.80        | 0.32      | 0.97        | 0.16      | 0.68        | 0.04    | 0.05    | 0.58       | 0.50     |

The updated table reveals that stack-wise alignment does increase the training time compared to N-BEATS.
However, Table 2 demonstrates that SD-based alignment, the chosen regularizer, exhibits the most enhanced generalizability while incurring marginal computational cost.
Please note that there is no difference in runtime between Feature-aligned N-BEATS and original N-BEATS during inference, since alignment is not applied during this stage.

Reviewer comment (Writing):
The clause on pg 1 "the proposed model is built a purely deep learning-based model which is N-BEATS proposed by [42,43]" parses a little awkwardly.

Author response:
Thanks for this careful advice. We have revised this sentence to 'In particular, the proposed model is built upon a deep learning model which is N-BEATS [45, 46]'.

Reviewer comment (Writing):
$\epsilon$ is overloaded as both the entropic regularization parameter and the expected risk in Proposition 2.1, which is never explicitly defined.

Author response:
Thanks for point out our mistake. We have replaced the notation for the expected risk to $R^T, R^k$. Furthermore, we denoted the notations before Proposition 2.1.

> Furthermore, denote by $R^k(\cdot):{\cal H}\rightarrow \mathbb{R}$ and $R^T(\cdot):{\cal H}\rightarrow \mathbb{R}$ the expected risk under the source measures $\mathbb{P}^k$, $k=1,\dots,K$, and the target measure $\mathbb{P}^T$, respectively.

Reviewer comment (Presentation):
The forecasting algorithms N-BEATS-I and N-BEATS-G are seen for the first time in Section 4 without context; a pointer to the first page of the appendix---where they are helpfully disambiguated---would be appreciated.

Author response:
Following this suggestion, we have added the pointer in the caption of Table 1.

Reviewer comment (Presentation):
Given the number of entries and small fonts in the tables (like Table 1), bolding the values of the best losses in each row may improve readability.

Author response:
To improve readability, we have bolded the best performance in each row of the all tables.
We appreciate the suggestion, which significantly improves the clarity of our paper.

Reviewer comment (Experiments):
Is there any intuition or experimental results that comment on the smoothness of $\sigma \circ g^m$? While the Lipschitzness bound in Lemma 3.4 suggests it could be highly non-smooth, I'd be interested to know more about whether the gradients were largely well-behaved.

Author response:
Thanks for pointing this out.
According to your suggestion, we have plotted the forecasting and alignment losses during both training and validation steps in Figure 5 in Appendix D.
This illustrates stable training process, indicating that the gradients are well-behaved during training.

> Feature-aligned N-BEATS is a complicated architecture based on the doubly residual stacking principle with an additional feature-aligning procedure. To investigate the stability of training, we analyze the training and validation loss plots. Figure 5 indicates that the gradients are well-behaved during training. This stable optimization is regarded to the Lemma 3.4 presented in Section 3.1.

Reviewer comment (Presentation):
What should I take away from Figure 1? It appears that the structure of the embeddings changes little between stacks and blocks, and it's unclear to me whether that should be the case or not. The structural differences between $\lambda = 1$ and $\lambda = 3$ are almost undetectable.

Author response:
In Figure 1 (which is Figure 2 in revised version), we demonstrate 1) the effects of FA on the embedding space and 2) its sensitivity to the hyperparameter $\lambda$, with a focus on the alignment aspect (We typically used $\lambda$ values of 1 and 3 for N-BEATS-G due to their superior performance in varied experimental settings.). N-BEATS operates as a recurrent framework, with blocks in stacks capturing distinct temporal features in time series data, such as trends and seasonality. Our goal was to assess whether FA successfully narrows the gap between source domains concerning these diverse features. This involved analyzing all embedding spaces from every block across all stacks. In this regard, it becomes evident that the domains are closely aligned for most features. Although the difference between $\lambda = 1$ and $\lambda = 3$ appears subtle, indicating that N-BEATS-G with FA has low sensitivity to $\lambda$ variations}, we observed that $\lambda$ can be a critical factor in specific cases, such as with N-BEATS-I, as detailed in Appendix F (Figure 8).

Reviewer comment (Presentation):
Overall presentation: The main contributions are rather buried in the technical exposition of sections 1-3. The main idea (adding stack-wise FA through minimising Sinkhorn div. added to the N-BEATS loss) could be simply laid out and perhaps visualised while some of the technical details may be put into Appendices. I would consider starting from algorithm and Section 3.3. and than relate it to principles in a backward manner as opposed to a current forward exposition that simply takes too long and has several leaps of faith anyways (see below). Authors are encouraged to lighten the exposition up in the main paper body and increase the rigour int he Appendix. More over, if paper claims “extraordinary results” bringing some results (e.g.,  Fig.4 in Appendix or simplified Fig.1) forward into the Introduction could increase motivation of the readers.

Author response:
We appreciate the valuable suggestions for our presentation. In response, we have provided a simplified illustration of Feature-aligned N-BEATS in Figure 1 in Section 3 of the revised script for better accessibility (noting that its detailed version is provided in Figure 3 in Appendix A). Additionally, we have relocated some of the more detailed technical aspects to the appendix for improved readability. Furthermore, for ease of understanding, before starting the main section (Sections 2 and 3), we have signposted how the training objective with the corresponding algorithm will be presented,

> The remainder of the article is organized as follows. In Section 2, we set the domain generalization problem in the context of time series forecasting, review the doubly residual stacking architecture of N-BEATS, and introduce the error analysis for domain generalization models. Section 3 is devoted to defining the marginal feature measures inspiring the stack-wise alignment, introducing the Sinkhorn divergence together with the corresponding representation learning bound, and presenting the training objective with the corresponding algorithm. In particular, Figure 1 therein illustrates the overall architecture of Feature-aligned N-BEATS. In Section 4, comprehensive experimental evaluations are provided. Section 5 concludes the paper. Other technical descriptions, visualized results, and ablation studies are given in Appendix.

We tried to lighten the exposition up in the main paper by putting some remarks (Remarks C.1 and C.2 in the revised script, which were Remarks 3.7 and 3.8 in the original script) in Appendix C.

Reviewer comment (Experiments):
Due to multitude of approximation and heuristics used, i.e., Sinkhorn instead of H-divergence, Definition 3.5, Remark 3.8,  “... Instead, we conduct the alignment by regularizing exclusively on feature extractors $\Psi$ ...” the work effectively strongly relies on experimental results. Experimental results seems only marginally better compared to selected alternatives (Tables 1 and 2), however. These should be extended to increase the validity of the results.

Author response:
We kindly ask to take a careful look at the case of CDG and ODG with MASE metric in Table 1. Since CDG represents the case where the source domain data are distinctive from one another, we can say that the representation learning through the alignment of distinctive source domains' feature measures can help to improve the original model's forecasting power. Furthermore, since the ODG represents that the source data and target data belong to the different super domains (that is, the source and target data are distinctive from each other), we can say that our FA-NBEATS improves the forecasting power on the (unknown) distinctive target sequential data (when domain shift is significant). For the case of IDG, as the source data and target data belong to the same super domain (that is, the source and target data are relatively not distinctive from each other), the improvement seems to be marginal. But this is quite intuitive because the model would not have much chance to learn domain invariant features through the similar types of the source domain and the model's prediction power at the not distinctive target domain could not be improved through the feature alignment of similar distributions. In this regard, we further emphasize that our experiment performances are verified in diverse domain generalization environments, and the performances' improvement also stands out.

Furthermore, our choice of SD over other divergences is substantiated by Table 2 in Section 4 and Table 6 in Appendix E for the following reasons: 1) While SD yields performance comparable to WD (noting that SD is an approximate of WD.), SD is more computationally efficient. 2) SD outperforms both MMD and KL (in the ODG context) in terms of forecasting, with MMD incurring higher computational costs compared to SD. 3) Despite KL being the most computationally efficient, Table 6 reveals its lack of robustness, as indicated by its frequent inability to produce a result, denoted as 'NaN'.

Reviewer comment (Evaluation):
I believe important but unclear computational cost increase compared to alternatives (cost increase is only assumed by me but expected since method adds “FA” and more on top).

Author response:
We appreciate the suggestion for evaluating the potential increase in computational cost due to the introduction of FA.
We have added a new row to Table 1, which presents the training time per iteration.

|               | N-HiTS | + FA (Ours) | N-BEATS-I | + FA (Ours) | N-BEATS-G | + FA (Ours) | NLinear | DLinear | Autoformer | Informer |
| ------------- | ------ | ----------- | --------- | ----------- | --------- | ----------- | ------- | ------- | ---------- | -------- |
| Runtime (sec) | 0.26   | 0.80        | 0.32      | 0.97        | 0.16      | 0.68        | 0.04    | 0.05    | 0.58       | 0.50     |

The updated table reveals that stack-wise alignment does increase the training time compared to N-BEATS.
However, Table 2 demonstrates that SD-based alignment, the chosen regularizer, exhibits the most enhanced generalizability while incurring marginal computational cost.
Please note that there is no difference in runtime between Feature-aligned N-BEATS and original N-BEATS during inference, since alignment is not applied during this stage.

Reviewer comment (Evaluation):
One of the major features that N-BEATS algorithm provides is an Interpretability. The paper avoids any analysis of interpretability after adding FA.

Author response:
We appreciate your concern about the interpretability of our model.
In this revision, we have provided a visualization (Figure 7) and additional explanation in Appendix F for interpretability analysis,

> Figure 7 exhibits the interpretability of the proposed method by presenting the final output of the model and intermediate stack forecasts. N-BEATS-I and N-HiTS presented in Appendix A have interpretability. More specifically, N-BEATS-I explicitly captures trend and seasonality information using polynomial and harmonic basis functions, respectively. N-HiTS employs Fourier decomposition and utilizes its stacks for hierarchical forecasting based on frequencies. Preserving these core architectures during the alignment procedure, Feature-aligned N-BEATS still retains interpretability.

Reviewer comment (Experiments):
Would naive feature alignment applied on N-BEATS work? Why not? That is, why should reader consider using proposed method?

Author response:
We highlight that under doubly residual stacking architecture, defining feature alignment is nontrivial, which is well-described in the 4-th paragraph of the Introduction in the revised script. 

> First, N-BEATS does not allow the feature alignment in a 'one-shot' unlike the aforementioned references. This is because it is a hierarchical multi-stacking architecture in which each stack consists of several blocks and is connected to each other by residual operations and feature extractions.

In this regard, we adopted a 'stack-wise' feature alignment strategy rather than 'naive' or 'block-wise' alignment. In particular, we believe that stack-wise alignment ensures wide feature coverage in the context of alignment and promotes efficient gradient descent. The N-BEATS framework, which employs a doubly residual stacking architecture, encodes diverse features across its blocks and stacks [45]. Our goal was to align these features by frequently calculating divergence (either block-wise or stack-wise) and incorporating this into the alignment loss. However, as [47] noted, overly frequent loss propagation can lead to issues of gradient vanishing or exploding. Consequently, we determined that 'stack-wise' alignment offers an optimal frequency for this process. A comparative analysis of block-wise and stack-wise alignment is detailed in Table 8 in Appendix G.1.

Reviewer comment (Experiments):
Paper already suggests using approximation to H-divergence which is computational expensive. So even more importantly, what are computation expenses related to proposed stack-wise approach compared to N-BEATS?

Author response:
We appreciate the suggestion for evaluating the potential increase in computational cost due to the introduction of FA.
We have added a new row to Table 1, which presents the training time per iteration.

|               | N-HiTS | + FA (Ours) | N-BEATS-I | + FA (Ours) | N-BEATS-G | + FA (Ours) | NLinear | DLinear | Autoformer | Informer |
| ------------- | ------ | ----------- | --------- | ----------- | --------- | ----------- | ------- | ------- | ---------- | -------- |
| Runtime (sec) | 0.26   | 0.80        | 0.32      | 0.97        | 0.16      | 0.68        | 0.04    | 0.05    | 0.58       | 0.50     |

The updated table reveals that stack-wise alignment does increase the training time compared to N-BEATS.
However, Table 2 demonstrates that SD-based alignment, the chosen regularizer, exhibits the most enhanced generalizability while incurring marginal computational cost.
Please note that there is no difference in runtime between Feature-aligned N-BEATS and original N-BEATS during inference, since alignment is not applied during this stage.

Reviewer comment (Presentation):
Consider bringing, possibly simplified, model architecture (Fig 2 from Appendix) to the main body instead of a rather cumbersome technical description Section 2.

Author response:
We appreciate the valuable suggestions for our presentation. In response, we have provided a simplified illustration of Feature-aligned N-BEATS in Figure 1 in Section 3 of the revised script for better accessibility (noting that its detailed version is provided in Figure 3 in Appendix A). Additionally, we have relocated some of the more detailed technical aspects to the appendix for improved readability. Furthermore, for ease of understanding, before starting the main section (Sections 2 and 3), we have signposted how the training objective with the corresponding algorithm will be presented,

> The remainder of the article is organized as follows. In Section 2, we set the domain generalization problem in the context of time series forecasting, review the doubly residual stacking architecture of N-BEATS, and introduce the error analysis for domain generalization models. Section 3 is devoted to defining the marginal feature measures inspiring the stack-wise alignment, introducing the Sinkhorn divergence together with the corresponding representation learning bound, and presenting the training objective with the corresponding algorithm. In particular, Figure 1 therein illustrates the overall architecture of Feature-aligned N-BEATS. In Section 4, comprehensive experimental evaluations are provided. Section 5 concludes the paper. Other technical descriptions, visualized results, and ablation studies are given in Appendix.

We tried to lighten the exposition up in the main paper by putting some remarks (Remarks C.1 and C.2 in the revised script, which were Remarks 3.7 and 3.8 in the original script) in Appendix C.

Reviewer comment (Novelty):
Focus more on novelty of the proposed approach in Sections 1-3.

Author response:
We already clarified our contributions explicitly in the 7-th paragraph of the Introduction (Section 1),

> To provide an informative procedure of stack-wise feature alignment, we introduce a concrete mathematical formulation of N-BEATS (Section 2), which enables to define the pushforward feature measures induced by the intricate residual operations and the feature extractions for each stack (Section 3.1). From this, we make use of theoretical properties of optimal transport problems to show a representation learning bound for the stack-wise feature alignment with the Sinkhorn divergence (Theorem 3.6), which justifies the feasibility of Feature-aligned N-BEATS. To embrace comprehensive domain generalization scenarios, we use real-world data and evaluate the proposed method under three distinct protocols based on the domain shift degree. We show that the model consistently outperforms other forecasting models. In particular, our method exhibits outstanding generalization capabilities under severe domain shift cases (Table 1). We further conduct ablation studies to support the choice of the Sinkhorn divergence in our model (Table 2).