Paper: The Logarithm Trick: achieve better long term forecast via Mean Logarithm Square Loss (ICLR-2024)

Reviewer comment (Experiments):
Given that this paper focuses on the study of loss functions for time series forecasting, there should be comparisons with more time series loss functions. For example, see [1].

Author response:
The MLSE loss function is specifically designed to address long-term forecasting problems by utilizing the concept of an 'error amplifier'. It is not a universal loss function for arbitrary timeseries tasks. We value your suggestions and are planning to extend our research to cover these interesting perspectives.

Reviewer comment (Writing):
On page 4, what do T850 and Z500 represent?

Author response:
We report the Root Mean Square Error (RMSE) in terms of two weather states, the T850 and the Z500, which are the crucial measurements in weather simulation tasks.

Reviewer comment (Experiments):
What is the difference between the experiments done in Sections 3.1 and 3.2? Why have the authors chosen different forecasting models for each section?

Author response:
We aim to test the effectiveness of MLSE across different models, various dataset scales, and disparate dataset resolutions.

Reviewer comment (Theory):
There should be more explanation about the penultimate equation on page 6, which is about the propagation matrix M.

Author response:
The $M_{t+N-1}^{N-1}$ is the propagation matrix produced by the Jacobian operator on multi-dimension Mean Value Theorem:.You can simply regard it as the $\nabla f_\theta({X}_{\delta})$. It is a matrix, if the input $X$ is a vector.

Reviewer comment (Experiments):
LMSE is the same as LMAE (up to a constant), so it would be fairer to include training under MAE loss as well.

Author response:
As far as the MAE function, we quickly make three extra experiments, the result below 
|  time_step   | T->T+2 |        |        | T->T+3 |        |        | 
| :----------: | :----: | :----: | :----: | :----: | :----: | :----: | 
|   loss_fn    |   L1   |   Ln   |  MAE   |   L1   |   Ln   |  MAE   | 
| 6H(01step)   | 46.315 | 33.953 | 35.531 | 49.954 | 34.873 | 37.126 | 
| 12H(02step)  | 58.529 | 47.613 | 48.234 | 60.805 | 47.730 | 49.233 |
| Day1(04step) | 81.847 | 71.277 | 69.627 | 81.749 | 70.288 | 70.215 | 
| Day3(12step) | 224.78 | 213.42 | 207.88 | 220.16 | 209.3  | 205.83 | 
| Day5(20step) | 436.9  | 421.95 | 414.41 | 424.26 | 412.11 | 408.82 | 

TL;DR:  The MLSE and MAE can both significantly outperform MSE. During the short-term stages, MLSE performs slightly better than MAE. Conversely, during the long-term stages, MAE tends to have a slight edge over MLSE.

Reviewer comment (Reproducibility):
For how many orders does the Dalpha strategy optimize for? Does it pick one order?

Author response:
>  For how many orders does the Dalpha strategy optimize for ? Does it pick one order ?

Figure 5 should the notation of Dalpha for first order and first + second order. Compute the Dalpha requires one-more forward operation (achieve t+N+1) compare to tradition long-term error (who only forward to t+N) and use the extra prediction compute the error. Thus, Dalpha-1 actually use the second order prediction (notice, not the error). And the Dalpha-2 use the third order prediction.

Reviewer comment (Novelty):
How is MASE an improved alternative over Dalpha? They seem two different things altogether. Dalpha optimizes the relative errors while MASE optimizes each error separately.

Author response:
> How is MASE an improved alternative over Dalpha? They seem two different things altogether. Dalpha optimizes the relative errors while MASE optimizes each error separately

Lets take the first order as example, the full Dalpha requires optimize the e1 and e2'/e1. It requires spontaneously minimum the e1 and maximum e1.  This make training hard. In MASE we just remove the denominator.

Reviewer comment (Theory):
I don't understand how the conclusion towards using the logarithm is made, in the paper it's just mentioned that "it should be", but that's not sufficient.

Author response:
> I don't understand how the conclusion towards using the logarithm is made, in the paper it's just mentioned that "it should be", but that's not sufficient.

In our paper, we approached the optimization problem by analyzing an amplifier due to the denominator. Therefore, we switched to optimizing its logarithmic form. Let's delve deeper into the derivation for **two timestamps optimization**:
$$
\min \\{\mathcal{E}^{I} \quad and \quad \alpha^{II}=\frac{\mathcal{E}^{II}}{\mathcal{E}^{I}}-1\\}
$$
  Notice minimize $\alpha^{II}$ equals to minimize $\frac{\mathcal{E}^{II}}{\mathcal{E}^{I}}$  which can be converted to log format as minimized  $\log(\mathcal{E}^{II}) - log(\mathcal{E}^{I})$. Remember we still need minimize $\mathcal{E}^{I}$ which is same as minimize $\log(\mathcal{E}^{I})$. These two part are independent implying that we need to assign two coefficients to combine them like:
$$
\text{final loss}=a*[\log(\mathcal{E}^{II}) - log(\mathcal{E}^{I})]+b*\log(\mathcal{E}^{I})
$$
  They are hyper-parameter and we just set the $a = 1, b = 2$ in this paper. This results in: 
$$
\text{final loss}=\log(\mathcal{E}^{II}) + \log(\mathcal{E}^{I}) ={MLSE}
$$

It's crucial to note there is no explicit theoretical formulation mandating the use of the logarithm. The MASE can also be the feasibly accommodation for using amplifier information. However, it need an additional forward computation for an extra peek into the future. And we want a zero-cost operation. Therefore we name this technology as a `trick` and leave its validation in a comprehensive experiments.

Reviewer comment (Novelty):
Although this paper proposes a simple and effective trick for autoregressive prediction, its significance is not enough for a ICLR publication.

Author response:
Thank you for your feedback. 
---
- We would like to emphasize that, to the best of our knowledge, this is the first paper that attempts to analyze the problem of error accumulation and proposes an effective and cost-free method to mitigate it.

Reviewer comment (Novelty):
The novelty seems somewhat limited. The only new thing introduced is adding a "log" transform in the fine tuning loss function – everything else is prior work. Mean squared log error is not a new metric or loss, and has been used for both training and evaluating forecast models before. One could argue the connection to error accumulation in this particular setting (and the application particularly to fine-tuning) is the novel part, but the justification for this does not seem so clear, aside from experiment results.

Author response:
We would like to clarify a misunderstanding: The loss function used in our study is the **Mean Log Squared Error** (MLSE)$\sum ln(x_i^2)$, **not the Mean Squared Logarithmic Error** (MSLE) $\sum(ln(1+x_i)-ln(1+\bar{x}))^2$. These are fundamentally different concepts with different applications and interpretations. Our choice of MLSE was deliberate and based on the specific objectives and demands of our research. To the best of our knowledge, there has been no explicit discussion of the influence of MLSE in similar contexts. Therefore, we believe our exploration of this area offers a unique contribution to the field.

Reviewer comment (Reproducibility):
The paper is not very clear or detailed. For example, the actual loss functions used in each training phase are not well defined, and there are many details left out, such as how hyperparameters are selected and models tuned. There are many incomplete sentences. The details in the explanations/formulation are lacking and not clear. For example, many details are left out in section 4 making it hard to follow. It is not clearly explained how the error metrics are computed in the reported results – for example, for table 2, in a particular range like 0–96 – is this computed averaged across predictions for all time steps 1 to 96, or just the last time step?

Author response:
We appreciate your interest in the detailed experimental setup. However, we believe there might be some confusion regarding the error calculation process we deployed. In  table 2, our methodology involves calculating the error as an average across all time step predictions, such as 0 -> 96, 0->196, etc. This approach aligns with the paradigm used in the previous work like TimesNet, **ensuring a fair basis for comparison** between our ATT results and E2E results.  A more clean clarify below:
- Train
  - the E2E model train on setting that use `seq[0:96]` to predicted `seq[96:96+720]`
  - the ATT model trained on setting that used `seq[0:96]` to predict `seq[96:192]` and autoregressive for seq[192:192+96]. 
- Testing:
  - the E2E use `seq[0:96]` to predicted `seq[96:96+720]` in one step
  - the ATT use `seq[0:96]` to predicted `seq[96:96+720]` in 720//96 + 1 times
  The error is compute via take average for each timestamp prediction for `label[96:96+720]`

Reviewer comment (Theory):
In the analyses, the authors claim the Nth order error can be seen as the 1st order error plus some error accumulation term, then state that it is most efficient to reduce/optimize for this Nth order error by optimizing for the 1st order error plus the "error amplifier" term (and subsequently drop the 1st order error part as well) – presumably as a way to bring in the other error terms – but the equating analysis is a bit of a stretch and requires multiple assumptions. Additionally, they also state that the 1st order error is already accounted for by the pre-training step, so fine-tuning can focus on the error amplifier, which is then roughly approximated by the mean log squared nth order error (bringing the loss back to the original objective anyway, just with a log term). This seems like an unjustified assumption, since fine-tuning a neural net can cause the original concept learned to be forgotten, but it would be suspected that since the nth order error depends on the 1st order error anyway, it would likely avoid the first order error getting worse, but it is not justified in this way here.

Author response:
Kindly notice the fine-tuning procedure optimizes both the $\mathcal{E}^I$ and $\mathcal{E}^{II}$ error terms. This is in contrast to the pretraining phase, which only optimizes $\mathcal{E}^I$. Therefore, the fine-tuning phase does not cause the model to forget the original concept learned but instead enriches it with additional information pertaining to $\mathcal{E}^{II}$.

Reviewer comment (Reproducibility):
The experimental procedure and results seem a bit lacking. As mentioned, specifics around hyperparameters and how they are selected are missing.

Author response:
To address your query, we have provided extensive information on the experiment setting, including the hyperparameters and the 'mean and variance' in the appendix. Specifically, the hyperparameters are outlined on pages 13 and 14, and a snapshot view of the 'mean and variance' can be found on pages 9, 10, and 11.