Paper: TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series (ICLR-2024)

Reviewer comment (Evaluation):
In the experiment section, the author evaluates the results using five datasets from the Monash Time Series Forecasting Repository based on dimensions, frequencies, and length. The chosen samples appear to be of a small size. Merely calculating the average rank might not provide an unbiased and comprehensive evaluation. The results would be more persuasive if the author utilized a broader range of datasets and presented a critical difference plot.

Author response:
**W2 / Q3: Justify the choice of datasets; a broader range of datasets may be utilized**

We take the opportunity to further explain why we used the five datasets - electricity, fred-md, kdd-cup, solar-10min, traffic. These were chosen due to their ubiquity in probabilistic forecasting benchmarks [1, 2, 3, 4]. TimeGrad [1] and TempFlow [2] use Solar, Electricity, Traffic in addition to two others, SPD [3] uses Electricity and Solar, and TACTiS [5] those five.

Nevertheless, we agree that the addition of more datasets will undoubtedly strengthen our contribution. In the short time available, we were able to evaluate the forecasting performance of TACTiS and TACTiS-2 on two new datasets, namely covid-deaths and rideshare, from the Monash Time Series Forecasting Repository [5]. We present these new results below, and add them to Appendix D.1 of the paper. As seen below, the NLL of TACTIS-2 is significantly better than that of TACTIS in both of the added datasets. 

Table: Mean NLL values on forecasting. Lower is better.
| Model    | covid-deaths | rideshare |
|----------|--------------|-----------|
| TACTiS   | -1.605 +- 0.121 | 14.815 +- 1.101 |
| TACTiS-2 | **-4.859 +- 0.093** | **10.451 +- 0.250** |

[1] Rasul, K., Seward, C., Schuster, I., & Vollgraf, R. (2021, July). Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning (pp. 8857-8868). PMLR.

[2] Rasul, K., Sheikh, A. S., Schuster, I., Bergmann, U. M., & Vollgraf, R. (2020, October). Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows. In International Conference on Learning Representations.

[3] Marin Bilos, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Gunnemann. Modeling temporal data as continuous functions with stochastic process diffusion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2452–2470. PMLR, 23–29 Jul 2023.

[4] Alexandre Drouin, Etienne Marcotte, and Nicolas Chapados. TACTiS: Transformer-attentional copulas for time series. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5447–5493. PMLR, 17–23 Jul 2022.

[5] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.

Reviewer comment (Experiments):
The author might consider employing other techniques, such as artificially creating uneven sampling frequencies, to garner more samples.

Author response:
**W3: The author might consider employing other techniques, such as artificially creating uneven sampling frequencies, to garner more samples.**

We thank the reviewer for the suggestion. We point to the reviewer that we have added new results to the paper in Appendix D.2 validating the ability of TACTiS-2 to work with uneven and unaligned data, artificially creating uneven and unaligned data from real-world datasets. We agree with the reviewer that it would be interesting to leverage the flexibility of using artificially created samples of unaligned and uneven data as a data augmentation method when training models. Such a method would be orthogonal to TACTiS and can potentially improve its performance.  

**Q2: How to capture the dimensional dependency using this architecture?**

One notable property of the TACTiS and TACTiS-2 models is that the multivariate dependencies are guaranteed to be captured by the copula component of the model, not in the others. Hence, if one is interested in inspecting dimensional dependencies to facilitate downstream decision-making tasks, one only needs to focus on the copula. For instance, one could draw samples from the copula and measure the statistical association between all dimensions. This technique was employed to inspect the learned associations in Drouin et al. (2022) [1]. Another interesting fact is that, in this work, we show that TACTiS-2 captures multivariate dependencies much better than TACTiS, as indicated by its significantly lower negative log likelihoods. Hence, this new model is poised to produce more accurate insights into multivariate dependencies, which could in turn lead to more accurate decision-making.

[1] Alexandre Drouin, Etienne Marcotte, and Nicolas Chapados. TACTiS: Transformer-attentional copulas for time series. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5447–5493. PMLR, 17–23 Jul 2022.

Reviewer comment (Experiments):
Why using autoencoders rather than other deep learning models such as CNN to estimate the probability distributions?

Author response:
**W1 / Q1: Choice of architecture rather than other deep learning models such as CNN to estimate the probability distributions**

The reviewer correctly states that alternative architectures, such as CNNs have been successful in time series tasks. Notable examples are included among our baselines. For instance, GPVar [1] and TimeGrad [2], which use LSTM-based architectures, and SPD [3] which uses a CNN-based architecture. In this work, our choice of using the transformer architecture is based on the added flexibility that it brings to the model [4].

In fact, a key feature of the transformer in TACTiS and TACTiS-2 is how the tokenization is performed for multivariate time series (as described in Sec 2 of our paper). We represent each value in the time series as a token paired with a timestamp and arbitrary stochastic covariates. We define our learning tasks with a mask that indicates which values are observed and which values are to be predicted. Note that this allows any learning task to be defined on the data based on the pattern of the mask. For instance, for forecasting, the values to be predicted would be at the end, while for interpolation, the values to be predicted will be in arbitrary positions in the data, and appropriate masks can be used for these tasks based on the pattern of the observed and predicted values. Arbitrary and more complex tasks can be defined using this approach. Further, as the tokenization does not impose any constraints on the data, this allows the architecture to consume arbitrarily uneven or unaligned data which often occur in the real-world. Thereby, adopting the transformer architecture in TACTiS-2 enables it as a general-purpose model satisfying the five desiderata for real-world time series problems as described in our introduction. We stress that these properties are enabled due to the flexibility of the transformer architecture.
 
[1] Salinas, D., Bohlke-Schneider, M., Callot, L., Medico, R., and Gasthaus, J. High-dimensional multivariate forecasting with low-rank Gaussian copula processes. Advances in Neural Information Processing Systems, 32: 6827–6837, 2019.

[2] Rasul, K., Seward, C., Schuster, I., & Vollgraf, R. (2021, July). Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning (pp. 8857-8868). PMLR.

[3] Marin Bilos, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Gunnemann. Modeling temporal data as continuous functions with stochastic process diffusion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2452–2470. PMLR, 23–29 Jul 2023.

[4] Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L. (2022). Transformers in time series: A survey. arXiv preprint arXiv:2202.07125.

Reviewer comment (Theory):
How to capture the dimensional dependency using this architecture?

Author response:
**W3: The author might consider employing other techniques, such as artificially creating uneven sampling frequencies, to garner more samples.**

We thank the reviewer for the suggestion. We point to the reviewer that we have added new results to the paper in Appendix D.2 validating the ability of TACTiS-2 to work with uneven and unaligned data, artificially creating uneven and unaligned data from real-world datasets. We agree with the reviewer that it would be interesting to leverage the flexibility of using artificially created samples of unaligned and uneven data as a data augmentation method when training models. Such a method would be orthogonal to TACTiS and can potentially improve its performance.  

**Q2: How to capture the dimensional dependency using this architecture?**

One notable property of the TACTiS and TACTiS-2 models is that the multivariate dependencies are guaranteed to be captured by the copula component of the model, not in the others. Hence, if one is interested in inspecting dimensional dependencies to facilitate downstream decision-making tasks, one only needs to focus on the copula. For instance, one could draw samples from the copula and measure the statistical association between all dimensions. This technique was employed to inspect the learned associations in Drouin et al. (2022) [1]. Another interesting fact is that, in this work, we show that TACTiS-2 captures multivariate dependencies much better than TACTiS, as indicated by its significantly lower negative log likelihoods. Hence, this new model is poised to produce more accurate insights into multivariate dependencies, which could in turn lead to more accurate decision-making.

[1] Alexandre Drouin, Etienne Marcotte, and Nicolas Chapados. TACTiS: Transformer-attentional copulas for time series. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5447–5493. PMLR, 17–23 Jul 2022.

Reviewer comment (Experiments):
Why do we take this particular subset of datasets?

Author response:
**W2 / Q3: Justify the choice of datasets; a broader range of datasets may be utilized**

We take the opportunity to further explain why we used the five datasets - electricity, fred-md, kdd-cup, solar-10min, traffic. These were chosen due to their ubiquity in probabilistic forecasting benchmarks [1, 2, 3, 4]. TimeGrad [1] and TempFlow [2] use Solar, Electricity, Traffic in addition to two others, SPD [3] uses Electricity and Solar, and TACTiS [5] those five.

Nevertheless, we agree that the addition of more datasets will undoubtedly strengthen our contribution. In the short time available, we were able to evaluate the forecasting performance of TACTiS and TACTiS-2 on two new datasets, namely covid-deaths and rideshare, from the Monash Time Series Forecasting Repository [5]. We present these new results below, and add them to Appendix D.1 of the paper. As seen below, the NLL of TACTIS-2 is significantly better than that of TACTIS in both of the added datasets. 

Table: Mean NLL values on forecasting. Lower is better.
| Model    | covid-deaths | rideshare |
|----------|--------------|-----------|
| TACTiS   | -1.605 +- 0.121 | 14.815 +- 1.101 |
| TACTiS-2 | **-4.859 +- 0.093** | **10.451 +- 0.250** |

[1] Rasul, K., Seward, C., Schuster, I., & Vollgraf, R. (2021, July). Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning (pp. 8857-8868). PMLR.

[2] Rasul, K., Sheikh, A. S., Schuster, I., Bergmann, U. M., & Vollgraf, R. (2020, October). Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows. In International Conference on Learning Representations.

[3] Marin Bilos, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Gunnemann. Modeling temporal data as continuous functions with stochastic process diffusion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2452–2470. PMLR, 23–29 Jul 2023.

[4] Alexandre Drouin, Etienne Marcotte, and Nicolas Chapados. TACTiS: Transformer-attentional copulas for time series. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5447–5493. PMLR, 17–23 Jul 2022.

[5] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.

Reviewer comment (Novelty):
The main weakness of this work is its limited significance. The key contribution is an incremental modification of the existing TACTIS model. While the work may be interesting for individuals specifically focusing on TACTIS, its significance for the broader time series community is unclear. One may argue on the basis of the empirical results; however, in their current form, the results are not exciting and comprehensive enough to fully support this argument.

Author response:
**W1: Limited significance, incremental modification of TACTiS, significance to the broader time series community is unclear, absence of enough methodological contributions**

We respectfully disagree. Our work introduces new theoretical results on two-stage optimization of non-parametric copulas, followed by a complete change of the optimization problem in TACTiS (along with necessary architectural changes), and much better predictive performance and training dynamics across a variety of tasks. Our work makes the desirable properties of TACTiS actually usable in practice, leading to what is currently the most accurate general-purpose model for multivariate probabilistic time series prediction. We kindly ask the reviewer to refer to the “Significance of Contribution” section of the general comment for an extended response to their comments and hope that they will reconsider their position.

Reviewer comment (Experiments):
In the absence of enough methodological contributions, the empirical contribution needs to be comprehensive. However, the experiments have only been conducted on 5 datasets from the Monash repository. The baseline selection also needs improvement for the _state of the art_ claim. CSDI and SSSD would be better baselines for the empirical comparison.

Author response:
**W2: Results are not that exciting and comprehensive, empirical contribution needs to be comprehensive**

We have broadened our empirical evaluation following each of the reviewer’s suggestions, as follows.

Reviewer comment (Experiments):
Adding better baselines such as CSDI and SSSD.

Author response:
**W2-a: Adding CSDI and SSSD as baselines**

We thank the reviewer for the suggestion. We planned to include CSDI [2] as part of our baselines, but our results did not reach the level of performance found in their paper despite our best efforts, and hence we decided to not include it in our forecasting baselines. However, as per the reviewer’s request, we present the results of CSDI and discuss them with respect to those of TACTiS-2 below. We are happy to add this comparison to the final revision of the paper. Regarding SSSD, we agree that it would make a meaningful baseline. It has been left out of our study as the published codebase [4] does not support forecasting out-of-the-box. However, we are happy to revisit this and include it in the final revision of our work. 

We hereby present results that compare the CRPS-Sum, CRPS, and Energy Score of CSDI with all other methods on 4 datasets of the forecasting benchmark. We were not able to include the largest dataset (traffic) due to limited time, but we are happy to add it to the final revision of the paper.

Table: Mean CRPS-Sum values for the forecasting experiments. Lower is better.
| Model      	| electricity 	| fred-md 	| kdd-cup 	| solar-10min 	|
|-----------------|----------------|-----------|-----------|----------------|
| Auto-ARIMA 	| 0.077 +- 0.016	| 0.043 +- 0.005	| 0.625 +- 0.066	| 0.994 +- 0.216      	|
| ETS        	| 0.059 +- 0.011	| 0.037 +- 0.010	| 0.408 +- 0.030 | 0.678 +- 0.097    	|
| TempFlow   	| 0.075 +- 0.024	| 0.095 +- 0.004 | 0.250 +- 0.010	| 0.507 +- 0.034    	|
| SPD        	| 0.062 +- 0.016	| 0.048 +- 0.011	| 0.319 +- 0.013	| 0.568 +- 0.061    	|
| TimeGrad   	| 0.067 +- 0.028  | 0.094 +- 0.030	| 0.326 +- 0.024	| 0.540 +- 0.044    	|
| GPVar      	| 0.035 +- 0.011  | 0.067 +- 0.008	| 0.290 +- 0.005	| 0.254 +- 0.028   	|
| CSDI       	| 0.165 +- 0.009 	| 0.043 +- 0.005	| 0.388 +- 0.038	| 1.063 +- 0.081   	|
| TACTiS     	| 0.021 +- 0.005 	| 0.042 +- 0.009	| 0.237 +- 0.013	| 0.311 +- 0.061	|
| TACTiS-2   	| **0.020 +-  0.005**	| **0.035 +- 0.005** 	| **0.234 +- 0.011** | **0.240 +- 0.027**   	|

Table: Mean CRPS values for the forecasting experiments. Lower is better.

| Model      	| electricity 	| fred-md 	| kdd-cup 	| solar-10min 	|
|----------------|-----------------|-----------|-----------|----------------|
| ETS        	| 0.094 +- 0.014 	| 0.050 +- 0.011 	| 0.560 +- 0.028 	| 0.844 +- 0.119      	|
| Auto-ARIMA 	| 0.129 +- 0.015	| 0.052 +- 0.005 	| 0.477 +- 0.015 	| 0.636 +- 0.060      	|
| TempFlow   	| 0.109 +- 0.024 	| 0.110 +- 0.003 	| 0.451 +- 0.005 	| 0.547 +- 0.036     	|
| TimeGrad   	| 0.101 +- 0.027 	| 0.142 +- 0.058 	| 0.495 +- 0.023 	| 0.560 +- 0.047      	|
| SPD        	| 0.099 +- 0.016 	| 0.058 +- 0.011 	| 0.465 +- 0.005 	| 0.585 +- 0.050       	|
| GPVar      	| 0.067 +- 0.010 	| 0.086 +- 0.009	| 0.459 +- 0.009 	| 0.298 +- 0.034       	|
| CSDI       	| 0.182 +- 0.011 	| 0.055 +- 0.006 	| 0.527 +- 0.028 	| 1.053 +- 0.067      	|
| TACTiS     	| 0.052 +- 0.006 	| 0.048 +- 0.010 	| 0.420 +- 0.007 	| 0.326 +- 0.049      	|
| TACTiS-2   	| **0.049 +- 0.006** 	| **0.043 +- 0.006**	| **0.413 +- 0.007** 	| **0.256 +- 0.029**       	|

Table: Mean Energy Score values for the forecasting experiments. Lower is better.
| Model      | electricity (x 10000) | fred-md (x 100000) | kdd-cup (x 1000) | solar-10min (x 100) |
|------------|-----------------------|--------------------|-----------------|---------------------|
| Auto-ARIMA | 44.59 +- 8.56         | 8.72  +- 0.81      | 18.76  +- 3.31  | 19.42 +- 3.37      |
| TempFlow   | 10.25 +- 2.03         | 20.16 +- 0.74      | 3.30   +- 0.19  | 4.25 +- 0.16         |
| ETS        | 7.94  +- 0.93         | 7.90  +- 1.88      | 3.60   +- 0.24  | 4.74  +- 0.17      |
| TimeGrad   | 9.69  +- 2.62         | 19.87  +- 7.23     | 3.30   +- 0.19  | 4.31 +- 0.23         |
| SPD        | 8.90  +- 1.18         | 8.94  +- 1.82      | 3.13   +- 0.24  | 3.68  +- 0.31      |
| GPVar      | 6.80  +- 0.62         | 11.43 +- 1.60      | 3.18  + 0.20    | 2.60 +- 0.10         |
| CSDI       | 19.96 +- 0.93         | 7.97  +- 0.01      | 3.37 +- 0.21    | 7.74 +- 0.66         |
| TACTiS     | 5.42  +- 0.57         | 8.18  +- 1.83      | 2.93 +- 0.22    | 2.88 +- 0.23         |
| TACTiS-2   | **4.91  +- 0.52**         | **6.72  +- 0.10**      | **2.81  +- 0.19**  | **2.37 +- 0.14**         |

Reviewer comment (Experiments):
Highlighting other aspects of the model. For instance, the "flexibility to handle unaligned/unevenly-sampled series" is mentioned multiple times in the related work but has only been studied in a toy setting.

Author response:
**W2-c: Highlight other aspects of the model. For instance, flexibility to handle unaligned/uneven data is only done on toy datasets.**

We thank the reviewer for suggesting the use of real-world data to evaluate the flexibility properties of TACTiS-2, as it rightfully improves the experimental quality of the work. We here report results on forecasting for real-world datasets where the observations have been rendered unevenly spaced and unaligned.

**Protocol and Results**: The datasets used in these experiments have been derived from real-world datasets with aligned and evenly sampled observations. To create uneven sampling, we randomly skipped the observations at some time steps in the original data. To create unalignment, we randomly selected a sampling frequency for each series in the dataset. The datasets resulting from such corruptions are faithful to real-world scenarios. Due to time constraints, we perform experiments on the kdd-cup and solar-10min datasets. These were chosen since they have the longest prediction horizon (48 and 72, respectively). A detailed discussion of the setting and implementational details has been added to the Appendix in Section D.2, and we will add results on the other datasets in the final version of the paper. As shown below, TACTiS-2 outperforms TACTiS in forecasting in real-world datasets where the data is uneven or unaligned.

Table: Mean NLL values of forecasting on uneven data. Lower is better.
| Model    | kdd-cup     | solar-10min   |
|----------|---------|---------|
| TACTiS   | 2.858 +- 0.476| 1.001 +- 0.783 |
| TACTiS-2 | **1.866 +- 0.230**| **0.261 +- 0.011** |

Table: Mean NLL values of forecasting on unaligned data. Lower is better.
| Model    | kdd-cup     | solar-10min   |
|----------|---------|---------|
| TACTiS   | 1.527 +- 0.072 | -0.038 +- 0.220 |
| TACTiS-2 | **0.722 +- 0.163** | **-3.218 +- 0.249** |

Reviewer comment (Evaluation):
Is there a reason why the numbers for electricity and traffic datasets differ so significantly from existing works [1]?

Author response:
**Q1: Why do numbers for electricity and traffic differ from existing work like in CSDI?**

In CSDI [1] the training protocol consists in using all data prior to a fixed date for training and testing on rolling windows over the subsequent data, adopting the protocol from GPVar [2]. Our work instead uses the backtesting protocol of TACTiS [3], which combines rolling-window evaluation with periodic retraining on a series of timestamps, averaging the results across these timestamps. Such a procedure mimics the use of the model in a real-world setting.

[1] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems, volume 34, 2021.

[2] Salinas, D., Bohlke-Schneider, M., Callot, L., Medico, R., and Gasthaus, J. High-dimensional multivariate forecasting with low-rank Gaussian copula processes. Advances in Neural Information Processing Systems, 32: 6827–6837, 2019.

[3] Alexandre Drouin, Etienne Marcotte, and Nicolas Chapados. TACTiS: Transformer-attentional copulas ´ for time series. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5447–5493. PMLR, 17–23 Jul 2022.

Reviewer comment (Theory):
Although TACTiS-2 is generally well motivated, it is still unclear how the two-stage solution in Sec 3.2 is derived. It should be clarified whether it is derived based on any assumptions/theorem or directly constructed.

Author response:
**W1: Unclear how the two-stage solution in Sec 3.2 is derived. Any assumptions/theorem or is directly constructed.**

Our proposed approach to learning non-parametric copulas is theoretically supported by Proposition 2 (Validity), which shows that any non-parametric copula learned using the proposed two-stage approach is valid. As stated in Sec. 3.2, the proof of this proposition is in App B.2. This proof relies heavily on Sklar's theorem [1]. It assumes that the random variables have continuous marginals and that the marginal and copula estimators have infinite expressivity (capacity in practical terms).

[1] Abe Sklar. Fonctions de répartition à n dimensions et leurs marges. Publications de l’Institut Statistique de l’Université de Paris, 8:229–231, 1959.

Reviewer comment (Theory):
Can the two-stage optimization achieve the optimal solution of the optimization problem in Eq. (7) and (8)? Although the optimal solution of each sub-problem in Eq (7) and (8) can be achieved, the gap between the theoretical optimal solution and the two-stage method should be discussed clearly.

Author response:
**W2: Can the two-stage optimization achieve the optimal solution of (7) and (8)? Although the optimal solution of each sub-problem in Eq (7) and (8) can be achieved, the gap between the theoretical optimal solution and the two-stage method should be discussed clearly.**

Yes, Proposition 2 shows that, given estimators with sufficient capacity and continuous random variables, the two-stage procedure is guaranteed to attain the optimum for both Problems (7) and (8) and that this also corresponds to the minimum of Problem (4). We refer the reviewer to App. B.2 for a detailed proof.

Beyond the theoretical guarantee provided by Proposition 2, Section 5.1 brings empirical evidence that the optimal solution can even be attained in conditions of finite amounts of data, model capacity, and training time. Using a distribution where the ground-truth copula is known, Fig 3 shows that our two-stage approach leads to a learned copula density that closely matches the ground truth. Fig 14 in the Appendix Section B.3 further shows that the learned marginal distributions closely match the ground truth. Hence, this experiment confirms that our two-stage procedure can reach the optimal solution of Problems (7) and (8), respectively, in a practical setting.

As for real-world datasets where the ground-truth copula is not known, we remark that it is not possible to verify that a solution is optimal. Yet, we now provide evidence that the two-stage procedure reaches solutions that are closer to the optimum (i.e., the minimum of Problem (4)) than alternative approaches: 

1. Using the permutation-based approach of Drouin et al. (2022) [1] (denoted as TACTiS), and 

2. Optimizing Problem (4) directly without the two-stage procedure (denoted as TACTiS-2 without the curriculum). 

This can be observed in the following table, which shows that, out of all considered approaches, the two-stage curriculum used in TACTiS-2 leads to the smallest negative log-likelihoods, indicating that TACTiS-2's solutions are the closest to optimality.

Table: Mean NLL values on forecasting. Results are over a single backtesting timestamp. Lower is better.
| Model                           | KDD      | Solar  |
|-----------------------------|----------|--------|
| TACTiS                          | 0.489 +- 0.095 | -1.458 +- 0.278 |
| TACTiS-2 without the curriculum | -0.086 +- 0.206| -2.654 +- 0.192 |
| TACTiS-2                        | **-1.343 +- 0.055**| **-5.107 +- 0.287** |

Reviewer comment (Evaluation):
The gap between TACTiS and TACTiS-2 can be further discussed.

Author response:
**W3: The gap between TACTiS and TACTiS-2 can be further discussed.**

In addressing this comment, we interpret “the gap” as “the empirical performance gap”. If it was meant as “methodological gap”, we invite the reviewer to refer to the “significance of contribution” paragraph in the general comment for a detailed exposition of the differences between TACTiS and TACTiS-2. We will be glad to further discuss any other interpretation.

TACTiS-2’s superior performance is mainly due to its improved optimization procedure, which ultimately allows it to reach better solutions, in particular better values for the copula parameters. In contrast, the convergence of TACTiS is significantly slower (see Fig. 4). This likely results from: i) optimizing a much more complex objective, with a factorial parameter complexity (see Sec. 3.1), ii) jointly optimizing the marginal and copula components, and iii) from the difficulty of achieving permutation invariance, which is essential to its convergence and for the validity of the copula. Additionally, the performance of TACTiS-2 is further enhanced by its use of a dual-encoder that learns representations specialized for each distributional component.

We added this valuable discussion on the empirical gap in the revised Section 7.

[1] Alexandre Drouin, Etienne Marcotte, and Nicolas Chapados. TACTiS: Transformer-attentional copulas for time series. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5447–5493. PMLR, 17–23 Jul 2022.

Reviewer comment (Evaluation):
May the author explain more about the difference between the task “solar-10min” and others, considering the different behavior on it, as shown in Figure 1.

Author response:
**W4: The difference between solar-10min and others:**

We thank the reviewer for the opportunity to discuss this difference. The solar-10min dataset taken from Lai et al. [1], contains solar power production records sampled every 10 minutes from 137 PV plants in Alabama. Among the considered datasets (see Appendix C.1 Table 6), this is the one with the highest sampling frequency (10 minutes). Therefore, such a dataset is rich in statistical dependencies due to the observations being very close in time. In addition, the geographical proximity of the PV plants leads to non-negligible correlations between the series due to shared factors, such as weather conditions. 

Indeed, the largest improvement in NLL is observed on the solar-10min dataset, as seen in Figure 1. We explain this observation by TACTiS-2’s improved ability to model multivariate dependencies. As for the drop in FLOPs, as shown in Table 2, TACTiS-2 still reduces the FLOPs by a significant amount. However the drop in FLOPs in less compared to other datasets, possibly because TACTiS was already quite efficient in FLOPs for this dataset, leaving less room for improvements by TACTiS-2. 

[1] Lai, Guokun, et al. "Modeling long-and short-term temporal patterns with deep neural networks." The 41st international ACM SIGIR conference on research & development in information retrieval. 2018.

Reviewer comment (Novelty):
The core innovation claimed by the paper is the reduction in computational complexity through a two-stage solution, first estimating marginals and then dependencies. However, this approach isn't novel, as seen in references [1,2]. The paper would benefit from a clearer distinction of how its methodology differs significantly from these existing methods.

Author response:
**W1: The core innovation is reduction in computational complexity through a two-stage solution. However this approach isn’t novel as seen in references [1] and [2].**

The reviewer is correct that such a two-stage procedure has previously been studied in the copula literature. In fact, we had attributed this approach to Joe and Xu (1996) [3] in the paper and have now added the relevant references that you suggested in the revision of the paper. As for the novelty component, we would like to emphasize that, to the best of our knowledge, previous work [1, 2, 3] has shown the validity of this approach in the context of parametric and semi-parametric copula estimation. In contrast, the use of neural networks for both the copula and marginals places us in the fully non-parametric setting. For this reason, we contribute Proposition 2 (and its proof in App. B.2), which shows that this approach is still valid in the non-parametric setting. We then proceed to using this result to make significant improvements over known approaches to learning transformer-based non-parametric copulas, constituting an additional element of novelty. We refer the reviewer to the “Significance of Contribution” in the General Comment for a detailed discussion of the significance of our work.

[1] Andersen, Elisabeth Wreford. “Two-stage estimation in copula models used in family studies.” Lifetime Data Analysis 11 (2005)

[2] Joe, Harry. “Asymptotic efficiency of the two-stage estimation method for copula-based models.” Journal of Multivariate Analysis 94.2 (2005).

[3] Joe, Harry, and James Jianmeng Xu. 1996. “The Estimation Method of Inference Functions for Margins for Multivariate Models.” R. Faculty Research and Publications. October 31.

Reviewer comment (Novelty):
The paper's primary contribution seems to be an incremental advancement in efficiency over the TACTiS approach. More substantial evidence or arguments are needed to establish this as a significant contribution to the field.

Author response:
**W2: Incremental advancement in efficiency over TACTIS. More substantial evidence or arguments are needed to establish this as a significant contribution to the field.**

The reviewer is correct that our contributions result in an improvement in efficiency in comparison with TACTiS. However, the main outcome of this work is not merely a minor speedup in model training. Theoretically, the reduced parameter complexity (from factorial to linear) is a strong result in itself. Empirically, Figure 4 and App. A.3 show that TACTiS-2 converges using a fraction of the compute required by TACTiS to converge over a period of **three days**. This fact has significant practical implications (much less wasted compute, enabling more frequent retrainings, among others). Further, in addition to being more efficient, TACTiS-2 converges to much better solutions. This is reflected by improved test metrics across all tasks, which have been acknowledged by all reviewers. In conclusion, our work addresses the drawbacks (sGoi) and key limitations (s9Tw) of TACTiS, making significant progress toward unlocking the full potential of such architectures for such general-purpose modeling of time series distributions. Rightly as the reviewer points out in the strengths, such a contribution is bound to have a broad applicability across numerous fields.

Reviewer comment (Evaluation):
When evaluating the model's efficacy, the improvement in terms of Negative Log-Likelihood (NLL) is notable. However, the Mean Continuous Ranked Probability Score (CRPS) metric indicates that these improvements are only marginal when compared to the TACTiS model.

Author response:
**W3: The improvement in NLL is notable, but the mean CRPS indicates that the improvement is only marginal when compared to TACTIS.**

We agree with the reviewer that TACTiS-2 is only marginally better than TACTiS in terms of CRPS(-Sum), while still largely outperforming other baselines. This is not surprising given that both models share the same architecture for modeling marginal distributions. In fact, by definition, the CRPS only measures errors in marginal distributions. The CRPS-Sum was introduced as an alternative that accounts for multivariate dependencies and has since been established as a community standard. However, recent work emphasized that this metric is more sensitive to errors in marginal distributions than multivariate dependencies [1, 2]. Our contributions mostly affect the learning of multivariate dependencies (copula), which explains the reviewer’s observation.

The NLL is known to be much more sensitive to errors in multivariate dependencies [2] and thus, we use it to further compare the models. As expected, TACTiS-2 achieves much smaller NLLs, indicating that it significantly outperforms TACTiS in learning the copula.

[1] Koochali, A., Schichtel, P., Dengel, A., and Ahmed, S. Random noise vs. state-of-the-art probabilistic forecasting methods: A case study on CRPS-Sum discrimination ability. Applied Sciences, 12(10) :5104, 2022.

[2] Étienne Marcotte, Valentina Zantedeschi, Alexandre Drouin, and Nicolas Chapados. Regions of reliability in the evaluation of multivariate probabilistic forecasts. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 23958–24004. PMLR, 23–29 Jul 2023.