Paper: CODA: Temporal Domain Generalization via Concept Drift Simulator (ICLR-2024)

Reviewer comment (Experiments):
The proposed algorithm can only work on low-dimensional data (as the authors also mentioned). It is intractable to learn the correlation matrix on the high-dimensional data. I guess that's why some dataset such as rotating NIST has been excluded from evaluation.

Author response:
Based on Theorem 1 and our analysis in Section 3.4, we agree that feature correlation matrices may not effectively represent high-dimensional data. 

**However, it does not mean the proposed framework can only work on low-dimensional data**. We conducted our CODA on a high-dimensional dataset (Rotate-MNIST) mentioned by the reviewer for performance comparison. We use the same encoder structure as [1] (MNIST ConvNet) and train three architecture predictors. **The results reveal all three different predictors trained on the dataset generated by CODA outperform other baselines**. Furthermore, the differences among the three trained predictors support one of our contributions that the proposed model-agnostic CODA framework is flexible for best architecture exploration towards different datasets and downstream tasks.

We have added the experimental results in Appendix G, and the added section title is highlighted in blue.


|      Frameworks       |     Rot-MNIST     |
|:---------------------:|:-----------------:|
|       LSSAE[1]        |  16.6 $\pm$ 0.7   |
|        DDA[2]         |  13.8 $\pm$ 0.3   |
|         GI            |   7.7 $\pm$ 1.3   |
|       DRAIN           |   7.5 $\pm$ 1.1   |
|      CODA (MLP)       | **6.0 $\pm$ 1.2** |
|    CODA (LightGBM)    | **5.8 $\pm$ 0.6** |
| CODA (FT-Transformer) | **6.3 $\pm$ 0.5** |

[1] Tiexin Qin, et al. "Generalizing to evolving domains with latent structure-aware sequential autoencoder." ICML 2022. 

[2] Qiuhao Zeng, et al. "Foresee what you will learn: Data augmentation for domain generalization in non-stationary environment." AAAI 2023.

Reviewer comment (Experiments):
The proposed method achieves the SOTA accuracy (which is great and I am impressed). But, it seems non-intuitive to me since it requires separate steps to solve the final task. In deep learning, usually, end-to-end learning for the final tasks works better. Also I am not sure how efficient this method is since it requires 3 or 4 steps: (1) learning the correlation matrix; (2) learning the data simulator; (3) tuning the data simulator; (4) learning the final classifier/regressor.

Author response:
We agree that end-to-end approaches usually is ideal due to the convenient training process. At the same time, **we also believe that end-to-end approaches may not always be the best solution for tackling the root cause of a problem**. The reasons are as follows:
- The main motivation behind our approach is to **"Nip the problem in the bud"**. In other words, **the root cause of concept drift lies in the temporal evolution of data**. Our solution is to directly tackle this problem from a data perspective, i.e., achieve TDG by training a prediction model for future data generation.
- When end-to-end approaches may be overfitting due to their comprehensive interaction between data and model, one of our main motivations is to offer the flexibility of model architecture exploration for different datasets and downstream tasks by providing high-quality and effective training data.
- It is evident that the most effective model architecture can vary across different datasets and downstream tasks. This observation is supported by the results in Table 1, which shows that the architecture yielding the best performance differs among the evaluations on the five datasets.

**W2-2 & Q1: Efficiency of CODA.**
"Requires separate steps to solve the final task is inefficient."

[AW2-2]:
In this work, we mainly focus on the effectiveness of achieving TDG rather than on efficiency.

Although efficiency is not our main goal, we would like to justify that our proposed framework achieves decent efficiency compared to the SOTA method DRAIN. 

The reason is that by splitting the whole temporal trend modeling and data generation process into three sub-processes (learning Correlation Predictor $H(\cdot)$, learning Data Simulator $G(\cdot)$, and predictor training), **each of the sub-processes is a manageable sub-problem and takes less training time than a whole end-to-end model**. We demonstrate the training time comparison to the SOTA DRAIN on Elec2 dataset in the table below, where we train the same MLP structure as the predictor.

We have added the experimental results in Appendix H, and the added section title is highlighted in blue.

| Framework & Components | Training Time (s) |
| :----: | :----: |
| **DRAIN** | **465.936** |
| **CODA (Total)** | **447.817** |
| CODA (Correlation Predictor) | 142.110 |
| CODA (Data Simulator) | 290.826 |
| CODA (MLP) | 14.880 |

Reviewer comment (Experiments):
I have another idea which seems simpler for future data simulation. Why not train a conditional data generator which uses the time index (encoded with the Position Embedding modules) as the condition. This idea is also used in diffusion models when $\epsilon(x_t, t)$ is modelled.

Author response:
Unfortunately, this idea cannot be implemented based on the **lack of sample indices for instances at each time domain**. The explanations are as follows:
- One of the key challenges of capturing temporal trends among multiple time points is that we don't have time indices for each data instance, so we cannot treat each instance as sequential data for modeling its temporal evolution pattern, as the diffusion model does.
- An alternative way is to capture the underlying temporal trend among multiple datasets (distributions), which is computationally infeasible and hard to generate effective training data (details of the analysis refer to Section 3.1). Therefore, our solution is to simplify the data distribution at each time domain to capture the underlying temporal trend better. We utilize feature correlation matrices to achieve simplification and provide theoretical analysis to prove the rationale of representing data distribution with a feature correlation matrix (refer to Section 3.4).
- Notes that the baseline GI proposed a time-sensitive model to extrapolate samples to the near future via the first-order Taylor expansion, which is an implicit way to use the time index as conditions for prediction. As shown in Table 1, three different architectures trained on the data generated by CODA outperform GI in all benchmarks.

Reviewer comment (Theory):
The formulation only uses the last domain and not all the previous domains.

Author response:
In our proposed CODA framework, the trained Data Simulator $G(\cdot)$ should learn the similar data distribution of the current domain $\mathcal{D}\_{T}$. This is based on the assumption that distribution shifts are smooth and closely related to domains in the near time domains (refer to the assumption (iii) in Theorem 1).

Therefore, based on the current data distribution $\mathcal{D}\_{T}$, ${G}(\mathcal{D}\_{T} ; \mathcal{\hat{C}}\_{T+1} | \theta\_{G})$ can simulate the future data distribution $\mathcal{\hat{D}}\_{T+1}$ that is subject to the predicted correlation matrix $\mathcal{\hat{C}}\_{T+1}$.

Reviewer comment (Novelty):
Constrained application in model-agnostic learning scenarios.

Author response:
We are also not sure about the content of this weakness. The reviewer's point is that "CODA can be only leveraged to model-agnostic tasks." If our understanding is correct, we believe **this is also a misunderstanding**. The reasons are as follows:
- "Model-agnostic" is a feature of "approaches" rather than a feature of "downstream tasks."[1] This merit provides the flexibility of the predictor to explore the best suitable model architecture for the tasks or scenarios you meet.
- The datasets generated by CODA are adaptable for training different backbone architectures, which is demonstrated in Table 1. For a detailed analysis of this adaptability, refer to 'Cross-Architecture Transferability' in Section 4.3.

[1] Daochen, Zha, et al., "Data-centric artificial intelligence: A survey," arXiv:2303.10158

Reviewer comment (Experiments):
Potential performance decline in handling high-dimensional data sets.

Author response:
Based on Theorem 1 and our analysis in Section 3.4, we agree that feature correlation matrices may not effectively represent high-dimensional data.

**However, it does not mean the proposed CODA framework can only work on low-dimensional data**. We conducted our CODA on a high-dimensional dataset (Rotate-MNIST) for performance comparison, shown in the table below. We use the same encoder structure as [1] (MNIST ConvNet) and train three architecture predictors. The results reveal all three different predictors trained on the dataset generated by CODA outperform other baselines. Furthermore, the differences among the three trained predictors support one of our contributions that the proposed model-agnostic CODA framework is flexible for best architecture exploration towards different datasets and downstream tasks.

We have added the experimental results in Appendix G, and the added section title is highlighted in blue.

| Frameworks | Sine | Rot-MNIST |
| :----: | :----: | :----: |
| LSSAE[2] | 36.8 $\pm$ 1.5 | 16.6 $\pm$ 0.7 |
| DDA[3] | 1.6 $\pm$ 0.9 | 13.8 $\pm$ 0.3 |
| GI | 33.2 $\pm$ 0.7 | 7.7 $\pm$ 1.3 |
| DRAIN | 3.0 $\pm$ 1.0 | 7.5 $\pm$ 1.1 |
| CODA (MLP) | 2.7 $\pm$ 0.9 | **6.0 $\pm$ 1.2** |
| CODA (LightGBM) | **1.2 $\pm$ 0.4** | **5.8 $\pm$ 0.6** |
| CODA (FT-Transformer) | **1.1 $\pm$ 0.4** | **6.3 $\pm$ 0.5** |

[2] Tiexin Qin, et al. "Generalizing to evolving domains with latent structure-aware sequential autoencoder." ICML 2022.

[3] Qiuhao Zeng, et al. "Foresee what you will learn: Data augmentation for domain generalization in non-stationary environment." AAAI 2023.

Reviewer comment (Experiments):
Exploration of CODA's effectiveness in diverse concept drift scenarios is insufficient.

Author response:
We already considered diverse concept drift patterns as follows: 
- Synthetic concept drifts:
    1. Cyclical change: the 2-Moons dataset is built with a cyclical concept drift pattern, and we conduct **Rot-MNIST** and **Sine** datasets as shown in the table above.
    2. Abrupt change: this type of temporal trend doesn't fit our assumption that **"the joint distribution of features and labels with smooth data shift over time"** (refer to Introduction).
- Real-world concept drift: The real-world datasets used in our experiments feature various and unknown patterns of concept drift. They have covered diverse realistic temporal trends, such as electricity demand changes (Elec2), space shuttle defects (Shuttle), and appliance energy usage changes (Appliance).

Reviewer comment (Theory):
In the context of high-dimensional data, how does CODA maintain performance efficiency? Is there an analysis within the study that discusses the computational complexity implications as the number of features increases?

Author response:
Based on Theorem 1 and our analysis in Section 3.4, we agree that feature correlation matrices may not effectively represent high-dimensional data.

**However, it does not mean the proposed CODA framework can only work on low-dimensional data**. We conducted our CODA on a high-dimensional dataset (Rotate-MNIST) for performance comparison, shown in the table below. We use the same encoder structure as [1] (MNIST ConvNet) and train three architecture predictors. The results reveal all three different predictors trained on the dataset generated by CODA outperform other baselines. Furthermore, the differences among the three trained predictors support one of our contributions that the proposed model-agnostic CODA framework is flexible for best architecture exploration towards different datasets and downstream tasks.

We have added the experimental results in Appendix G, and the added section title is highlighted in blue.

| Frameworks | Sine | Rot-MNIST |
| :----: | :----: | :----: |
| LSSAE[2] | 36.8 $\pm$ 1.5 | 16.6 $\pm$ 0.7 |
| DDA[3] | 1.6 $\pm$ 0.9 | 13.8 $\pm$ 0.3 |
| GI | 33.2 $\pm$ 0.7 | 7.7 $\pm$ 1.3 |
| DRAIN | 3.0 $\pm$ 1.0 | 7.5 $\pm$ 1.1 |
| CODA (MLP) | 2.7 $\pm$ 0.9 | **6.0 $\pm$ 1.2** |
| CODA (LightGBM) | **1.2 $\pm$ 0.4** | **5.8 $\pm$ 0.6** |
| CODA (FT-Transformer) | **1.1 $\pm$ 0.4** | **6.3 $\pm$ 0.5** |

[2] Tiexin Qin, et al. "Generalizing to evolving domains with latent structure-aware sequential autoencoder." ICML 2022.

[3] Qiuhao Zeng, et al. "Foresee what you will learn: Data augmentation for domain generalization in non-stationary environment." AAAI 2023.

Reviewer comment (Experiments):
The generated data is then used to train models, which raises the question of whether it would be unfair for comparison methods. Should the comparison methods also use the same generated data to fine-tune?

Author response:
**Yes, the performance comparison is fair**. In our experiments, all the MLP models trained on the data generated by CODA are the same as one of the baselines DRAIN. Instead of designing predictor model structures, our approach focuses on the quality and efficacy of the generated training data.

**Other baselines cannot be trained on only one dataset at a single time domain.**
- CODA simulates the one domain ahead data for model training. In contrast, other baselines require all training domains for fine-tuning their whole models or frameworks and, therefore, cannot be trained using data from a single domain alone.

Reviewer comment (Theory):
I have concerns about the motivation of this paper. In particular, the authors have emphasized that existing TDG methods are model-centric, which are *unnecessarily comprehensive*, and therefore, TDG should be addressed via a data-centric approach. I doubt this point, as generating samples, in principle, is more challenging than discriminating them. With that said, I am not against the approach itself, but the paper presents it in a way that the data-centric itself is superior to model-centric, which I cannot agree with.

Author response:
We consider model-centric and data-centric approaches as parallel strategies, and our goal is not to position one approach against the other. Our main motivations are as follows:
1. **Nip the problem in the bud**
    - We believe the fundamental cause of concept drift is the underlying temporal trend of data distribution over time. Therefore, with the generated future data, TDG can be achieved by training a prediction model on an i.i.d. dataset.
    - The main motivation behind our approach is to "Nip the problem in the bud". In other words, **the root cause of concept drift lies in the temporal evolution of data**. Our solution is to directly tackle this problem from data perspective, i.e., achieve TDG by training a prediction model for future data generation.
2. **Flexibility and transferability for architecture-type exploration**
    - Furthermore, it is evident that the most effective model architecture can vary across different datasets and downstream tasks, such as MLPs, tree-based, and Transformer-based backbones. This observation is supported by the results in Table 1 of our paper, which shows that the architecture yielding the best performance differs among the evaluations on the five datasets.
    - However, existing model-centric methods are limited to specific model architectures. In contrast, our data-centric CODA framework offers flexibility for exploring various architectures by providing transferable training datasets. These datasets are adaptable for training different backbone architectures. For a detailed analysis of this adaptability, refer to 'Cross-Architecture Transferability' in Section 4.3.

Reviewer comment (Novelty):
In addition, generating the instances itself is challenging, but not necessary -- one may generate the feature samples in the representation space. After all, the ultimate goal is to train a predictor that generalizes well on the future domain rather than to generate the instances themselves.

Author response:
Although the ultimate goal is to train a predictor, as mentioned in the response for W1. The reason we generate instances for simulating the future data distribution is to pursue model-agnostic methods due to the efficacy of model architecture selection, such as Tree-based or Transformer-based models, varying across different datasets and downstream tasks, which is supported by the results in Table 1. Generating data in representation space can not guarantee model agnostic because the fixed encoder part is required for the efficacy of representation.

Reviewer comment (Novelty):
TDG has been studied recently. However, the authors only review two of them (DRAIN and GI), missing a few related works in the literature (e.g., [1, 2]). In fact, I think [2] also adopted a generative approach that *predicts* the feature domain. The authors should clarify the contributions and novelties given these works.

Author response:
Our main contribution and novelty is that we propose a data-centric (model-agnostic) TDG framework by using feature correlation matrices **to simplify the challenges of capturing the temporal trend**. The main challenges of capturing temporal trend among multiple time points is two-fold:
1. In most of the real-world benchmarks, **we don't have a sample index for each data instance at different time domains**, so we cannot treat each instance as a time series data for modeling its temporal evolution pattern. Therefore, it is impossible to predict the future features for each sequence. It is only durable to capture the trend of data distribution along time domains and generate future datasets/samples.
2. An alternative way is to capture the underlying temporal trend among multiple datasets (distributions) using the kernel data distribution estimation method, which is **computationally infeasible and hard to generate effective training data** (details of the analysis refer to Section 3.1).

**Novelties and contributions**
- For the two missed related works[1][2], we have added them to our references. Although they are also generative-based methods, both data augmentation in latent space[1] and the generation of augmented features within latent space[2] are **not model-agnostic** because different predictors, which are not trained with the same encoder, cannot identify the augmented embeddings or features.
- We also conducted our CODA on Rot-MNIST for performance comparison. We use the same encoder structure as [1] (MNIST ConvNet) and train three different architecture predictors. The results reveal all three different predictors trained on the dataset generated by CODA outperform [1, 2] and other baselines, which again proves the **effectiveness and transferability of CODA**.

We have added the experimental results in Appendix G, and the added section title is highlighted in blue.

| Frameworks | Sine | Rot-MNIST |
| :----: | :----: | :----: |
| LSSAE[1] | 36.8 $\pm$ 1.5 | 16.6 $\pm$ 0.7 |
| DDA[2] | 1.6 $\pm$ 0.9 | 13.8 $\pm$ 0.3 |
| GI | 33.2 $\pm$ 0.7 | 7.7 $\pm$ 1.3 |
| DRAIN | 3.0 $\pm$ 1.0 | 7.5 $\pm$ 1.1 |
| CODA (MLP) | 2.7 $\pm$ 0.9 | **6.0 $\pm$ 1.2** |
| CODA (LightGBM) | **1.2 $\pm$ 0.4** | **5.8 $\pm$ 0.6** |
| CODA (FT-Transformer) | **1.1 $\pm$ 0.4** | **6.3 $\pm$ 0.5** |

[1] Tiexin Qin, et al. "Generalizing to evolving domains with latent structure-aware sequential autoencoder." ICML 2022.
[2] Qiuhao Zeng, et al. "Foresee what you will learn: Data augmentation for domain generalization in non-stationary environment." AAAI 2023.

Reviewer comment (Theory):
Why modeling the correlation between two consecutive domains is not clear to me. After all, it only captures the second-order information of data.

Author response:
We would like to clarify this misunderstanding. We don't model the correlation between two consecutive time domains. We use an LSTM to model the temporal trend of **feature correlation matrices over all the time domains in the training datasets**. Eq.(3) aims to optimize the loss between the predicted $\mathcal{\hat{C}}\_t$ and the groudtruth $\mathcal{C}\_{t}$ given $\mathcal{C}\_{1}$ to $\mathcal{C}\_{t-1}$.
- As mentioned in the response to your W3, the two challenges of capturing the temporal trend from datasets are: 
    1. We don't have a sample index for each data instance at different time domains, so we cannot capture the temporal trend of each sample independently. Therefore, we need to capture the temporal trend among multiple data distributions.
    2. However, our preliminary experiments and analysis show the infeasibility of directly modeling the temporal evolution among data distributions (refer to Section 3.1).
- To this end, the core idea of our solution is **to simplify the data distribution at each time domain to capture the underlying temporal trend better**. In this work, we utilize feature correlation matrices to achieve simplification and provide theoretical analysis to prove the rationale of representing data distribution with a feature correlation matrix (refer to Section 3.4).
- We want to emphasize that while numerous methods exist to simplify dataset information, **our pioneering research introduces a natural and theoretically supported data-centric approach for this purpose**.

Reviewer comment (Reproducibility):
The form of the simulator $\mathcal{G}$ is not clearly defined in the paper. In particular, from Sec. 3.3, it is still not clear to me how the synthetic dataset $\hat{\mathcal{D}}_{T+1}$ is generated from Eq (4). In Appendix B.1 it only says that $\mathcal{G}$ is a generative model, but how the estimated correlation matrix and $\mathcal{D}_T$ are incorporated in the generation process is not clear to me.

Author response:
Based on the current data distribution $\mathcal{D}\_{T}$, Data Simulator ${G}(\mathcal{D}\_{T} ; \mathcal{\hat{C}}\_{T+1} | \theta\_{G})$ can simulate the future data distribution $\mathcal{\hat{D}}\_{T+1}$ that is subject to the predicted correlation matrix $\mathcal{\hat{C}}\_{T+1}$. We further explain the details as follows:

- Specifically, our CODA framework comprises two replaceable components: Correlation Predictor ${H}(\cdot)$ (refers to Section 3.2) and Data Simulator $G(\cdot)$ (refers to Section 3.3). Essentially, they can be substituted by other models that perform similar functions, where $G(\cdot)$ should be a generative model that can incorporate prior knowledge into account for data generation. In our case, the prior knowledge is the predicted future correlation matrix $\mathcal{\hat{C}}\_{T+1}$, as described in Eq.(4) and Eq.(5).

- Simultaneously, the trained $G(\cdot)$ should learn the similar data distribution of the current domain $\mathcal{D}\_{T}$. This is based on the assumption that distribution shifts are smooth and closely related to domains in the near time domains (refer to the assumption (iii) in Theorem 1).

- In our experiments, we instantiate $G(\cdot)$ with a generative model that jointly learns the encoder and decoder of a VAE-based generative model and a learnable graph. Thus, it can **treat prior knowledge as an adjacency matrix and encourage the learned graph to be similar to the given prior knowledge (refer to Sections 3.3 and 4.1)**.

Therefore, based on the current data distribution $\mathcal{D}\_{T}$, ${G}(\mathcal{D}\_{T} ; \mathcal{\hat{C}}\_{T+1} | \theta\_{G})$ can simulate the future data distribution $\mathcal{\hat{D}}\_{T+1}$ that is subject to the predicted correlation matrix $\mathcal{\hat{C}}\_{T+1}$.

Reviewer comment (Experiments):
The empirical analysis is weak. In addition to the baseline algorithms mentioned above. Several commonly used benchmark data sets are also missing, including both synthetic (e.g., Circle, Sine) and real (e.g., RMNIST, Portraits, Ocular, Caltran, WILDS) data sets.

Author response:
We have considered diverse concept drift patterns as follows show: 
- Synthetic datasets: Besides the 2-Moons, we experimented on Sine dataset shown in the table below.
- Real datasets: Besides the real-world datasets used in our experiments (Elec2, ONT, Shuttle, and Appliance), we conducted one more experiment on Rot-MNIST, using the same encoder structure as [1] (MNIST ConvNet) and train three architecture predictors, and the results are shown as the table below. The results reveal all three different predictors trained on the dataset generated by CODA outperform other baselines. Furthermore, the differences among the three trained predictors support one of our contributions that the proposed model-agnostic CODA framework is flexible for best architecture exploration towards different datasets and downstream tasks.

We have added the experimental results in Appendix G, and the added section title is highlighted in blue.

| Frameworks | Sine | Rot-MNIST |
| :----: | :----: | :----: |
| LSSAE[1] | 36.8 $\pm$ 1.5 | 16.6 $\pm$ 0.7 |
| DDA[2] | 1.6 $\pm$ 0.9 | 13.8 $\pm$ 0.3 |
| GI | 33.2 $\pm$ 0.7 | 7.7 $\pm$ 1.3 |
| DRAIN | 3.0 $\pm$ 1.0 | 7.5 $\pm$ 1.1 |
| CODA (MLP) | 2.7 $\pm$ 0.9 | **6.0 $\pm$ 1.2** |
| CODA (LightGBM) | **1.2 $\pm$ 0.4** | **5.8 $\pm$ 0.6** |
| CODA (FT-Transformer) | **1.1 $\pm$ 0.4** | **6.3 $\pm$ 0.5** |

Reviewer comment (Theory):
The definition of correlation matrices is not clear to me. Do they include the label information? If yes, how? If not, how to incorporate the label information to generate discriminative features? Also, what is the dimension of the matrices? If it is high, how to guarantee the consistency of the estimation?

Author response:
**Yes, it includes label information**. In feature correlation matrices, each row and column corresponds to a specific feature. The final row and column represent label information. Each cell within the matrix indicates the degree of correlation between a pair of features.

Furthermore, **Section 3.4** presents a theoretical analysis that guarantees the consistency of our feature correlation estimation with **three assumptions that can be easily satisfied in reality**.

Reviewer comment (Theory):
In Eq (3), how the cross entropy between two matrices is defined? Why both $\ell_1$ and $\ell_2$ norm regularizations are imposed?

Author response:
The three regularization terms in Eq.(3) are explained as follows:
1. The $\ell_1$-norm encourages sparsity in the predicted $\mathcal{\hat{C}}\_t$ because it can effectively "zero out" less important features while the correlation matrices are generally sparse (as shown in Appendix Figure 9).
2. The $\ell_2$-norm is sensitive to significant errors and imposes a penalty on $\mathcal{\hat{C}}_t$ for substantial errors, promoting overall accuracy in the reconstruction.
3. The cross-entropy loss $\mathcal{L}\_{CE}$ can measure how well the distribution of $\mathcal{\hat{C}}\_t$ matches the groundtruth distribution of $\mathcal{C}\_t$.

Despite the errors between predicted future $\mathcal{\hat{C}}\_{t+1}$ and the ground truth $\mathcal{C}\_{t+1}$ is minimal (as shown in Figure 10 in the Appendix), there is huge potential to enhance the Correlation Predictor. As one of the future directions, it could be achieved using a more sophisticated sequential prediction framework than LSTM.

Reviewer comment (Theory):
I cannot see the connection between Theorem 1 and the proposed method (e.g., Eq (5)). From my understanding, Theorem 1 states that for two random vectors, if they are bounded and their distributions are close, then the difference between their correlation matrices are also bounded. But how this is related to the algorithm? In Eq (5), the distance is already constrained by the regularization term.

Author response:
Theorem 1 serves as the theoretical foundation for the usage of prior knowledge (predicted feature correlation matrix $\mathcal{\hat{C}}\_{T+1}$) in Data Simulator.
- As mentioned in our response to W3 and W4, one of the main challenges of capturing the temporal trend among multiple time domains is computationally infeasible (refer to Section 3.1). Our framework addresses this by representing the data distribution at each time domain by its feature correlation matrix to capture the temporal trend effectively. This simplification can effectively represent the original distribution information only if Theorem 1 holds.
- Based on the analysis in Section 3.4, we conclude that the three assumptions in Theorem 1 can be easily satisfied in reality, which serves as the theoretical foundation for the simplification.