Paper: Long-Term Typhoon Trajectory Prediction: A Physics-Conditioned Approach Without Reanalysis Data (ICLR-2024)

Reviewer comment (Evaluation):
Since real-time computation is the goal, it would be important to give computation times values.

Author response:
All deep learning methodologies can infer within 3 seconds with A100 single gpu. Indeed, additional time is required to obtain input data; however, compared to other methods, it is sufficiently close to real-time. Other models using ERA5 have more than 3 days of data acquisition delays, while our model has only a 3-hour delay. Therefore, it takes about 3 hours to calculate the typhoon's trajectory 72 hours ahead.

Reviewer comment (Reproducibility):
It is not clear how the 'probability cones' are obtained: stochasticity is mentioned only in the result part, not in the method.

Author response:
The probability cones are obtained through kernel density estimation (KDE). We have added this information to Figure 3.

Reviewer comment (Experiments):
I understand that the number of data is limited, but I would like to know if a validation set was used to fine-tune the hyper-parameters or if it was done using the 2019-2021 years. Please explain better.

Author response:
We have included the following content in the main text to address this: The training data spans the years 1950-2018 and includes 1,334 typhoons, while the test dataset covers the years 2019-2021 and comprises 90 typhoons.  Note that for hyperparameter tuning, we train on data from 1950 to 2016 and use the data from 2017 to 2018 as the validation set.

Reviewer comment (Presentation):
It would be interesting to see one of the 'worst' cases also in Figure 3, with a comment on it.

Author response:
We have added an analysis of a failure case in Appendix A.4.

Reviewer comment (Writing):
Many typos are present, see below.

Author response:
Thank you very much. We have corrected all typos and revised the sentences with a native speaker.

Reviewer comment (Experiments):
In your state-of-the art comparison, which one uses UN, which uses only trajectory information?

Author response:
Only LT3P uses the UM data. To the best of our knowledge, LT3P is the first work that combines real-time NWP data with ERA5 data. For the use of data type, we add the clarifications for all comparison methods in Table 2.

Reviewer comment (Evaluation):
Could you comment a bit more on the Figure 4? It looks like the bias-correction is actually losing the details, just fitting better to the distribution of the ERA5. Maybe there is more to see? The values (SSIM, mean) are not very convincing.

Author response:
Based on your comment, we need to provide better analysis by focusing on typhoon trajectory in the time axis. We thus replace the SSIM with the temporal correlation coefficient (TCC), which measures the Pearson correlation coefficient of motion patterns between a predicted and ground-truth trajectory. Detailed analysis can be found in Figure 9&10 and the description of Appendix A.5 section.

Reviewer comment (Presentation):
Figure 1: What is the scale of the image? How many km (or give the lat/lon) in both directions?

Author response:
We have added a description on this in the caption of Figure 1. The area covers latitudes from 0 to 59.75°N and longitudes from 100°E to 179.75°E, whose distance is about 59.75 degrees×111 km/degree≈6632.25 km and (179.75−100) degrees×111 km/degree≈8842.75 km, respectively.

Reviewer comment (Reproducibility):
If I read correctly, several explanations about network architecture, losses, etc. are missing. This would prevent the re-production by fellow researchers.

Author response:
We have utilized three different baselines: GAN, CVAE, and Diffusion. Due to the extensive nature of the contents, we have added detailed explanations in Appendix A.1.

Reviewer comment (Evaluation):
The manuscript says that one needs to wait for three hours to obtain the UM reanalysis data. This means the proposed method loses the lead time of three hours, so the score of "6-hour leadtime" should be understood as "3-hour leadtime"?

Author response:
Considering a real-world, real-time inference scenario, UM data has a 3-hour delay at data acquisition, the first lead time is +3 hours, and the rest are at +6-hour intervals.  For example, if we assume that we request the UM prediction dataset of 00:00, we can access the data at 03:00. This data represents intervals of UM = {00, +06, +12, +18 … +72}, and though it can be accessed 3 hours later, all the data are at 6-hour intervals. Therefore, while there is a 3-hour delay for inference, from the dataset's perspective, the intervals are consistently every 6 hours

Reviewer comment (Experiments):
Dependence on Real-Time NWP Dataset: The model's performance is dependent on the availability and accuracy of the real-time NWP dataset, which could be a potential limitation.

Author response:
Our model focuses on leveraging the advantages of real-time NWP datasets, and therefore, it does not function properly in situations where NWP datasets are entirely unavailable. As the reviewer point out, this can be seen as a potential limitation of our study. However, our model requires only three basic types of meteorological variables, which in most cases can be easily obtained from freely available outputs of NWP models. A prime example is the use of the publicly available dataset from ECMWF's IFS forecasts. Known for their high accuracy, these IFS forecasts are a reliable source. To account for the practical scenario of using NWP model outputs with varying accuracies, we opt for the UM forecasts, known to have a larger bias. This decision is made to demonstrate that our model can yield impressive results even with NWP datasets of somewhat lower accuracy.

Reviewer comment (Experiments):
Need for Future Work: While the paper outlines plans for future work, including extending the application to all kinds of tropical cyclones, these aspects have not yet been addressed or tested.

Author response:
In our future work, we will validate LT3P on all types of tropical cyclones occurring globally. We have added future work in Appendix A.7.

Reviewer comment (Experiments):
How well does the LT3P model generalize to different regions and conditions of typhoon occurrences? Have there been any specific challenges in adapting the model to various geographical locations?

Author response:
We have applied our LT3P model to tropical cyclones occurring in different regions. Appendix A.3 shows the results of applying LT3P to different areas. In conclusion, LT3P does not suffer significant performance degradation because it uses the UM prediction, a global model without a domain gap. However, additional training appears necessary for optimal performance.

Reviewer comment (Evaluation):
How does the dependency on real-time NWP datasets affect the model’s performance, especially in scenarios where real-time data might be sparse or inaccurate?

Author response:
Our current work essentially requires NWP datasets. Therefore, it cannot be utilized when access to NWP data is not possible. However, since we have chosen UM among many NWPs, it is expected to perform well with other higher-performing NWPs like IFS and GFS.

Reviewer comment (Evaluation):
How does the LT3P model handle uncertainties in typhoon trajectory prediction, and what measures are in place to ensure the reliability of its predictions?

Author response:
Currently, we minimize the uncertainties through multi-modal prediction which produces a set of outputs from one observation trajectory. The multi-modal trajectories are averaged for better probabilistic forecasting of one dominant path, which is conventionally used in meteorology. As another aspect, we can represent the uncertainty, which is visualized in Figure 3. As the probability map of typhoons, we show the coverage of potential pathways using kernel density estimation. And, we do not have measures for the reliability of the predictions. We believe that there is an optimal way to select one dominant path from them, which has been described in our future work. Please refer to Appendix A.7.

Reviewer comment (Reproducibility):
The method seems to have been tested using a GAN, a CVAE, and a diffusion model, but implementation details are largely lacking.

Author response:
We thank the reviewer for the insightful feedback concerning clarity and details. Your comment makes our paper polish up. Based on comments, we have revised it, marked in orange. Below, we address each of your questions with corresponding answers.

**[W2 - Detail of implementation]** Unclear meaning and implementation details of "so it is matched through the latitude and longitude mapping and the interpolation method.

The original meaning of the sentence is that we just utilize a bilinear interpolation to adjust a resolution of UM data and ERA5. Instead of the expression that the reviewer point out, we have changed it to “Note that the resolution has been adjusted using bi-linear interpolation”. Please check it on page 7 of the main text.

Reviewer comment (Reproducibility):
Unclear meaning and implementation details of *"so it is matched through the latitude and longitude mapping and the interpolation method"*.

Author response:
We thank the reviewer for the insightful feedback concerning clarity and details. Your comment makes our paper polish up. Based on comments, we have revised it, marked in orange. Below, we address each of your questions with corresponding answers.

**[W3 - Definition of ‘ensemble method’]** Unclear what exactly the "ensemble method" refers to and how exactly it is performed in practice for all the models.

We have employed the ensemble average method for comparison, similar to the approach used by meteorological offices. Since each inference from the GAN, CVAE, and Diffusion models produces different outputs due to random noise, we average the generated 20 trajectory samples from our model for better probabilistic forecasting. On page 7 of the main text, this method is clearly referred to as the ‘ensemble average '.

Reviewer comment (Reproducibility):
Unclear what exactly the *"ensemble method"* refers to and how exactly it is performed in practice for all the models.

Author response:
We thank the reviewer for the insightful feedback concerning clarity and details. Your comment makes our paper polish up. Based on comments, we have revised it, marked in orange. Below, we address each of your questions with corresponding answers.

**[W4 - More experiments and details on ablation study].** Please include more ablations, e.g. with UM and joint training and bias-correction but without pre-training. The whole ablation study section is a bit hard to understand and should be expanded (in the appendix if needed).

Unfortunately, due to the inherent characteristics of our model structure, conducting bias-correction independently without the pre-training phase is not feasible.
The ERA5 dataset has been accumulating data from 1950 to the present, whereas the UM dataset has been available since 2010. Therefore, if only the UM dataset is used, the best track typhoon dataset is also only available from 2010. The number of typhoons since 2010 is approximately 200, which is a small fraction compared to the total of 1,424 typhoons that have occurred since 1950. Consequently, the performance of UM-only training is significantly limited. In contrast, a joint training setting, which uses ERA5 data from 1950 to 2009 and the UM dataset from 2010 to 2021, can utilize the entire typhoon dataset.This dataset is about six times larger than UM-only dataset, which seems to significantly improve the performance of data-driven models due to the increased amount of data
Lastly, we have added more content on the analysis of ablation studies in Appendix A.6 section.

Reviewer comment (Experiments):
Please include more ablations, e.g. with UM and joint training and bias-correction but without pre-training. The whole ablation study section is a bit hard to understand and should be expanded (in the appendix if needed).

Author response:
We thank the reviewer for the insightful feedback concerning clarity and details. Your comment makes our paper polish up. Based on comments, we have revised it, marked in orange. Below, we address each of your questions with corresponding answers.

**[W5 - Implementation details].** Are the baselines in the first set of rows of Table 2 (e.g SocialGAN) all off-the-shelf pre-trained human trajectory prediction models? If so, it is not surprising at all that they perform so badly, and it would be good to retrain them on your data.

We note all baselines in Table 2 are trained and evaluated using the best track dataset for strictly fair comparison. To avoid misunderstanding, we have specified the training dataset of the baselines on page 7.

Reviewer comment (Experiments):
Are the baselines in the first set of rows of Table 2 (e.g SocialGAN) all off-the-shelf pre-trained human trajectory prediction models? If so, it is not surprising at all that they perform so badly, and it would be good to retrain them on your data.

Author response:
We thank the reviewer for the insightful feedback concerning clarity and details. Your comment makes our paper polish up. Based on comments, we have revised it, marked in orange. Below, we address each of your questions with corresponding answers.

**[W5 - Implementation details].** Are the baselines in the first set of rows of Table 2 (e.g SocialGAN) all off-the-shelf pre-trained human trajectory prediction models? If so, it is not surprising at all that they perform so badly, and it would be good to retrain them on your data.

We note all baselines in Table 2 are trained and evaluated using the best track dataset for strictly fair comparison. To avoid misunderstanding, we have specified the training dataset of the baselines on page 7.

Reviewer comment (Experiments):
Why is Ruttgers et al. not included in the benchmarking?

Author response:
We thank the reviewer for the insightful feedback concerning clarity and details. Your comment makes our paper polish up. Based on comments, we have revised it, marked in orange. Below, we address each of your questions with corresponding answers.

**[W5 - Selection of comparison models].** Additionally, why is Ruttgers et al. not included in the benchmarking?

We did not include Ruttgers et al.'s work in our benchmarking efforts due to differences in both the training dataset and methodology. Ruttgers et al. track typhoon paths by reconstructing them after marking the typhoon center coordinates on satellite images. In contrast, we do not use satellite images. 

However, it would be valuable to conduct an experiment based on their work. We have now requested the authors for their codes and datasets. Should these sources become available, we will carry out an additional comparison with Ruttgers et al.'s work, and include the result in our paper.

Reviewer comment (Writing):
First sentence of methods section: 1) There is no index *i* for none of the variables (e.g. it should be $c_i$, I think); 2) $t_p$ is the input sequence **length** (not the input sequence). Similarly for $t_f$; 3) $p$ is given different meanings for $C_p$ and the pressure levels $p \in P$. This is confusing, can you use different letters, please?

Author response:
**[Q1 - Notation Issues].** Confusing notation.

Thank you for pointing out our mistakes. We have specified the index 'i' on page 4 and corrected the duplicated notation 'p.' To avoid confusion, we have changed all the small ‘p’ to ‘o’ which represents observations in descriptions and figures. “”

Reviewer comment (Writing):
Why are geopotential height and wind vectors inputs taken from the output timesteps $i=t_p+1, \dots, t_p+t_f$? Should it not be $i=1, \dots, t_p$?

Author response:
**[Q2 - Timestamps of UM and ERA5].** Why are geopotential height and wind vectors inputs taken from the output timesteps.

The output time steps represent the intervals at which we aim to predict the typhoon paths, and leveraging atmospheric information for these time frames is crucial for accurate prediction. This goal is achieved by utilizing the Numerical Weather Prediction (NWP) forecast fields, especially the UM dataset. 
We use physics-based predicted values from UM as inputs, offering a significant advantage over using ERA5. Unlike ERA5, which not only experiences delays but also lacks access to future data, the UM forecast incorporates future atmospheric conditions through physics-based modeling. This makes it a more suitable choice for our input values.

Reviewer comment (Writing):
3.1 Preliminaries (not preliminarily).

Author response:
**[Q3 - Grammar error]** . 3.1 Preliminaries (not preliminarily)

As the reviewer pointed out, we have corrected the typo. Do you think any further modifications are needed for the content of this section? We believe explaining how our dataset is derived from the NWP model is essential. Therefore, we have composed this section to convey that information.

Reviewer comment (Writing):
Tone down "is too large scale to train a model from scratch". There exist models trained from scratch on ERA5.

Author response:
**[Q4 - Tone].** Tone down "is too large scale to train a model from scratch". There exist models trained from scratch on ERA5…

We have toned down the sentence as “The ERA5 dataset, accumulated at 6-hour intervals from 1950 to the present, requires a significant amount of time to train a model from scratch. To address this challenge, we have adopted a strategy of constructing a foundational model for typhoon trajectory prediction”. This is reflected in Section 3.2 on page 5.

Reviewer comment (Writing):
"$\tilde X$ is the forecasting values for ERA5 from time $t$ to $t+t_f$". Shouldn't it be $t+1$ to $t+t_f$?

Author response:
**[Q5. - Timestamp of bias-correction output]** Timestamp of \tilde{X} 

The output time frame of \( \tilde{x} \) is from \( t_{o}+1 \) to \( t_{o} + t_{f} \).

Reviewer comment (Reproducibility):
I don't completely understand how the bias correction phase works, is the UM data matched with ERA5 based on the timestamp?

Author response:
**[Q6 - Timestamp of bias-correction output].** I don't completely understand how the bias correction phase works, is the UM data matched with ERA5 based on the timestamp?

Yes, you are correct. In our bias correction phase, we train the model on ERA5 data for the period from 1950 to 2010, ensuring it outputs ERA5 data. Simultaneously, for the period from 2010 to 2021, UM data is used as an input, and the bias correction guides the model to produce ERA5-like output from UM data.