Original: Earthformer: Exploring Space-Time Transformers for Earth System Forecasting
Revised: Earthformer: Exploring Space-Time Transformers for Earth System Forecasting

Original: Abstract
Revised: Abstract

Original: Conventionally, Earth system (e.g., weather and climate) forecasting relies on numerical simulation with complex physical models and hence is both expensive in computation and demanding on domain expertise.
Revised: Conventionally, Earth system (e.g., weather and climate) forecasting relies on numerical simulation with complex physical models and hence is both expensive in computation and demanding on domain expertise.

Original: With the explosive growth of spatiotemporal Earth observation data in the past decade, data-driven models that apply Deep Learning (DL) are demonstrating impressive potential for various Earth system forecasting tasks.
Revised: With the explosive growth of spatiotemporal Earth observation data in the past decade, data-driven models that apply Deep Learning (DL) are demonstrating impressive potential for various Earth system forecasting tasks.

Original: The Transformer as an emerging DL architecture, despite its broad success in other domains, has limited adoption in this area.
Revised: The Transformer as an emerging DL architecture, despite its broad success in other domains, has limited adoption in this area.

Original: In this paper, we propose Earthformer , a space-time Transformer for Earth system forecasting.
Revised: In this paper, we propose Earthformer , a space-time Transformer for Earth system forecasting.

Original: Earthformer is based on a generic, flexible and efficient space-time attention block, named Cuboid Attention .
Revised: Earthformer is based on a generic, flexible and efficient space-time attention block, named Cuboid Attention .

Original: The idea is to decompose the data into cuboids and apply cuboid-level self-attention in parallel.
Revised: The idea is to decompose the data into cuboids and apply cuboid-level self-attention in parallel.

Original: These cuboids are further connected with a collection of global vectors.
Revised: These cuboids are further connected with a collection of global vectors.

Original: We conduct experiments on the MovingMNIST dataset and a newly proposed chaotic N -body MNIST dataset to verify the effectiveness of cuboid attention and figure out the best design of Earthformer.
Revised: We conduct experiments on the MovingMNIST dataset and a newly proposed chaotic N -body MNIST dataset to verify the effectiveness of cuboid attention and figure out the best design of Earthformer.

Original: Experiments on two real-world benchmarks about precipitation nowcasting and El Niño/Southern
Revised: Experiments on two real-world benchmarks about precipitation nowcasting and El Niño/Southern

Original: Oscillation (ENSO) forecasting show that Earthformer achieves state-of-the-art performance.
Revised: Oscillation (ENSO) forecasting show that Earthformer achieves state-of-the-art performance.

Original:  Introduction
Revised:  Introduction

Original: The Earth is a complex system.
Revised: The Earth is a complex system.

Original: Variabilities of the Earth system, ranging from regular events like temperature fluctuation to extreme events like drought, hail storm, and El Niño/Southern Oscillation (ENSO), impact our daily life.
Revised: Variabilities of the Earth system, ranging from regular events like temperature fluctuation to extreme events like drought, hail storm, and El Niño/Southern Oscillation (ENSO), impact our daily life.

Original: Among all the consequences, Earth system variabilities can influence crop yields, delay airlines, cause floods and forest fires.
Revised: Among all the consequences, Earth system variabilities can influence crop yields, delay airlines, cause floods and forest fires.

Original: Precise and timely forecasting of these variabilities can help people take necessary precautions to avoid crisis, or better utilize natural resources such as wind and solar energy.
Revised: Precise and timely forecasting of these variabilities can help people take necessary precautions to avoid crisis, or better utilize natural resources such as wind and solar energy.

Original: Thus, improving forecasting models for Earth variabilities (e.g., weather and climate) has a huge socioeconomic impact.
Revised: Thus, improving forecasting models for Earth variabilities (e.g., weather and climate) has a huge socioeconomic impact.

Original: Despite its importance, the operational weather and climate forecasting systems have not fundamentally changed for almost 50 years [34].
Revised: Despite its importance, the operational weather and climate forecasting systems have not fundamentally changed for almost 50 years [34].

Original: These operational models, including the state-of-the-art High Resolution Ensemble Forecast (HREF)
Revised: These operational models, including the state-of-the-art High Resolution Ensemble Forecast (HREF)

Original: [Figure]
Revised: [Figure]

Original: rainfall nowcasting model used in National Oceanic and Atmospheric Administration (NOAA) [32], rely on meticulous numerical simulation of physical models.
Revised: rainfall nowcasting model used in National Oceanic and Atmospheric Administration (NOAA) [32], rely on meticulous numerical simulation of physical models.

Original: Such simulation-based systems inevitably fall short in the ability to incorporate signals from newly emerging geophysical observation systems [12], or take advantage of the Petabytes-scale Earth observation data [43].
Revised: Such simulation-based systems inevitably fall short in the ability to incorporate signals from newly emerging geophysical observation systems [12], or take advantage of the Petabytes-scale Earth observation data [43].

Original: As an appealing alternative, deep learning (DL) is offering a new approach for Earth system forecasting [34].
Revised: As an appealing alternative, deep learning (DL) is offering a new approach for Earth system forecasting [34].

Original: Instead of explicitly incorporating physical rules, DL-based forecasting models are trained on the Earth observation data [36].
Revised: Instead of explicitly incorporating physical rules, DL-based forecasting models are trained on the Earth observation data [36].

Original: By learning from a large amount of observations, DL models are able to figure out the system’s intrinsic physical rules and generate predictions that outperform simulation-based models [9].
Revised: By learning from a large amount of observations, DL models are able to figure out the system’s intrinsic physical rules and generate predictions that outperform simulation-based models [9].

Original: Such technique has demonstrated success in several applications, including precipitation nowcasting [32, 6] and ENSO forecasting [15].
Revised: Such technique has demonstrated success in several applications, including precipitation nowcasting [32, 6] and ENSO forecasting [15].

Original: Because the Earth system is chaotic [21], high-dimensional, and spatiotemporal, designing appropriate DL architecture for modeling the system is particularly challenging.
Revised: Because the Earth system is chaotic [21], high-dimensional, and spatiotemporal, designing appropriate DL architecture for modeling the system is particularly challenging.

Original: Previous works relied on the combination of Recurrent
Revised: Previous works relied on the combination of Recurrent

Original: Neural Networks (RNN) and Convolutional Neural Networks (CNN) [36, 37, 43, 13, 45].
Revised: Neural Networks (RNN) and Convolutional Neural Networks (CNN) [36, 37, 43, 13, 45].

Original: These two architectures impose temporal and spatial inductive biases that help capturing spatiotemporal patterns.
Revised: These two architectures impose temporal and spatial inductive biases that help capturing spatiotemporal patterns.

Original: However, as a chaotic system, variabilities of the Earth system, such as rainfall and ENSO, are highly sensitive to the system’s initial conditions and can respond abruptly to internal changes.
Revised: However, as a chaotic system, variabilities of the Earth system, such as rainfall and ENSO, are highly sensitive to the system’s initial conditions and can respond abruptly to internal changes.

Original: It is unclear whether the inductive biases in RNN and CNN models still hold for such complex systems.
Revised: It is unclear whether the inductive biases in RNN and CNN models still hold for such complex systems.

Original: On the other hand, recent years have witnessed major breakthroughs in DL brought by the wide adoption of Transformer.
Revised: On the other hand, recent years have witnessed major breakthroughs in DL brought by the wide adoption of Transformer.

Original: The model was originally proposed for natural language processing [42, 7], and later has been extended to computer vision [8, 22], multimodal text-image generation [31], graph learning [52], etc.
Revised: The model was originally proposed for natural language processing [42, 7], and later has been extended to computer vision [8, 22], multimodal text-image generation [31], graph learning [52], etc.

Original: Transformer relies on the attention mechanism to capture data correlations and is powerful at modeling complex and long-range dependencies, both of which appear in Earth systems (See Fig.
Revised: Transformer relies on the attention mechanism to capture data correlations and is powerful at modeling complex and long-range dependencies, both of which appear in Earth systems (See Fig.

Original: 1 for an example of Earth observation data).
Revised: 1 for an example of Earth observation data).

Original: Despite being suitable for the problem,
Revised: Despite being suitable for the problem,

Original: Transformer sees limited adoption for Earth system forecasting.
Revised: Transformer sees limited adoption for Earth system forecasting.

Original: Naively applying the Transformer architecture is infeasible because the O ( N 2 ) attention mechanism is too computationally expensive for the high-dimensional Earth observation data.
Revised: Naively applying the Transformer architecture is infeasible because the O ( N 2 ) attention mechanism is too computationally expensive for the high-dimensional Earth observation data.

Original: How to design a space-time Transformer that is good at predicting the future of the Earth systems is largely an open problem to the community.
Revised: How to design a space-time Transformer that is good at predicting the future of the Earth systems is largely an open problem to the community.

Original: In this paper, we propose Earthformer , a space-time Transformer for Earth system forecasting.
Revised: In this paper, we propose Earthformer , a space-time Transformer for Earth system forecasting.

Original: To better explore the design of space-time attention, we propose Cuboid Attention , which is a generic building block for efficient space-time attention.
Revised: To better explore the design of space-time attention, we propose Cuboid Attention , which is a generic building block for efficient space-time attention.

Original: The idea is to decompose the input tensor to non-overlapping cuboids and apply cuboid-level self-attention in parallel.
Revised: The idea is to decompose the input tensor to non-overlapping cuboids and apply cuboid-level self-attention in parallel.

Original: Since we limit the O ( N 2 ) self-attention inside the local cuboids, the overall complexity is greatly reduced.
Revised: Since we limit the O ( N 2 ) self-attention inside the local cuboids, the overall complexity is greatly reduced.

Original: Different types of correlations can be captured via different cuboid decompositions.
Revised: Different types of correlations can be captured via different cuboid decompositions.

Original: By stacking multiple cuboid attention layers with different hyperparameters, we are able to subsume several previously proposed video Transformers [19, 23, 4] as special cases, and also come up with new attention patterns that were not studied before.
Revised: By stacking multiple cuboid attention layers with different hyperparameters, we are able to subsume several previously proposed video Transformers [19, 23, 4] as special cases, and also come up with new attention patterns that were not studied before.

Original: A limitation of this design is the lack of a mechanism for the local cuboids to communicate with each other.
Revised: A limitation of this design is the lack of a mechanism for the local cuboids to communicate with each other.

Original: Thus, we introduce a collection of global vectors that attend to all the local cuboids, thereby gathering the overall status of the system.
Revised: Thus, we introduce a collection of global vectors that attend to all the local cuboids, thereby gathering the overall status of the system.

Original: By attending to the global vectors, the local cuboids can grasp the general dynamics of the system and share information with each other.
Revised: By attending to the global vectors, the local cuboids can grasp the general dynamics of the system and share information with each other.

Original: To verify the effectiveness of cuboid attention and figure out the best design under the Earth system forecasting scenario, we conducted extensive experiments on two synthetic datasets: the MovingMNIST [36] dataset and a newly proposed N -body MNIST dataset.
Revised: To verify the effectiveness of cuboid attention and figure out the best design under the Earth system forecasting scenario, we conducted extensive experiments on two synthetic datasets: the MovingMNIST [36] dataset and a newly proposed N -body MNIST dataset.

Original: Digits in the N -body MNIST follow the chaotic 3 -body motion pattern [25], which makes the dataset not only more challenging than MovingMNIST but also more relevant to Earth system forecasting.
Revised: Digits in the N -body MNIST follow the chaotic 3 -body motion pattern [25], which makes the dataset not only more challenging than MovingMNIST but also more relevant to Earth system forecasting.

Original: The synthetic experiments reveal the following findings: 1) stacking cuboid attention layers with the Axial attention pattern is both efficient and effective, achieving the best overall performance, 2) adding global vectors provides consistent performance gain without increasing the computational cost, 3) adding hierarchy in the encoder-decoder architecture can improve performance.
Revised: The synthetic experiments reveal the following findings: 1) stacking cuboid attention layers with the Axial attention pattern is both efficient and effective, achieving the best overall performance, 2) adding global vectors provides consistent performance gain without increasing the computational cost, 3) adding hierarchy in the encoder-decoder architecture can improve performance.

Original: Based on these findings, we figured out the optimal design for Earthformer and made comparisons with other baselines on the SEVIR [43] benchmark for precipitation nowcasting and the ICAR-ENSO dataset [15] for ENSO forecasting.
Revised: Based on these findings, we figured out the optimal design for Earthformer and made comparisons with other baselines on the SEVIR [43] benchmark for precipitation nowcasting and the ICAR-ENSO dataset [15] for ENSO forecasting.

Original: Experiments show that Earthformer achieves state-of-the-art (SOTA) performance on both tasks.
Revised: Experiments show that Earthformer achieves state-of-the-art (SOTA) performance on both tasks.

Original:  Related Work
Revised:  Related Work

Original: Deep learning architectures for Earth system forecasting.
Revised: Deep learning architectures for Earth system forecasting.

Original: Conventional DL models for Earth system forecasting are based on CNN and RNN.
Revised: Conventional DL models for Earth system forecasting are based on CNN and RNN.

Original: U-Net with either 2D CNN or 3D CNN have been used for precipitation nowcasting [43], Seasonal Arctic Sea ice prediction [1], and ENSO forecasting [15].
Revised: U-Net with either 2D CNN or 3D CNN have been used for precipitation nowcasting [43], Seasonal Arctic Sea ice prediction [1], and ENSO forecasting [15].

Original: Shi et al.
Revised: Shi et al.

Original: [36] proposed the ConvLSTM network that combines CNN and LSTM for precipitation nowcasting.
Revised: [36] proposed the ConvLSTM network that combines CNN and LSTM for precipitation nowcasting.

Original: Wang et al.
Revised: Wang et al.

Original: [45] proposed PredRNN which adds the spatiotemporal memory flow structure to ConvLSTM.
Revised: [45] proposed PredRNN which adds the spatiotemporal memory flow structure to ConvLSTM.

Original: To better learn long-term high-level relations, Wang et al.
Revised: To better learn long-term high-level relations, Wang et al.

Original: [44] proposed E3D-LSTM that integrates 3D CNN to LSTM.
Revised: [44] proposed E3D-LSTM that integrates 3D CNN to LSTM.

Original: To disentangle PDE dynamics from unknown complementary information, PhyDNet [13] incorporates a new recurrent physical cell to perform
Revised: To disentangle PDE dynamics from unknown complementary information, PhyDNet [13] incorporates a new recurrent physical cell to perform

Original: PDE-constrained prediction in latent space.
Revised: PDE-constrained prediction in latent space.

Original: Espeholt et al.
Revised: Espeholt et al.

Original: [9] proposed MetNet-2 that outperforms
Revised: [9] proposed MetNet-2 that outperforms

Original: HREF for forecasting precipitation.
Revised: HREF for forecasting precipitation.

Original: The architecture is based on ConvLSTM and dilated CNN.
Revised: The architecture is based on ConvLSTM and dilated CNN.

Original: Very recently, there are works that tried to apply Transformer for solving Earth system forecasting problems.
Revised: Very recently, there are works that tried to apply Transformer for solving Earth system forecasting problems.

Original: Pathak et al.
Revised: Pathak et al.

Original: [28] proposed the FourCastNet for global weather forecasting, which is based on Adaptive Fourier Neural Operators (AFNO) [14].
Revised: [28] proposed the FourCastNet for global weather forecasting, which is based on Adaptive Fourier Neural Operators (AFNO) [14].

Original: Bai et al.
Revised: Bai et al.

Original: [3] proposed Rainformer for precipitation nowcasting, which is based on an architecture that combines CNN and SwinTransformer [22].
Revised: [3] proposed Rainformer for precipitation nowcasting, which is based on an architecture that combines CNN and SwinTransformer [22].

Original: In our experiments, we can see that Earthformer outperforms Rainformer.
Revised: In our experiments, we can see that Earthformer outperforms Rainformer.

Original: Space-time Transformers for video modeling.
Revised: Space-time Transformers for video modeling.

Original: Inspired by the success of ViT [8] for image classification, space-time Transformer is adopted for improved video understanding.
Revised: Inspired by the success of ViT [8] for image classification, space-time Transformer is adopted for improved video understanding.

Original: In order to bypass the huge memory consumption brought by joint spatiotemporal attention, several pioneering work proposed efficient alternatives, such as divided attention [4], axial attention [19, 4], factorized encoder [27, 2] and separable attention [54].
Revised: In order to bypass the huge memory consumption brought by joint spatiotemporal attention, several pioneering work proposed efficient alternatives, such as divided attention [4], axial attention [19, 4], factorized encoder [27, 2] and separable attention [54].

Original: Beyond minimal adaptation from ViT, some recent work introduced more prior to the design of space-time transformers, including trajectory [29], multi-scale [23, 11] and multi-view [49].
Revised: Beyond minimal adaptation from ViT, some recent work introduced more prior to the design of space-time transformers, including trajectory [29], multi-scale [23, 11] and multi-view [49].

Original: However, no prior work focuses on exploring the design of space-time Transformers for Earth system forecasting.
Revised: However, no prior work focuses on exploring the design of space-time Transformers for Earth system forecasting.

Original: Global and local attention in vision Transformers.
Revised: Global and local attention in vision Transformers.

Original: To make self-attention more efficient in terms of both memory consumption and speed, recent works have adapted the essence of CNN to perform local attention in transformers [16, 52].
Revised: To make self-attention more efficient in terms of both memory consumption and speed, recent works have adapted the essence of CNN to perform local attention in transformers [16, 52].

Original: HaloNets [41] develops a new self-attention model family consisting of simple local self-attention and convolutional hybrids, which outperform both CNN and vanilla ViT on a range of downstream vision tasks.
Revised: HaloNets [41] develops a new self-attention model family consisting of simple local self-attention and convolutional hybrids, which outperform both CNN and vanilla ViT on a range of downstream vision tasks.

Original: GLiT [5] introduces a locality module and uses neural architecture search to find an efficient backbone.
Revised: GLiT [5] introduces a locality module and uses neural architecture search to find an efficient backbone.

Original: Focal transformer [51] proposes focal self-attention that can incorporate both fine-grained local and coarse-grained global interactions.
Revised: Focal transformer [51] proposes focal self-attention that can incorporate both fine-grained local and coarse-grained global interactions.

Original: However, these architectures are not directly applicable to spatiotemporal forecasting.
Revised: However, these architectures are not directly applicable to spatiotemporal forecasting.

Original: Besides, they are also different from our design because we keep K global vectors to summarize the statistics of the dynamic system and connect the local cuboids; experiments show that such a global vector design is crucial to successful spatiotemporal forecasting.
Revised: Besides, they are also different from our design because we keep K global vectors to summarize the statistics of the dynamic system and connect the local cuboids; experiments show that such a global vector design is crucial to successful spatiotemporal forecasting.

Original:  Model
Revised:  Model

Original: Similar to previous works [36, 43, 3], we formulate Earth system forecasting as a spatiotemporal sequence forecasting problem.
Revised: Similar to previous works [36, 43, 3], we formulate Earth system forecasting as a spatiotemporal sequence forecasting problem.

Original: The Earth observation data, such as radar echo maps from
Revised: The Earth observation data, such as radar echo maps from

Original: NEXRAD [17] and climate data from CIMP6 [10], are represented as a spatiotemporal sequence [ X i ]
Revised: NEXRAD [17] and climate data from CIMP6 [10], are represented as a spatiotemporal sequence [ X i ]

Original: Ti =1 , X i ∈
Revised: Ti =1 , X i ∈

Original: R H × W × C in .
Revised: R H × W × C in .

Original: Based on these observations, the model predicts the K -step-ahead future
Revised: Based on these observations, the model predicts the K -step-ahead future

Original: [ Y T
Revised: [ Y T

Original: + i ]
Revised: + i ]

Original: Ki =1 , Y T + i ∈
Revised: Ki =1 , Y T + i ∈

Original: R H × W × C out .
Revised: R H × W × C out .

Original: Here, H, W denote the spatial resolution, and C in , C out denote the number of measurements available at each space-time coordinate from the input and target sequences, respectively.
Revised: Here, H, W denote the spatial resolution, and C in , C out denote the number of measurements available at each space-time coordinate from the input and target sequences, respectively.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 2, our proposed Earthformer is a hierarchical Transformer encoderdecoder based on Cuboid Attention .
Revised: 2, our proposed Earthformer is a hierarchical Transformer encoderdecoder based on Cuboid Attention .

Original: The input observations are encoded as a hierarchy of hidden states and then decoded to the prediction target.
Revised: The input observations are encoded as a hierarchy of hidden states and then decoded to the prediction target.

Original: In what follows, we will present the detailed design of cuboid attention and the hierarchical encoder-decoder architecture adopted in Earthformer.
Revised: In what follows, we will present the detailed design of cuboid attention and the hierarchical encoder-decoder architecture adopted in Earthformer.

Original: 3.1 Cuboid Attention
Revised: 3.1 Cuboid Attention

Original: Compared with images and text, spatiotemporal data in Earth systems usually have higher dimensionality.
Revised: Compared with images and text, spatiotemporal data in Earth systems usually have higher dimensionality.

Original: As a consequence, applying Transformers to this task is challenging.
Revised: As a consequence, applying Transformers to this task is challenging.

Original: For example, for a 3 D
Revised: For example, for a 3 D

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: tensor with shape ( T, H, W ) , the complexity of the vanilla self-attention is O
Revised: tensor with shape ( T, H, W ) , the complexity of the vanilla self-attention is O

Original: ( T 2 H 2 W 2 ) and can be computationally infeasible.
Revised: ( T 2 H 2 W 2 ) and can be computationally infeasible.

Original: Previous literature proposed various structure-aware space-time attention mechanisms to reduce the complexity [19, 23, 4].
Revised: Previous literature proposed various structure-aware space-time attention mechanisms to reduce the complexity [19, 23, 4].

Original: These space-time attention mechanisms share the common design of stacking multiple elementary attention layers that focus on different types of data correlations (e.g., temporal correlation and spatial correlation).
Revised: These space-time attention mechanisms share the common design of stacking multiple elementary attention layers that focus on different types of data correlations (e.g., temporal correlation and spatial correlation).

Original: Steming from this observation, we propose the generic cuboid attention layer that involves three steps: “decompose”, “attend”, and “merge”.
Revised: Steming from this observation, we propose the generic cuboid attention layer that involves three steps: “decompose”, “attend”, and “merge”.

Original: Decompose.
Revised: Decompose.

Original: We first decompose the input spatiotemporal tensor X ∈ R T × H × W × C into a sequence of cuboids { x ( n ) } .
Revised: We first decompose the input spatiotemporal tensor X ∈ R T × H × W × C into a sequence of cuboids { x ( n ) } .

Original: [Equation]
Revised: [Equation]

Original: where cuboid_size = ( b T , b H , b W ) is the size of the local cuboid, strategy ∈ { “ local ” , “ dilated ” } controls whether to adopt the local decomposition strategy or the dilated decomposition strategy [4], shift =
Revised: where cuboid_size = ( b T , b H , b W ) is the size of the local cuboid, strategy ∈ { “ local ” , “ dilated ” } controls whether to adopt the local decomposition strategy or the dilated decomposition strategy [4], shift =

Original: ( s T , s H , s W ) is the window shift offset [22].
Revised: ( s T , s H , s W ) is the window shift offset [22].

Original: Fig.
Revised: Fig.

Original: 4 provides three examples showing how an input tensor will be decomposed following different hyperparameters of Decompose ( · ) .
Revised: 4 provides three examples showing how an input tensor will be decomposed following different hyperparameters of Decompose ( · ) .

Original: There are a total number of ⌈ Tb T ⌉⌈
Revised: There are a total number of ⌈ Tb T ⌉⌈

Original: Hb H ⌉⌈
Revised: Hb H ⌉⌈

Original: Wb W ⌉ cuboids in { x ( n ) } .
Revised: Wb W ⌉ cuboids in { x ( n ) } .

Original: To simplify the notation, we assume that T, H, W are divisible by b T , b H , b W .
Revised: To simplify the notation, we assume that T, H, W are divisible by b T , b H , b W .

Original: In the implementation, we pad the input tensor if it is not divisible.
Revised: In the implementation, we pad the input tensor if it is not divisible.

Original: Assume x ( n ) is the ( n T , n H , n W ) -th cuboid in { x ( n ) } .
Revised: Assume x ( n ) is the ( n T , n H , n W ) -th cuboid in { x ( n ) } .

Original: The ( i, j, k ) -th element of x ( n ) can be mapped to the ( i ′ , j ′ , k ′ ) -th
Revised: The ( i, j, k ) -th element of x ( n ) can be mapped to the ( i ′ , j ′ , k ′ ) -th

Original: element of X via Eqn.
Revised: element of X via Eqn.

Original: 2 if the strategy is “local” or Eqn.
Revised: 2 if the strategy is “local” or Eqn.

Original: 3 if the strategy is “dilated”.
Revised: 3 if the strategy is “dilated”.

Original: [Equation]
Revised: [Equation]

Original: Since the mapping is bijective, one can then map the elements from X to { x ( n ) } via the inverse operation.
Revised: Since the mapping is bijective, one can then map the elements from X to { x ( n ) } via the inverse operation.

Original: Attend.
Revised: Attend.

Original: After decomposing the input tensor into a sequence of non-overlapping cuboids { x ( n ) } , we apply self-attention within each cuboid in parallel.
Revised: After decomposing the input tensor into a sequence of non-overlapping cuboids { x ( n ) } , we apply self-attention within each cuboid in parallel.

Original: [Equation]
Revised: [Equation]

Original: The query, key, and value matrices Q , K , and V of Attention
Revised: The query, key, and value matrices Q , K , and V of Attention

Original: [Equation]
Revised: [Equation]

Original: Softmax (cid:16) ( W Q Q )( W K K ) T / √ C (cid:17) ( W V V ) are all flattened versions of x ( n ) , and we unravel the resulting matrix back to a 3 D tensor.
Revised: Softmax (cid:16) ( W Q Q )( W K K ) T / √ C (cid:17) ( W V V ) are all flattened versions of x ( n ) , and we unravel the resulting matrix back to a 3 D tensor.

Original: W Q , W K and W V are linear projection weights and are abbreviated together as Θ .
Revised: W Q , W K and W V are linear projection weights and are abbreviated together as Θ .

Original: The self-attention parameter Θ are shared across all cuboids.
Revised: The self-attention parameter Θ are shared across all cuboids.

Original: The computational complexity of the “attend” step is O (cid:16) ⌈
Revised: The computational complexity of the “attend” step is O (cid:16) ⌈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 (cid:17) ≈
Revised: 2 (cid:17) ≈

Original: [Equation]
Revised: [Equation]

Original: H b W ) , which scales linearly with the cuboid size.
Revised: H b W ) , which scales linearly with the cuboid size.

Original: Since the cuboid size can be much smaller than the size of the input tensor, the layer is more efficient than full attention.
Revised: Since the cuboid size can be much smaller than the size of the input tensor, the layer is more efficient than full attention.

Original: Merge.
Revised: Merge.

Original: Merge ( · ) is the inverse operation of Decompose ( · ) .
Revised: Merge ( · ) is the inverse operation of Decompose ( · ) .

Original: The sequence of cuboids obtained after the attention step { x ( n ) out } are merged back to the original input shape to produce the final output of cuboid attention, as shown in Eqn. 5.
Revised: The sequence of cuboids obtained after the attention step { x ( n ) out } are merged back to the original input shape to produce the final output of cuboid attention, as shown in Eqn. 5.

Original: The mapping follows the same bijections in Eqn.
Revised: The mapping follows the same bijections in Eqn.

Original: 2 and Eqn. 3.
Revised: 2 and Eqn. 3.

Original: [Equation]
Revised: [Equation]

Original: We combine the “decompose”, “attend” and “merge” steps described in Eqn.
Revised: We combine the “decompose”, “attend” and “merge” steps described in Eqn.

Original: 1,4,5 to construct the generic cuboid attention as in Eqn. 6.
Revised: 1,4,5 to construct the generic cuboid attention as in Eqn. 6.

Original: [Equation]
Revised: [Equation]

Original: Explore cuboid attention patterns.
Revised: Explore cuboid attention patterns.

Original: By stacking multiple cuboid attention layers with different choices of “cuboid_size”, “strategy” and “shift”, we are able to efficiently explore existing and potentially more effective space-time attention.
Revised: By stacking multiple cuboid attention layers with different choices of “cuboid_size”, “strategy” and “shift”, we are able to efficiently explore existing and potentially more effective space-time attention.

Original: In this paper, we explore the cuboid attention patterns as listed in Table 1.
Revised: In this paper, we explore the cuboid attention patterns as listed in Table 1.

Original: From the table, we can see that cuboid attention subsumes previously proposed space-time attention methods like axial attention, video swin-Transformer, and divided space-time attention.
Revised: From the table, we can see that cuboid attention subsumes previously proposed space-time attention methods like axial attention, video swin-Transformer, and divided space-time attention.

Original: Also, we manually picked the patterns that are reasonable and not computationally expensive as our search space.
Revised: Also, we manually picked the patterns that are reasonable and not computationally expensive as our search space.

Original: The flexibility of cuboid attention allows us to conduct Neural
Revised: The flexibility of cuboid attention allows us to conduct Neural

Original: Architecture Search (NAS) to automatically search for a pattern but we will leave it as future work.
Revised: Architecture Search (NAS) to automatically search for a pattern but we will leave it as future work.

Original: 3.2 Global Vectors
Revised: 3.2 Global Vectors

Original: One limitation of the previous formulation is that the cuboids do not communicate with each other.
Revised: One limitation of the previous formulation is that the cuboids do not communicate with each other.

Original: This is undesirable because each cuboid is not capable of understanding the global dynamics of the system.
Revised: This is undesirable because each cuboid is not capable of understanding the global dynamics of the system.

Original: Thus, inspired by the [CLS] token adopted in BERT [7, 53], we propose to introduce a collection of P global vectors G ∈ R P × C to help cuboids scatter and gather crucial global information.
Revised: Thus, inspired by the [CLS] token adopted in BERT [7, 53], we propose to introduce a collection of P global vectors G ∈ R P × C to help cuboids scatter and gather crucial global information.

Original: When each cuboid is performing the self-attention, the elements will not only attend to the other elements within the same cuboid but also attend to the global vectors G .
Revised: When each cuboid is performing the self-attention, the elements will not only attend to the other elements within the same cuboid but also attend to the global vectors G .

Original: We revise Eqn.
Revised: We revise Eqn.

Original: 4 to Eqn.to enable local-global information exchange.
Revised: 4 to Eqn.to enable local-global information exchange.

Original: We also use Eqn. 8 to update the global vectors G by aggregating the information from all elements of the input tensor X .
Revised: We also use Eqn. 8 to update the global vectors G by aggregating the information from all elements of the input tensor X .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: Here, Cat ( · ) flattens and concatenates its input tensors.
Revised: Here, Cat ( · ) flattens and concatenates its input tensors.

Original: By combining Eqn. 1,7,8,5, we abbreviate the overall computation of the cuboid attention layer with global vectors as in Eqn. 9.
Revised: By combining Eqn. 1,7,8,5, we abbreviate the overall computation of the cuboid attention layer with global vectors as in Eqn. 9.

Original: [Equation]
Revised: [Equation]

Original: The additional complexity caused by the global vectors is approximately O (cid:0) THW · P + P 2 (cid:1) .
Revised: The additional complexity caused by the global vectors is approximately O (cid:0) THW · P + P 2 (cid:1) .

Original: Given that P is usually small (in our experiments, P is at most 8 ), the computational overhead induced by the global structure is negligible.
Revised: Given that P is usually small (in our experiments, P is at most 8 ), the computational overhead induced by the global structure is negligible.

Original: The architecture of the cuboid attention layer is illustrated in Fig.
Revised: The architecture of the cuboid attention layer is illustrated in Fig.

Original: 3.3 Hierarchical Encoder-Decoder Architecture
Revised: 3.3 Hierarchical Encoder-Decoder Architecture

Original: Earthformer adopts a hierarchical encoder-decoder architecture illustrated in Fig.
Revised: Earthformer adopts a hierarchical encoder-decoder architecture illustrated in Fig.

Original: The hierarchical architecture gradually encodes the input sequence to multiple levels of representations and generates the prediction via a coarse-to-fine procedure.
Revised: The hierarchical architecture gradually encodes the input sequence to multiple levels of representations and generates the prediction via a coarse-to-fine procedure.

Original: Each hierarchy stacks D cuboid attention blocks.
Revised: Each hierarchy stacks D cuboid attention blocks.

Original: The cuboid attention block in the encoder uses one of the patterns described in Table 1, and each cuboid block in the decoder adopts the “Axial” pattern.
Revised: The cuboid attention block in the encoder uses one of the patterns described in Table 1, and each cuboid block in the decoder adopts the “Axial” pattern.

Original: To reduce the spatial resolution of the input to cuboid attention layers, we include a pair of initial downsampling and upsampling modules that consist of stacked 2D-CNN and Nearest Neighbor Interpolation (NNI) layers.
Revised: To reduce the spatial resolution of the input to cuboid attention layers, we include a pair of initial downsampling and upsampling modules that consist of stacked 2D-CNN and Nearest Neighbor Interpolation (NNI) layers.

Original: Different from other papers that adopt Transformer for video prediction [19, 30], we generate the predictions in a non-auto-regressive fashion rather than an auto-regressive patch-by-patch fashion.
Revised: Different from other papers that adopt Transformer for video prediction [19, 30], we generate the predictions in a non-auto-regressive fashion rather than an auto-regressive patch-by-patch fashion.

Original: This means that our decoder directly generates the predictions from the initial learned positional embeddings.
Revised: This means that our decoder directly generates the predictions from the initial learned positional embeddings.

Original: We also conducted experiments with an auto-regressive decoder based on visual codebook [33].
Revised: We also conducted experiments with an auto-regressive decoder based on visual codebook [33].

Original: However, the auto-regressive decoder underperforms the non-auto-regressive decoder in terms of forecasting skill scores.
Revised: However, the auto-regressive decoder underperforms the non-auto-regressive decoder in terms of forecasting skill scores.

Original: The comparison between non-auto-regressive decoder and auto-regressive decoder is shown in Appendix C.
Revised: The comparison between non-auto-regressive decoder and auto-regressive decoder is shown in Appendix C.

Original:  Experiments
Revised:  Experiments

Original: We first conducted experiments on two synthetic datasets, MovingMNIST and a newly proposed
Revised: We first conducted experiments on two synthetic datasets, MovingMNIST and a newly proposed

Original: N -body MNIST, to verify the effectiveness of Earthformer and conduct ablation study on our design choices.
Revised: N -body MNIST, to verify the effectiveness of Earthformer and conduct ablation study on our design choices.

Original: Results on these two datasets lead to the following findings: 1) Among all patterns listed in Table 1, “Axial” achieves the best overall performance; 2)
Revised: Results on these two datasets lead to the following findings: 1) Among all patterns listed in Table 1, “Axial” achieves the best overall performance; 2)

Original: Global vectors bring consistent performance gain with negligible increase in computational cost; 3) Using a hierarchical coarseto-fine structure can boost the performance.
Revised: Global vectors bring consistent performance gain with negligible increase in computational cost; 3) Using a hierarchical coarseto-fine structure can boost the performance.

Original: Based on these findings, we figured out the optimal design of Earthformer and compared it with other state-of-the-art models on two real-world datasets:
Revised: Based on these findings, we figured out the optimal design of Earthformer and compared it with other state-of-the-art models on two real-world datasets:

Original: SEVIR [43] and ICAR-ENSO 2 .
Revised: SEVIR [43] and ICAR-ENSO 2 .

Original: On both datasets, Earthformer achieved the best overall performance.
Revised: On both datasets, Earthformer achieved the best overall performance.

Original: The statistics of all the datasets used in the experiments are shown in Table 2.
Revised: The statistics of all the datasets used in the experiments are shown in Table 2.

Original: We normalized the data to the range [0 , 1] and trained all the models with the Mean-Squared Error (MSE) loss.
Revised: We normalized the data to the range [0 , 1] and trained all the models with the Mean-Squared Error (MSE) loss.

Original: More implementation details are shown in Appendix A.
Revised: More implementation details are shown in Appendix A.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.1 Experiments on Synthetic Datasets
Revised: 4.1 Experiments on Synthetic Datasets

Original: MovingMNIST.
Revised: MovingMNIST.

Original: We follow [38] to use the public MovingMNIST dataset 3 .
Revised: We follow [38] to use the public MovingMNIST dataset 3 .

Original: The dataset contains 10,000 sequences.
Revised: The dataset contains 10,000 sequences.

Original: Each sequence shows 2 digits moving inside a 64 × 64 frame.
Revised: Each sequence shows 2 digits moving inside a 64 × 64 frame.

Original: We split the dataset to use 8,100 samples for training, 900 samples for validation and 1,000 samples for testing.
Revised: We split the dataset to use 8,100 samples for training, 900 samples for validation and 1,000 samples for testing.

Original: The task is to predict the future 10 frames for each sequence conditioned on the first 10 frames.
Revised: The task is to predict the future 10 frames for each sequence conditioned on the first 10 frames.

Original: N -body MNIST.
Revised: N -body MNIST.

Original: The Earth is a complex system in which an extremely large number of variables interact with each other.
Revised: The Earth is a complex system in which an extremely large number of variables interact with each other.

Original: Compared with the Earth system, the dynamics of the synthetic MovingMNIST dataset, in which the digits move independently with constant speed, is over-simplified.
Revised: Compared with the Earth system, the dynamics of the synthetic MovingMNIST dataset, in which the digits move independently with constant speed, is over-simplified.

Original: Thus, achieving good performance on MovingMNIST does not imply that the model is capable of modeling complex interactions in the Earth system.
Revised: Thus, achieving good performance on MovingMNIST does not imply that the model is capable of modeling complex interactions in the Earth system.

Original: On the other hand, the real-world Earth observation data, though experiencing rapid development, are still noisy and may not provide useful insights for model development.
Revised: On the other hand, the real-world Earth observation data, though experiencing rapid development, are still noisy and may not provide useful insights for model development.

Original: Therefore, we extend MovingMNIST to N -body MNIST, where N digits are moving with the N -body motion pattern inside a 64 × 64 frame.
Revised: Therefore, we extend MovingMNIST to N -body MNIST, where N digits are moving with the N -body motion pattern inside a 64 × 64 frame.

Original: Each digit has its mass and is subjected to the gravity from other digits.
Revised: Each digit has its mass and is subjected to the gravity from other digits.

Original: We choose N = 3 in the experiments so that the digits will follow the chaotic 3 -body motion [25].
Revised: We choose N = 3 in the experiments so that the digits will follow the chaotic 3 -body motion [25].

Original: The highly non-linear interactions in N -body MNIST make it much more challenging than the original MovingMNIST.
Revised: The highly non-linear interactions in N -body MNIST make it much more challenging than the original MovingMNIST.

Original: We generate 20,000 sequences for training, 1, for validation and 1,000 for testing.
Revised: We generate 20,000 sequences for training, 1, for validation and 1,000 for testing.

Original: Perceptual examples of the dataset can be found at the first two rows of Fig.
Revised: Perceptual examples of the dataset can be found at the first two rows of Fig.

Original: In Appendix D, we demonstrate the chaotic behavior of N -body MNIST.
Revised: In Appendix D, we demonstrate the chaotic behavior of N -body MNIST.

Original: Hierarchical v.s. non
Revised: Hierarchical v.s. non

Original: -hierarchical.
Revised: -hierarchical.

Original: We choose “Axial” without global vectors as our cuboid attention pattern and compare the performance of non-hierarchical and hierarchical architectures on
Revised: We choose “Axial” without global vectors as our cuboid attention pattern and compare the performance of non-hierarchical and hierarchical architectures on

Original: MovingMNIST.
Revised: MovingMNIST.

Original: The ablation study on the importance of adopting a hierarchical encoder-decoder is shown in Table 3.
Revised: The ablation study on the importance of adopting a hierarchical encoder-decoder is shown in Table 3.

Original: We can see that the hierarchical architecture has similar FLOPS with the nonhierarchical architectures while being better in MSE.
Revised: We can see that the hierarchical architecture has similar FLOPS with the nonhierarchical architectures while being better in MSE.

Original: This observation is consistent as we increase the depth until the performance saturates.
Revised: This observation is consistent as we increase the depth until the performance saturates.

Original: Cuboid pattern search.
Revised: Cuboid pattern search.

Original: The design of cuboid attention greatly facilitates the search for optimal space-time attention.
Revised: The design of cuboid attention greatly facilitates the search for optimal space-time attention.

Original: We compare the patterns listed in Table 1 on both MovingMNIST and N -body
Revised: We compare the patterns listed in Table 1 on both MovingMNIST and N -body

Original: MNIST to investigate the effectiveness and efficiency of different space-time attention methods on spatiotemporal forecasting tasks.
Revised: MNIST to investigate the effectiveness and efficiency of different space-time attention methods on spatiotemporal forecasting tasks.

Original: Besides the previously proposed space-time attention methods, we also include new configurations that are reasonable and not computationally expensive in our search space.
Revised: Besides the previously proposed space-time attention methods, we also include new configurations that are reasonable and not computationally expensive in our search space.

Original: For each pattern, we also compare the variant that uses global vectors.
Revised: For each pattern, we also compare the variant that uses global vectors.

Original: Results are summarized in Table 4.
Revised: Results are summarized in Table 4.

Original: We find that the “Axial” pattern is both effective and efficient and adding global vectors improves performance for all patterns while having similar FLOPS.
Revised: We find that the “Axial” pattern is both effective and efficient and adding global vectors improves performance for all patterns while having similar FLOPS.

Original: We thus pick “Axial + global” as the pattern in Earthformer when conducting experiments on real-world datasets.
Revised: We thus pick “Axial + global” as the pattern in Earthformer when conducting experiments on real-world datasets.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Comparison to the state of the art.
Revised: Comparison to the state of the art.

Original: We evaluate six spatiotemporal forecasting algorithms:
Revised: We evaluate six spatiotemporal forecasting algorithms:

Original: UNet [43], ConvLSTM [36], PredRNN [45], PhyDNet [13], E3D-LSTM [44] and Rainformer [3].
Revised: UNet [43], ConvLSTM [36], PredRNN [45], PhyDNet [13], E3D-LSTM [44] and Rainformer [3].

Original: The results are in Table 5.
Revised: The results are in Table 5.

Original: Note that the MovingMNIST performance on several papers [13] is obtained by training the model with on-the-fly generated digits while we pre-generate the digits and train all models on a fixed dataset.
Revised: Note that the MovingMNIST performance on several papers [13] is obtained by training the model with on-the-fly generated digits while we pre-generate the digits and train all models on a fixed dataset.

Original: Comparing the numbers in the table with numbers shown in these papers are not fair.
Revised: Comparing the numbers in the table with numbers shown in these papers are not fair.

Original: We train all baselines from scratch on both MovingMNIST and N -body MNIST using the default hyperparameters and configurations in their officially released code 4 .
Revised: We train all baselines from scratch on both MovingMNIST and N -body MNIST using the default hyperparameters and configurations in their officially released code 4 .

Original: Qualitative results on N -body MNIST.
Revised: Qualitative results on N -body MNIST.

Original: Fig.
Revised: Fig.

Original: 5 shows the generation results of different methods on a sample sequence from the N -body MNIST test set.
Revised: 5 shows the generation results of different methods on a sample sequence from the N -body MNIST test set.

Original: The qualitative example demonstrates that
Revised: The qualitative example demonstrates that

Original: our
Revised: our

Original: Earthformer is capable of learning long-range interactions among digits and correctly predicting their future motion trajectories.
Revised: Earthformer is capable of learning long-range interactions among digits and correctly predicting their future motion trajectories.

Original: Also, we can see that Earthformer is able to more accurately predict the position of the digits with the help of global vectors.
Revised: Also, we can see that Earthformer is able to more accurately predict the position of the digits with the help of global vectors.

Original: On the contrary, none of the baseline algorithms that achieved solid performance on MovingMNIST gives the correct and precise position of the digit “0” in the last frame.
Revised: On the contrary, none of the baseline algorithms that achieved solid performance on MovingMNIST gives the correct and precise position of the digit “0” in the last frame.

Original: They either predict incorrect motion trajectories (PredRNN and E3D-LSTM), or generate highly blurry predictions (Rainformer, UNet and PhyDNet) to accommodate the uncertainty about the future.
Revised: They either predict incorrect motion trajectories (PredRNN and E3D-LSTM), or generate highly blurry predictions (Rainformer, UNet and PhyDNet) to accommodate the uncertainty about the future.

Original: 4.2 SEVIR Precipitation Nowcasting
Revised: 4.2 SEVIR Precipitation Nowcasting

Original: Storm EVent ImageRy (SEVIR) [43] is a spatiotemporally aligned dataset containing over 10, weather events.
Revised: Storm EVent ImageRy (SEVIR) [43] is a spatiotemporally aligned dataset containing over 10, weather events.

Original: Each event consists of 384 km × 384 km image sequences spanning over 4 hours.
Revised: Each event consists of 384 km × 384 km image sequences spanning over 4 hours.

Original: Images in SEVIR were sampled and aligned across five different data types: three channels (C02,4 Except for Rainformer which originally has 212M parameters and thus suffers from overfitting severely.
Revised: Images in SEVIR were sampled and aligned across five different data types: three channels (C02, and thus suffers from overfitting severely.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: C09, C13) from the GOES-16 advanced baseline imager, NEXRAD Vertically Integrated Liquid (VIL) mosaics, and GOES-16 Geostationary Lightning Mapper (GLM) flashes.
Revised: C09, C13) from the GOES-16 advanced baseline imager, NEXRAD Vertically Integrated Liquid (VIL) mosaics, and GOES-16 Geostationary Lightning Mapper (GLM) flashes.

Original: The SEVIR benchmark supports scientific research on multiple meteorological applications including precipitation nowcasting, synthetic radar generation, front detection, etc.
Revised: The SEVIR benchmark supports scientific research on multiple meteorological applications including precipitation nowcasting, synthetic radar generation, front detection, etc.

Original: We adopt SEVIR for benchmarking precipitation nowcasting, i.e., to predict the future VIL up to 60 minutes (12 frames) given 65 minutes context VIL (13 frames).
Revised: We adopt SEVIR for benchmarking precipitation nowcasting, i.e., to predict the future VIL up to 60 minutes (12 frames) given 65 minutes context VIL (13 frames).

Original: Fig.
Revised: Fig.

Original: 1 shows an example of VIL observation sequences in SEVIR.
Revised: 1 shows an example of VIL observation sequences in SEVIR.

Original: Besides MSE, we also include the Critical Success Index ( CSI ), which is commonly used in precipitation nowcasting and is defined as CSI =
Revised: Besides MSE, we also include the Critical Success Index ( CSI ), which is commonly used in precipitation nowcasting and is defined as CSI =

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: Alarms .
Revised: Alarms .

Original: To count the # Hits (truth=1, pred=1), # Misses (truth=1, pred=0) and # F .
Revised: To count the # Hits (truth=1, pred=1), # Misses (truth=1, pred=0) and # F .

Original: Alarms (truth=0, pred=1), the prediction and the ground-truth are rescaled back to the range 0-255 and binarized at thresholds [16 , 74 , 133 , 160 , 181 , 219] .
Revised: Alarms (truth=0, pred=1), the prediction and the ground-truth are rescaled back to the range 0-255 and binarized at thresholds [16 , 74 , 133 , 160 , 181 , 219] .

Original: We report CSI at different thresholds and also their mean CSI - M .
Revised: We report CSI at different thresholds and also their mean CSI - M .

Original: SEVIR is much larger than MovingMNIST and N -body MNIST and has higher resolution.
Revised: SEVIR is much larger than MovingMNIST and N -body MNIST and has higher resolution.

Original: We thus slightly adjust the configurations of baselines based on those for MovingMNIST for fair comparison.
Revised: We thus slightly adjust the configurations of baselines based on those for MovingMNIST for fair comparison.

Original: Detailed configurations are shown in Appendix A.
Revised: Detailed configurations are shown in Appendix A.

Original: The experiment results are listed in Table 6.
Revised: The experiment results are listed in Table 6.

Original: Earthformer consistently outperforms baselines on almost all metrics and brings significant performance gain especially at high thresholds like CSI - 219 , which are more valued by the communities.
Revised: Earthformer consistently outperforms baselines on almost all metrics and brings significant performance gain especially at high thresholds like CSI - 219 , which are more valued by the communities.

Original: [Table]
Revised: [Table]

Original: 4.3 ICAR-ENSO Sea Surface Temperature Anomalies Forecasting
Revised: 4.3 ICAR-ENSO Sea Surface Temperature Anomalies Forecasting

Original: El Niño/Southern Oscillation (ENSO) has a wide range of associations with regional climate extremes and ecosystem impacts.
Revised: El Niño/Southern Oscillation (ENSO) has a wide range of associations with regional climate extremes and ecosystem impacts.

Original: ENSO sea surface temperature
Revised: ENSO sea surface temperature

Original: SST)
Revised: SST)

Original: anomalies forecasting for lead times up to one year (12 steps) is a valuable and challenging problem.
Revised: anomalies forecasting for lead times up to one year (12 steps) is a valuable and challenging problem.

Original: Nino3.4 index, which is the areaaveraged SST anomalies across a certain area ( 170 ◦ - 120 ◦ W , 5 ◦ S - 5 ◦ N ) of the Pacific, serves as a crucial indicator of this climate event.
Revised: Nino3.4 index, which is the areaaveraged SST anomalies across a certain area ( 170 ◦ - 120 ◦ W , 5 ◦ S - 5 ◦ N ) of the Pacific, serves as a crucial indicator of this climate event.

Original: The forecast quality is evaluated by the correlation skill [15] of the three-month-moving-averaged Nino3.4 index C Nino3 .
Revised: The forecast quality is evaluated by the correlation skill [15] of the three-month-moving-averaged Nino3.4 index C Nino3 .

Original: 4 =
Revised: 4 =

Original: [Equation]
Revised: [Equation]

Original: calculated on the whole test set of size N , where Y ∈ R N × K is the ground-truth of K -step Nino3.
Revised: calculated on the whole test set of size N , where Y ∈ R N × K is the ground-truth of K -step Nino3.

Original: index, X ∈ R N × K is the corresponding prediction of Nino3.4 index.
Revised: index, X ∈ R N × K is the corresponding prediction of Nino3.4 index.

Original: ICAR-ENSO consists of historical climate observation and stimulation data provided by Institute for
Revised: ICAR-ENSO consists of historical climate observation and stimulation data provided by Institute for

Original: Climate and Application Research (ICAR).
Revised: Climate and Application Research (ICAR).

Original: We forecast the SST anomalies up to 14 steps (2 steps more than one year for calculating three-month-moving-average) given a context of 12 steps of SST anomalies observations.
Revised: We forecast the SST anomalies up to 14 steps (2 steps more than one year for calculating three-month-moving-average) given a context of 12 steps of SST anomalies observations.

Original: Table 7 compares the performance of our Earthformer with baselines on the
Revised: Table 7 compares the performance of our Earthformer with baselines on the

Original: ICAR-ENSO dataset.
Revised: ICAR-ENSO dataset.

Original: We report the mean correlation skill C - Nino3 .
Revised: We report the mean correlation skill C - Nino3 .

Original: 4 - M = 1 K (cid:80) k C Nino3 .
Revised: 4 - M = 1 K (cid:80) k C Nino3 .

Original: 4 k and the weighted mean correlation skill C - Nino3 .
Revised: 4 k and the weighted mean correlation skill C - Nino3 .

Original: 4 - WM = 1 K (cid:80) k
Revised: 4 - WM = 1 K (cid:80) k

Original: a k · C Nino3 .
Revised: a k · C Nino3 .

Original: 4 k over K = 12 forecasting steps 5 , as well as the MSE between the spatiotemporal SST anomalies prediction and the corresponding ground-truth.
Revised: 4 k over K = 12 forecasting steps 5 , as well as the MSE between the spatiotemporal SST anomalies prediction and the corresponding ground-truth.

Original: We can find that Earthformer consistently outperforms the baselines in all concerned evaluation metrics and that using global vectors further improves the performance.
Revised: We can find that Earthformer consistently outperforms the baselines in all concerned evaluation metrics and that using global vectors further improves the performance.

Original:  Conclusions and Broader Impact
Revised:  Conclusions and Broader Impact

Original: In this paper, we propose Earthformer, a space-time Transformer for Earth system forecasting.
Revised: In this paper, we propose Earthformer, a space-time Transformer for Earth system forecasting.

Original: Earthformer is based on a generic and efficient building block called Cuboid Attention .
Revised: Earthformer is based on a generic and efficient building block called Cuboid Attention .

Original: It achieves
Revised: It achieves

Original: SOTA on MovingMNIST, our newly proposed N -body MNIST, SEVIR, and ICAR-ENSO.
Revised: SOTA on MovingMNIST, our newly proposed N -body MNIST, SEVIR, and ICAR-ENSO.

Original: Our work has certain limitations.
Revised: Our work has certain limitations.

Original: The first one is that Earthformer is a deterministic model that does not model uncertainty.
Revised: The first one is that Earthformer is a deterministic model that does not model uncertainty.

Original: This may result in predicting the average of all plausible futures, causing the model to generate blurry predictions of low perceptual quality and be lack of valuable small-scale details.
Revised: This may result in predicting the average of all plausible futures, causing the model to generate blurry predictions of low perceptual quality and be lack of valuable small-scale details.

Original: In fact, the community lacks appropriate metrics that measure the uncertainty component in
Revised: In fact, the community lacks appropriate metrics that measure the uncertainty component in

Original: Earth system forecasting models.
Revised: Earth system forecasting models.

Original: Extending Earthformer to a probabilistic forecasting model can be an exciting future direction.
Revised: Extending Earthformer to a probabilistic forecasting model can be an exciting future direction.

Original: We include more detailed discussions and preliminary experiments about handling uncertainty in Appendix C.
Revised: We include more detailed discussions and preliminary experiments about handling uncertainty in Appendix C.

Original: The second one is that the model is purely data-driven and does not take advantage of the physical knowledge of the Earth system.
Revised: The second one is that the model is purely data-driven and does not take advantage of the physical knowledge of the Earth system.

Original: Recent studies on adding physical constraints [20, 26] and ensembling the predictions from a data-driven model and a physics-based model [32] imply that it is an active and promising research direction to pursue.
Revised: Recent studies on adding physical constraints [20, 26] and ensembling the predictions from a data-driven model and a physics-based model [32] imply that it is an active and promising research direction to pursue.

Original: We plan to study how to incorporate physical knowledge into Earthformer in the future.
Revised: We plan to study how to incorporate physical knowledge into Earthformer in the future.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work has been made possible by a Research Impact Fund project (R6003-21) and an Innovation and Technology Fund project (ITS/004/21FP) funded by the Hong Kong Government.
Revised: This work has been made possible by a Research Impact Fund project (R6003-21) and an Innovation and Technology Fund project (ITS/004/21FP) funded by the Hong Kong Government.

Original: a k = b k · ln k , where b k = 1 . 5 , for k ≤ 4 ; b k = 2 , for 4 < k ≤ 11 ; b k = 3 , for k > 11 .
Revised: a k = b k · ln k , where b k = 1 . 5 , for k ≤ 4 ; b k = 2 , for 4 < k ≤ 11 ; b k = 3 , for k > 11 .

Original: A Implementation Details
Revised: A Implementation Details

Original: All experiments are conducted on machines with NVIDIA V100 GPUs.
Revised: All experiments are conducted on machines with NVIDIA V100 GPUs.

Original: All models including
Revised: All models including

Original: Earthformer and the baselines can fit in a single GPU (with gradient checkpointing) and get trained without model parallelization.
Revised: Earthformer and the baselines can fit in a single GPU (with gradient checkpointing) and get trained without model parallelization.

Original: A.1 Earthformer
Revised: A.1 Earthformer

Original: Cuboid attention block.
Revised: Cuboid attention block.

Original: A cuboid attention block consists of L cuboid attention layers with different hyperparameters, as shown in 1.
Revised: A cuboid attention block consists of L cuboid attention layers with different hyperparameters, as shown in 1.

Original: For example, as illustrated in the “Configuration Values” of Table 1, “Axial” has L = 3 since it has 3 consecutive patterns ( T , 1 , 1 ) → ( 1 , H , 1 ) →
Revised: For example, as illustrated in the “Configuration Values” of Table 1, “Axial” has L = 3 since it has 3 consecutive patterns ( T , 1 , 1 ) → ( 1 , H , 1 ) →

Original: ( 1 , 1 , W ) , “Divided Space-Time” and “Video Swin P × M ” have L = 2 .
Revised: ( 1 , 1 , W ) , “Divided Space-Time” and “Video Swin P × M ” have L = 2 .

Original: We also add Layer Normalization (LN) and Feed-Forward Networks (FFNs) in the same way as Pre-LN Transformer [48].
Revised: We also add Layer Normalization (LN) and Feed-Forward Networks (FFNs) in the same way as Pre-LN Transformer [48].

Original: Given the input tensor X d − 1 and global vectors G d − 1 , the output ( X d , G d ) of the d -th cuboid attention block (there are D cuboid attention blocks in total for each hierarchy as illustrated in Fig. 2), is computed as Eqn. 10:
Revised: Given the input tensor X d − 1 and global vectors G d − 1 , the output ( X d , G d ) of the d -th cuboid attention block (there are D cuboid attention blocks in total for each hierarchy as illustrated in Fig. 2), is computed as Eqn. 10:

Original: [Equation]
Revised: [Equation]

Original: Here, FFN local d,l , LN local d,l means the FFN an LN layers applied on the tensor X d,l − 1 , and
Revised: Here, FFN local d,l , LN local d,l means the FFN an LN layers applied on the tensor X d,l − 1 , and

Original: FFN global d,l , LN global d,l means those applied on the global vectors G d,l − 1 .
Revised: FFN global d,l , LN global d,l means those applied on the global vectors G d,l − 1 .

Original: CubAttn local d,l is the combination of “decompose”, “attend” and “merge” pipeline described in Eqn.
Revised: CubAttn local d,l is the combination of “decompose”, “attend” and “merge” pipeline described in Eqn.

Original: We omit cuboid_size, strategy and shift for brevity.
Revised: We omit cuboid_size, strategy and shift for brevity.

Original: Attn global d,l is equivalent to Eqn. 8.
Revised: Attn global d,l is equivalent to Eqn. 8.

Original: For the l -th cuboid pattern in a cuboid attention block, the input X d,l − 1 and G d,l − 1 go through the LN layer, cuboid attention layer and FFN layer with residual connections sequentially, produce the output X d,l and G d,l .
Revised: For the l -th cuboid pattern in a cuboid attention block, the input X d,l − 1 and G d,l − 1 go through the LN layer, cuboid attention layer and FFN layer with residual connections sequentially, produce the output X d,l and G d,l .

Original: The output of the final cuboid pattern X d,L and G d,L serve as the output of the whole cuboid attention block X d and G d .
Revised: The output of the final cuboid pattern X d,L and G d,L serve as the output of the whole cuboid attention block X d and G d .

Original: Hyperparameters of Earthformer architecture.
Revised: Hyperparameters of Earthformer architecture.

Original: The detailed architecture configurations of Earthformer are described in Table 8 and Table 9.
Revised: The detailed architecture configurations of Earthformer are described in Table 8 and Table 9.

Original: We adopt the same configurations for MovingMNIST,
Revised: We adopt the same configurations for MovingMNIST,

Original: N -body MNIST and ICAR-ENSO datasets, as shown in Table 8.
Revised: N -body MNIST and ICAR-ENSO datasets, as shown in Table 8.

Original: We slightly adjust the configurations on SEVIR as shown in Table 9, due to its high resolution and large dataset size.
Revised: We slightly adjust the configurations on SEVIR as shown in Table 9, due to its high resolution and large dataset size.

Original: Optimization.
Revised: Optimization.

Original: We train all Earthformer variants using the AdamW optimizer.
Revised: We train all Earthformer variants using the AdamW optimizer.

Original: Detailed configurations are shown in Table 10.
Revised: Detailed configurations are shown in Table 10.

Original: We train for 100 epochs on all datasets and early-stop model training according to the validation score with tolerance = 20 .
Revised: We train for 100 epochs on all datasets and early-stop model training according to the validation score with tolerance = 20 .

Original: We adopt 20% linear warm-up and Cosine learning rate scheduler that decays the lr from its maximum to zero after warm-up.
Revised: We adopt 20% linear warm-up and Cosine learning rate scheduler that decays the lr from its maximum to zero after warm-up.

Original: We adopt data parallel and gradient accumulation to use a total batch size of 64 while the 16GB GPU can only afford a smaller batch size like 2 or 4 .
Revised: We adopt data parallel and gradient accumulation to use a total batch size of 64 while the 16GB GPU can only afford a smaller batch size like 2 or 4 .

Original: A.2 Baselines
Revised: A.2 Baselines

Original: We train baseline algorithms following their officially released configurations and tune the learning rate, learning rate scheduler, working resolution, etc., to optimize their performance on each dataset.
Revised: We train baseline algorithms following their officially released configurations and tune the learning rate, learning rate scheduler, working resolution, etc., to optimize their performance on each dataset.

Original: We list the modifications we applied to the baselines for each dataset in Table 11.
Revised: We list the modifications we applied to the baselines for each dataset in Table 11.

Original: B More Ablation Analysis on Synthetic Datasets
Revised: B More Ablation Analysis on Synthetic Datasets

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 4.1, the design of cuboid attention greatly facilitates the search for optimal space-time attention.
Revised: 4.1, the design of cuboid attention greatly facilitates the search for optimal space-time attention.

Original: Table 4 summarizes the cuboid attention pattern search results with the model depth equal to 4 on both MovingMNIST and N -body MNIST.
Revised: Table 4 summarizes the cuboid attention pattern search results with the model depth equal to 4 on both MovingMNIST and N -body MNIST.

Original: We further investigate if it is feasible to accelerate the pattern search using shallower models.
Revised: We further investigate if it is feasible to accelerate the pattern search using shallower models.

Original: The results with model depth equal to 2 are shown in Table 12.
Revised: The results with model depth equal to 2 are shown in Table 12.

Original: We find that the “Axial” pattern is still the optimal among the patterns listed in Table 1, and adding global vectors improves performance for all the patterns while having similar
Revised: We find that the “Axial” pattern is still the optimal among the patterns listed in Table 1, and adding global vectors improves performance for all the patterns while having similar

Original: FLOPS.
Revised: FLOPS.

Original: This observation implies that in the future work, it might be feasible to conduct NAS using smaller and shallower models with affordable cost and transfer the results to larger and deeper models.
Revised: This observation implies that in the future work, it might be feasible to conduct NAS using smaller and shallower models with affordable cost and transfer the results to larger and deeper models.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: + global 49.
Revised: + global 49.

Original: [Figure]
Revised: [Figure]

Original: C Non-Auto-Regressive v.s. Auto-Regressive
Revised: C Non-Auto-Regressive v.s. Auto-Regressive

Original: As mentioned in Sec. 3.3, besides generating the future predictions directly in a non-auto-regressive way, previous works generate the predictions patch-by-patch in raster-scan ordering [46, 50, 30].
Revised: As mentioned in Sec. 3.3, besides generating the future predictions directly in a non-auto-regressive way, previous works generate the predictions patch-by-patch in raster-scan ordering [46, 50, 30].

Original: In fact, the pros and cons of the auto-regressive and non-auto-regressive approaches are not clear under the setting of Earth system forecasting.
Revised: In fact, the pros and cons of the auto-regressive and non-auto-regressive approaches are not clear under the setting of Earth system forecasting.

Original: Therefore, we replace the decoder of the non-auto-regressive
Revised: Therefore, we replace the decoder of the non-auto-regressive

Original: Earthformer with the auto-regressive decoder shown in Fig.
Revised: Earthformer with the auto-regressive decoder shown in Fig.

Original: 6 and propose an Earthformer variant called Earthformer AR .
Revised: 6 and propose an Earthformer variant called Earthformer AR .

Original: The auto-regressive approach uses discrete visual tokens as the input and target.
Revised: The auto-regressive approach uses discrete visual tokens as the input and target.

Original: Here, we pretrain a
Revised: Here, we pretrain a

Original: VQ-VAE [40] with a codebook e ∈ R Q × C code , where Q is the codebook size.
Revised: VQ-VAE [40] with a codebook e ∈ R Q × C code , where Q is the codebook size.

Original: The discrete latent space is hence Q -way categorical.
Revised: The discrete latent space is hence Q -way categorical.

Original: The VQ-VAE encoder downsamples an input frame X i ∈
Revised: The VQ-VAE encoder downsamples an input frame X i ∈

Original: R H × W × C to
Revised: R H × W × C to

Original: Enc ( X i ) ∈
Revised: Enc ( X i ) ∈

Original: R H ′
Revised: R H ′

Original: × W ′ ×
Revised: × W ′ ×

Original: C code , then maps each element to its nearest code e i learned in the codebook, and produces the encoding Enc code
Revised: C code , then maps each element to its nearest code e i learned in the codebook, and produces the encoding Enc code

Original: ( X i ) ∈
Revised: ( X i ) ∈

Original: [ Q ] H ′
Revised: [ Q ] H ′

Original: × W ′ × 1 .
Revised: × W ′ × 1 .

Original: The decoder does it reversely: it takes the latent visual codes Z code i ∈
Revised: The decoder does it reversely: it takes the latent visual codes Z code i ∈

Original: [ Q ] H ′
Revised: [ Q ] H ′

Original: ×
Revised: ×

Original: W ′ × 1 , indexes the codes in the codebook to convert the discrete codes to real-valued vectors Z i ∈ R H ′
Revised: W ′ × 1 , indexes the codes in the codebook to convert the discrete codes to real-valued vectors Z i ∈ R H ′

Original: × W ′ ×
Revised: × W ′ ×

Original: C code , and then upsamples the frame back to the pixel space Y
Revised: C code , and then upsamples the frame back to the pixel space Y

Original: i ∈
Revised: i ∈

Original: R H × W × C .
Revised: R H × W × C .

Original: Fig.
Revised: Fig.

Original: 6 illustrates the architecture of the Earthformer AR.
Revised: 6 illustrates the architecture of the Earthformer AR.

Original: The input is a sequence of frames containing discrete visual codes [ Enc code ( X i )] i , which are the output of the VQ-VAE encoder.
Revised: The input is a sequence of frames containing discrete visual codes [ Enc code ( X i )] i , which are the output of the VQ-VAE encoder.

Original: The target sequence is [ Enc code ( Y i )] i .
Revised: The target sequence is [ Enc code ( Y i )] i .

Original: Earthformer AR generates the target code one-by-one in raster-scan ordering.
Revised: Earthformer AR generates the target code one-by-one in raster-scan ordering.

Original: The generation at the current step Z code ( a,b,c ) is conditioned on the context X and all the previously generated codes [ Z code <
Revised: The generation at the current step Z code ( a,b,c ) is conditioned on the context X and all the previously generated codes [ Z code <

Original: ( a,b,c ) ] :
Revised: ( a,b,c ) ] :

Original: [Equation]
Revised: [Equation]

Original: The VQ-VAE decoder decodes the generated visual codes Z code to the final output in the pixel space.
Revised: The VQ-VAE decoder decodes the generated visual codes Z code to the final output in the pixel space.

Original: We conduct preliminary experiments of Earthformer AR on the SEVIR dataset and find that Earthformer AR is able to give more perceptually satisfying predictions than Earthformer, but is inferior in terms of skill scores.
Revised: We conduct preliminary experiments of Earthformer AR on the SEVIR dataset and find that Earthformer AR is able to give more perceptually satisfying predictions than Earthformer, but is inferior in terms of skill scores.

Original: Fig.
Revised: Fig.

Original: 7 to Fig.
Revised: 7 to Fig.

Original: 10 show the qualitative results of Earthformer and Earthformer AR on several challenging test cases.
Revised: 10 show the qualitative results of Earthformer and Earthformer AR on several challenging test cases.

Original: The outputs of Earthformer AR look more like “real”
Revised: The outputs of Earthformer AR look more like “real”

Original: VIL images and do not suffer from producing blurry predictions.
Revised: VIL images and do not suffer from producing blurry predictions.

Original: However, the performance of
Revised: However, the performance of

Original: Earthformer AR is much worse than Earthformer (even worse than some simple baselines including
Revised: Earthformer AR is much worse than Earthformer (even worse than some simple baselines including

Original: Inputs -
Revised: Inputs -

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: UNet [43]) in the concerned evaluation metrics.
Revised: UNet [43]) in the concerned evaluation metrics.

Original: We further investigate the effect of the sampling algorithm for Earthformer AR.
Revised: We further investigate the effect of the sampling algorithm for Earthformer AR.

Original: We compared 1) generating the codes with argmax step-by-step and 2) randomly drawing samples from p (cid:16) Z code ( a,b,c ) |Z code < ( a,b,c ) , X i (cid:17) specified by Eqn. 11.
Revised: We compared 1) generating the codes with argmax step-by-step and 2) randomly drawing samples from p (cid:16) Z code ( a,b,c ) |Z code < ( a,b,c ) , X i (cid:17) specified by Eqn. 11.

Original: We denote the argmax -variant as Earthformer AR argmax .
Revised: We denote the argmax -variant as Earthformer AR argmax .

Original: We find that argmax sampling can give better skill scores but the generated results have less perceptual similarity than the “real” VIL images.
Revised: We find that argmax sampling can give better skill scores but the generated results have less perceptual similarity than the “real” VIL images.

Original: Based on these results, we pick the non-auto-regressive decoder for experiments in the main paper.
Revised: Based on these results, we pick the non-auto-regressive decoder for experiments in the main paper.

Original: The fact that Earthformer AR gives more perceptually satisfying predictions than Earthformer while having worse skill scores triggers future work.
Revised: The fact that Earthformer AR gives more perceptually satisfying predictions than Earthformer while having worse skill scores triggers future work.

Original: So far, there is no well-established metric for evaluating the preceptual quality of the predictions generated by Earth system forecasting models.
Revised: So far, there is no well-established metric for evaluating the preceptual quality of the predictions generated by Earth system forecasting models.

Original: The Fréchet
Revised: The Fréchet

Original: Inception Distance (FID) and Inception Score (IS) adopted in evaluating image GANs [24] rely on a pretrained network.
Revised: Inception Distance (FID) and Inception Score (IS) adopted in evaluating image GANs [24] rely on a pretrained network.

Original: We are able to pretrain Earthformer on Earth observations and adopt the pretrained network for both calculating the FID and initializing the GAN discriminator.
Revised: We are able to pretrain Earthformer on Earth observations and adopt the pretrained network for both calculating the FID and initializing the GAN discriminator.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: D Chaos in N-Body MNIST Dataset
Revised: D Chaos in N-Body MNIST Dataset

Original: The Earth system is chaotic [39], meaning that the future is very sensitive to the initial conditions.
Revised: The Earth system is chaotic [39], meaning that the future is very sensitive to the initial conditions.

Original: In Fig.
Revised: In Fig.

Original: 11, we illustrate the chaotic effect of N -body MNIST.
Revised: 11, we illustrate the chaotic effect of N -body MNIST.

Original: By only slightly changing the initial velocities of the digits, the positions of the digits after 20 steps change significantly.
Revised: By only slightly changing the initial velocities of the digits, the positions of the digits after 20 steps change significantly.

Original: N -body MNIST is thus much more challenging than MovingMNIST.
Revised: N -body MNIST is thus much more challenging than MovingMNIST.

Original: E Evaluation Metrics in SEVIR
Revised: E Evaluation Metrics in SEVIR

Original: In Sec.
Revised: In Sec.

Original: 4.2 and Table 6, we follow [43] to use the Critical Success Index ( CSI ) for prediction quality evaluation 6 .
Revised: 4.2 and Table 6, we follow [43] to use the Critical Success Index ( CSI ) for prediction quality evaluation 6 .

Original: Besides [43], the SEVIR team holds the SEVIR Dataset Challenge 7 , where the data involved are exactly the same while the evaluation metrics are slightly different 8 .
Revised: Besides [43], the SEVIR team holds the SEVIR Dataset Challenge 7 , where the data involved are exactly the same while the evaluation metrics are slightly different 8 .

Original: Specifically, [43] used six precipitation thresholds which correspond to pixel values [219 , 181 , 160 , 133 , 74 , 16] .
Revised: Specifically, [43] used six precipitation thresholds which correspond to pixel values [219 , 181 , 160 , 133 , 74 , 16] .

Original: The prediction and the ground-truth are rescaled back to the range 0-255.
Revised: The prediction and the ground-truth are rescaled back to the range 0-255.

Original: The # Hits ( τ ) (truth ≥ τ , pred ≥ τ ), # Misses ( τ ) (truth ≥ τ , pred < τ ) and # F .
Revised: The # Hits ( τ ) (truth ≥ τ , pred ≥ τ ), # Misses ( τ ) (truth ≥ τ , pred < τ ) and # F .

Original: Alarms ( τ ) (truth < τ , pred ≥ τ ) at certain threshold τ are counted over all test pixels as shown in Eqn.
Revised: Alarms ( τ ) (truth < τ , pred ≥ τ ) at certain threshold τ are counted over all test pixels as shown in Eqn.

Original: # Hits ( τ ) =
Revised: # Hits ( τ ) =

Original: N (cid:88) n =
Revised: N (cid:88) n =

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: H (cid:88) h
Revised: H (cid:88) h

Original: =
Revised: =

Original: W (cid:88) w =
Revised: W (cid:88) w =

Original: Hits ( τ )[ n, t, h, w ] # Misses ( τ ) =
Revised: Hits ( τ )[ n, t, h, w ] # Misses ( τ ) =

Original: N (cid:88) n =
Revised: N (cid:88) n =

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: H (cid:88) h
Revised: H (cid:88) h

Original: =
Revised: =

Original: W (cid:88) w =
Revised: W (cid:88) w =

Original: Misses ( τ )[ n, t, h, w ] # F .
Revised: Misses ( τ )[ n, t, h, w ] # F .

Original: Alarms ( τ ) =
Revised: Alarms ( τ ) =

Original: N (cid:88) n =
Revised: N (cid:88) n =

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: H (cid:88) h
Revised: H (cid:88) h

Original: =
Revised: =

Original: W (cid:88) w =
Revised: W (cid:88) w =

Original: F .
Revised: F .

Original: Alarms ( τ )[ n, t, h, w ]
Revised: Alarms ( τ )[ n, t, h, w ]

Original: CSI
Revised: CSI

Original: [Equation]
Revised: [Equation]

Original: = # Hits ( τ ) # Hits
Revised: = # Hits ( τ ) # Hits

Original: [Equation]
Revised: [Equation]

Original: + # Misses (
Revised: + # Misses (

Original: [Equation]
Revised: [Equation]

Original: + # F .
Revised: + # F .

Original: Alarms ( τ )
Revised: Alarms ( τ )

Original: CSI
Revised: CSI

Original: [Equation]
Revised: [Equation]

Original: M =
Revised: M =

Original: [Equation]
Revised: [Equation]

Original: (cid:88) τ ∈ [219 , 181 , 160 , 133 , 74 , 16]
Revised: (cid:88) τ ∈ [219 , 181 , 160 , 133 , 74 , 16]

Original: CSI - τ (12) τ ∈ [219 , 181 , 160 , 133 , 74 , 16] is one of the thresholds.
Revised: CSI - τ (12) τ ∈ [219 , 181 , 160 , 133 , 74 , 16] is one of the thresholds.

Original: Hits ( τ ) , Misses ( τ ) , F .
Revised: Hits ( τ ) , Misses ( τ ) , F .

Original: Alarms ( τ ) ∈ [0 , 1] N × T × H × W , where N is the test dataset size, T is the forecasting horizon, H and W are the height and width, respectively.
Revised: Alarms ( τ ) ∈ [0 , 1] N × T × H × W , where N is the test dataset size, T is the forecasting horizon, H and W are the height and width, respectively.

Original: We denote the average CSI - τ over the thresholds [219 , 181 , 160 , 133 , 74 , 16] as CSI - M .
Revised: We denote the average CSI - τ over the thresholds [219 , 181 , 160 , 133 , 74 , 16] as CSI - M .

Original: The results using the above evaluation metrics are demonstrated in Sec.
Revised: The results using the above evaluation metrics are demonstrated in Sec.

Original: 4.2 and Table 6.
Revised: 4.2 and Table 6.

Original: The calculation of CSI is slightly different in the SEVIR Dataset Challenge, as shown in Eqn. 13.
Revised: The calculation of CSI is slightly different in the SEVIR Dataset Challenge, as shown in Eqn. 13.

Original: [Table]
Revised: [Table]

Original: 2.5023 Rainformer[3] 184.
Revised: 2.5023 Rainformer[3] 184.

Original: 3.0711 Earthformerw/oglobal 13.
Revised: 3.0711 Earthformerw/oglobal 13.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 5306 Earthformer 15.
Revised: 5306 Earthformer 15.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: # Hits ( τ, t ) =
Revised: # Hits ( τ, t ) =

Original: N (cid:88) n =
Revised: N (cid:88) n =

Original: H (cid:88) h
Revised: H (cid:88) h

Original: =
Revised: =

Original: W (cid:88) w =
Revised: W (cid:88) w =

Original: Hits ( τ )[ n, t, h, w ] # Misses ( τ, t ) =
Revised: Hits ( τ )[ n, t, h, w ] # Misses ( τ, t ) =

Original: N (cid:88) n =
Revised: N (cid:88) n =

Original: H (cid:88) h
Revised: H (cid:88) h

Original: =
Revised: =

Original: W (cid:88) w =
Revised: W (cid:88) w =

Original: Misses ( τ )[ n, t, h, w ] # F
Revised: Misses ( τ )[ n, t, h, w ] # F

Original: Alarms ( τ, t ) =
Revised: Alarms ( τ, t ) =

Original: N (cid:88) n =
Revised: N (cid:88) n =

Original: H (cid:88) h
Revised: H (cid:88) h

Original: =
Revised: =

Original: W (cid:88) w =
Revised: W (cid:88) w =

Original: F . Alarms (
Revised: F . Alarms (

Original: τ
Revised: τ

Original: [ n, t, h, w ]
Revised: [ n, t, h, w ]

Original: CSI - τ ( t ) =
Revised: CSI - τ ( t ) =

Original: # Hits ( τ, t ) # Hits ( τ, t ) + #
Revised: # Hits ( τ, t ) # Hits ( τ, t ) + #

Original: Misses (
Revised: Misses (

Original: τ t ) + #
Revised: τ t ) + #

Original: F . Alarms (
Revised: F . Alarms (

Original: τ
Revised: τ

Original: , t
Revised: , t

Original: CSI - τ = 1 T
Revised: CSI - τ = 1 T

Original: T (cid:88) t
Revised: T (cid:88) t

Original: CSI - τ ( t )
Revised: CSI - τ ( t )

Original: CSI - M 3 = 1(cid:88)
Revised: CSI - M 3 = 1(cid:88)

Original: τ ∈ [133 , 74 , 16]
Revised: τ ∈ [133 , 74 , 16]

Original: [Equation]
Revised: [Equation]

Original: The CSI - τ , at certain threshold, is the mean of CSI - τ ( t ) over forecasting horizon T .
Revised: The CSI - τ , at certain threshold, is the mean of CSI - τ ( t ) over forecasting horizon T .

Original: We denote the average CSI - τ over the three thresholds [133 , 74 , 16] used in the SEVIR Dataset Challenge as
Revised: We denote the average CSI - τ over the three thresholds [133 , 74 , 16] used in the SEVIR Dataset Challenge as

Original: CSI - M 3 , and the average CSI - τ over the six thresholds [219 , 181 , 160 , 133 , 74 , 16] used in [43] as
Revised: CSI - M 3 , and the average CSI - τ over the six thresholds [219 , 181 , 160 , 133 , 74 , 16] used in [43] as

Original: CSI - M 6 .
Revised: CSI - M 6 .

Original: The experiment results evaluated using the metrics in the SEVIR Dataset Challenge are listed in Table 13.
Revised: The experiment results evaluated using the metrics in the SEVIR Dataset Challenge are listed in Table 13.

Original: Earthformer still consistently outperforms the baselines in almost all the metrics.
Revised: Earthformer still consistently outperforms the baselines in almost all the metrics.