Original: HT-N ET : H IERARCHICAL T RANSFORMER BASED O PERATOR L EARNING M ODEL FOR M ULTISCALE PDE S
Revised: HT-N ET : H IERARCHICAL T RANSFORMER BASED O PERATOR L EARNING M ODEL FOR M ULTISCALE PDE S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Complex nonlinear interplays of multiple scales give rise to many interesting physical phenomena and pose significant difficulties for the computer simulation of multiscale PDE models in areas such as reservoir simulation, high-frequency scattering, and turbulence modeling.
Revised: Complex nonlinear interplays of multiple scales give rise to many interesting physical phenomena and pose significant difficulties for the computer simulation of multiscale PDE models in areas such as reservoir simulation, high-frequency scattering, and turbulence modeling.

Original: In this paper, we introduce a hierarchical transformer (HT) scheme to efficiently learn the solution operator for multiscale PDEs.
Revised: In this paper, we introduce a hierarchical transformer (HT) scheme to efficiently learn the solution operator for multiscale PDEs.

Original: We construct a hierarchical architecture with a scale-adaptive interaction range, such that the features can be computed in a nested manner and with a controllable linear cost.
Revised: We construct a hierarchical architecture with a scale-adaptive interaction range, such that the features can be computed in a nested manner and with a controllable linear cost.

Original: Self-attentions over a hierarchy of levels can be used to encode and decode the multiscale solution space across all scales.
Revised: Self-attentions over a hierarchy of levels can be used to encode and decode the multiscale solution space across all scales.

Original: In addition, we adopt an empirical H 1 loss function to counteract the spectral bias of the neural network approximation for multiscale functions.
Revised: In addition, we adopt an empirical H 1 loss function to counteract the spectral bias of the neural network approximation for multiscale functions.

Original: In the numerical experiments, we demonstrate the superior performance of the HT scheme compared with state-of-the-art (SOTA) methods for representative multiscale problems.
Revised: In the numerical experiments, we demonstrate the superior performance of the HT scheme compared with state-of-the-art (SOTA) methods for representative multiscale problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Partial differential equation (PDE) models with multiple temporal/spatial scales are ubiquitous in physics, engineering, and other disciplines.
Revised: Partial differential equation (PDE) models with multiple temporal/spatial scales are ubiquitous in physics, engineering, and other disciplines.

Original: They are of tremendous importance in making predictions for challenging practical problems such as reservoir modeling, high-frequency scattering, and atmosphere circulation, to name a few.
Revised: They are of tremendous importance in making predictions for challenging practical problems such as reservoir modeling, high-frequency scattering, and atmosphere circulation, to name a few.

Original: The complex nonlinear interplays of characteristic scales cause major difficulties in the computer simulation of multiscale PDEs.
Revised: The complex nonlinear interplays of characteristic scales cause major difficulties in the computer simulation of multiscale PDEs.

Original: While the resolution of all characteristic scales is prohibitively expensive, sophisticated multiscale methods have been developed to efficiently and accurately solve the multiscale
Revised: While the resolution of all characteristic scales is prohibitively expensive, sophisticated multiscale methods have been developed to efficiently and accurately solve the multiscale

Original: PDEs by incorporating microscopic information.
Revised: PDEs by incorporating microscopic information.

Original: However, most of them are designed for problems with fixed input parameters.
Revised: However, most of them are designed for problems with fixed input parameters.

Original: Recently, several novel methods such as Fourier neural operator (FNO) (Li et al., 2021), Galerkin transformer (GT) (Cao, 2021) and deep operator network (DeepONet) (Lu et al., 2021) are developed to directly learn the operator (mapping) between infinite dimensional spaces for PDE problems, by taking advantages of the enhanced expressibility of deep neural networks and advanced architectures such as feature embedding, channel mixing, and self-attentions.
Revised: Recently, several novel methods such as Fourier neural operator (FNO) (Li et al., 2021), Galerkin transformer (GT) (Cao, 2021) and deep operator network (DeepONet) (Lu et al., 2021) are developed to directly learn the operator (mapping) between infinite dimensional spaces for PDE problems, by taking advantages of the enhanced expressibility of deep neural networks and advanced architectures such as feature embedding, channel mixing, and self-attentions.

Original: Such methods can deal with an ensemble of input parameters and have great potential for the efficient forward and inverse solvers of PDE problems.
Revised: Such methods can deal with an ensemble of input parameters and have great potential for the efficient forward and inverse solvers of PDE problems.

Original: However, for multiscale problems, most existing operator learning schemes essentially capture the smooth part of the solution space, and how to resolve the intrinsic multiscale features remains to be a major challenge.
Revised: However, for multiscale problems, most existing operator learning schemes essentially capture the smooth part of the solution space, and how to resolve the intrinsic multiscale features remains to be a major challenge.

Original: In this paper, we design a hierarchical transformer based operator learning method, so that the accurate, efficient, and robust computer simulation of multiscale PDE problems with an ensemble of input parameters becomes feasible.
Revised: In this paper, we design a hierarchical transformer based operator learning method, so that the accurate, efficient, and robust computer simulation of multiscale PDE problems with an ensemble of input parameters becomes feasible.

Original: Our main contribution can be summarized in the following,
Revised: Our main contribution can be summarized in the following,

Original: • we develop a novel transformer architecture that allows the decomposition of the input-output mapping to a hierarchy of levels, the features can be updated in a nested manner based on the hierarchical local aggregation of self-attentions with linear computational cost; • we adopt the empirical H 1 loss which avoids the spectral bias and enhances the ability to capture the oscillatory features of the multiscale solution space; • the resulting scheme has significantly better accuracy and generalization properties for multiscale input parameters, compared with state-of-the-art (SOTA) models.
Revised: • we develop a novel transformer architecture that allows the decomposition of the input-output mapping to a hierarchy of levels, the features can be updated in a nested manner based on the hierarchical local aggregation of self-attentions with linear computational cost; • we adopt the empirical H 1 loss which avoids the spectral bias and enhances the ability to capture the oscillatory features of the multiscale solution space; • the resulting scheme has significantly better accuracy and generalization properties for multiscale input parameters, compared with state-of-the-art (SOTA) models.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: We briefly introduce multiscale PDEs in Section § 2.1, then summarize relevant multiscale numerical methods in § 2.2, and neural solvers, in particular neural operators in § 2.3.
Revised: We briefly introduce multiscale PDEs in Section § 2.1, then summarize relevant multiscale numerical methods in § 2.2, and neural solvers, in particular neural operators in § 2.3.

Original: 2. M ULTISCALE PDE S
Revised: 2. M ULTISCALE PDE S

Original: Multiscale PDEs, in a narrower sense, refer to PDEs with rapidly varying coefficients, which may arise from a wide range of applications in heterogeneous and random media.
Revised: Multiscale PDEs, in a narrower sense, refer to PDEs with rapidly varying coefficients, which may arise from a wide range of applications in heterogeneous and random media.

Original: In a broader sense, they may form a hierarchy of models at different scales by a systematic derivation, starting from fundamental laws of physics such as quantum mechanics (E & Engquist, 2003).
Revised: In a broader sense, they may form a hierarchy of models at different scales by a systematic derivation, starting from fundamental laws of physics such as quantum mechanics (E & Engquist, 2003).

Original: Some outstanding multiscale PDE models may include:
Revised: Some outstanding multiscale PDE models may include:

Original: Multiscale elliptic equations is an important class of prototypical examples, such as the following secondorder elliptic equation of divergence form,
Revised: Multiscale elliptic equations is an important class of prototypical examples, such as the following secondorder elliptic equation of divergence form,

Original: [Equation]
Revised: [Equation]

Original: where 0 < a min ≤ a ( x ) ≤ a max , ∀ x ∈ D , and the forcing term f ∈ H − 1 ( D ; R ) .
Revised: where 0 < a min ≤ a ( x ) ≤ a max , ∀ x ∈ D , and the forcing term f ∈ H − 1 ( D ; R ) .

Original: The coefficient to solution map is S : L ∞
Revised: The coefficient to solution map is S : L ∞

Original: ( D ; R + )
Revised: ( D ; R + )

Original: → H 10 ( D ; R ) , such that u = S ( a ) .
Revised: → H 10 ( D ; R ) , such that u = S ( a ) .

Original: In (Li et al., 2021), smooth coefficient a ( x ) is considered and S can be well resolved by the FNO parameterization.
Revised: In (Li et al., 2021), smooth coefficient a ( x ) is considered and S can be well resolved by the FNO parameterization.

Original: The setup a
Revised: The setup a

Original: ( x ) ∈ L ∞ allows rough coefficients with fast oscillation (e.g. a ( x ) = a ( x/ε ) with ε ≪ 1 ), high contrast ratio with a max /a min ≫ 1 , and even a continuum of non-separable scales.
Revised: ( x ) ∈ L ∞ allows rough coefficients with fast oscillation (e.g. a ( x ) = a ( x/ε ) with ε ≪ 1 ), high contrast ratio with a max /a min ≫ 1 , and even a continuum of non-separable scales.

Original: The rough coefficient case is much harder from both scientific computing (Branets et al., 2009) and operator learning perspectives.
Revised: The rough coefficient case is much harder from both scientific computing (Branets et al., 2009) and operator learning perspectives.

Original: Navier-Stokes equation models the flow of incompressible fluids, which becomes turbulent due to the simultaneous interaction of a wide range of temporal and spatial scales of motion.
Revised: Navier-Stokes equation models the flow of incompressible fluids, which becomes turbulent due to the simultaneous interaction of a wide range of temporal and spatial scales of motion.

Original: Helmholtz equation models time-harmonic acoustic waves.
Revised: Helmholtz equation models time-harmonic acoustic waves.

Original: Its numerical solution exhibits severe difficulties in the high wave number regime due to the interaction of high-frequency waves and numerical mesh.
Revised: Its numerical solution exhibits severe difficulties in the high wave number regime due to the interaction of high-frequency waves and numerical mesh.

Original: 2. M ULTISCALE S OLVERS FOR M ULTISCALE PDE S
Revised: 2. M ULTISCALE S OLVERS FOR M ULTISCALE PDE S

Original: For multiscale PDEs, the computational cost of classical numerical methods, such as finite element methods, finite difference methods, etc., usually scales proportionally to 1 /ε ≫ 1 .
Revised: For multiscale PDEs, the computational cost of classical numerical methods, such as finite element methods, finite difference methods, etc., usually scales proportionally to 1 /ε ≫ 1 .

Original: Multiscale solvers have been developed such that their computational costs are independent of ε by incorporating microscopic information.
Revised: Multiscale solvers have been developed such that their computational costs are independent of ε by incorporating microscopic information.

Original: Asymptotic and numerical homogenization Asymptotic homogenization (Bensoussan et al., 1978) is an elegant analytical approach for multiscale PDEs with scale separation, e.g., a ( x ) = a ( x/ε ) with ε ≪ 1 in equation 2.1.
Revised: Asymptotic and numerical homogenization Asymptotic homogenization (Bensoussan et al., 1978) is an elegant analytical approach for multiscale PDEs with scale separation, e.g., a ( x ) = a ( x/ε ) with ε ≪ 1 in equation 2.1.

Original: For general multiscale PDEs with possibly a continuum of scales, numerical homogenization (Engquist & Souganidis, 2008) offers an effective numerical approach that aims to identify low dimensional approximation spaces such that the approximation bases are adapted to the corresponding multiscale operator, and can be efficiently constructed (e.g. with localized bases).
Revised: For general multiscale PDEs with possibly a continuum of scales, numerical homogenization (Engquist & Souganidis, 2008) offers an effective numerical approach that aims to identify low dimensional approximation spaces such that the approximation bases are adapted to the corresponding multiscale operator, and can be efficiently constructed (e.g. with localized bases).

Original: See Appendix A for details.
Revised: See Appendix A for details.

Original: Multilevel and multiresolution methods Multilevel/multigrid methods (Hackbusch, 1985; Xu &
Revised: Multilevel and multiresolution methods Multilevel/multigrid methods (Hackbusch, 1985; Xu &

Original: Zikatanov, 2017), and multiresolution methods such as wavelets (Brewster & Beylkin, 1995; Beylkin &
Revised: Zikatanov, 2017), and multiresolution methods such as wavelets (Brewster & Beylkin, 1995; Beylkin &

Original: Coult, 1998) have been successfully applied to PDEs.
Revised: Coult, 1998) have been successfully applied to PDEs.

Original: However, the convergence of those methods for multiscale problems can be severely affected by the regularity of coefficients (Branets et al., 2009).
Revised: However, the convergence of those methods for multiscale problems can be severely affected by the regularity of coefficients (Branets et al., 2009).

Original: Recently, the introduction of gamblets (Owhadi, 2017) can be seen as a multilevel extension of numerical homogenization, and it opens an avenue to automatically discover scalable multilevel algorithms and operator-adapted wavelets for linear PDEs with rough coefficients.
Revised: Recently, the introduction of gamblets (Owhadi, 2017) can be seen as a multilevel extension of numerical homogenization, and it opens an avenue to automatically discover scalable multilevel algorithms and operator-adapted wavelets for linear PDEs with rough coefficients.

Original: See also Appendix A.2.
Revised: See also Appendix A.2.

Original: Low-rank decomposition based methods It is well-known that the elliptic Green’s function has lowrank approximation (Bebendorf, 2005), which lays the theoretical foundation of the (near-)linear complexity methods such as fast multipole method (Greengard & Rokhlin, 

Revised: Low-rank decomposition based methods It is well-known that the elliptic Green’s function has lowrank approximation (Bebendorf, 2005), which lays the theoretical foundation of the (near-)linear complexity methods such as fast multipole method (Greengard & Rokhlin,

Original: ; Ying et al., 2004), hierarchical matrices ( H and H 2 matrices) (Hackbusch et al., 2002; Bebendorf, 2008), and hierarchical interpolative factorization (Ho & Ying, 2016).
Revised: ; Ying et al., 2004), hierarchical matrices ( H and H 2 matrices) (Hackbusch et al., 2002; Bebendorf, 2008), and hierarchical interpolative factorization (Ho & Ying, 2016).

Original: See Appendix B for the connection with our method presented in this paper.
Revised: See Appendix B for the connection with our method presented in this paper.

Original: Tensor numerical methods Analytical approaches such as periodic unfolding (Cioranescu et al., 2008) suggest that a low-dimensional multiscale PDE can be transformed into a high-dimensional PDE.
Revised: Tensor numerical methods Analytical approaches such as periodic unfolding (Cioranescu et al., 2008) suggest that a low-dimensional multiscale PDE can be transformed into a high-dimensional PDE.

Original: Therefore, tensor numerical methods, e.g., the sparse tensor finite element method (Harbrecht & Schwab, 2011) and the quantic tensor train (QTT) method (Kazeev et al., 2022), provide efficient numerical procedures to find the low-rank tensor representation of multiscale solutions.
Revised: Therefore, tensor numerical methods, e.g., the sparse tensor finite element method (Harbrecht & Schwab, 2011) and the quantic tensor train (QTT) method (Kazeev et al., 2022), provide efficient numerical procedures to find the low-rank tensor representation of multiscale solutions.

Original: 2. N EURAL O PERATOR FOR PDE S
Revised: 2. N EURAL O PERATOR FOR PDE S

Original: Neural PDE solvers and Spectral Bais Various neural solvers have been proposed in (E et al., 2017;
Revised: Neural PDE solvers and Spectral Bais Various neural solvers have been proposed in (E et al., 2017;

Original: Sirignano & Spiliopoulos, 2018; Raissi et al., 2019) to solve PDEs with fixed parameters.
Revised: Sirignano & Spiliopoulos, 2018; Raissi et al., 2019) to solve PDEs with fixed parameters.

Original: The difficulty to solve multiscale PDEs has been exhibited by the so-called spectral bias or frequency principle (Rahaman et al., 2019; Ronen et al., 2019), which shows that DNN-based algorithms are often inefficient to learn highfrequency components of multiscale functions.
Revised: The difficulty to solve multiscale PDEs has been exhibited by the so-called spectral bias or frequency principle (Rahaman et al., 2019; Ronen et al., 2019), which shows that DNN-based algorithms are often inefficient to learn highfrequency components of multiscale functions.

Original: A series of algorithms have been developed to overcome the high-frequency curse of DNNs and to solve multiscale PDEs (Cai et al., 2020; Wang et al., 2021).
Revised: A series of algorithms have been developed to overcome the high-frequency curse of DNNs and to solve multiscale PDEs (Cai et al., 2020; Wang et al., 2021).

Original: Operator learning for PDEs DNN algorithms demonstrate more potential to learn the input-output mapping of parametric PDEs.
Revised: Operator learning for PDEs DNN algorithms demonstrate more potential to learn the input-output mapping of parametric PDEs.

Original: Finite-dimensional operator learning methods such as (Zhu & Zabaras, 2018; Fan et al., 2019a;b; Khoo et al., 2020) can be applied to problems with fixed discretization.
Revised: Finite-dimensional operator learning methods such as (Zhu & Zabaras, 2018; Fan et al., 2019a;b; Khoo et al., 2020) can be applied to problems with fixed discretization.

Original: In particular, (Fan et al., 2019a;b) combines H or H 2 matrices linear operations with nonlinear activation functions, though the nonlinear geometrical interaction/aggregation is absent, which limits the expressivity of the resulting neural operator.
Revised: In particular, (Fan et al., 2019a;b) combines H or H 2 matrices linear operations with nonlinear activation functions, though the nonlinear geometrical interaction/aggregation is absent, which limits the expressivity of the resulting neural operator.

Original: Infinite-dimensional operator learning methods (Li et al., 2021; Gupta et al., 2021) aim to learn the mapping between infinite-dimensional Banach spaces, and the convolutions in the neural operator construction are parametrized by Fourier or wavelet transform.
Revised: Infinite-dimensional operator learning methods (Li et al., 2021; Gupta et al., 2021) aim to learn the mapping between infinite-dimensional Banach spaces, and the convolutions in the neural operator construction are parametrized by Fourier or wavelet transform.

Original: Nevertheless, those methods do not always work even for multiscale PDEs with fixed parameters.
Revised: Nevertheless, those methods do not always work even for multiscale PDEs with fixed parameters.

Original: On the other hand, while the universal approximation can be rigorously proved for FNO operators (Kovachki et al., 2021), “extra smoothness” is required to achieve a meaningful decay rate, which either is absent or gives rise to large constants for multiscale PDEs.
Revised: On the other hand, while the universal approximation can be rigorously proved for FNO operators (Kovachki et al., 2021), “extra smoothness” is required to achieve a meaningful decay rate, which either is absent or gives rise to large constants for multiscale PDEs.

Original: This motivates us to construct a new architecture for multiscale operator learning.
Revised: This motivates us to construct a new architecture for multiscale operator learning.

Original: Efficient Attention
Revised: Efficient Attention

Original: The vanilla multi-head self-attention in (Vaswani et al., 2017) scales quadratically with the number of tokens, which is prohibitive for high-resolution problems.
Revised: The vanilla multi-head self-attention in (Vaswani et al., 2017) scales quadratically with the number of tokens, which is prohibitive for high-resolution problems.

Original: Many efficient attention methods are proposed using the kernel trick (low-rank projection) (Choromanski et al., 2020;
Revised: Many efficient attention methods are proposed using the kernel trick (low-rank projection) (Choromanski et al., 2020;

Original: Wang et al., 2020; Peng et al., 2021; Nguyen et al., 2021) to reduce the computation cost arising in the dense matrix operation.
Revised: Wang et al., 2020; Peng et al., 2021; Nguyen et al., 2021) to reduce the computation cost arising in the dense matrix operation.

Original: In the context of operator learning, Galerkin transformer (Cao, 2021) proposed to remove the softmax normalization in self-attention and introduced a linearized self-attention variant with Petrov-Galerkin projection normalization.
Revised: In the context of operator learning, Galerkin transformer (Cao, 2021) proposed to remove the softmax normalization in self-attention and introduced a linearized self-attention variant with Petrov-Galerkin projection normalization.

Original: Furthermore, hierarchical transformers using local window aggregation are proposed in Liu et al.
Revised: Furthermore, hierarchical transformers using local window aggregation are proposed in Liu et al.

Original: (2021b)
Revised: (2021b)

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2022) for NLP and vision applications.
Revised: (2022) for NLP and vision applications.

Original:  M ETHODS
Revised:  M ETHODS

Original: We introduce the hierarchical attention model in this section, motivated by the hierarchical matrix, in particular the H 2 matrix variant.
Revised: We introduce the hierarchical attention model in this section, motivated by the hierarchical matrix, in particular the H 2 matrix variant.

Original: We follow the setup in (Li et al., 2021; Lu et al., 2021) to approximate the operator S : a (cid:55)→ u := S ( a ) , with the input a ∈ A drawn from a distribution µ and the output u ∈ U , where A and U are infinite dimensional Banach spaces respectively.
Revised: We follow the setup in (Li et al., 2021; Lu et al., 2021) to approximate the operator S : a (cid:55)→ u := S ( a ) , with the input a ∈ A drawn from a distribution µ and the output u ∈ U , where A and U are infinite dimensional Banach spaces respectively.

Original: We aim to learn the operator S from a finite collection of finitely observed input-output pairs through a parametric map N : A× Θ → U and a loss functional L : U × U → R , such that the optimal parameter θ ∗ =
Revised: We aim to learn the operator S from a finite collection of finitely observed input-output pairs through a parametric map N : A× Θ → U and a loss functional L : U × U → R , such that the optimal parameter θ ∗ =

Original: arg min θ ∈ Θ E a ∼ µ [ L ( N ( a , θ ) , S ( a ))] .
Revised: arg min θ ∈ Θ E a ∼ µ [ L ( N ( a , θ ) , S ( a ))] .

Original: The input a can then be mapped to features f through patch embedding (Dosovitskiy et al., 2020).
Revised: The input a can then be mapped to features f through patch embedding (Dosovitskiy et al., 2020).

Original: Hierarchical Discretization We introduce the hierarchical discretization of the spatial domain D , which can be used directly for time-independent PDEs.
Revised: Hierarchical Discretization We introduce the hierarchical discretization of the spatial domain D , which can be used directly for time-independent PDEs.

Original: Time-dependent PDEs can be treated by taking time-sliced data as feature channels.
Revised: Time-dependent PDEs can be treated by taking time-sliced data as feature channels.

Original: Let I ( r ) be the finest level index set,
Revised: Let I ( r ) be the finest level index set,

Original: such that each index i =
Revised: such that each index i =

Original: ( i 1 , . .
Revised: ( i 1 , . .

Original: , i r ) ∈
Revised: , i r ) ∈

Original: I ( r ) denotes the finest level spatial objects such as image pixels, finite difference points, etc.
Revised: I ( r ) denotes the finest level spatial objects such as image pixels, finite difference points, etc.

Original: For any i
Revised: For any i

Original: = ( i 1 , . .
Revised: = ( i 1 , . .

Original: , i r ) ∈
Revised: , i r ) ∈

Original: I ( r ) , and 1 ≤ m ≤ r , i ( m ) =
Revised: I ( r ) , and 1 ≤ m ≤ r , i ( m ) =

Original: ( i 1 , . .
Revised: ( i 1 , . .

Original: , i m ) represents i
Revised: , i m ) represents i

Original: ’s m -th
Revised: ’s m -th

Original: level parent node which is the aggregate of finer level objects.
Revised: level parent node which is the aggregate of finer level objects.

Original: I ( m ) :
Revised: I ( m ) :

Original: = { i ( m ) : i ∈ I ( r ) } is the m -th level index set.
Revised: = { i ( m ) : i ∈ I ( r ) } is the m -th level index set.

Original: We can similarly define the natural parent-child relationship between coarser and finer nodes, which induces the index tree I together with the index sets I ( m ) (1 ≤ m ≤ r ) .
Revised: We can similarly define the natural parent-child relationship between coarser and finer nodes, which induces the index tree I together with the index sets I ( m ) (1 ≤ m ≤ r ) .

Original: Each index i ∈ I ( m ) corresponds to a token, which can be characterized e.g. by its position x ( m ) i and a feature vector f ( m ) i with number of channels C ( m ) .
Revised: Each index i ∈ I ( m ) corresponds to a token, which can be characterized e.g. by its position x ( m ) i and a feature vector f ( m ) i with number of channels C ( m ) .

Original: In the following, we will restrict our presentation to the quadtree setting as illustrated in Fig 3.1.
Revised: In the following, we will restrict our presentation to the quadtree setting as illustrated in Fig 3.1.

Original: Reduce Operation
Revised: Reduce Operation

Original: The reduce operation defines the map from finer-level features to coarser-level features.
Revised: The reduce operation defines the map from finer-level features to coarser-level features.

Original: For i ∈ I ( m ) , we denote i ( m,m +1) as the set of ( m + 1) -th level child nodes of i .
Revised: For i ∈ I ( m ) , we denote i ( m,m +1) as the set of ( m + 1) -th level child nodes of i .

Original: In the quadtree setting, i ( m,m +1) = { ( i, 0) , ( i, 1) , ( i, 2) , ( i, 3) } 1 .
Revised: In the quadtree setting, i ( m,m +1) = { ( i, 0) , ( i, 1) , ( i, 2) , ( i, 3) } 1 .

Original: The reduce map can be abstractly defined as f ( m )  i 
Revised: The reduce map can be abstractly defined as f ( m ) ,t i =

Original: [Equation] i
Revised: R ( m ) ( { f ( m +1) ,

Original: [Equation]
Revised: [Equation]

Original:  m,m +1) ) , which maps the ( m + 1) -th level features with indices in i ( m,m +1) to the m -th level feature with index i ( m ) .
Revised: j } j ∈ i ( m,m +1) ) , which maps the ( m +1) -th level features with indices in i ( m,m +1) to the m -th level feature with index i ( m ) .

Original: For simplicity, we use a simple linear layer for the operator R ( m ) in our current implementation, namely, f ( m )  i
Revised: For simplicity, we use a simple linear layer for the operator R ( m ) in our current implementation, namely, f ( m ) ,t i

Original: [Equation]
Revised: [Equation]

Original: where R ( m ) 0 , R ( m ) 1 , R ( m ) 2 , R ( m )∈ R C ( m − 1) ×C ( m ) are parametrized by linear layers.
Revised: where R ( m ) 0 , R ( m ) 1 , R ( m ) 2 , R ( m )∈ R C ( m − 1) ×C ( m ) are parametrized by linear layers.

Original: From the perspective of H matrices, those parametrized matrices R ( m ) j embed low-rank approximation to the attention kernel.
Revised: From the perspective of H matrices, those parametrized matrices R ( m ) j embed low-rank approximation to the attention kernel.

Original: Please refer to Appendix B for more details.
Revised: Please refer to Appendix B for more details.

Original: Multilevel Local Aggregation
Revised: Multilevel Local Aggregation

Original: The finest level features f ( r ) ,t i ∈ R C ( r ) at the evolution step t , with index i ∈ I ( r ) , can be updated by the following token aggregation formula through vanilla attention, atten : f ( r ) ,t +
Revised: The finest level features f ( r ) ,t i ∈ R C ( r ) at the evolution step t , with index i ∈ I ( r ) , can be updated by the following token aggregation formula through vanilla attention, atten : f ( r ) ,t +1 i =

Original: [Equation]
Revised: N ( r ) (cid:88) j =

Original:  i [Equation] f ( r ) ,t j ) , for i ∈ I ( r ) , (3.1) with N ( r ) :
Revised: G ( f ( r ) ,t i , f ( r ) ,t j ) v ( f ( r ) ,t j ) , for i ∈ I ( r ) , (3.1) with N ( r ) :

Original: = |I
Revised: = |I

Original: ( r ) | .
Revised: ( r ) | .

Original: For simplicity, we ignore the softmax normalizing factor and only consider single-head attention, and we assume that the interaction kernel G is of the form G ( f i , f j ) =
Revised: For simplicity, we ignore the softmax normalizing factor and only consider single-head attention, and we assume that the interaction kernel G is of the form G ( f i , f j ) =

Original: exp( W Q f i · W K f j ) =
Revised: exp( W Q f i · W K f j ) =

Original: exp( q i · k j ) , where q i :=
Revised: exp( q i · k j ) , where q i :=

Original: W Q f i , k i :=
Revised: W Q f i , k i :=

Original: W K f i , v i := v ( f i ) =
Revised: W K f i , v i := v ( f i ) =

Original: W V f i , and W Q , W K , W V ∈ R C ( r ) ×C ( r ) are learnable matrices.
Revised: W V f i , and W Q , W K , W V ∈ R C ( r ) ×C ( r ) are learnable matrices.

Original: Instead of computing equation 3.1 explicitly with O ( N 2 ) cost, we propose a self-attention based local aggregation scheme, inspired by the H 2 matrices (Hackbusch, 2015), see also
Revised: Instead of computing equation 3.1 explicitly with O ( N 2 ) cost, we propose a self-attention based local aggregation scheme, inspired by the H 2 matrices (Hackbusch, 2015), see also

Original: Appendix B.
Revised: Appendix B.

Original: The local aggregation at the r -th level writes,
Revised: The local aggregation at the r -th level writes,

Original: Figure dex tree.
Revised: Figure dex tree.

Original: In this example, the 2D unit square is hierarchically discretized into three levels which are indexed by I (1) , I (2) and I (3) , respectively.
Revised: In this example, the 2D unit square is hierarchically discretized into three levels which are indexed by I (1) , I (2) and I (3) , respectively.

Original: For example, we denote by (1) (1 , 2) =
Revised: For example, we denote by (1) (1 , 2) =

Original: { (1 , 0) , (1 , 1) , (1 , 2) , (1 , 3) } the set of the second level child nodes of the node (1) .
Revised: { (1 , 0) , (1 , 1) , (1 , 2) , (1 , 3) } the set of the second level child nodes of the node (1) .

Original: [Equation]
Revised: [Equation]

Original: where N ( r ) ( i ) is the set of the r -th level neighbors of i
Revised: where N ( r ) ( i ) is the set of the r -th level neighbors of i

Original: ∈ I ( r ) , q ( r ) ,t i
Revised: ∈ I ( r ) , q ( r ) ,t i

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: K f
Revised: K f

Original: [Equation]
Revised: [Equation]

Original: v ( r ) ,t i :
Revised: v ( r ) ,t i :

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: with learnable matrices ˆ W Q , ˆ W K , ˆ W V ∈ R C ( r ) ×C ( r ) .
Revised: with learnable matrices ˆ W Q , ˆ W K , ˆ W V ∈ R C ( r ) ×C ( r ) .

Original: Then for m = r − 1 , ..., 1 , we calculate the local attention in each level with nested q ( m ) ,t
Revised: Then for m = r − 1 , ..., 1 , we calculate the local attention in each level with nested q ( m ) ,t

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where q ( m ) ,t i =
Revised: where q ( m ) ,t i =

Original: R ( m ) ( { q ( m +1) ,t j ) } j ∈
Revised: R ( m ) ( { q ( m +1) ,t j ) } j ∈

Original: i
Revised: i

Original: ( m,m +1) ) , k ( m ) ,t i =
Revised: ( m,m +1) ) , k ( m ) ,t i =

Original: R ( m ) ( { k ( m +1) ,t j ) } j ∈
Revised: R ( m ) ( { k ( m +1) ,t j ) } j ∈

Original: i
Revised: i

Original: ( m,m +1) ) , v ( m ) ,t i =
Revised: ( m,m +1) ) , v ( m ) ,t i =

Original: R ( m ) ( { v ( m +1) ,t j ) } j ∈
Revised: R ( m ) ( { v ( m +1) ,t j ) } j ∈

Original: i
Revised: i

Original: ( m,m +1) ) , N ( m ) ( i ) is the set of the m -th level neighbors of i ∈
Revised: ( m,m +1) ) , N ( m ) ( i ) is the set of the m -th level neighbors of i ∈

Original: I ( m ) .
Revised: I ( m ) .

Original: Decompose Operation To mix the multilevel features f ( m ) ,t i , m = 1 , ..., r , we propose a decompose operation that reverses the reduce operation.
Revised: Decompose Operation To mix the multilevel features f ( m ) ,t i , m = 1 , ..., r , we propose a decompose operation that reverses the reduce operation.

Original: The decompose operator D ( m ) : f ( m ) ,t i (cid:55)→ { f ( m +1) ,t + 12 j } j ∈
Revised: The decompose operator D ( m ) : f ( m ) ,t i (cid:55)→ { f ( m +1) ,t + 12 j } j ∈

Original: i
Revised: i

Original: ( m,m +1) , maps the m -th level feature with index i to ( m + 1) -th level features associated to its child set i ( m,m +1) .
Revised: ( m,m +1) , maps the m -th level feature with index i to ( m + 1) -th level features associated to its child set i ( m,m +1) .

Original: In the current implementation, we use a simple linear layer such that f ( m +1) ,t + 12 ( i,j ) =
Revised: In the current implementation, we use a simple linear layer such that f ( m +1) ,t + 12 ( i,j ) =

Original: D ( m ) ,T j f ( m ) ,t i , for j = 0 , 1 , 2 , 3 , with parameter matrices D ( m ) j ∈ R C ( m ) ×C ( m +1) .
Revised: D ( m ) ,T j f ( m ) ,t i , for j = 0 , 1 , 2 , 3 , with parameter matrices D ( m ) j ∈ R C ( m ) ×C ( m +1) .

Original: f ( m +1) ,t + 1 

Revised: f ( m +1) ,t + 1

Original: j is further aggregated to ¯ f ( m +1) ,t +1 j such that f ( m +1) ,t +1 j = ¯ f ( m +1) ,t +1 j + f ( m +1) ,t + 1 2 j .
Revised: j is further aggregated to ¯ f ( m +1) ,t +1 j such that f ( m +1) ,t +1 j = ¯ f ( m +1) ,t +1 j + f ( m +1) ,t + 1 2 j .

Original: Remark 

Revised: Remark

Original: In our current implementation, the operators R ( m ) and D ( m ) only consist of linear layers.
Revised: In our current implementation, the operators R ( m ) and D ( m ) only consist of linear layers.

Original: In general, these operators may contain nonlinear activation functions.
Revised: In general, these operators may contain nonlinear activation functions.

Original: Remark 

Revised: Remark

Original: The nested learnable operators R ( m ) and R ( m ) induce the channel mixing and a structured parameterization of W Q , W V , W K matrix for the coarse level tokens.
Revised: The nested learnable operators R ( m ) and R ( m ) induce the channel mixing and a structured parameterization of W Q , W V , W K matrix for the coarse level tokens.

Original: See Appendix B for details.
Revised: See Appendix B for details.

Original: We summarize the hierarchical attention algorithm in the following.
Revised: We summarize the hierarchical attention algorithm in the following.

Original: Proposition 3.1 (Complexity of Algorithm 1)
Revised: Proposition 3.1 (Complexity of Algorithm 1)

Original: The reduce operation, multilevel aggregation, and decomposition operation together form a V-cycle for the update of features, as illustrated in Figure 3.2.
Revised: The reduce operation, multilevel aggregation, and decomposition operation together form a V-cycle for the update of features, as illustrated in Figure 3.2.

Original: The cost of one V-cycle is O ( N ) if I is a quadtree as implemented in the paper.
Revised: The cost of one V-cycle is O ( N ) if I is a quadtree as implemented in the paper.

Original: See Appendix C for the proof.
Revised: See Appendix C for the proof.

Original: Remark 3 (Technical contribution)
Revised: Remark 3 (Technical contribution)

Original: In Liu et al.
Revised: In Liu et al.

Original: (2021b)
Revised: (2021b)

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2022), spatial resolutions are sequentially reduced, and attentions are performed at each level separately.
Revised: (2022), spatial resolutions are sequentially reduced, and attentions are performed at each level separately.

Original: However, attentions are only used for a fixed level, and there is no multilevel attention-based aggregation.
Revised: However, attentions are only used for a fixed level, and there is no multilevel attention-based aggregation.

Original: The coarse latent variables are used for classification and generation tasks, and fine-scale information may get lost in the reduce(coarsening) process.
Revised: The coarse latent variables are used for classification and generation tasks, and fine-scale information may get lost in the reduce(coarsening) process.

Original: While in the HT-Net architecture, we are motivated by multiscale numerical methods such as numerical homogenization and hierarchical matrix method.
Revised: While in the HT-Net architecture, we are motivated by multiscale numerical methods such as numerical homogenization and hierarchical matrix method.

Original: Attention-based local aggregations are constructed at each level, and features from all levels are summed up to form the updated fine-scale features, which enables the recovery of the fine details with linear cost.
Revised: Attention-based local aggregations are constructed at each level, and features from all levels are summed up to form the updated fine-scale features, which enables the recovery of the fine details with linear cost.

Original: Decoder The decoder maps the features f (at the last update step) to the solution u .
Revised: Decoder The decoder maps the features f (at the last update step) to the solution u .

Original: The decoder is often chosen with prior knowledge of the PDE.
Revised: The decoder is often chosen with prior knowledge of the PDE.

Original: A simple feedforward neural network (FFN) is used in Lu et al.
Revised: A simple feedforward neural network (FFN) is used in Lu et al.

Original: (2021) to learn a basis set as the decoder.
Revised: (2021) to learn a basis set as the decoder.

Original: A good decoder can also be learned from the data using SVD (Bhattacharya et al., 2021).
Revised: A good decoder can also be learned from the data using SVD (Bhattacharya et al., 2021).

Original: In this paper, we use the spectral convolution layers in Li et al.
Revised: In this paper, we use the spectral convolution layers in Li et al.

Original: Loss functions Loss functions are crucial for efficient training and robust generalization of neural network models.
Revised: Loss functions Loss functions are crucial for efficient training and robust generalization of neural network models.

Original: For multiscale problems considered in this paper, we prefer to use H 1 loss instead of the usual Lloss function, as it puts more ”weights” on high-frequency components.
Revised: For multiscale problems considered in this paper, we prefer to use H 1 loss instead of the usual Lloss function, as it puts more ”weights” on high-frequency components.

Original: (Yu et al., 2022) adopted Sobolev norm based loss function for function approximation, where H 1 loss is defined for the neural network function itself.
Revised: (Yu et al., 2022) adopted Sobolev norm based loss function for function approximation, where H 1 loss is defined for the neural network function itself.

Original: On the contrary, we define the H 1 loss on the target solution space, which measures the distance between the prediction ˆ u :=
Revised: On the contrary, we define the H 1 loss on the target solution space, which measures the distance between the prediction ˆ u :=

Original: N ( a ) and the ground truth u .
Revised: N ( a ) and the ground truth u .

Original: We refer the readers to Adams & Fournier (2003) for the definition of Sobolev spaces and H 1 norm, and to the next section for the implementation.
Revised: We refer the readers to Adams & Fournier (2003) for the definition of Sobolev spaces and H 1 norm, and to the next section for the implementation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We demonstrate the effectiveness of the hierarchical transformer model (HT-Net) on multiscale operator learning.
Revised: We demonstrate the effectiveness of the hierarchical transformer model (HT-Net) on multiscale operator learning.

Original: All our examples are in 2D. We first study the ability of the model on a multiscale elliptic equation with two-phase coefficients which is a widely used benchmark problem for operator learning (Li et al., 2021).
Revised: All our examples are in 2D. We first study the ability of the model on a multiscale elliptic equation with two-phase coefficients which is a widely used benchmark problem for operator learning (Li et al., 2021).

Original: As the “multiscaleness” of the coefficients such as the oscillation, contrast, and roughness of the two-phase interface can be fine-tuned, we showcase the accuracy and the robustness of the HT-NET model for both smooth and rough coefficients.
Revised: As the “multiscaleness” of the coefficients such as the oscillation, contrast, and roughness of the two-phase interface can be fine-tuned, we showcase the accuracy and the robustness of the HT-NET model for both smooth and rough coefficients.

Original: We also show that HT-NET is able to provide better generalization errors for out-of-distribution input parameters.
Revised: We also show that HT-NET is able to provide better generalization errors for out-of-distribution input parameters.

Original: We also test the Navier-Stokes equation with a large Reynolds number as an example with non-linearity and time dependence.
Revised: We also test the Navier-Stokes equation with a large Reynolds number as an example with non-linearity and time dependence.

Original: 4. S ETUP
Revised: 4. S ETUP

Original: In our experiments, the spatial domain is D :=
Revised: In our experiments, the spatial domain is D :=

Original: [0 , 1] 2 and is discretized uniformly with h = 1 /n .
Revised: [0 , 1] 2 and is discretized uniformly with h = 1 /n .

Original: Let the uniform grid be G 2 :
Revised: Let the uniform grid be G 2 :

Original: = { ( x i , x j ) =
Revised: = { ( x i , x j ) =

Original: ( ih, jh ) | i, j = 0 , ..., n − 1 } .
Revised: ( ih, jh ) | i, j = 0 , ..., n − 1 } .

Original: { ( a j , u j ) } Nj =1 are functions pairs such that u j = S ( a j ) , and a j is drown from some probability measure µ .
Revised: { ( a j , u j ) } Nj =1 are functions pairs such that u j = S ( a j ) , and a j is drown from some probability measure µ .

Original: The actual data pairs for training and testing are pointwise evaluations of a j and u j on the grid G 2 , denoted by a j and u j , respectively.
Revised: The actual data pairs for training and testing are pointwise evaluations of a j and u j on the grid G 2 , denoted by a j and u j , respectively.

Original: The comparison with all the baselines is consistent with (most time better than) references.
Revised: The comparison with all the baselines is consistent with (most time better than) references.

Original: The hierarchical index tree I can be generated by the corresponding quadtree representation of the nodes with depth r , such that the finest level objects are pixels or patches aggregated by pixels.
Revised: The hierarchical index tree I can be generated by the corresponding quadtree representation of the nodes with depth r , such that the finest level objects are pixels or patches aggregated by pixels.

Original: See Appendix D for more details.
Revised: See Appendix D for more details.

Original: Empirical H 1 loss
Revised: Empirical H 1 loss

Original: function
Revised: function

Original: Empirical L 2 loss function is defined as L L ( { ( a j , u j ) } Nj =1 ; θ ) :
Revised: Empirical L 2 loss function is defined as L L ( { ( a j , u j ) } Nj =1 ; θ ) :

Original: = 1 N (cid:80) Ni =1 ∥
Revised: = 1 N (cid:80) Ni =1 ∥

Original: u j − N ( a j ; θ ) ∥
Revised: u j − N ( a j ; θ ) ∥

Original: l 2 / ∥ u j ∥
Revised: l 2 / ∥ u j ∥

Original: l 2 , where ∥ · ∥ l 2 is the canonical l 2 vector norm.
Revised: l 2 , where ∥ · ∥ l 2 is the canonical l 2 vector norm.

Original: For any ξ ∈
Revised: For any ξ ∈

Original: Z 2 n := (cid:8)
Revised: Z 2 n := (cid:8)

Original: ξ ∈
Revised: ξ ∈

Original: Z 2 | − n/ 2 + 1 ⩽ ξ j ⩽ n/ 2 , j = 1 , 2 (cid:9) , the normalized discrete Fourier transform (DFT) coefficients of f writes F ( f )( ξ ) :
Revised: Z 2 | − n/ 2 + 1 ⩽ ξ j ⩽ n/ 2 , j = 1 , 2 (cid:9) , the normalized discrete Fourier transform (DFT) coefficients of f writes F ( f )( ξ ) :

Original: = 1 √ n (cid:80) x ∈ G 2 f ( x ) e − 2 iπx · ξ .
Revised: = 1 √ n (cid:80) x ∈ G 2 f ( x ) e − 2 iπx · ξ .

Original: The empirical H 1 loss function is given by,
Revised: The empirical H 1 loss function is given by,

Original: L H ( { ( a j , u j ) } Nj =1 ; θ ) :
Revised: L H ( { ( a j , u j ) } Nj =1 ; θ ) :

Original: = 1 N (cid:80) Ni =1 ∥
Revised: = 1 N (cid:80) Ni =1 ∥

Original: u j − N ( a j ; θ ) ∥
Revised: u j − N ( a j ; θ ) ∥

Original: h / ∥
Revised: h / ∥

Original: u j ∥ h , where ∥ u ∥ h := (cid:113)(cid:80) ξ ∈ Z 2 n | ξ | 2 ( F ( u )( ξ )) 2 .
Revised: u j ∥ h , where ∥ u ∥ h := (cid:113)(cid:80) ξ ∈ Z 2 n | ξ | 2 ( F ( u )( ξ )) 2 .

Original: L H can be viewed as a weighted L L loss with weights | ξ | 2 , which forces the operator to capture highfrequency components in the solution.
Revised: L H can be viewed as a weighted L L loss with weights | ξ | 2 , which forces the operator to capture highfrequency components in the solution.

Original: In this work, we use the equivalent frequency space representation of the discrete H 1 norm, and in general, discrete H 1 norm in real space can also be adopted.
Revised: In this work, we use the equivalent frequency space representation of the discrete H 1 norm, and in general, discrete H 1 norm in real space can also be adopted.

Original: 4. M ULTISCALE E LLIPTIC E QUATION
Revised: 4. M ULTISCALE E LLIPTIC E QUATION

Original: [Figure]
Revised: [Figure]

Original: We apply the HT-Net model to learn the coefficient to solution operator for equation 2.1.
Revised: We apply the HT-Net model to learn the coefficient to solution operator for equation 2.1.

Original: We use the twophase coefficient model in Li et al.
Revised: We use the twophase coefficient model in Li et al.

Original: (2021), which is also benchmarked in Gupta et al.
Revised: (2021), which is also benchmarked in Gupta et al.

Original: (2021); Cao (2021).
Revised: (2021); Cao (2021).

Original: In previous works, the coefficients do not oscillate much and the solutions look smooth.
Revised: In previous works, the coefficients do not oscillate much and the solutions look smooth.

Original: We change the parameters which control the smoothness and contrast of the coefficients, such that the solutions contain more roughness than the Darcy benchmark in Li et al.
Revised: We change the parameters which control the smoothness and contrast of the coefficients, such that the solutions contain more roughness than the Darcy benchmark in Li et al.

Original: (2021), see Figure 4.1.
Revised: (2021), see Figure 4.1.

Original: We refer the readers to Appendix
Revised: We refer the readers to Appendix

Original: E for data generation details.
Revised: E for data generation details.

Original: We also include experiments for multiscale trigonometric coefficients with higher contrast.
Revised: We also include experiments for multiscale trigonometric coefficients with higher contrast.

Original: Detailed description on multiscale trigonometric coefficients is in Appendix F.1.
Revised: Detailed description on multiscale trigonometric coefficients is in Appendix F.1.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Spectral Bias in Operator Learning
Revised: Spectral Bias in Operator Learning

Original: We demonstrate the spectral bias phenomena in the neural operator training in Figure 4.2.
Revised: We demonstrate the spectral bias phenomena in the neural operator training in Figure 4.2.

Original: We compare HT-Net trained with H 1 and L 2 losses, and show the evolution of errors in the frequency domain as well as the loss curves.
Revised: We compare HT-Net trained with H 1 and L 2 losses, and show the evolution of errors in the frequency domain as well as the loss curves.

Original: The comparison shows that HT-Net with H 1 loss has faster decay for high frequencies, and the errors over all the frequencies decay more uniformly.
Revised: The comparison shows that HT-Net with H 1 loss has faster decay for high frequencies, and the errors over all the frequencies decay more uniformly.

Original: The HT-Net trained with H 1 loss also has better testing and generalization errors.
Revised: The HT-Net trained with H 1 loss also has better testing and generalization errors.

Original: Generalization Errors HT-Net generalizes better on test data from samples out of distribution.
Revised: Generalization Errors HT-Net generalizes better on test data from samples out of distribution.

Original: In addition to what we have shown in (c,d) of Figure 4.2, we compare HT-Net with other models in Table 2.
Revised: In addition to what we have shown in (c,d) of Figure 4.2, we compare HT-Net with other models in Table 2.

Original: The models are trained with the rough two-phase dataset in Table 4.1 and tested on out-of-distribution data.
Revised: The models are trained with the rough two-phase dataset in Table 4.1 and tested on out-of-distribution data.

Original: FNO2D His FNO model but trained using our H 1 loss.
Revised: FNO2D His FNO model but trained using our H 1 loss.

Original: HT-Net outperforms other models by order of magnitude.
Revised: HT-Net outperforms other models by order of magnitude.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 4. N AVIER -S TOKES E QUATION
Revised: 4. N AVIER -S TOKES E QUATION

Original: We consider the 2D Navier-Stokes equation in vorticity form on the unit torus, which is also benchmarked in Li et al.
Revised: We consider the 2D Navier-Stokes equation in vorticity form on the unit torus, which is also benchmarked in Li et al.

Original: [Equation]
Revised: [Equation]

Original: with velocity u , vorticity w = ∇ × u , initial vorticity w 0 , viscosity ν > 0 ( ∼ Re − 1 with Re being the
Revised: with velocity u , vorticity w = ∇ × u , initial vorticity w 0 , viscosity ν > 0 ( ∼ Re − 1 with Re being the

Original: Reynolds number), and forcing term f .
Revised: Reynolds number), and forcing term f .

Original: We learn the operator S : w ( · , 0 ≤ t ≤ 10) → w ( · , 10 ≤ t ≤ T ) , mapping the vorticity up to time 10 to the vorticity up to some later time T > 10 .
Revised: We learn the operator S : w ( · , 0 ≤ t ≤ 10) → w ( · , 10 ≤ t ≤ T ) , mapping the vorticity up to time 10 to the vorticity up to some later time T > 10 .

Original: We experiment with viscosities ν = 1e − 3 , 1e − 4 , 1e − 5 , and decrease the final time T accordingly as the Reynolds numbers increase and the dynamics become chaotic.
Revised: We experiment with viscosities ν = 1e − 3 , 1e − 4 , 1e − 5 , and decrease the final time T accordingly as the Reynolds numbers increase and the dynamics become chaotic.

Original: Time dependent neural operator Following the setup in Li et al.
Revised: Time dependent neural operator Following the setup in Li et al.

Original: (2021), we fix the resolution as 64 ×for both training and testing.
Revised: (2021), we fix the resolution as 64 ×for both training and testing.

Original: The ten time-slices of solutions w ( · , t ) at t = 0 , ..., 9 are taken as the input data to the neural operator N which maps the solutions at the previous 10 timesteps to the next time step (2D functions to 2D functions).
Revised: The ten time-slices of solutions w ( · , t ) at t = 0 , ..., 9 are taken as the input data to the neural operator N which maps the solutions at the previous 10 timesteps to the next time step (2D functions to 2D functions).

Original: This procedure, often referred to as the rolled-out prediction, can be repeated recurrently until the final time T .
Revised: This procedure, often referred to as the rolled-out prediction, can be repeated recurrently until the final time T .

Original: We list the results of HT-Net, FNO-3D (convolution in space-time),
Revised: We list the results of HT-Net, FNO-3D (convolution in space-time),

Original: FNO2D, U-Net in Ronneberger et al.
Revised: FNO2D, U-Net in Ronneberger et al.

Original: (2015) in Table 3, and HT-Net is significantly better than other methods.
Revised: (2015) in Table 3, and HT-Net is significantly better than other methods.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have built HT-Net, a hierarchical transformer based operator learning model for multiscale PDEs, which allows nested computation of features and self-attentions, and provides a hierarchical representation for the multiscale solution space.
Revised: We have built HT-Net, a hierarchical transformer based operator learning model for multiscale PDEs, which allows nested computation of features and self-attentions, and provides a hierarchical representation for the multiscale solution space.

Original: The reduce, multilevel local aggregation, and decompose operations form a finecoarse-fine V-cycle for the feature update.
Revised: The reduce, multilevel local aggregation, and decompose operations form a finecoarse-fine V-cycle for the feature update.

Original: We also introduced the empirical H 1 loss to reduce the spectral bias in the multiscale operator learning.
Revised: We also introduced the empirical H 1 loss to reduce the spectral bias in the multiscale operator learning.

Original: HT-Net provides much better accuracy and robustness compared with state-of-the-art (SOTA) neural operators, which is demonstrated by several multiscale benchmarks.
Revised: HT-Net provides much better accuracy and robustness compared with state-of-the-art (SOTA) neural operators, which is demonstrated by several multiscale benchmarks.

Original: Limitation and outlook (1)
Revised: Limitation and outlook (1)

Original: The current implementation of HT-Net relies on a regular grid.
Revised: The current implementation of HT-Net relies on a regular grid.

Original: The extension to data clouds and graph neural networks will offer more opportunities to take advantage of the hierarchical representation.
Revised: The extension to data clouds and graph neural networks will offer more opportunities to take advantage of the hierarchical representation.

Original: The current implementation of the attention-based operator lacks the flexibility to generalize to a different resolution as FNO (Li et al., 2021)).
Revised: The current implementation of the attention-based operator lacks the flexibility to generalize to a different resolution as FNO (Li et al., 2021)).

Original: However, for multiscale problems, the discretization invariance of FNO and MWT models may be hampered by the aliasing error in the frequency domain, which are validated by the experiment results.
Revised: However, for multiscale problems, the discretization invariance of FNO and MWT models may be hampered by the aliasing error in the frequency domain, which are validated by the experiment results.

Original: Recent convergence analysis of operator learning methods such as
Revised: Recent convergence analysis of operator learning methods such as

Original: Kovachki et al.
Revised: Kovachki et al.

Original: Lanthaler et al.
Revised: Lanthaler et al.

Original: (2022) leverage the smoothness of the solutions, however, multiscale
Revised: (2022) leverage the smoothness of the solutions, however, multiscale

Original: PDEs usually have lower regularity and do not fall in such categories.
Revised: PDEs usually have lower regularity and do not fall in such categories.

Original: For HT-Net, though it might be possible to achieve (approximate) discretization invariance by using proper sampling and interpolation modules, we believe a full treatment will be operator adaptive.
Revised: For HT-Net, though it might be possible to achieve (approximate) discretization invariance by using proper sampling and interpolation modules, we believe a full treatment will be operator adaptive.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This work proposes a hierarchical transformer operator learning model for multiscale PDEs.
Revised: This work proposes a hierarchical transformer operator learning model for multiscale PDEs.

Original: As stated in the introduction, solving multiscale PDEs is associated with some of the most challenging practical problems, such as reservoir modeling, fracture and fatigue prediction, high frequency scattering, weather forecasting, etc.
Revised: As stated in the introduction, solving multiscale PDEs is associated with some of the most challenging practical problems, such as reservoir modeling, fracture and fatigue prediction, high frequency scattering, weather forecasting, etc.

Original: Potentially, HT-Net solvers could help to reduce the prohibitively expensive computational cost of those simulations.
Revised: Potentially, HT-Net solvers could help to reduce the prohibitively expensive computational cost of those simulations.

Original: The negative consequences are not obvious.
Revised: The negative consequences are not obvious.

Original: Though, in theory, any technique can be misused, it is not likely to happen at the current stage.
Revised: Though, in theory, any technique can be misused, it is not likely to happen at the current stage.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The datasets for Section 4 are either downloaded (for smooth two-phase coefficients and Navier-Stokes) from https://github.com/zongyi-li/fourier_neural_operator , or generated (for rough
Revised: The datasets for Section 4 are either downloaded (for smooth two-phase coefficients and Navier-Stokes) from https://github.com/zongyi-li/fourier_neural_operator , or generated (for rough

Original: Darcy coefficients) using the code from the same website.
Revised: Darcy coefficients) using the code from the same website.

Original: We implemented P 1 finite element method in
Revised: We implemented P 1 finite element method in

Original: MATLAB to solve equation 2.1 with multiscale trigonometric coefficients in Appendix F.1, and in FreeFEM to solve the Helmholtz equation in Appendix F.2.
Revised: MATLAB to solve equation 2.1 with multiscale trigonometric coefficients in Appendix F.1, and in FreeFEM to solve the Helmholtz equation in Appendix F.2.

Original: We have included introductions of the relevant mathematical and data generation concepts in the appendix.
Revised: We have included introductions of the relevant mathematical and data generation concepts in the appendix.

Original: We put the code for and also a link for datasets at the anonymous Github page https://github.com/
Revised: We put the code for and also a link for datasets at the anonymous Github page https://github.com/

Original: Shengren-Kato/VFMM-ICLR2023.git .
Revised: Shengren-Kato/VFMM-ICLR2023.git .

Original: Supplementary descriptions of the code are also provided in the page.
Revised: Supplementary descriptions of the code are also provided in the page.

Original: A M ULTISCALE PDE
Revised: A M ULTISCALE PDE

Original: In this section we introduce some mathematical and numerical concepts related to multiscale PDEs.
Revised: In this section we introduce some mathematical and numerical concepts related to multiscale PDEs.

Original: A. A SYMPTOTIC H OMOGENIZATION
Revised: A. A SYMPTOTIC H OMOGENIZATION

Original: We introduce some basic formulation of asymptotic homogenization, by assuming that a ( x ) = a ( xε ) with 1-periodic funciton a ( · ) in equation 2.1, namely,
Revised: We introduce some basic formulation of asymptotic homogenization, by assuming that a ( x ) = a ( xε ) with 1-periodic funciton a ( · ) in equation 2.1, namely,

Original: [Equation]
Revised: [Equation]

Original: It can be derived from the asymptotic expansion of the two-scale function u ε = u ( x, x/ε ) in ε , and justified rigorously that u ε ⇀ u 0 in H 1 , with u 0 ( x ) the solution of the homogenized problem
Revised: It can be derived from the asymptotic expansion of the two-scale function u ε = u ( x, x/ε ) in ε , and justified rigorously that u ε ⇀ u 0 in H 1 , with u 0 ( x ) the solution of the homogenized problem

Original: [Equation]
Revised: [Equation]

Original: which contains only coarse scale information.
Revised: which contains only coarse scale information.

Original: The homogenized coefficient a 0 can be computed by the formula a 0 ij = (cid:82) Y ( e i + ∇ χ
Revised: The homogenized coefficient a 0 can be computed by the formula a 0 ij = (cid:82) Y ( e i + ∇ χ

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: T a ( y
Revised: T a ( y

Original: [Equation]
Revised: [Equation]

Original: e j + ∇ χ j
Revised: e j + ∇ χ j

Original: [Equation]
Revised: [Equation]

Original: dy , where χ i solves the cell problems given by
Revised: dy , where χ i solves the cell problems given by

Original: [Equation]
Revised: [Equation]

Original: with Y being the d dimensional torus.
Revised: with Y being the d dimensional torus.

Original: Asymptotic homogenization provides an approximate solution ˆ u ε = u 0 − ε (cid:80) χ
Revised: Asymptotic homogenization provides an approximate solution ˆ u ε = u 0 − ε (cid:80) χ

Original: i (cid:0) xε (cid:1) ∂u 0 ∂x i such that ∥ u ε − ˆ u ε ∥ H 1 ≤ cε 12 .
Revised: i (cid:0) xε (cid:1) ∂u 0 ∂x i such that ∥ u ε − ˆ u ε ∥ H 1 ≤ cε 12 .

Original: We note that to find the homogenization approximation ˆ u ε only requires the coarse-scale solution u 0 and precomputation of d cell problems for χ i which do not depend on f and D .
Revised: We note that to find the homogenization approximation ˆ u ε only requires the coarse-scale solution u 0 and precomputation of d cell problems for χ i which do not depend on f and D .

Original: A. N UMERICAL H OMOGENIZATION AND M ULTILEVEL /M ULTIGRID M ETHODS
Revised: A. N UMERICAL H OMOGENIZATION AND M ULTILEVEL /M ULTIGRID M ETHODS

Original: Given the smallest scale of the multiscale problem ε and a coarse computational scale determined by the available computational power and the desired precision, with ε ≪ H ≪ 1 .
Revised: Given the smallest scale of the multiscale problem ε and a coarse computational scale determined by the available computational power and the desired precision, with ε ≪ H ≪ 1 .

Original: The goal of numerical homogenization is to construct a finite-dimensional approximation space V H and to seek an approximate solution u H ∈ V H , such that, the accuracy estimate ∥
Revised: The goal of numerical homogenization is to construct a finite-dimensional approximation space V H and to seek an approximate solution u H ∈ V H , such that, the accuracy estimate ∥

Original: u − u H ∥ ≤ CH α holds for optimal choices of the norm ∥·∥ and the exponent α , and optimal computational cost holds with V H constructed via pre-computed subproblems which are localized, independent and do not depend on the RHS and boundary condition of the problem.
Revised: u − u H ∥ ≤ CH α holds for optimal choices of the norm ∥·∥ and the exponent α , and optimal computational cost holds with V H constructed via pre-computed subproblems which are localized, independent and do not depend on the RHS and boundary condition of the problem.

Original: In recent two decades, great progress has been made in this area (Hou et al., 1999; M ˚ alqvist & Peterseim, 2014; Owhadi & Zhang, 2007), approximation spaces with optimal accuracy (in the sense of Kolmogorov
Revised: In recent two decades, great progress has been made in this area (Hou et al., 1999; M ˚ alqvist & Peterseim, 2014; Owhadi & Zhang, 2007), approximation spaces with optimal accuracy (in the sense of Kolmogorov

Original: N -width (Berlyand & Owhadi, 2010)) and cost can be constructed for elliptic equations with fixed rough coefficients.
Revised: N -width (Berlyand & Owhadi, 2010)) and cost can be constructed for elliptic equations with fixed rough coefficients.

Original: For multiscale PDEs, operator learning methods can be seen as a step forward from numerical homogenization, since they can be applied to an ensemble of coefficients, and the decoder can be interpreted as the basis of the underlying problem as well.
Revised: For multiscale PDEs, operator learning methods can be seen as a step forward from numerical homogenization, since they can be applied to an ensemble of coefficients, and the decoder can be interpreted as the basis of the underlying problem as well.

Original: For multiscale PDEs, multilevel/multigrid methods can be seen as the multilevel generalization of the numerical homogenization methods.
Revised: For multiscale PDEs, multilevel/multigrid methods can be seen as the multilevel generalization of the numerical homogenization methods.

Original: Numerical homogenization can provide coarse spaces with optimal approximation and localization properties.
Revised: Numerical homogenization can provide coarse spaces with optimal approximation and localization properties.

Original: Recently, operator-adapted wavelets (gamblets) (Owhadi, 2017; Xie et al., 2019) have been developed, and enjoy three properties that are ideal for the construction of efficient direct methods: scale orthogonality, well-conditioned multi-resolution decomposition, and localization.
Revised: Recently, operator-adapted wavelets (gamblets) (Owhadi, 2017; Xie et al., 2019) have been developed, and enjoy three properties that are ideal for the construction of efficient direct methods: scale orthogonality, well-conditioned multi-resolution decomposition, and localization.

Original: Gamblets has been generalized to solve Navier-Stokes equation (Budninskiy et al., 2019) and Helmholtz equation (Hauck & Peterseim, 2022) efficiently.
Revised: Gamblets has been generalized to solve Navier-Stokes equation (Budninskiy et al., 2019) and Helmholtz equation (Hauck & Peterseim, 2022) efficiently.

Original: B H IERARCHICAL M ATRIX P ERSPECTIVE
Revised: B H IERARCHICAL M ATRIX P ERSPECTIVE

Original: The hierarchically nested attention in Algorithm 1 resembles the celebrated hierarchical matrix method (Hackbusch, 2015), in particular, the H 2 matrix from the perspective of matrix operations.
Revised: The hierarchically nested attention in Algorithm 1 resembles the celebrated hierarchical matrix method (Hackbusch, 2015), in particular, the H 2 matrix from the perspective of matrix operations.

Original: In the following,
Revised: In the following,

Original: During we take the binary tree-like hierarchical discretization shown in Fig B.1 as an example to illustrate the reduce operation, decompose operation, and multilevel token aggregation in Algorithm 1 in matrix formulas.
Revised: During we take the binary tree-like hierarchical discretization shown in Fig B.1 as an example to illustrate the reduce operation, decompose operation, and multilevel token aggregation in Algorithm 1 in matrix formulas.

Original: STEP 0:
Revised: STEP 0:

Original: Given features f ( r ) ,t , compute the queries q ( r ) ,t i , keys k ( r ) ,t j , and values v ( r ) ,t i for j ∈
Revised: Given features f ( r ) ,t , compute the queries q ( r ) ,t i , keys k ( r ) ,t j , and values v ( r ) ,t i for j ∈

Original: I ( r ) .
Revised: I ( r ) .

Original: Starting from the finest level features f ( r ) ,t i , i ∈ I ( r ) , the queries q ( r ) ,t can be obtained by   ...
Revised: Starting from the finest level features f ( r ) ,t i , i ∈ I ( r ) , the queries q ( r ) ,t can be obtained by   ...

Original: q ( r ) ,t i ...
Revised: q ( r ) ,t i ...

Original:   =  
Revised:   =  

Original: W Q, ( r )
Revised: W Q, ( r )

Original: W Q, ( r ) ...
Revised: W Q, ( r ) ...

Original: W Q, ( r )   (cid:124) (cid:123)(cid:122) (cid:125) |I
Revised: W Q, ( r )   (cid:124) (cid:123)(cid:122) (cid:125) |I

Original: ( r ) |   ...
Revised: ( r ) |   ...

Original: f ( r ) ,t i ...
Revised: f ( r ) ,t i ...

Original:     |I
Revised:     |I

Original: ( r ) | , the keys k ( r ) ,t and values v ( r ) ,t can follow the similar procedure.
Revised: ( r ) | , the keys k ( r ) ,t and values v ( r ) ,t can follow the similar procedure.

Original: STEP 1:
Revised: STEP 1:

Original: For m = r − 1 : 1 , Do the reduce operations q ( m ) ,t i =
Revised: For m = r − 1 : 1 , Do the reduce operations q ( m ) ,t i =

Original: R ( m ) ( { q ( m +1) ,t j } j ∈ i ( m,m +1) ) and also for k ( m ) ,t i and v ( m ) ,t i , for any i ∈ I ( m ) .
Revised: R ( m ) ( { q ( m +1) ,t j } j ∈ i ( m,m +1) ) and also for k ( m ) ,t i and v ( m ) ,t i , for any i ∈ I ( m ) .

Original: If R ( m ) is linear, the reduce operations correspond to   ...
Revised: If R ( m ) is linear, the reduce operations correspond to   ...

Original: q ( m ) ,t i ...
Revised: q ( m ) ,t i ...

Original: 
Revised: 

Original:  = R ( m ) ,t   ...
Revised:  = R ( m ) ,t   ...

Original: q ( m +1) ,t i ...
Revised: q ( m +1) ,t i ...

Original:   .The
Revised:   .The

Original: reduce matrix is given by
Revised: reduce matrix is given by

Original: R ( m ) :=  
Revised: R ( m ) :=  

Original: R ( m ) R ( m )
Revised: R ( m ) R ( m )

Original: R ( m ) R ( m )...
Revised: R ( m ) R ( m )...

Original: R ( m ) R ( m )
Revised: R ( m ) R ( m )

Original:  (cid:124) (cid:123)(cid:122) (cid:125) |I
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) |I

Original: ( m +1)
Revised: ( m +1)

Original: |   |I ( m ) | , and R ( m ) 0 , R ( m )∈ R C ( m − 1) ×C ( m ) are matrices parametrized by linear layers in our paper (in practice, queries, keys, and values use different R ( m ) 0 , R ( m )to enhance the expressivity).
Revised: |   |I ( m ) | , and R ( m ) 0 , R ( m )∈ R C ( m − 1) ×C ( m ) are matrices parametrized by linear layers in our paper (in practice, queries, keys, and values use different R ( m ) 0 , R ( m )to enhance the expressivity).

Original: In general, these operators R ( m ) are not limited to linear operators.
Revised: In general, these operators R ( m ) are not limited to linear operators.

Original: The composition of nonlinear activation functions would help increase the expressivity.
Revised: The composition of nonlinear activation functions would help increase the expressivity.

Original: The nested learnable operators R ( m ) also induce the channel mixing and is equivalent to a structured parameterization of W Q , W V , W K matrix for the coarse level tokens, in the sense that, inductively,    ...
Revised: The nested learnable operators R ( m ) also induce the channel mixing and is equivalent to a structured parameterization of W Q , W V , W K matrix for the coarse level tokens, in the sense that, inductively,    ...

Original: q ( m ) ,t i ...
Revised: q ( m ) ,t i ...

Original: 
Revised: 

Original:   =
Revised:   =

Original: R ( m ) · · · R ( r − 1)    ...
Revised: R ( m ) · · · R ( r − 1)    ...

Original: q ( r ) ,t i ...
Revised: q ( r ) ,t i ...

Original: 
Revised: 

Original:   =
Revised:   =

Original: R ( m ) · · · R ( r − 1)  
Revised: R ( m ) · · · R ( r − 1)  

Original: W Q, ( r )
Revised: W Q, ( r )

Original: W Q, ( r ) . . .
Revised: W Q, ( r ) . . .

Original: W Q, ( r ) 
Revised: W Q, ( r ) 

Original:    ...
Revised:    ...

Original: f ( r ) ,t i ...
Revised: f ( r ) ,t i ...

Original:   .
Revised:   .

Original: STEP 2:
Revised: STEP 2:

Original: With the m -th level queries and keys, we can calculate the local attention matrix G ( m ) ,t loc at m -th level with G ( m ) ,t loc , i , j := exp( q ( m ) ,t i · k ( m ) ,t j ) for i ∈ N ( m ) ( j ) , or i ∼ j .
Revised: With the m -th level queries and keys, we can calculate the local attention matrix G ( m ) ,t loc at m -th level with G ( m ) ,t loc , i , j := exp( q ( m ) ,t i · k ( m ) ,t j ) for i ∈ N ( m ) ( j ) , or i ∼ j .

Original: STEP 3: The decompose operations, opposite to the reduce operations, correspond to the transpose of the following matrix in the linear case,
Revised: STEP 3: The decompose operations, opposite to the reduce operations, correspond to the transpose of the following matrix in the linear case,

Original: D ( m ) :=  
Revised: D ( m ) :=  

Original: D ( m ) D ( m )
Revised: D ( m ) D ( m )

Original: D ( m ) D ( m )...
Revised: D ( m ) D ( m )...

Original: D ( m ) D ( m )
Revised: D ( m ) D ( m )

Original:  (cid:124) (cid:123)(cid:122) (cid:125) |I
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) |I

Original: ( m +1)
Revised: ( m +1)

Original: |   |I ( m ) | ,
Revised: |   |I ( m ) | ,

Original: The m -th level aggregation in Figure 3.2 contributes to the final output f ( r ) ,t +1 in the form
Revised: The m -th level aggregation in Figure 3.2 contributes to the final output f ( r ) ,t +1 in the form

Original: D ( r − 1) , T · · · D ( m ) , T G ( m ) loc R ( m ) · · · R ( r − 1)     ... v ( r ) ,t i ...
Revised: D ( r − 1) , T · · · D ( m ) , T G ( m ) loc R ( m ) · · · R ( r − 1)     ... v ( r ) ,t i ...

Original: 
Revised: 

Original:    .
Revised:    .

Original: Eventually, aggregations at all r levels in one V cycle can be summed up as
Revised: Eventually, aggregations at all r levels in one V cycle can be summed up as

Original: [Equation]
Revised: [Equation]

Original: The hierarchical attention matrix G h := (cid:80) r − 1 m =1 ( D ( r − 1) , T · · · D ( m ) , T G ( m ) loc R ( m ) · · · R ( r − 1) )
Revised: The hierarchical attention matrix G h := (cid:80) r − 1 m =1 ( D ( r − 1) , T · · · D ( m ) , T G ( m ) loc R ( m ) · · · R ( r − 1) )

Original: + G ( r ) loc in equation B.1 resembles the three-level H 2 matrix decomposition illustrated in the following Figure B.2, we also refer to (Hackbusch, 2015) for a detailed description.
Revised: + G ( r ) loc in equation B.1 resembles the three-level H 2 matrix decomposition illustrated in the following Figure B.2, we also refer to (Hackbusch, 2015) for a detailed description.

Original: The sparsity of the matrix lies in the fact that the attention matrix is only computed for pairs of tokens within the neighbor set.
Revised: The sparsity of the matrix lies in the fact that the attention matrix is only computed for pairs of tokens within the neighbor set.

Original: The H 2 matrix-vector multiplication in B.1 also implies O ( N ) complexity of the Algorithm 1.
Revised: The H 2 matrix-vector multiplication in B.1 also implies O ( N ) complexity of the Algorithm 1.

Original: Note that, the local attention matrix at level I (1) (pink), level I (2) (blue) and level I (3) (green ) are G (1) loc ,
Revised: Note that, the local attention matrix at level I (1) (pink), level I (2) (blue) and level I (3) (green ) are G (1) loc ,

Original: G (2) loc and G (3) loc , respectively.
Revised: G (2) loc and G (3) loc , respectively.

Original: However, when considering their contributions to the finest level, they are equivalent to the attention matrix D (2) , T D (1) , T G (1) loc R (1) R (2)
Revised: However, when considering their contributions to the finest level, they are equivalent to the attention matrix D (2) , T D (1) , T G (1) loc R (1) R (2)

Original: ∈
Revised: ∈

Original: R I (3) × R I (3) (pink), D (2) , T G (2) loc R (2) ∈
Revised: R I (3) × R I (3) (pink), D (2) , T G (2) loc R (2) ∈

Original: R I (3) × R I (3) (blue) and G (3) loc (green), as demonstrated in Figure B.2.
Revised: R I (3) × R I (3) (blue) and G (3) loc (green), as demonstrated in Figure B.2.

Original: Each pink block and blue block are actually low-rank sub-matrix with rank C (1) and rank C (2) , respectively, by definition.
Revised: Each pink block and blue block are actually low-rank sub-matrix with rank C (1) and rank C (2) , respectively, by definition.

Original: C P ROOF OF P ROPOSITION 3. 
Revised: C P ROOF OF P ROPOSITION 3.

Original: Proof
Revised: Proof

Original: For each level m , the cost to compute equation 3.3 is c ( |I ( m ) |C ( m ) )
Revised: For each level m , the cost to compute equation 3.3 is c ( |I ( m ) |C ( m ) )

Original: since for each i ∈ I ( m ) the cardinality of the neighbour set N ( m ) ( i ) is bounded by a constant c .
Revised: since for each i ∈ I ( m ) the cardinality of the neighbour set N ( m ) ( i ) is bounded by a constant c .

Original: The reduce operation f ( k − 1) i =
Revised: The reduce operation f ( k − 1) i =

Original: R ( k − 1) ( { f ( m ) j } j ∈ i ( k − 1 ,k ) ) costs at most |I ( m ) |C ( m ) C
Revised: R ( k − 1) ( { f ( m ) j } j ∈ i ( k − 1 ,k ) ) costs at most |I ( m ) |C ( m ) C

Original: ( k − 1) flops and so does the decompose operation at the same level.
Revised: ( k − 1) flops and so does the decompose operation at the same level.

Original: Therefore, for each level, the operation cost is c ( |I ( m ) |C ( m ) )
Revised: Therefore, for each level, the operation cost is c ( |I ( m ) |C ( m ) )

Original: + 2 |I ( m )
Revised: + 2 |I ( m )

Original: |C ( m ) C ( m − 1) .
Revised: |C ( m ) C ( m − 1) .

Original: When I is a quadtree, I ( r ) =
Revised: When I is a quadtree, I ( r ) =

Original: N, I ( r − 1) = N/ 4 , · · · , I (1) = 4 , therefore the total computational cost ∼ O ( N ) .
Revised: N, I ( r − 1) = N/ 4 , · · · , I (1) = 4 , therefore the total computational cost ∼ O ( N ) .

Original: □
Revised: □

Original: D I MPLEMENTATION D ETAILS
Revised: D I MPLEMENTATION D ETAILS

Original: In the HT-Net implementation, we follow the window attention scheme in (Liu et al., 2021b) for the definition of the neighborhood N ( · ) ( · ) in equation 3.2 and equation 3.3.
Revised: In the HT-Net implementation, we follow the window attention scheme in (Liu et al., 2021b) for the definition of the neighborhood N ( · ) ( · ) in equation 3.2 and equation 3.3.

Original: In this paper, we choose r = 3 as the depth of the HT-Net, GeLU as the activation function, and a CNN-based patch embedding module to transfer the input data into features.
Revised: In this paper, we choose r = 3 as the depth of the HT-Net, GeLU as the activation function, and a CNN-based patch embedding module to transfer the input data into features.

Original: For a dataset with resolution n f × n f , such as in the multiscale elliptic equation benchmark 4.2, the input feature f (3) is represented as a tensor of size n × n × C via patch embedding.
Revised: For a dataset with resolution n f × n f , such as in the multiscale elliptic equation benchmark 4.2, the input feature f (3) is represented as a tensor of size n × n × C via patch embedding.

Original: The self-attention is first computed within a local window on level 3.
Revised: The self-attention is first computed within a local window on level 3.

Original: Then the reduce layer concatenates the features of each group of 2 × 2 neighboring tokens and applies a linear transformation on the 4 C -dimensional concatenated features on n 2 × n 2 level 2 tokens, to obtain level 2 features f (2) as a tensor of the size n 2 × n 2 × 2 C .
Revised: Then the reduce layer concatenates the features of each group of 2 × 2 neighboring tokens and applies a linear transformation on the 4 C -dimensional concatenated features on n 2 × n 2 level 2 tokens, to obtain level 2 features f (2) as a tensor of the size n 2 × n 2 × 2 C .

Original: The procedure is repeated from level 2 to level 1 with f (1) of size n 4 × n 4 × 4 C .
Revised: The procedure is repeated from level 2 to level 1 with f (1) of size n 4 × n 4 × 4 C .

Original: We adopt the window attention scheme from (Liu et al., 2021b).
Revised: We adopt the window attention scheme from (Liu et al., 2021b).

Original: One can refer to the implementation details there.
Revised: One can refer to the implementation details there.

Original: For the decompose process, starting at level 1, a linear layer is applied to transform the 4 Cdimensional features f (1) into 8 C -dimensional features.
Revised: For the decompose process, starting at level 1, a linear layer is applied to transform the 4 Cdimensional features f (1) into 8 C -dimensional features.

Original: Each level 1 token with 8 C -dimensional features is decomposed into four level 2 tokens with 2 C -dimensional features and added to the level 2 feature f (2) with output size of n 2 × n 2 × 2 C .
Revised: Each level 1 token with 8 C -dimensional features is decomposed into four level 2 tokens with 2 C -dimensional features and added to the level 2 feature f (2) with output size of n 2 × n 2 × 2 C .

Original: The procedure is repeated from level 2 level to level 3 with the output f (3) of size n × n × C .
Revised: The procedure is repeated from level 2 level to level 3 with the output f (3) of size n × n × C .

Original: We use the Adam optimizer with learning rate 1e − 3 , weight decay 1e − 4 and the 1-cycle schedule as in (Cao, 2021).
Revised: We use the Adam optimizer with learning rate 1e − 3 , weight decay 1e − 4 and the 1-cycle schedule as in (Cao, 2021).

Original: We use batch size 8 for experiments in Sections 4.2 and 4.3, and batch size 4 for experiments in Appendices F.1 and F.2.
Revised: We use batch size 8 for experiments in Sections 4.2 and 4.3, and batch size 4 for experiments in Appendices F.1 and F.2.

Original: Evaluation setup For Darcy rough in table 1, we use a train-val-test split of the dataset with sizes of 1280, 112, and 112, respectively.
Revised: Evaluation setup For Darcy rough in table 1, we use a train-val-test split of the dataset with sizes of 1280, 112, and 112, respectively.

Original: Each run is carried out with different seeds.
Revised: Each run is carried out with different seeds.

Original: For the Darcy smooth and multiscale trigonometric cases, the training size, validation size, and testing size are 1000, 100 and 100, respectively.
Revised: For the Darcy smooth and multiscale trigonometric cases, the training size, validation size, and testing size are 1000, 100 and 100, respectively.

Original: For the Navier-Stokes experiment in Section 4.3, we have tried N = 1000 and N =training samples as shown in Table 3, and the number of testing samples is 100 and 1000 respectively.
Revised: For the Navier-Stokes experiment in Section 4.3, we have tried N = 1000 and N =training samples as shown in Table 3, and the number of testing samples is 100 and 1000 respectively.

Original: The baseline’s implementations are based on their official implementation if provided publicly.
Revised: The baseline’s implementations are based on their official implementation if provided publicly.

Original: The results are comparable with the ones reported in the references.
Revised: The results are comparable with the ones reported in the references.

Original: All experiments are run on a NVIDIA A100 GPU.
Revised: All experiments are run on a NVIDIA A100 GPU.

Original: E D ATA G ENERATION FOR THE T WO -P HASE C OEFFICIENT IN S ECTION 4. 
Revised: E D ATA G ENERATION FOR THE T WO -P HASE C OEFFICIENT IN S ECTION 4.

Original: The two-phase coefficients and solutions are generated according to https://github.com/ zongyi-li/fourier_neural_operator/tree/master/data_generation , and used as operator learning benchmarks in Li et al.
Revised: The two-phase coefficients and solutions are generated according to https://github.com/ zongyi-li/fourier_neural_operator/tree/master/data_generation , and used as operator learning benchmarks in Li et al.

Original: ; Gupta et al.
Revised: ; Gupta et al.

Original: (2021); Cao (2021).
Revised: (2021); Cao (2021).

Original: The coefficients a ( x ) are generated according to a ∼ µ :=
Revised: The coefficients a ( x ) are generated according to a ∼ µ :=

Original: ψ # N (cid:0) 0 , ( − ∆ + cI ) − 2 (cid:1) with zero Neumann boundary conditions on the Laplacian.
Revised: ψ # N (cid:0) 0 , ( − ∆ + cI ) − 2 (cid:1) with zero Neumann boundary conditions on the Laplacian.

Original: The mapping ψ : R → R takes the value a max on the positive part of the real line and a min on the negative part.
Revised: The mapping ψ : R → R takes the value a max on the positive part of the real line and a min on the negative part.

Original: The push-forward is defined in a pointwise manner.
Revised: The push-forward is defined in a pointwise manner.

Original: The forcing term is fixed as f ( x ) ≡ 1 .
Revised: The forcing term is fixed as f ( x ) ≡ 1 .

Original: Solutions u are obtained by using a second-order finite difference scheme on a suitable grid.
Revised: Solutions u are obtained by using a second-order finite difference scheme on a suitable grid.

Original: The parameters a max and a min can control the contrast of the coefficient.
Revised: The parameters a max and a min can control the contrast of the coefficient.

Original: The parameter c can control the roughness (oscillation) of the coefficient, and the coefficient with a larger c has rougher two-phase interfaces, as shown in
Revised: The parameter c can control the roughness (oscillation) of the coefficient, and the coefficient with a larger c has rougher two-phase interfaces, as shown in

Original: Figure 4.1.
Revised: Figure 4.1.

Original: F A DDITIONAL E XPERIMENTS F. M ULTISCALE TRIGONOMETRIC COEFFICIENT
Revised: F A DDITIONAL E XPERIMENTS F. M ULTISCALE TRIGONOMETRIC COEFFICIENT

Original: In this experiment, we consider equation 2.1 with multiscale trigonometric coefficient adapted from Owhadi (2017), such that D
Revised: In this experiment, we consider equation 2.1 with multiscale trigonometric coefficient adapted from Owhadi (2017), such that D

Original: =
Revised: =

Original: [ − 1 , 1] 2 , a ( x ) =
Revised: [ − 1 , 1] 2 , a ( x ) =

Original: (cid:81) k
Revised: (cid:81) k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (1 + 12 cos( a k π ( x 1 + x 2 )))
Revised: (1 + 12 cos( a k π ( x 1 + x 2 )))

Original: (1 + 1 2 sin( a k π ( x 2 − 3 x 1 ))) , with a k = uniform(2 k − 1 , 1 . 5 × 2 k − 1 ) , and fixed f ( x ) ≡ 1 .
Revised: (1 + 1 2 sin( a k π ( x 2 − 3 x 1 ))) , with a k = uniform(2 k − 1 , 1 . 5 × 2 k − 1 ) , and fixed f ( x ) ≡ 1 .

Original: The reference solutions are obtained using P 1 FEM on a 1023 × 1023 grid.
Revised: The reference solutions are obtained using P 1 FEM on a 1023 × 1023 grid.

Original: Datasets of lower resolution are sampled from the higher resolution dataset by linear interpolation.
Revised: Datasets of lower resolution are sampled from the higher resolution dataset by linear interpolation.

Original: The experiment results for the multiscale trigonometric case with different resolutions are shown in Table 4.
Revised: The experiment results for the multiscale trigonometric case with different resolutions are shown in Table 4.

Original: HT-Net obtains the best relative L 2 error compared to other neural operators at various resolutions at 600 epochs.
Revised: HT-Net obtains the best relative L 2 error compared to other neural operators at various resolutions at 600 epochs.

Original: Multiwavelet neural operator MWT has the second best performance as it also possesses a multiresolution structure, but it does not adapt to the PDE.
Revised: Multiwavelet neural operator MWT has the second best performance as it also possesses a multiresolution structure, but it does not adapt to the PDE.

Original: It is not surprising that FNO, an excellent smoother which filters higher frequency modes, fails to capture the high-frequency oscillations of the solution.
Revised: It is not surprising that FNO, an excellent smoother which filters higher frequency modes, fails to capture the high-frequency oscillations of the solution.

Original: In contrast, our method has a better performance in this respect.
Revised: In contrast, our method has a better performance in this respect.

Original: See Figure F.1 for illustrations of the coefficient and comparison of the solutions at the slice x = 0 .
Revised: See Figure F.1 for illustrations of the coefficient and comparison of the solutions at the slice x = 0 .

Original: [Table]
Revised: [Table]

Original: F. H ELMHOLTZ EQUATION
Revised: F. H ELMHOLTZ EQUATION

Original: We test the performance of HT-Net for the acoustic Helmholtz equation in highly heterogeneous media as an example of multiscale wave phenomena, whose solution is considerably expensive for complicated and large geological models.
Revised: We test the performance of HT-Net for the acoustic Helmholtz equation in highly heterogeneous media as an example of multiscale wave phenomena, whose solution is considerably expensive for complicated and large geological models.

Original: We adopt the setup from De Hoop et al.
Revised: We adopt the setup from De Hoop et al.

Original: (2022), for the Helmholtz equation on the domain D
Revised: (2022), for the Helmholtz equation on the domain D

Original: = [0 , 1] 2 .
Revised: = [0 , 1] 2 .

Original: Given frequency ω = 10 3 and wavespeed field c : Ω → R , the excitation field u : Ω → R solves the equation
Revised: Given frequency ω = 10 3 and wavespeed field c : Ω → R , the excitation field u : Ω → R solves the equation

Original: [Equation]
Revised: [Equation]

Original: where ∂ Ω 3 is the top side of the boundary, and ∂ Ω 1 , 2 , 4 are other sides.
Revised: where ∂ Ω 3 is the top side of the boundary, and ∂ Ω 1 , 2 , 4 are other sides.

Original: The wave speed field is c ( x ) = 

Revised: The wave speed field is c ( x ) =

Original: + tanh(˜ c ( x )) , where ˜ c is sampled from the Gaussian field ˜ c ∼ N (0 , (cid:0) − ∆ + τ 2 (cid:1) − d ) , where τ = 3 and d = 2 are chosen to control the roughness.
Revised: + tanh(˜ c ( x )) , where ˜ c is sampled from the Gaussian field ˜ c ∼ N (0 , (cid:0) − ∆ + τ 2 (cid:1) − d ) , where τ = 3 and d = 2 are chosen to control the roughness.

Original: The Helmholtz equation is solved on a 100 × 100 grid by finite element methods.
Revised: The Helmholtz equation is solved on a 100 × 100 grid by finite element methods.

Original: We aim to learn the mapping from c ∈ R 100 × 100 to u ∈ R 100 × 100 as shown in Fig F.2.
Revised: We aim to learn the mapping from c ∈ R 100 × 100 to u ∈ R 100 × 100 as shown in Fig F.2.

Original: [Figure]
Revised: [Figure]

Original: The Helmholtz equation is notorious for solving.
Revised: The Helmholtz equation is notorious for solving.

Original: One reason is the so-called resonance phenomenon as the frequency ω is close to an eigenfrequency of the Helmholtz operator for some particular c .
Revised: One reason is the so-called resonance phenomenon as the frequency ω is close to an eigenfrequency of the Helmholtz operator for some particular c .

Original: We found that it is necessary to use a large training dataset of size 8000 .
Revised: We found that it is necessary to use a large training dataset of size 8000 .

Original: The test dataset is of size 800 .
Revised: The test dataset is of size 800 .

Original: All models are trained within 500 epochs.
Revised: All models are trained within 500 epochs.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We also compare the evaluation time of the trained models in 5.
Revised: We also compare the evaluation time of the trained models in 5.

Original: Compared to HT-Net, FNO has both a larger error and a longer evaluation time.
Revised: Compared to HT-Net, FNO has both a larger error and a longer evaluation time.

Original: UNet, as a CNN based method, has much faster evaluation time (30 times faster than HT-
Revised: UNet, as a CNN based method, has much faster evaluation time (30 times faster than HT-

Original: ) but the worst error (60 times as much as HT-
Revised: ) but the worst error (60 times as much as HT-

Original: F. C OMPARISON WITH CLASSICAL METHODS (FDM) AND MULTISCALE METHODS (GRPS)
Revised: F. C OMPARISON WITH CLASSICAL METHODS (FDM) AND MULTISCALE METHODS (GRPS)

Original: We compare the accuracy and efficiency of HT-Net with two conventional solvers - finite difference method (FDM) as a typical classical method, and generalized rough polyharmonic splines (GRPS)
Revised: We compare the accuracy and efficiency of HT-Net with two conventional solvers - finite difference method (FDM) as a typical classical method, and generalized rough polyharmonic splines (GRPS)

Original: (Liu et al., 2021a) as a typical multiscale method, for the Darcy rough benchmark.
Revised: (Liu et al., 2021a) as a typical multiscale method, for the Darcy rough benchmark.

Original: The results are listed in Table 6.
Revised: The results are listed in Table 6.

Original: [Table]
Revised: [Table]

Original: F.
Revised: F.

Original: S TUDY FOR HYPERPARAMETERS
Revised: S TUDY FOR HYPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: We conduct studies for hyperparameters such as number of hierarchical levels (depth), window size
Revised: We conduct studies for hyperparameters such as number of hierarchical levels (depth), window size

Original: [Table]
Revised: [Table]

Original: and feature dimension.
Revised: and feature dimension.

Original: We list the results for the Darcy rough benchmark in Table
Revised: We list the results for the Darcy rough benchmark in Table

Original: [Table]
Revised: [Table]

Original: where the notation HTNet [3, 8, 80] represents HT-Net with 3 levels, window size
Revised: where the notation HTNet [3, 8, 80] represents HT-Net with 3 levels, window size

Original: [Table]
Revised: [Table]

Original: and feature dimension 80.
Revised: and feature dimension 80.

Original: The experiments are run for 100 epochs.
Revised: The experiments are run for 100 epochs.

Original: The results show that larger values of hierarchical level, window size
Revised: The results show that larger values of hierarchical level, window size

Original: [Table]
Revised: [Table]

Original: and feature dimension might be helpful to reduce errors, though with a higher computational cost.
Revised: and feature dimension might be helpful to reduce errors, though with a higher computational cost.

Original: To balance the model size, computational cost
Revised: To balance the model size, computational cost

Original: [Table]
Revised: [Table]

Original: and performance, we choose the hyperparameters as [3, 8, 80].
Revised: and performance, we choose the hyperparameters as [3, 8, 80].

Original: F. M EMORY USAGE
Revised: F. M EMORY USAGE

Original: [Table]
Revised: [Table]

Original: The CUDA mem(GB) is the sum of self cuda memory usuage from the PyTorch autograd profiler for 1 backpropagation.
Revised: The CUDA mem(GB) is the sum of self cuda memory usuage from the PyTorch autograd profiler for 1 backpropagation.

Original: The mem(GB) is recorded from nvidia − smi of the memory allocated for the active Python process during profiling.
Revised: The mem(GB) is recorded from nvidia − smi of the memory allocated for the active Python process during profiling.

Original: We report the memory usage of different models for the Darcy smooth (with resolution 211 × 211) and Darcy rough (with resolution 256 × 256) benchmarks in Table F.5.
Revised: We report the memory usage of different models for the Darcy smooth (with resolution 211 × 211) and Darcy rough (with resolution 256 × 256) benchmarks in Table F.5.

Original: The table shows that the memory usage of HTNet is stable.
Revised: The table shows that the memory usage of HTNet is stable.

Original: For the resolution of 256 × 256, both MWT and GT consume larger or comparable CUDA memory compared with HT-Net, albeit HT-Net has much better accuracy.
Revised: For the resolution of 256 × 256, both MWT and GT consume larger or comparable CUDA memory compared with HT-Net, albeit HT-Net has much better accuracy.