Original: HT-N ET : H IERARCHICAL T RANSFORMER BASED O PERATOR L EARNING M ODEL FOR M ULTISCALE PDE S
Revised: HT-N ET : H IERARCHICAL T RANSFORMER BASED O PERATOR L EARNING M ODEL FOR M ULTISCALE PDE S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Complex nonlinear interplays of multiple scales give rise to many interesting physical phenomena and pose major difficulties for the computer simulation of multiscale PDE models in areas such as reservoir simulation, high frequency scattering and turbulence modeling.
Revised: Complex nonlinear interplays of multiple scales give rise to many interesting physical phenomena and pose significant difficulties for the computer simulation of multiscale PDE models in areas such as reservoir simulation, high-frequency scattering, and turbulence modeling.

Original: In this paper, we introduce a hierarchical transformer (HT-Net) scheme to efficiently learn the solution operator for multiscale PDEs.
Revised: In this paper, we introduce a hierarchical transformer (HT) scheme to efficiently learn the solution operator for multiscale PDEs.

Original: We construct a hierarchical architecture with scale adaptive interaction range, such that the features can be computed in a nested manner and with a controllable linear cost.
Revised: We construct a hierarchical architecture with a scale-adaptive interaction range, such that the features can be computed in a nested manner and with a controllable linear cost.

Original: Self-attentions over a hierarchy of levels can be used to encode and decode the multiscale solution space over all scale ranges.
Revised: Self-attentions over a hierarchy of levels can be used to encode and decode the multiscale solution space across all scales.

Original: In addition, we adopt an empirical H 1 loss function to counteract the spectral bias of the neural network approximation for multiscale functions.
Revised: In addition, we adopt an empirical H 1 loss function to counteract the spectral bias of the neural network approximation for multiscale functions.

Original: In the numerical experiments, we demonstrate the superior performance of the HT-Net scheme compared with state-of-the-art (SOTA) methods for representative multiscale problems.
Revised: In the numerical experiments, we demonstrate the superior performance of the HT scheme compared with state-of-the-art (SOTA) methods for representative multiscale problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Partial differential equation (PDE) models with multiple temporal/spatial scales are ubiquitous in physics, engineering, and other disciplines.
Revised: Partial differential equation (PDE) models with multiple temporal/spatial scales are ubiquitous in physics, engineering, and other disciplines.

Original: They are of tremendous importance in making predictions for challenging practical problems such as reservoir modeling, fracture propagation, high frequency scattering,atmosphere and ocean circulation, to name a few.
Revised: They are of tremendous importance in making predictions for challenging practical problems such as reservoir modeling, high-frequency scattering, and atmosphere circulation, to name a few.

Original: The complex non-linear interplays of characteristic scales cause major difficulties for the computer simulation of multiscale PDEs.
Revised: The complex nonlinear interplays of characteristic scales cause major difficulties in the computer simulation of multiscale PDEs.

Original: While the resolution of all characteristic scales is prohibitively expensive, sophisticated multiscale methods have been developed to efficiently and accurately solve the multiscale PDEs by incorporating microscopic information, though most of them are designed for problems with fixed input parameters.
Revised: While the resolution of all characteristic scales is prohibitively expensive, sophisticated multiscale methods have been developed to efficiently and accurately solve the multiscale PDEs by incorporating microscopic information. However, most of them are designed for problems with fixed input parameters.

Original: Recently, several novel methods such as Fourier neural operator (FNO) Li et al. (2021), Galerkin transformer (GT) Cao (2021) and deep operator network (DeepONet) Lu et al. (2021) are developed to directly learn the operator (mapping) between infinite dimensional spaces for PDE problems, by taking advantages of the enhanced expressibility of deep neural networks and advanced architectures such as feature embedding, channel mixing and self-attentions.
Revised: Recently, several novel methods such as Fourier neural operator (FNO) (Li et al., 2021), Galerkin transformer (GT) (Cao, 2021) and deep operator network (DeepONet) (Lu et al., 2021) are developed to directly learn the operator (mapping) between infinite dimensional spaces for PDE problems, by taking advantages of the enhanced expressibility of deep neural networks and advanced architectures such as feature embedding, channel mixing, and self-attentions.

Original: Such methods can deal with an ensemble of input parameters, and have great potential for the fast forward and inverse solves of PDE problems.
Revised: Such methods can deal with an ensemble of input parameters and have great potential for the efficient forward and inverse solvers of PDE problems.

Original: However, for multiscale problems, most existing operator learning schemes essentially capture the smooth part of the solution space, and how to resolve the intrinsic multiscale features remains to be a major challenge.
Revised: However, for multiscale problems, most existing operator learning schemes essentially capture the smooth part of the solution space, and how to resolve the intrinsic multiscale features remains to be a major challenge.

Original: In this paper, we design a hierarchical transformer based operator learning method, so that the accurate, efficient and robust computer simulation of multiscale PDE problems with an ensemble of input parameters becomes feasible.
Revised: In this paper, we design a hierarchical transformer based operator learning method, so that the accurate, efficient, and robust computer simulation of multiscale PDE problems with an ensemble of input parameters becomes feasible.

Original: Our main contribution can be summarized in the following,
Revised: Our main contribution can be summarized in the following,

Original: • we develop a novel transformer architecture which allows the decomposition of the input-output mapping to a hierarchy of levels, the features can be updated in a nested manner based on the hierarchical local aggregation of self-attentions with linear computational cost; • we adopt the empirical H 1 loss which avoid the spectral bias and enhance the ability to capture the oscillatory features of the multiscale solution space; • the resulting scheme has significantly better accuracy as well as better generalization properties for multiscale input parameters, compared with state-of-the-art (SOTA) models.
Revised: • we develop a novel transformer architecture that allows the decomposition of the input-output mapping to a hierarchy of levels, the features can be updated in a nested manner based on the hierarchical local aggregation of self-attentions with linear computational cost; • we adopt the empirical H 1 loss which avoids the spectral bias and enhances the ability to capture the oscillatory features of the multiscale solution space; • the resulting scheme has significantly better accuracy and generalization properties for multiscale input parameters, compared with state-of-the-art (SOTA) models.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: We briefly introduce multiscale PDEs in Section § 2.1, then summarize relevant multiscale numerical methods in § 2.2, and neural solvers, in particular neural operators in § 2.3.
Revised: We briefly introduce multiscale PDEs in Section § 2.1, then summarize relevant multiscale numerical methods in § 2.2, and neural solvers, in particular neural operators in § 2.3.

Original: 2. M ULTISCALE PDE S
Revised: 2. M ULTISCALE PDE S

Original: Multiscale PDEs, in a narrower sense, refer to PDEs with rapidly varying coefficients, which may arise from a wide range of applications in heterogenous and random media.
Revised: Multiscale PDEs, in a narrower sense, refer to PDEs with rapidly varying coefficients, which may arise from a wide range of applications in heterogeneous and random media.

Original: In a broader sense, they may form a hierarchy of models at different scales by a systematic derivation, starting from fundamental laws of physics such as quantum mechanics E & Engquist (2003).
Revised: In a broader sense, they may form a hierarchy of models at different scales by a systematic derivation, starting from fundamental laws of physics such as quantum mechanics (E & Engquist, 2003).

Revised: Some outstanding multiscale PDE models may include:

Original: Multiscale elliptic equations is an important class of prototypical examples, such as the following second order elliptic equation of divergence form,
Revised: Multiscale elliptic equations is an important class of prototypical examples, such as the following secondorder elliptic equation of divergence form,

Original: [Equation]
Revised: [Equation]

Original: where 0 < a min ≤ a ( x ) ≤ a max , ∀ x ∈ D , and the forcing term f ∈ H − 1 ( D ; R ) .
Revised: where 0 < a min ≤ a ( x ) ≤ a max , ∀ x ∈ D , and the forcing term f ∈ H − 1 ( D ; R ) .

Original: The coefficient to solution map is S : L ∞
Revised: The coefficient to solution map is S : L ∞

Original: ( D ; R + )
Revised: ( D ; R + )

Original: → H 10 ( D ; R ) , such that u = S ( a ) .
Revised: → H 10 ( D ; R ) , such that u = S ( a ) .

Original: In Li et al. (2021), smooth coefficient a ( x ) is considered and S can be well resolved by the FNO parameterization.
Revised: In (Li et al., 2021), smooth coefficient a ( x ) is considered and S can be well resolved by the FNO parameterization.

Original: The setup a
Revised: The setup a

Original: ( x ) ∈ L ∞ allows rough coefficients with fast oscillation (e.g. a ( x ) = a ( x/ε ) with ε ≪ 1 ), high contrast ratio with a max /a min ≫ 1 , and even a continuum of non-separable scales.
Revised: ( x ) ∈ L ∞ allows rough coefficients with fast oscillation (e.g. a ( x ) = a ( x/ε ) with ε ≪ 1 ), high contrast ratio with a max /a min ≫ 1 , and even a continuum of non-separable scales.

Original: Rough coefficient case is much harder from both scientific computing Branets et al. (2009) and operator learning perspectives.
Revised: The rough coefficient case is much harder from both scientific computing (Branets et al., 2009) and operator learning perspectives.

Original: Other outstanding multiscale PDE models may include:

Original: • Incompressible Navier-Stokes equation, which models thefluid dynamics, when the Reynolds number is large, the flow  becomes turbulent due to the simultaneous interaction of a wide range of  scales of motion, both in time and in space.
Revised: Navier-Stokes equation models the flow of incompressible fluids, which becomes turbulent due to the simultaneous interaction of a wide range of temporal and spatial scales of motion.

Original: • Helmholtz equation, which modelsthe time-harmonic acoustic wave-propagation, its numerical solution exhibits severe difficulties in the high wave number regime due to the interaction of high frequency waves and numerical mesh;
Revised: Helmholtz equation models time-harmonic acoustic waves. Its numerical solution exhibits severe difficulties in the high wave number regime due to the interaction of high-frequency waves and numerical mesh.

Original:  M ULTISCALE S OLVERS FOR M ULTISCALE PDE S
Revised: 2. M ULTISCALE S OLVERS FOR M ULTISCALE PDE S

Original: For multiscale PDEs,  computational cost of classical numerical methods such as finite element methods, finite difference methods etc. usually scales proportional to 1 /ε ≫ 1 .
Revised: For multiscale PDEs, the computational cost of classical numerical methods, such as finite element methods, finite difference methods, etc., usually scales proportionally to 1 /ε ≫ 1 .

Original: Multiscale solvers have been developed such that their computational costs are independent of ε by incorporating microscopic information.
Revised: Multiscale solvers have been developed such that their computational costs are independent of ε by incorporating microscopic information.

Original: Asymptotic and numerical homogenization Asymptotic homogenization Bensoussan et al. (1978) is an elegant analytical approach for multiscale PDEs with scale separation, e.g., a ( x ) = a ( x/ε ) with ε ≪  in equation 2.1.
Revised: Asymptotic and numerical homogenization Asymptotic homogenization (Bensoussan et al., 1978) is an elegant analytical approach for multiscale PDEs with scale separation, e.g., a ( x ) = a ( x/ε ) with ε ≪ 1 in equation 2.1.

Original: Formore general multiscale PDEs with possibly a continuum of scales, numerical homogenization Engquist & Souganidis (2008) offers an effective numerical approach which aims to identify low dimensional approximation spaces such that the approximation bases are adapted to the corresponding multiscale operator, and can be efficiently constructed (e.g. localized).
Revised: For general multiscale PDEs with possibly a continuum of scales, numerical homogenization (Engquist & Souganidis, 2008) offers an effective numerical approach that aims to identify low dimensional approximation spaces such that the approximation bases are adapted to the corresponding multiscale operator, and can be efficiently constructed (e.g. with localized bases).

Original: See Appendix A for details.
Revised: See Appendix A for details.

Original: Multilevel and multiresolution method Multilevel/multigrid methods Hackbusch (1985); Xu &
Revised: Multilevel and multiresolution methods Multilevel/multigrid methods (Hackbusch, 1985; Xu &

Original: Zikatanov (2017) have been successfully applied to PDEs, and classical wavelet based methods Brewster & Beylkin (1995); Beylkin & Coult (1998) enable a multi-resolution decomposition.
Revised: Zikatanov, 2017), and multiresolution methods such as wavelets (Brewster & Beylkin, 1995; Beylkin & Coult, 1998) have been successfully applied to PDEs.

Original: However, the convergence of those methods for multiscale problems can be severely affected by the regularity of coefficients Branets et al.
Revised: However, the convergence of those methods for multiscale problems can be severely affected by the regularity of coefficients (Branets et al., 2009).

Original: (2009) because they are not operator adapted. Recently, the introduction of gamblets Owhadi (2017) opens an avenue to automatically discover scalable multilevel algorithms and operator adapted wavelets for linear PDEs with rough coefficients.
Revised: Recently, the introduction of gamblets (Owhadi, 2017) can be seen as a multilevel extension of numerical homogenization, and it opens an avenue to automatically discover scalable multilevel algorithms and operator-adapted wavelets for linear PDEs with rough coefficients.

Original: Gamblets can be seen as the multilevel extension of numerical homogenization, see also Appendix A.2.
Revised: See also Appendix A.2.

Original: Low Rank Decomposition based Methods It is well-known that the elliptic Green’s function has low rank approximation Bebendorf (2005), which lays the theoretical foundation of the low rank decomposition based numerical methods such as fast multipole method Greengard & Rokhlin (1987);{+Rokhlin, 
+}
Revised: Low-rank decomposition based methods It is well-known that the elliptic Green’s function has lowrank approximation (Bebendorf, 2005), which lays the theoretical foundation of the (near-)linear complexity methods such as fast multipole method (Greengard & {+Rokhlin, 
+}

Original:  Ying et al. hierarchical matrices ( H and H 2 matrices) Hackbusch et al. ; Bebendorf (2008) and hierarchical interpolative factorization Ho & Ying (2016).
Revised: ; Ying et al., 2004), hierarchical matrices ( H and H 2 matrices) (Hackbusch et al., 2002; Bebendorf, 2008), and hierarchical interpolative factorization (Ho & Ying, 2016).

Original: Numerical evidence suggests the robustness and effectiveness of low rank matrix decomposition based methods.

Original: See Appendix B for the connection with our method presented in this paper.
Revised: See Appendix B for the connection with our method presented in this paper.

Original: Tensor Numerical Method Analytical approaches such as periodic unfolding Cioranescu et al. (2008) suggest that low dimensional multiscale operator can be transformed to a high dimensional PDE, and tensor numerical methods, e.g, the sparse tensor finite element method Harbrecht & Schwab (2011) and the quantic tensor train (QTT) method Kazeev et al. (2022), provide efficient numerical procedures to find the low rank tensor representation ofthe multiscale solution.
Revised: Tensor numerical methods Analytical approaches such as periodic unfolding (Cioranescu et al., 2008) suggest that a low-dimensional multiscale PDE can be transformed into a high-dimensional PDE. Therefore, tensor numerical methods, e.g., the sparse tensor finite element method (Harbrecht & Schwab, 2011) and the quantic tensor train (QTT) method (Kazeev et al., 2022), provide efficient numerical procedures to find the low-rank tensor representation of multiscale solutions.

Original: 2. N EURAL O PERATOR FOR PDE S
Revised: 2. N EURAL O PERATOR FOR PDE S

Original: Neural PDE solvers and Spectral Bais Various neural solvers have been proposed in E et al.
Revised: Neural PDE solvers and Spectral Bais Various neural solvers have been proposed in (E et al., 2017;

Original: Sirignano & Spiliopoulos (2018); Raissi et al. (2019) to solve PDEs with fixed parameters.
Revised: Sirignano & Spiliopoulos, 2018; Raissi et al., 2019) to solve PDEs with fixed parameters.

Original: The difficulty to solve multiscale PDEs has already been exhibited by the so-called spectral bias or frequency principle

Original: Rahaman et al. ; Ronen et al. ; Xu et al. (2019), which shows that DNN-based algorithmsoften suffer from the “curse of high-frequency” as they are  inefficient to learn high-frequency components of multi-scale functions.
Revised: The difficulty to solve multiscale PDEs has been exhibited by the so-called spectral bias or frequency principle (Rahaman et al., 2019; Ronen et al., 2019), which shows that DNN-based algorithms are often inefficient to learn highfrequency components of multiscale functions.

Original: A series of algorithms have been developed tosolve multi-scale PDEs and to overcome the high-frequency curse ofgeneral DNNs  (Cai et al., 2020;Li et al., 2020; Wang et al., 2021).
Revised: A series of algorithms have been developed to overcome the high-frequency curse of DNNs and to solve multiscale PDEs (Cai et al., 2020; Wang et al., 2021).

Original: Operator learning for PDEsInstead of solving PDEs with fixed parameters, DNN algorithms demonstrate more potential to learn the input-output mapping of parametric PDEs.
Revised: Operator learning for PDEs DNN algorithms demonstrate more potential to learn the input-output mapping of parametric PDEs.

Original: Finite dimensional operator learning methods such as Zhu & Zabaras (2018); Fan et al. (2019a;b); Khoo et al. (2020) can be applied to problems with fixed discretization.
Revised: Finite-dimensional operator learning methods such as (Zhu & Zabaras, 2018; Fan et al., 2019a;b; Khoo et al., 2020) can be applied to problems with fixed discretization.

Original: In particular, Fan et al. (2019a;b) combines H or H 2 matrices linear operations with nonlinear activation functions, while thecomplex nonlinear geometrical interaction/aggregation is absent, which limits the expressivity of the resulting neural operator.
Revised: In particular, (Fan et al., 2019a;b) combines H or H 2 matrices linear operations with nonlinear activation functions, though the nonlinear geometrical interaction/aggregation is absent, which limits the expressivity of the resulting neural operator.

Original: The infinite dimensional operator learning methods Li et al. ; Gupta et al. (2021) aim to learn the mapping between infinite dimensional Banach spaces, and the convolutions in theconstruction of neural operators are parametrized by, e.g., Fourier or wavelet transform, which is efficient to implement.
Revised: Infinite-dimensional operator learning methods (Li et al., 2021; Gupta et al., 2021) aim to learn the mapping between infinite-dimensional Banach spaces, and the convolutions in the neural operator construction are parametrized by Fourier or wavelet transform.

Original: Nevertheless,even in the linear case, those methods do not always work if multiscale features are present.
Revised: Nevertheless, those methods do not always work even for multiscale PDEs with fixed parameters.

Original: On the other hand,  universal approximation can be rigorously proved for FNOrigorously for FNO operators Kovachki et al. (2021), while “extra smoothness”of input parameters is required to achieve  meaningful decay rate, which either is absent or gives rise to large constants for multiscale PDEs.
Revised: On the other hand, while the universal approximation can be rigorously proved for FNO operators (Kovachki et al., 2021), “extra smoothness” is required to achieve a meaningful decay rate, which either is absent or gives rise to large constants for multiscale PDEs.

Original: This motivates us to construct a new architecture for multiscale operator learning.
Revised: This motivates us to construct a new architecture for multiscale operator learning.

Original: Efficient Attention
Revised: Efficient Attention

Original: The vanilla multi-head self-attention in Vaswani et al. (2017) scales quadratically with the number of tokens, which is prohibitive for high-resolution problems.
Revised: The vanilla multi-head self-attention in (Vaswani et al., 2017) scales quadratically with the number of tokens, which is prohibitive for high-resolution problems.

Original: Many efficient attention methods are proposed using the kernel trick (low rank projection) Choromanski et al.
Revised: Many efficient attention methods are proposed using the kernel trick (low-rank projection) (Choromanski et al., 2020;

Original: ; Wang et al. Peng et al. ; Nguyen et al. (2021) to reduce the computation cost arisen in the dense matrix operation.
Revised: Wang et al., 2020; Peng et al., 2021; Nguyen et al., 2021) to reduce the computation cost arising in the dense matrix operation.

Original: In the context of operator learning, Galerkin transformer Cao (2021) proposed to remove the softmax normalization in self-attention and introduced a linearized self-attention variant with Petrov-Galerkin projection normalization.
Revised: In the context of operator learning, Galerkin transformer (Cao, 2021) proposed to remove the softmax normalization in self-attention and introduced a linearized self-attention variant with Petrov-Galerkin projection normalization.

Original: Furthermore, hierarchical attention using local window aggregation is proposed in Liu et al.
Revised: Furthermore, hierarchical transformers using local window aggregation are proposed in Liu et al.

Original: (2021); Zhang et al.
Revised: (2021b) ; Zhang et al.

Original: (2022) for NLP and vision applications.
Revised: (2022) for NLP and vision applications.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we will introduce our hierarchical transformer model, in particular, thecomputing process over hierarchical levels of complexity, as shown in Figure 3.1.
Revised: We introduce the hierarchical attention model in this section, motivated by the hierarchical matrix, in particular the H 2 matrix variant.

Original: We follow the setupfor operator learning in the pioneering work Li et al. ; Lu et al. (2021) to approximate the operator S : f (cid:55)→ u := S ( f ) , with the input f ∈ A drawn from a distribution µ and the output u ∈ U , where A and U are infinite dimensional Banach spaces respectively.
Revised: We follow the setup in (Li et al., 2021; Lu et al., 2021) to approximate the operator S : a (cid:55)→ u := S ( a ) , with the input a ∈ A drawn from a distribution µ and the output u ∈ U , where A and U are infinite dimensional Banach spaces respectively.

Original: We aim to learn the operator S from a finite collection of finitely observed input-output pairs by constructing a parametric map N : A × Θ → U andusing a loss functional L : U × U → R , such that the optimal parameter θ ∗ =arg min θ
Revised: We aim to learn the operator S from a finite collection of finitely observed input-output pairs through a parametric map N : A× Θ → U and a loss functional L : U × U → R , such that the optimal parameter θ ∗ =

Original:  ∈ Θ E f ∼ µ [ L ( N ( f , θ ) , S ( f ))] .
Revised: arg min θ ∈ Θ E a ∼ µ [ L ( N ( a , θ ) , S ( a ))] .

Original: Our method is motivated by the celebrated hierarchical matrix method, which is an efficient numerical method for the solution of integral and differential equations.

Original: Unlike the kernel trick used in efficient attentions, the H and H 2 matrices use the divide-and-conqure strategy and treat the interactions in space separately by geometrical scales and locality.

Original: The idea is reflected in Liu et al. ; Zhang et al. to a certain degree. In the following, we are going to give a hierarchical nested feature update scheme and rethink the transformer from the hierarchical matrix perspective.
Revised: The input a can then be mapped to features f through patch embedding (Dosovitskiy et al., 2020).

Original: Hierarchical Discretization Wefirst introduce the hierarchical discretization of the spatial domain D , which can be used directly for time independent PDEs such as equation 2.1.
Revised: Hierarchical Discretization We introduce the hierarchical discretization of the spatial domain D , which can be used directly for time-independent PDEs.

Original: Time dependent PDEs can be treated by taking time sliced data as feature channels.
Revised: Time-dependent PDEs can be treated by taking time-sliced data as feature channels.

Original: Let I ( r ) be the finest level index set,
Revised: Let I ( r ) be the finest level index set,

Original: such that each index i =
Revised: such that each index i =

Original: ( i 1 , . .
Revised: ( i 1 , . .

Original: , i r ) ∈
Revised: , i r ) ∈

Original: I ( r ) denotes the finest level spatial objects such as image pixels, finite difference points, etc.
Revised: I ( r ) denotes the finest level spatial objects such as image pixels, finite difference points, etc.

Original: For any i
Revised: For any i

Original: = ( i 1 , . .
Revised: = ( i 1 , . .

Original: , i r ) ∈
Revised: , i r ) ∈

Original: I ( r ) , and 1 ≤ m ≤ r ,we denote i ’s m -th level parent node as i ( m ) =
Revised: I ( r ) , and 1 ≤ m ≤ r , i ( m ) =

Original: ( i 1 , . .
Revised: ( i 1 , . .

Original: , i m )which represents  m -th level objects such as superpixels/aggregates/patches of finer level objects.
Revised: , i m ) represents i ’s m -th level parent node which is the aggregate of finer level objects.

Original: The m -th level index set is I ( m ) : = { i ( m ) : i ∈ I ( r ) } . For i ( m ) ∈
Revised: I ( m ) : = { i ( m ) : i ∈ I ( r ) } is the m -th level index set.

Original: I ( m ) , we denote i ( m,m ′ ) as the m ′ -th

Original: level parent node of i ( m ) if m > m ′ ≥ 1 , and i ( m,m ′ ) :

Original: = { j ∈ I ( m ′ ) | j ( m,m ′ ) = i ( m ) } as the set of them ′ -th level child nodes of i ( m ) if m ≤ m ′ ≤ r . The index(cluster) tree I is induced by the index sets I ( m ) , 1 ≤ m ≤ r , and the parent/child relation.
Revised: We can similarly define the natural parent-child relationship between coarser and finer nodes, which induces the index tree I together with the index sets I ( m ) (1 ≤ m ≤ r ) .

Original: Each index i ∈ I ( m ) corresponds to a spatial object (sometimes denoted as token) which can be characterized e.g. by its position x ( m ) i and a feature vector f ( m ) i with number of channels C ( m ) . 
Revised: Each index i ∈ I ( m ) corresponds to a token, which can be characterized e.g. by its position x ( m ) i and a feature vector f ( m ) i with number of channels C ( m ) . In the following, we will restrict our presentation to the quadtree setting as illustrated in Fig 3.1.

Original: Reduce Operation
Revised: Reduce Operation

Original: The reduce operation defines the map from finer level features to coarser level features.
Revised: The reduce operation defines the map from finer-level features to coarser-level features.

Original: Given i( m ) ∈ I ( m ) and its child nodes i ( m,m +1) ⊂ I ( m +1) , the operator R ( m) maps the m + 1 -th level features with indices in i ( m,m +1) to the m -th level feature with index i ( m ) , namely, f ( m )i =
Revised: For i ∈ I ( m ) , we denote i ( m,m +1) as the set of ( m + 1) -th level child nodes of i . In the quadtree setting, i ( m,m +1) = { ( i, 0) , ( i, 1) , ( i, 2) , ( i, 3) } 1 . The reduce map can be abstractly defined as f ( m )

Original: R (
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ) ( { f ( m +
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ) j } j ∈ i ( m,m +1) ) . The reduce operations can be done recursively from the finest level  to the coarsest level, which leads to a nested representation.
Revised: m,m +1) ) , which maps the ( m + 1) -th level features with indices in i ( m,m +1) to the m -th level feature with index i ( m ) . For simplicity, we use a simple linear layer for the operator R ( m ) in our current implementation, namely, f ( m ) i

Original: In the following, we present a self-attention based local aggregation scheme and a token mixing scheme across multiple levels to go from the coarsest level back to the finest level, which completes a V-cycle of feature update.
Revised: [Equation] where R ( m ) 0 , R ( m ) 1 , R ( m ) 2 , R ( m )∈ R C ( m − 1) ×C ( m ) are parametrized by linear layers. From the perspective of H matrices, those parametrized matrices R ( m ) j embed low-rank approximation to the attention kernel.

Original:  Multilevel Local AggregationWe introduce the self-attention based aggregation.
Revised: Please refer to Appendix B for more details. Multilevel Local Aggregation

Original: The update of the feature f ( r ) ,t i ∈ R C ( r )  with index i ∈ I ( r )  can be given by the followingr -th level token aggregation formula,
Revised: The finest level features f ( r ) ,t i ∈ R C ( r ) at the evolution step t , with index i ∈ I ( r ) , can be updated by the following token aggregation formula through vanilla attention, atten : f ( r ) ,t +

Original: [Equation]
Revised: [Equation]

Original: where the index t denotes the evolution of features, and N ( r ) :
Revised: i [Equation] f ( r ) ,t j ) , for i ∈ I ( r ) , (3.1) with N ( r ) :

Original: = |I
Revised: = |I

Original: ( r ) |is the number of features at level r .
Revised: ( r ) | .

Original: In the context of vanilla attention, we ignore the softmax normalizing factor and only consider 1 head attention for simplicity of notations, the interaction kernel G can be simplified as G ( f i , f j ) =
Revised: For simplicity, we ignore the softmax normalizing factor and only consider single-head attention, and we assume that the interaction kernel G is of the form G ( f i , f j ) =

Original: exp( W Q f i · W K f j ) =
Revised: exp( W Q f i · W K f j ) =

Original: exp( q i · k j ) , where q i :=
Revised: exp( q i · k j ) , where q i :=

Original: W Q f i , k i :=
Revised: W Q f i , k i :=

Original: W K f i , v i := v ( f i ) =
Revised: W K f i , v i := v ( f i ) =

Original: W V f i , and W Q , W K , W V ∈ R C ( r ) ×C ( r ) are learnable matrices.
Revised: W V f i , and W Q , W K , W V ∈ R C ( r ) ×C ( r ) are learnable matrices.

Original: Instead of calculating equation 3.1 explicitly with O ( N 2 ) cost, we propose a self-attention based local aggregation scheme, inspired by the H 2 matrices Hackbusch (2015), see also
Revised: Instead of computing equation 3.1 explicitly with O ( N 2 ) cost, we propose a self-attention based local aggregation scheme, inspired by the H 2 matrices (Hackbusch, 2015), see also

Original: Appendix B.
Revised: Appendix B.

Original: The local aggregation at the r -th level writes,
Revised: The local aggregation at the r -th level writes,

Revised: Figure dex tree.

Revised: In this example, the 2D unit square is hierarchically discretized into three levels which are indexed by I (1) , I (2) and I (3) , respectively.

Revised: For example, we denote by (1) (1 , 2) =

Revised: { (1 , 0) , (1 , 1) , (1 , 2) , (1 , 3) } the set of the second level child nodes of the node (1) .

Original: [Equation]
Revised: [Equation]

Original: where N ( r ) ( i ) is the set of the r -th level neighbor tokens of i
Revised: where N ( r ) ( i ) is the set of the r -th level neighbors of i

Original: ∈ I ( r ) , q ( r ) ,t i :=
Revised: ∈ I ( r ) , q ( r ) ,t i [Equation]

Original: ˆ W Q f( r ) ,t i , k ( r ) ,t i :=
Revised: ˆ [Equation] K f

Original: ˆ W K f ( r ) ,t i , v ( r ) ,t i :=
Revised: [Equation] v ( r ) ,t i : [Equation] ˆ

Original: ˆ W V f ( r ) ,t i with learnable matrices ˆ W Q , ˆ W K , ˆ W V ∈ R C ( r ) ×C ( r ) .
Revised: [Equation] with learnable matrices ˆ W Q , ˆ W K , ˆ W V ∈ R C ( r ) ×C ( r ) .

Original: Then for m = r − 1 , ..., 1 , we calculate the aggregation in each level with nested q ( m ) ,t i, k ( m ) ,t j , v ( m ) ,t j , atten ( m ) loc : f ( m ) ,t +1 i = (cid:88) j ∈N ( m ) ( i ) exp( q ( m ) ,t i · k ( m ) ,t j ) v ( m ) ,t j , for i ∈
Revised: Then for m = r − 1 , ..., 1 , we calculate the local attention in each level with nested q ( m ) ,t i

Original: I ( m ) (3.3) where q ( m ) ,t i = R ( m ) ( { q ( m +1) ,t j ) } 
Revised: [Equation] where q ( m ) ,t i = R ( m ) ( { q ( m +1) ,t j ) } j ∈

Original: j ∈
Revised: i

Original: i ( m,m +1) ) , k ( m ) ,t i =
Revised: ( m,m +1) ) , k ( m ) ,t i =

Original: R ( m ) ( { k ( m +1) ,t j ) } j ∈
Revised: R ( m ) ( { k ( m +1) ,t j ) } j ∈

Original: i
Revised: i

Original: ( m,m +1) ) , v ( m ) ,t i =
Revised: ( m,m +1) ) , v ( m ) ,t i =

Original: R ( m ) ( { v ( m +1) ,t j ) } j ∈
Revised: R ( m ) ( { v ( m +1) ,t j ) } j ∈

Original: i
Revised: i

Original: ( m,m +1) ) , N ( m ) ( i ) is the set of the m -th level neighbor tokens of i ∈
Revised: ( m,m +1) ) , N ( m ) ( i ) is the set of the m -th level neighbors of i ∈

Original: I ( m ) .
Revised: I ( m ) .

Original: [-Remark 
-]
Revised: [-Remark 
-]Decompose Operation To mix the multilevel features f ( m ) ,t i , m = 1 , ..., r , we propose a decompose operation that reverses the reduce operation. The decompose operator D ( m ) : f ( m ) ,t i (cid:55)→ { f ( m +1) ,t + 12 j } j ∈ i ( m,m +1) , maps the m -th level feature with index i to ( m + 1) -th level features associated to its child set i ( m,m +1) .

Original: Inour paper, for simplicity, the learnable operators R ( m  ) and D ( m ) only consist of fully connected linear layers or convolution layers without nonlinear activation function.{+,T j f ( m ) ,t i , for j = 0 , 1 , 2 , 3 , with parameter matrices D ( m ) j ∈ R C ( m ) ×C ( m +1) . f ( m +1) ,t + 1 
 j is further aggregated to ¯ f ( m +1) ,t +1 j such that f ( m +1) ,t +1 j = ¯ f ( m +1) ,t +1 j + f ( m +1) ,t + 1 2 j .+}
Revised: In the current implementation, we use a simple linear layer such that f ( m +1) ,t + 12 ( i,j ) = D ( m ) {+,T j f ( m ) ,t i , for j = 0 , 1 , 2 , 3 , with parameter matrices D ( m ) j ∈ R C ( m ) ×C ( m +1) . f ( m +1) ,t + 1 
 j is further aggregated to ¯ f ( m +1) ,t +1 j such that f ( m +1) ,t +1 j = ¯ f ( m +1) ,t +1 j + f ( m +1) ,t + 1 2 j .+}

Original: {+Remark 
+}
Revised: {+Remark 
+}

Revised: In our current implementation, the operators R ( m ) and D ( m ) only consist of linear layers.

Original: In general, these operators are not limited to linear operators.
Revised: In general, these operators may contain nonlinear activation functions.

Original: {+Remark 
+} The nested learnable operators R ( m ) also induce the channel mixing andis equivalent to a structured parameterization of W Q , W V , W K matrix for the coarse level tokens.
Revised: {+Remark 
+} The nested learnable operators R ( m ) and R ( m ) induce the channel mixing and a structured parameterization of W Q , W V , W K matrix for the coarse level tokens.

Original: See Appendix B for the discussion.
Revised: See Appendix B for details.

Original: Decompose Operation To mix the multilevel features f ( m ) i , m = 1 , ..., r , we propose a decompose operation which reverses the reduce operation.

Original: The decompose operator D ( m ) : f ( m ) ,t i (cid:55)→ { f ( m +1) ,t + 12 j } j ∈

Original: i

Original: ( m,m +1) , maps the m -th level feature with index i to m + 1 -th level features associated to its child set i ( m,m +1) .

Original: f ( m +1) ,t + 12 j is further aggregated to f ( m +1) ,t +1 j + = f ( m +1) ,t + 12 j .

Original: In summary, we have thefollowing algorithm for hierarchical attention, which can drop-in substitute the canonical attention. j ,t + 12 i , for any i ∈ I ( m +1) Output : f ( r ) ,t +1 i for any i ∈ I ( r ) .
Revised: We summarize the hierarchical attention algorithm in the following.

Original: Proposition 3.1 (Complexity of Algorithm 1)
Revised: Proposition 3.1 (Complexity of Algorithm 1)

Original: The reduce operation, multilevel aggregation, and decomposition operation together form a V-cycle for the update of features, as illustrated in Figure 3.1.
Revised: The reduce operation, multilevel aggregation, and decomposition operation together form a V-cycle for the update of features, as illustrated in Figure 3.2.

Original: The cost of one V-cycle is O ( N ) if I is a quadtree as implemented in the paper.
Revised: The cost of one V-cycle is O ( N ) if I is a quadtree as implemented in the paper.

Original: See Appendix C for the proof.
Revised: See Appendix C for the proof.

Revised: Remark 3 (Technical contribution)

Original: Decoder The decoder maps the features f (at the last update step) to the solution u .
Revised: In Liu et al. (2021b) ; Zhang et al. (2022), spatial resolutions are sequentially reduced, and attentions are performed at each level separately.

Revised: However, attentions are only used for a fixed level, and there is no multilevel attention-based aggregation.

Revised: The coarse latent variables are used for classification and generation tasks, and fine-scale information may get lost in the reduce(coarsening) process.

Revised: While in the HT-Net architecture, we are motivated by multiscale numerical methods such as numerical homogenization and hierarchical matrix method.

Revised: Attention-based local aggregations are constructed at each level, and features from all levels are summed up to form the updated fine-scale features, which enables the recovery of the fine details with linear cost.

Revised: Decoder The decoder maps the features f (at the last update step) to the solution u .

Original: The decoder is often chosen with prior knowledge of the PDE.
Revised: The decoder is often chosen with prior knowledge of the PDE.

Original: A simple feedforward neural network (FFN) is used in Lu et al.
Revised: A simple feedforward neural network (FFN) is used in Lu et al.

Original: (2021) to learn a basis set as the decoder.
Revised: (2021) to learn a basis set as the decoder.

Original: A good decoder can also be learned from the data using SVD Bhattacharya et al.
Revised: A good decoder can also be learned from the data using SVD (Bhattacharya et al., 2021).

Original: In this paper, we use the spectral convolution layers in Li et al.
Revised: In this paper, we use the spectral convolution layers in Li et al.

Original: Loss functions The choice of loss functions is crucial forthe efficient trainingprocess and the generalization.
Revised: Loss functions Loss functions are crucial for efficient training and robust generalization of neural network models.

Original: Due to the multiscale nature of the problem considered in this paper, we prefer to choose H 1 loss instead of the usual L 2 loss function, as it puts more ”weights” on the high frequency components. 
Revised: For multiscale problems considered in this paper, we prefer to use H 1 loss instead of the usual Lloss function, as it puts more ”weights” on high-frequency components. (Yu et al., 2022) adopted Sobolev norm based loss function for function approximation, where H 1 loss is defined for the neural network function itself. On the contrary, we define the H 1 loss on the target solution space, which measures the distance between the prediction ˆ u :=

Original:  We refer the readers to Adams & Fournier (2003) for detailed definition of Sobolev space and H 1 norm, and to the next section for the implementation.
Revised: N ( a ) and the ground truth u . We refer the readers to Adams & Fournier (2003) for the definition of Sobolev spaces and H 1 norm, and to the next section for the implementation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We demonstrate the effectiveness of the hierarchical transformer model (HT-Net) onoperator learning tasks for multiscale PDEs to showcase its qualities.
Revised: We demonstrate the effectiveness of the hierarchical transformer model (HT-Net) on multiscale operator learning.

Original: All our examples are in 2D. We first study the ability of the model on the multiscale elliptic equation with two phase coefficients which is a widely used benchmark problem for operator learning Li et al.
Revised: All our examples are in 2D. We first study the ability of the model on a multiscale elliptic equation with two-phase coefficients which is a widely used benchmark problem for operator learning (Li et al., 2021).

Original: (2021), and the “multiscaleness” of the coefficients such as the oscillation, contrast and roughness of the two phase interface can be tuned in the model. We demonstrate the accuracy and  robustness of the HT-NET  for both smooth and rough coefficients, and also show that HTNET is able to provide better generalization error for out-of-distribution input parameters.
Revised: As the “multiscaleness” of the coefficients such as the oscillation, contrast, and roughness of the two-phase interface can be fine-tuned, we showcase the accuracy and the robustness of the HT-NET model for both smooth and rough coefficients. We also show that HT-NET is able to provide better generalization errors for out-of-distribution input parameters.

Original: We also test the Navier-Stokes equation with  large Reynolds number, as an example with non-linearity and time dependence.
Revised: We also test the Navier-Stokes equation with a large Reynolds number as an example with non-linearity and time dependence.

Original: 4. S ETUP
Revised: 4. S ETUP

Original: In our experiments, the spatial domain is D :=
Revised: In our experiments, the spatial domain is D :=

Original: [0 , 1] 2 and is discretized uniformly with h = 1 /n .
Revised: [0 , 1] 2 and is discretized uniformly with h = 1 /n .

Original: Let the uniform grid be G 2 :
Revised: Let the uniform grid be G 2 :

Original: = { ( x i , x j ) =
Revised: = { ( x i , x j ) =

Original: ( ih, jh ) | i, j = 0 , ..., n − 1 } .
Revised: ( ih, jh ) | i, j = 0 , ..., n − 1 } .

Original: { ( a j , u j ) } Nj =1 are functions pairs such that u j = S ( a j ) , and a j is drown from some probability measure µ .
Revised: { ( a j , u j ) } Nj =1 are functions pairs such that u j = S ( a j ) , and a j is drown from some probability measure µ .

Original: The actual data pairs for training and testing are pointwise evaluations of a j and u j on the grid G 2 , denoted by a j and u j  respectively.
Revised: The actual data pairs for training and testing are pointwise evaluations of a j and u j on the grid G 2 , denoted by a j and u j , respectively.

Original: The comparison with all the baselines are consistent with (in most time better than) references.
Revised: The comparison with all the baselines is consistent with (most time better than) references.

Original: The hierarchical index tree I can be generated by the corresponding quadtree representation of the nodes with depth r , such that the finest level objects are pixels or patches aggregated by pixels.
Revised: The hierarchical index tree I can be generated by the corresponding quadtree representation of the nodes with depth r , such that the finest level objects are pixels or patches aggregated by pixels.

Original: See Appendix D for more detail.
Revised: See Appendix D for more details.

Original: Empirical H 1 loss function Empirical L 2 loss function is defined as L L ( { ( a j , u j ) } Nj =1 ; θ ) :
Revised: Empirical H 1 loss function Empirical L 2 loss function is defined as L L ( { ( a j , u j ) } Nj =1 ; θ ) :

Original: = 1 N (cid:80) Ni =1 ∥
Revised: = 1 N (cid:80) Ni =1 ∥

Original: u j − N ( a j ; θ ) ∥
Revised: u j − N ( a j ; θ ) ∥

Original: l 2 / ∥ u j ∥
Revised: l 2 / ∥ u j ∥

Original: l 2 , where ∥ · ∥ l 2 is the canonical l 2 vector norm.
Revised: l 2 , where ∥ · ∥ l 2 is the canonical l 2 vector norm.

Original: For any ξ ∈ Z 2 n := (cid:8) ξ ∈ Z 2 | − n/ 2 + 1 ⩽ ξ j ⩽ n/ 2 , j = 1 , 2 (cid:9) , the normalized discrete Fourier transform (DFT) coefficients of f writes F ( f )( ξ ) :
Revised: For any ξ ∈ Z 2 n := (cid:8) ξ ∈ Z 2 | − n/ 2 + 1 ⩽ ξ j ⩽ n/ 2 , j = 1 , 2 (cid:9) , the normalized discrete Fourier transform (DFT) coefficients of f writes F ( f )( ξ ) :

Original: = 1 √ n (cid:80) x ∈ G 2 f ( x ) e − 2 iπx · ξ .
Revised: = 1 √ n (cid:80) x ∈ G 2 f ( x ) e − 2 iπx · ξ .

Original: Then we define ∥ u

Original: ∥ h := (cid:113)(cid:80) ξ ∈ Z 2 n | ξ | 2 ( F ( u )( ξ )) 2 .

Original: The empirical H 1 loss function is given by,
Revised: The empirical H 1 loss function is given by,

Original: [Equation]
Revised: L H ( { ( a j , u j ) } Nj =1 ; θ ) : = 1 N (cid:80) Ni =1 ∥ u j − N ( a j ; θ ) ∥ h / ∥

Revised: u j ∥ h , where ∥ u ∥ h := (cid:113)(cid:80) ξ ∈ Z 2 n | ξ | 2 ( F ( u )( ξ )) 2 .

Original: L H can be viewed as a weighted L L loss with weights | ξ | 2 , which force the operator to capture the high frequency components in the solution.
Revised: L H can be viewed as a weighted L L loss with weights | ξ | 2 , which forces the operator to capture highfrequency components in the solution.

Original: In this work, we use the equivalent frequency space representation of  discrete H 1 norm, and in general, discrete H 1 norm in real space can also be adapted.
Revised: In this work, we use the equivalent frequency space representation of the discrete H 1 norm, and in general, discrete H 1 norm in real space can also be adopted.

Original: 4. M ULTISCALE E LLIPTIC E QUATION
Revised: 4. M ULTISCALE E LLIPTIC E QUATION

Original: [Figure]
Revised: [Figure]

Original: We apply the HT-Net  to learn the coefficient to solution operator for equation 2.1.
Revised: We apply the HT-Net model to learn the coefficient to solution operator for equation 2.1.

Original: We use the two-phase coefficient model inthe FNO paper Li et al.
Revised: We use the twophase coefficient model in Li et al.

Original: (2021), which is also benchmarked in Gupta et al.
Revised: (2021), which is also benchmarked in Gupta et al.

Original: (2021); Cao (2021).
Revised: (2021); Cao (2021).

Original: In previous works, the coefficients do not oscillate much and the solutions look smooth.
Revised: In previous works, the coefficients do not oscillate much and the solutions look smooth.

Original: We change the parameters which control the smoothness and contrast of the coefficients, such that the solutions contain more roughness, see Figure 4.1.
Revised: We change the parameters which control the smoothness and contrast of the coefficients, such that the solutions contain more roughness than the Darcy benchmark in Li et al. (2021), see Figure 4.1.

Original: We refer the readers to Appendix E for data generation details.
Revised: We refer the readers to Appendix E for data generation details.

Original: We also include experiments for multiscale trigonometric coefficients with higher contrast in Appendix F.1.
Revised: We also include experiments for multiscale trigonometric coefficients with higher contrast.

Original: We list the relative H 1 and L 2 testing errors for the cases with smooth and rough two-phase coefficients  in Table 1, which demonstrate that HT-Net outperforms other methods by a huge margin, for both smooth and rough cases. In particular, the comparison between HT-Net and its two level version indicates that increasing the number of levels can increase accuracy
Revised: Detailed description on multiscale trigonometric coefficients is in Appendix F.1.

Original: [Table]
Revised: [Table]

Original:  Spectral Bias in Operator Learning We have observed the spectral bias phenomena in thetraining of neural operators.
Revised: [Figure] Spectral Bias in Operator Learning We demonstrate the spectral bias phenomena in the neural operator training in Figure 4.2.

Original: In Figure 4.2, we compare HT-Net trained with H 1 and L 2 losses, and show the evolutions of errors in the frequency domain as well as the loss curves.
Revised: We compare HT-Net trained with H 1 and L 2 losses, and show the evolution of errors in the frequency domain as well as the loss curves.

Original: Comparison shows that HT-Net with H 1 loss has faster decay for high frequencies, and the errors over all the frequencies decay more uniformly.
Revised: The comparison shows that HT-Net with H 1 loss has faster decay for high frequencies, and the errors over all the frequencies decay more uniformly.

Original: Also, the

Original:  HT-Net  with H 1 loss  has better testing and generalization errors, although the training errors are comparable.
Revised: The HT-Net trained with H 1 loss also has better testing and generalization errors.

Original: [Figure]

Original: Generalization Errors HT-Net generalizes better on test data from samples out of distribution.
Revised: Generalization Errors HT-Net generalizes better on test data from samples out of distribution.

Original: In addition to what we have showed in (c,d) of Figure 4.2, we compare HT-Net with other models in Table 2.
Revised: In addition to what we have shown in (c,d) of Figure 4.2, we compare HT-Net with other models in Table 2.

Original: The models are trained with the rough two-phase dataset in Table 4.1, and tested on out-of-distribution data.
Revised: The models are trained with the rough two-phase dataset in Table 4.1 and tested on out-of-distribution data.

Revised: FNO2D His FNO model but trained using our H 1 loss.

Original: HT-Net outperforms other models byan order of magnitude. 
Revised: HT-Net outperforms other models by order of magnitude. [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original:  N AVIER S TOKES 
Revised: 4. N AVIER -S TOKES E QUATION

Original: We consider the 2D Navier-Stokes equation in vorticity form on the unit torus, which is also benchmarked in Li et al.
Revised: We consider the 2D Navier-Stokes equation in vorticity form on the unit torus, which is also benchmarked in Li et al.

Original: [Equation]
Revised: [Equation]

Original: with velocity u , vorticity w = ∇ × u , initial vorticity w 0 , viscosity ν > 0 ( ∼ Re − 1 with Re being the
Revised: with velocity u , vorticity w = ∇ × u , initial vorticity w 0 , viscosity ν > 0 ( ∼ Re − 1 with Re being the

Original: Reynolds number), and forcing term f .
Revised: Reynolds number), and forcing term f .

Original: We learn the operator S : w ( · , 0 ≤ t ≤ 10) → w ( · , 10 ≤ t ≤ T ) , mapping the vorticity up to time 10 to the vorticity up to some later time T > 10 .
Revised: We learn the operator S : w ( · , 0 ≤ t ≤ 10) → w ( · , 10 ≤ t ≤ T ) , mapping the vorticity up to time 10 to the vorticity up to some later time T > 10 .

Original: We experiment with viscosities ν = 1e − 3 , 1e − 4 , 1e − 5 , and decrease the final time T accordingly as the Reynolds numbers increase and the dynamics become chaotic.
Revised: We experiment with viscosities ν = 1e − 3 , 1e − 4 , 1e − 5 , and decrease the final time T accordingly as the Reynolds numbers increase and the dynamics become chaotic.

Original: Time dependent neural operator Following the set up in Li et al.
Revised: Time dependent neural operator Following the setup in Li et al.

Original: (2021), we fix the resolution to be 64 × 64 for both training and testing.
Revised: (2021), we fix the resolution as 64 ×for both training and testing.

Original: The ten time slices of solution w ( · , t ) at t = 0 , ..., 9 are taken as the input data to the neural operator N which maps the solution at the previous 10 time steps to the next time step (2D functions to 2D functions).
Revised: The ten time-slices of solutions w ( · , t ) at t = 0 , ..., 9 are taken as the input data to the neural operator N which maps the solutions at the previous 10 timesteps to the next time step (2D functions to 2D functions).

Original: This procedure can be repeated recurrently until the final time T , which is often called the rolled-out prediction.
Revised: This procedure, often referred to as the rolled-out prediction, can be repeated recurrently until the final time T .

Original: We list the results of HT-Net together with FNO-3D (convolution in space-time), FNO-2D, U-Net in Ronneberger et al. (2015) in Table 3, andwe observe that HT-Net is significantly better than other methods. 
Revised: We list the results of HT-Net, FNO-3D (convolution in space-time), FNO2D, U-Net in Ronneberger et al. (2015) in Table 3, and HT-Net is significantly better than other methods. [Equation]

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have built HT-Net, a hierarchical transformer based operator learning model for multiscale PDEs, which allows nested computation of features and self-attentions, and in turn provide a  representation for the multiscale solution space.
Revised: We have built HT-Net, a hierarchical transformer based operator learning model for multiscale PDEs, which allows nested computation of features and self-attentions, and provides a hierarchical representation for the multiscale solution space.

Original: The reduce operations, multilevel local aggregations [Table] and decompose operations form a fine-coarse-fine V-cycle for the feature update.
Revised: The reduce, multilevel local aggregation, and decompose operations form a finecoarse-fine V-cycle for the feature update.

Original: We also introduced the empirical H 1 loss to reduce the spectral bias for theapproximation of multiscale functions.
Revised: We also introduced the empirical H 1 loss to reduce the spectral bias in the multiscale operator learning.

Original: HT-Net can provide much better accuracy and robustness compared with state-of-the-art (SOTA) neural operators, which is demonstrated by the multiscaleelliptic and Naiver-Stokes benchmarks. 
Revised: HT-Net provides much better accuracy and robustness compared with state-of-the-art (SOTA) neural operators, which is demonstrated by several multiscale benchmarks. Limitation and outlook (1)

Original: In this paper, we use regular grid for the discretization ofPDE, HT-Net can be extended to more flexible setups such as scattered points and graph neural networks, which offers more opportunities to take advantage of the hierarchical representation. 
Revised: The current implementation of HT-Net relies on a regular grid. The extension to data clouds and graph neural networks will offer more opportunities to take advantage of the hierarchical representation. The current implementation of the attention-based operator lacks the flexibility to generalize to a different resolution as FNO (Li et al., 2021)).

Revised: However, for multiscale problems, the discretization invariance of FNO and MWT models may be hampered by the aliasing error in the frequency domain, which are validated by the experiment results.

Revised: Recent convergence analysis of operator learning methods such as

Revised: Kovachki et al.

Revised: Lanthaler et al.

Revised: (2022) leverage the smoothness of the solutions, however, multiscale

Revised: PDEs usually have lower regularity and do not fall in such categories.

Revised: For HT-Net, though it might be possible to achieve (approximate) discretization invariance by using proper sampling and interpolation modules, we believe a full treatment will be operator adaptive.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This work proposes a hierarchical transformer operator learning model for multiscale PDEs.
Revised: This work proposes a hierarchical transformer operator learning model for multiscale PDEs.

Original: As stated in the introduction, solving multiscale PDEs is associated with some of the most challenging practical problems such as reservoir modeling, fracture and fatigue prediction, high frequency scattering, weather forecasting, etc.
Revised: As stated in the introduction, solving multiscale PDEs is associated with some of the most challenging practical problems, such as reservoir modeling, fracture and fatigue prediction, high frequency scattering, weather forecasting, etc.

Original: Potentially, HT-Net solvers could help to reduce the prohibitively expensive computational cost of those simulations.
Revised: Potentially, HT-Net solvers could help to reduce the prohibitively expensive computational cost of those simulations.

Original: The negative consequences are not obvious.
Revised: The negative consequences are not obvious.

Original: Though in theory any technique can be misused, it is not likely to happen at the current stage.
Revised: Though, in theory, any technique can be misused, it is not likely to happen at the current stage.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The datasets for Section 4 are either downloaded (for smooth two-phase coefficients and Navier-Stokes) from https://github.com/zongyi-li/fourier_neural_operator , or generated (for rough
Revised: The datasets for Section 4 are either downloaded (for smooth two-phase coefficients and Navier-Stokes) from https://github.com/zongyi-li/fourier_neural_operator , or generated (for rough

Original: Darcy coefficients) using the code from the same website.
Revised: Darcy coefficients) using the code from the same website.

Original: We implemented P 1 finite element method in
Revised: We implemented P 1 finite element method in

Original: MATLAB to solve equation 2.1 with multiscale trigonometric coefficients in Appendix F.1, and in FreeFEM to solve the Helmholtz equation in Appendix F.2.
Revised: MATLAB to solve equation 2.1 with multiscale trigonometric coefficients in Appendix F.1, and in FreeFEM to solve the Helmholtz equation in Appendix F.2.

Original: We have included introductions of the relevant mathematical and data generation concepts in the appendix.
Revised: We have included introductions of the relevant mathematical and data generation concepts in the appendix.

Original: We put the code for and also a link for datasets at the anonymous Github page https://github.com/
Revised: We put the code for and also a link for datasets at the anonymous Github page https://github.com/

Original: Shengren-Kato/VFMM-ICLR2023.git .
Revised: Shengren-Kato/VFMM-ICLR2023.git .

Original: Supplementary descriptions of the code are also provided in the page.
Revised: Supplementary descriptions of the code are also provided in the page.

Original: A M ULTISCALE PDE
Revised: A M ULTISCALE PDE

Original: In this section, we introduce some mathematical and numerical concepts related to multiscale PDEs.
Revised: In this section we introduce some mathematical and numerical concepts related to multiscale PDEs.

Original: A. A SYMPTOTIC H OMOGENIZATION
Revised: A. A SYMPTOTIC H OMOGENIZATION

Original: Here we introduce some basic formulation of asymptotic homogenization. We suppose that a ( x ) in equation 2.1 takes a special form a ( xε ) , namely,
Revised: We introduce some basic formulation of asymptotic homogenization, by assuming that a ( x ) = a ( xε ) with 1-periodic funciton a ( · ) in equation 2.1, namely,

Original: [Equation]
Revised: [Equation]

Original: It can be derived from  asymptotic expansion of the two-scale function u ε = u ( x, x/ε ) in ε , and justified rigorously that u ε ⇀ u 0 in H 1 , with u 0 ( x ) the solution of the homogenized problem
Revised: It can be derived from the asymptotic expansion of the two-scale function u ε = u ( x, x/ε ) in ε , and justified rigorously that u ε ⇀ u 0 in H 1 , with u 0 ( x ) the solution of the homogenized problem

Original: [Equation]
Revised: [Equation]

Original: which contains only coarse scale information.
Revised: which contains only coarse scale information.

Original: The homogenized coefficient a 0 can be computed by the formula a 0 ij = (cid:82) Y ( e i + ∇ χ i ) T a ( y)
Revised: The homogenized coefficient a 0 can be computed by the formula a 0 ij = (cid:82) Y ( e i + ∇ χ i [Equation] T a ( y

Revised: [Equation]

Original: ( e j + ∇ χ j ) dy , where χ i solves the cell problems given by
Revised: e j + ∇ χ j [Equation] dy , where χ i solves the cell problems given by

Original: [Equation]
Revised: [Equation]

Original: with Y being the d dimensional torus.
Revised: with Y being the d dimensional torus.

Original: Asymptotic homogenization provides an approximate solution ˆ u ε = u 0 − ε (cid:80) χ
Revised: Asymptotic homogenization provides an approximate solution ˆ u ε = u 0 − ε (cid:80) χ

Original: i (cid:0) xε (cid:1) ∂u 0 ∂x i such that ∥ u ε − ˆ u ε ∥ H 1 ≤ cε 12 .
Revised: i (cid:0) xε (cid:1) ∂u 0 ∂x i such that ∥ u ε − ˆ u ε ∥ H 1 ≤ cε 12 .

Original: We note that to find the homogenization approximation ˆ u ε only requires the coarse scale solution u 0 and precomputation of d cell problems for χ i which do not depend on f and D .
Revised: We note that to find the homogenization approximation ˆ u ε only requires the coarse-scale solution u 0 and precomputation of d cell problems for χ i which do not depend on f and D .

Original: A. N UMERICAL H OMOGENIZATION AND M ULTILEVEL /M ULTIGRID M ETHODS
Revised: A. N UMERICAL H OMOGENIZATION AND M ULTILEVEL /M ULTIGRID M ETHODS

Original: Given the smallest scale of the multiscale problem ε and a coarse computational scale determined by the available computational power and the desired precision, with ε ≪ H ≪ 1 .
Revised: Given the smallest scale of the multiscale problem ε and a coarse computational scale determined by the available computational power and the desired precision, with ε ≪ H ≪ 1 .

Original: The goal of numerical homogenization is to construct a finite dimensional approximation space V H and to seek an approximate solution u H ∈ V H , such that, the accuracy estimate ∥
Revised: The goal of numerical homogenization is to construct a finite-dimensional approximation space V H and to seek an approximate solution u H ∈ V H , such that, the accuracy estimate ∥

Original: u − u H ∥ ≤ CH α holds for optimal choices of the norm ∥·∥ and the exponent α , and optimal computational cost holds with V H constructed via precomputed subproblems which are localized, independent and do not depend on the RHS and boundary condition of the problem.
Revised: u − u H ∥ ≤ CH α holds for optimal choices of the norm ∥·∥ and the exponent α , and optimal computational cost holds with V H constructed via pre-computed subproblems which are localized, independent and do not depend on the RHS and boundary condition of the problem.

Original: In recent two decades, great progress have been made in this area Hou et al. (1999); M ˚ alqvist & Peterseim (2014); Owhadi & Zhang (2007), approximation spaces with optimal accuracy (in the sense of Kolmogorov
Revised: In recent two decades, great progress has been made in this area (Hou et al., 1999; M ˚ alqvist & Peterseim, 2014; Owhadi & Zhang, 2007), approximation spaces with optimal accuracy (in the sense of Kolmogorov

Original: N -width Berlyand & Owhadi (2010)) and cost can be constructed for elliptic equation with fixed rough coefficients.
Revised: N -width (Berlyand & Owhadi, 2010)) and cost can be constructed for elliptic equations with fixed rough coefficients.

Original: For multiscale PDEs, operator learning methods can be seen as a step forward from numerical homogenization, since they can be applied to an ensemble of coefficients, and the decoder can be interpreted as the basis of the underlying problem as well.
Revised: For multiscale PDEs, operator learning methods can be seen as a step forward from numerical homogenization, since they can be applied to an ensemble of coefficients, and the decoder can be interpreted as the basis of the underlying problem as well.

Original: For multiscale PDEs, multilevel/multigrid methods can be seen as the multilevel generalization of the numerical homogenization methods.
Revised: For multiscale PDEs, multilevel/multigrid methods can be seen as the multilevel generalization of the numerical homogenization methods.

Original: Numerical homogenization can provide coarse spaces with optimal approximation and localization properties.
Revised: Numerical homogenization can provide coarse spaces with optimal approximation and localization properties.

Original: Recently, operator-adapted wavelets (gamblets) Owhadi (2017); Xie et al. (2019) have been developed, and enjoy three properties that are ideal for the construction of efficient direct methods: scale orthogonality, well-conditioned multi-resolution decomposition, and localization.
Revised: Recently, operator-adapted wavelets (gamblets) (Owhadi, 2017; Xie et al., 2019) have been developed, and enjoy three properties that are ideal for the construction of efficient direct methods: scale orthogonality, well-conditioned multi-resolution decomposition, and localization.

Original: Gamblets has been generalized to solve Navier-Stokes equation Budninskiy et al. and Helmholtz equation Hauck & Peterseim (2022) efficiently.
Revised: Gamblets has been generalized to solve Navier-Stokes equation (Budninskiy et al., 2019) and Helmholtz equation (Hauck & Peterseim, 2022) efficiently.

Original: B H IERARCHICAL M ATRIX P ERSPECTIVE
Revised: B H IERARCHICAL M ATRIX P ERSPECTIVE

Original: The hierarchical nested attention  Algorithm 1 resembles the celebrated hierarchical matrix method Hackbusch (2015), in particular, the H 2 matricies from the perspective of matrix operations.
Revised: The hierarchically nested attention in Algorithm 1 resembles the celebrated hierarchical matrix method (Hackbusch, 2015), in particular, the H 2 matrix from the perspective of matrix operations.

Original: In the following,  we demonstrate Algorithm 1 in matrix formulas.
Revised: In the following, During we take the binary tree-like hierarchical discretization shown in Fig B.1 as an example to illustrate the reduce operation, decompose operation, and multilevel token aggregation in Algorithm 1 in matrix formulas.

Original: STEP 0:
Revised: STEP 0:

Original: Get the  q ( r ) ,t i ,  k ( r ) ,t j ,  v ( r ) ,t i for j ∈
Revised: Given features f ( r ) ,t , compute the queries q ( r ) ,t i , keys k ( r ) ,t j , and values v ( r ) ,t i for j ∈

Original: I ( r ) .
Revised: I ( r ) .

Original: Starting from the finest level features f ( r )  i , i ∈ I ( r ) , the queries q ( r )  can be obtained by   ...
Revised: Starting from the finest level features f ( r ) ,t i , i ∈ I ( r ) , the queries q ( r ) ,t can be obtained by   ...

Original: q ( r )  i ...
Revised: q ( r ) ,t i ...

Original:   =  
Revised:   =  

Original: W Q
Revised: W Q, ( r )

Original: W Q . . .
Revised: W Q, ( r ) ...

Original: W Q   (cid:124) (cid:123)(cid:122) (cid:125) |I
Revised: W Q, ( r )   (cid:124) (cid:123)(cid:122) (cid:125) |I

Original: ( r ) |   ...
Revised: ( r ) |   ...

Original: f ( r )  i ...
Revised: f ( r ) ,t i ...

Original:     |I
Revised:     |I

Original: ( r ) | , the keys k ( r )  and values v ( r )  can follow the similar procedure.
Revised: ( r ) | , the keys k ( r ) ,t and values v ( r ) ,t can follow the similar procedure.

Original: STEP 1:
Revised: STEP 1:

Original: For m = r − 1 : 1 , Do the reduce operations q ( m ) ,t i =
Revised: For m = r − 1 : 1 , Do the reduce operations q ( m ) ,t i =

Original: R ( m ) ( { q ( m +1) ,t j } j ∈ i ( m,m +1) ) and also for k ( m ) ,t i and v ( m ) ,t i , for any i ∈ I ( m ) .
Revised: R ( m ) ( { q ( m +1) ,t j } j ∈ i ( m,m +1) ) and also for k ( m ) ,t i and v ( m ) ,t i , for any i ∈ I ( m ) .

Original: The reduce operations corresponds to   ...
Revised: If R ( m ) is linear, the reduce operations correspond to   ...

Original: q ( m )  i ...
Revised: q ( m ) ,t i ...

Original:   = R ( m )    ... q ( m +1)  i ...
Revised:   = R ( m ) ,t   ... q ( m +1) ,t i ...

Original:   , where the reduce matrix is given by
Revised:   .The reduce matrix is given by

Original: R ( m ) :=  
Revised: R ( m ) :=  

Original: R ( m ) R ( m )R ( m )
Revised: R ( m ) R ( m )

Original: R ( m )R ( m ) R ( m ) R ( m )
Revised: R ( m )

Original: R ( m )...
Revised: R ( m )...

Original: R ( m )R ( m ) R ( m )
Revised: R ( m )

Original: R ( m )
Revised: R ( m )

Original:  (cid:124) (cid:123)(cid:122) (cid:125) |I
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) |I

Original: ( m +1)
Revised: ( m +1)

Original: |   |I ( m ) | , and R ( m ) 0 , R ( m) 1 , R ( m ) 2 , R ( m )∈ R C ( m − 1) × C ( m ) are learnable parameter matrix (in practice, queries, keys, and values use different R ( m ) 0 , R ( m) 1 , R ( m ) 2 , R ( m )to enhance the expressivity). 
Revised: |   |I ( m ) | , and R ( m ) 0 , R ( m )∈ R C ( m − 1) ×C ( m ) are matrices parametrized by linear layers in our paper (in practice, queries, keys, and values use different R ( m ) 0 , R ( m )to enhance the expressivity). In general, these operators R ( m ) are not limited to linear operators.

Original:  and  inductively,   ... q ( m )  i ...
Revised: The composition of nonlinear activation functions would help increase the expressivity. The nested learnable operators R ( m ) also induce the channel mixing and is equivalent to a structured parameterization of W Q , W V , W K matrix for the coarse level tokens, in the sense that, inductively,    ... q ( m ) ,t i ...

Original: 
Revised: 

Original:  = R ( m ) · · · R ( r )   ...
Revised:   = R ( m ) · · · R ( r − 1)    ...

Original: q ( r )  i ...
Revised: q ( r ) ,t i ...

Original:   , and also   ... v ( m ) i ...

Original: 
Revised: 

Original:  = R ( m ) · · · R ( r )   ... v ( r ) i ...
Revised:   = R ( m ) · · · R ( r − 1)   W Q, ( r ) W Q, ( r ) . . .

Revised: W Q, ( r ) 

Revised:    ...

Revised: f ( r ) ,t i ...

Original:   .
Revised:   .

Original: STEP 2:
Revised: STEP 2:

Original: With the m -th level queries and keys, we can calculate the  attention matrix G ( m )  loc at m -th level with G ( m )  loc , i , j := exp( q ( m ) ,t i · k ( m ) ,t j ) for i ∈ N ( m ) ( j ) , or i ∼ j .
Revised: With the m -th level queries and keys, we can calculate the local attention matrix G ( m ) ,t loc at m -th level with G ( m ) ,t loc , i , j := exp( q ( m ) ,t i · k ( m ) ,t j ) for i ∈ N ( m ) ( j ) , or i ∼ j .

Original: STEP 3: The decompose operations, opposite to the reduce operations, corresponds to the transpose of the following matrix 
Revised: STEP 3: The decompose operations, opposite to the reduce operations, correspond to the transpose of the following matrix in the linear case,

Original: D ( m ) :=  
Revised: D ( m ) :=  

Original: D ( m ) D ( m )
Revised: D ( m ) D ( m )

Original: D ( m )D ( m ) D ( m ) D ( m )
Revised: D ( m )

Original: D ( m) D ( m )...D ( m ) D ( m )
Revised: D ( m )...

Original: D ( m ) R ( m )
Revised: D ( m ) D ( m )

Original:  (cid:124) (cid:123)(cid:122) (cid:125) |I
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) |I

Original: ( m +1)
Revised: ( m +1)

Original: |   |I ( m ) | ,
Revised: |   |I ( m ) | ,

Original: Eventually, the m -th level aggregation in Figure 3.1 contributes to the final output f 
Revised: The m -th level aggregation in Figure 3.2 contributes to the final output f ( r ) ,t +1 in the form

Original: ( r,t +1) in the form D ( r ) , T · · · D ( m ) , T G ( m ) loc R ( m ) · · · R ( r )   ... v ( r ) ,t i ...  and   ... f ( r ) ,t +1 i ...
Revised: D ( r − 1) , T · · · D ( m ) , T G ( m ) loc R ( m ) · · · R ( r − 1)     ... v ( r ) ,t i ...

Original: 
Revised: 

Original: [Equation]
Revised:    .

Original: (cid:32)
Revised: Eventually, aggregations at all r levels in one V cycle can be summed up as

Original: [Equation]
Revised: [Equation]

Original: (cid:88) [Equation]  [Equation]
Revised: The hierarchical attention matrix G h := (cid:80) r − 1 m =1 ( D ( r − 1) , T · · · D ( m ) , T G ( m ) loc R ( m ) · · · R ( r − 1) )

Original: B.1) The matrix in equation B.1 resembles the three level H 2 matrix decomposition illustrated in the following Figure B.1 for a three-level decomposition, we also refer to Hackbusch (2015) for  detailed description.
Revised: + G ( r ) loc in equation B.1 resembles the three-level H 2 matrix decomposition illustrated in the following Figure B.2, we also refer to (Hackbusch, 2015) for a detailed description.

Original: The sparsity of the matrix lies in the fact that the attention matrix is only computed for pairs of tokens within the neighbor set.
Revised: The sparsity of the matrix lies in the fact that the attention matrix is only computed for pairs of tokens within the neighbor set.

Original: The H 2 matrix vector multiplication in B.1 also implies O ( N ) complexity of the Algorithm 1.
Revised: The H 2 matrix-vector multiplication in B.1 also implies O ( N ) complexity of the Algorithm 1.

Original: To this end, we connect our hierarchical attention with H 2 matrix.
Revised: Note that, the local attention matrix at level I (1) (pink), level I (2) (blue) and level I (3) (green ) are G (1) loc ,

Original: From the perspective of H 2 matrix, we view equation B.1 as an approximation (in the matrix form) of the finest level global attention equation 3.1. But one should note that, we take W Q f ( m ) i and R [-( m ) 
-]
Revised: G (2) loc and G (3) loc , respectively. However, when considering their contributions to the finest level, they are equivalent to the attention matrix D (2) , T D (1) , T G (1) loc R [-( m ) 
-](1) R (2)

Original: q ( m ) i as the matrix vector multiplications with q ( m ) i ∈
Revised: ∈

Original: R C ( m ) and R ( m )∈ R C ( m − 1) ×C ( m ) , while in the H 2 matrix setting, W Q , R ( m ) 0 , f ( m ) i are all scalars.
Revised: R I (3) × R I (3) (pink), D (2) , T G (2) loc R (2) ∈ R I (3) × R I (3) (blue) and G (3) loc (green), as demonstrated in Figure B.2. Each pink block and blue block are actually low-rank sub-matrix with rank C (1) and rank C (2) , respectively, by definition.

Original: C P ROOF OF P ROPOSITION 3. 
Revised: C P ROOF OF P ROPOSITION 3.

Original: Proof
Revised: Proof

Original: For each level m , the cost to compute equation 3.3 is c ( |I ( m ) |C ( m ) )
Revised: For each level m , the cost to compute equation 3.3 is c ( |I ( m ) |C ( m ) )

Original: since for each i ∈ I ( m ) the cardinality of the neighbour set N ( m ) ( i ) is bounded by a constant c .
Revised: since for each i ∈ I ( m ) the cardinality of the neighbour set N ( m ) ( i ) is bounded by a constant c .

Original: The reduce operation f ( k − 1) i =
Revised: The reduce operation f ( k − 1) i =

Original: R ( k − 1) ( { f ( m ) j } j ∈ i ( k − 1 ,k ) ) costs at most |I ( m ) |C ( m ) C
Revised: R ( k − 1) ( { f ( m ) j } j ∈ i ( k − 1 ,k ) ) costs at most |I ( m ) |C ( m ) C

Original: ( k − 1) flops and so does the decompose operation at the same level.
Revised: ( k − 1) flops and so does the decompose operation at the same level.

Original: Therefore, for each level, the operation cost is c ( |I ( m ) |C ( m ) )
Revised: Therefore, for each level, the operation cost is c ( |I ( m ) |C ( m ) )

Original: + 2 |I ( m )
Revised: + 2 |I ( m )

Original: |C ( m ) C ( m − 1) .
Revised: |C ( m ) C ( m − 1) .

Original: When I is a quadtree, I ( r ) =
Revised: When I is a quadtree, I ( r ) =

Original: N, I ( r − 1) = N/ 4 , · · · , I (1) = 4 , therefore the total computational cost ∼ O ( N ) .
Revised: N, I ( r − 1) = N/ 4 , · · · , I (1) = 4 , therefore the total computational cost ∼ O ( N ) .

Original: □
Revised: □

Original: D I MPLEMENTATION D ETAILS
Revised: D I MPLEMENTATION D ETAILS

Original: In the HT-Net implementation, we follow the window attention scheme in Liu et al. (2021) for the definition of the neighborhood N ( · ) ( · ) in equation 3.2 and equation 3.3.
Revised: In the HT-Net implementation, we follow the window attention scheme in (Liu et al., 2021b) for the definition of the neighborhood N ( · ) ( · ) in equation 3.2 and equation 3.3.

Original: In this paper, we choose r = 3 for the depth of the HT-Net.
Revised: In this paper, we choose r = 3 as the depth of the HT-Net, GeLU as the activation function, and a CNN-based patch embedding module to transfer the input data into features.

Original: For  dataset with resolution n f × n f , such as in example 4.2, the input feature f (3) is represented as a tensor of size n × n × C .
Revised: For a dataset with resolution n f × n f , such as in the multiscale elliptic equation benchmark 4.2, the input feature f (3) is represented as a tensor of size n × n × C via patch embedding.

Original: The self-attention is first computed within a local window on level 3.
Revised: The self-attention is first computed within a local window on level 3.

Original: Then the reduce layer concatenates the features of each group of 2 × 2 neighboring tokens and applies a linear transformation on the 4 C -dimensional concatenated features on n 2 × n 2 level 2 tokens, to obtain level 2 features f (2) as a tensor of the size n 2 × n 2 × 2 C .
Revised: Then the reduce layer concatenates the features of each group of 2 × 2 neighboring tokens and applies a linear transformation on the 4 C -dimensional concatenated features on n 2 × n 2 level 2 tokens, to obtain level 2 features f (2) as a tensor of the size n 2 × n 2 × 2 C .

Original: The procedure is repeated from level 2 to level 1 with f (1) of size n 4 × n 4 × 4 C .
Revised: The procedure is repeated from level 2 to level 1 with f (1) of size n 4 × n 4 × 4 C .

Revised: We adopt the window attention scheme from (Liu et al., 2021b).

Original:  For the decompose process, starting at level 1, a linear layer is applied to transform the 4 C -dimensional features f (1) into 8 C -dimensional features.
Revised: One can refer to the implementation details there. For the decompose process, starting at level 1, a linear layer is applied to transform the 4 Cdimensional features f (1) into 8 C -dimensional features.

Original: Each level 1 token with 8 C -dimensional features is decomposed into four level 2 tokens with 2 C -dimensional features and added to the level 2 feature f (2) with output size of n 2 × n 2 × 2 C .
Revised: Each level 1 token with 8 C -dimensional features is decomposed into four level 2 tokens with 2 C -dimensional features and added to the level 2 feature f (2) with output size of n 2 × n 2 × 2 C .

Original: The procedure is repeated from level 2 level to level 3 with the output f (3) of size n × n × C .
Revised: The procedure is repeated from level 2 level to level 3 with the output f (3) of size n × n × C .

Original: We optimize models using the Adam optimizer with learning rate 1e − 3  and the 1-cycle schedule as in Cao (2021).
Revised: We use the Adam optimizer with learning rate 1e − 3 , weight decay 1e − 4 and the 1-cycle schedule as in (Cao, 2021).

Original: We use batch size 8 for experiments in Sections 4.2 and 4.3, and batch size 4 for experiments in Appendices F.1 and F.2.
Revised: We use batch size 8 for experiments in Sections 4.2 and 4.3, and batch size 4 for experiments in Appendices F.1 and F.2.

Original:  For the roughcoefficient experiment in Section 4.2, there are 1280 samples in the training set andsamples in  the testing set.
Revised: Evaluation setup For Darcy rough in table 1, we use a train-val-test split of the dataset with sizes of 1280, 112, and 112, respectively.

Revised: Each run is carried out with different seeds.

Original: For the  Navier-Stokes experiment in Section 4.3, we have tried N = 1000 and N = 10000 training samples as shown in Table 3, and the number of testing samples is 100andrespectively. For other experiments, the number of training samples is 1000, and the number of testing samples is 100.
Revised: For the Darcy smooth and multiscale trigonometric cases, the training size, validation size, and testing size are 1000, 100 and 100, respectively. For the Navier-Stokes experiment in Section 4.3, we have tried N = 1000 and N =training samples as shown in Table 3, and the number of testing samples is 100 and 1000 respectively.

Revised: The baseline’s implementations are based on their official implementation if provided publicly.

Revised: The results are comparable with the ones reported in the references.

Original: All experiments are run on one NVIDIA A100 GPU.
Revised: All experiments are run on a NVIDIA A100 GPU.

Original: E D ATA G ENERATION FOR THE T WO -P HASE C OEFFICIENT IN S ECTION 4. 
Revised: E D ATA G ENERATION FOR THE T WO -P HASE C OEFFICIENT IN S ECTION 4.

Original: The two-phase coefficients and solutions are generated according to https://github.com/ zongyi-li/fourier_neural_operator/tree/master/data_generation , and used as operator learning benchmarks in Li et al.
Revised: The two-phase coefficients and solutions are generated according to https://github.com/ zongyi-li/fourier_neural_operator/tree/master/data_generation , and used as operator learning benchmarks in Li et al.

Original: ; Gupta et al.
Revised: ; Gupta et al.

Original: (2021); Cao (2021).
Revised: (2021); Cao (2021).

Original: The coefficients a ( x ) are generated according to a ∼ µ :=
Revised: The coefficients a ( x ) are generated according to a ∼ µ :=

Original: ψ # N (cid:0) 0 , ( − ∆ + cI ) − 2 (cid:1) with zero Neumann boundary conditions on the Laplacian.
Revised: ψ # N (cid:0) 0 , ( − ∆ + cI ) − 2 (cid:1) with zero Neumann boundary conditions on the Laplacian.

Original: The mapping ψ : R → R takes the value a max on the positive part of the real line and a min on the negative part.
Revised: The mapping ψ : R → R takes the value a max on the positive part of the real line and a min on the negative part.

Original: The push-forward is defined in a pointwise manner.
Revised: The push-forward is defined in a pointwise manner.

Original: The forcing term is fixed as f ( x ) ≡ 1 .
Revised: The forcing term is fixed as f ( x ) ≡ 1 .

Original: Solutions u are obtained by using a second-order finite difference scheme on a suitable grid.
Revised: Solutions u are obtained by using a second-order finite difference scheme on a suitable grid.

Original: The parameters a max and a min can control the contrast of the coefficient.
Revised: The parameters a max and a min can control the contrast of the coefficient.

Original: The parameter c can control the roughness (oscillation) of the coefficient, and the coefficient with a larger c has rougher two-phase interfaces, as shown in
Revised: The parameter c can control the roughness (oscillation) of the coefficient, and the coefficient with a larger c has rougher two-phase interfaces, as shown in

Original: Figure 4.1.
Revised: Figure 4.1.

Original: F A DDITIONAL E XPERIMENTS F. M ULTISCALE TRIGONOMETRIC COEFFICIENT
Revised: F A DDITIONAL E XPERIMENTS F. M ULTISCALE TRIGONOMETRIC COEFFICIENT

Original: In this experiment, we consider equation 2.1 with multiscale trigonometric coefficient adapted from Owhadi (2017), such that D
Revised: In this experiment, we consider equation 2.1 with multiscale trigonometric coefficient adapted from Owhadi (2017), such that D

Original: =
Revised: =

Original: [ − 1 , 1] 2 , a ( x ) =
Revised: [ − 1 , 1] 2 , a ( x ) =

Original: (cid:81) 
Revised: (cid:81) k

Original:  [Equation]
Revised: = [Equation]

Original: )))(1 + 12  sin( a k π ( x 2 − 3 x 1 ))) , with a k = rand(2 k − 1 , 1 . 5 × 2 k − 1 ) , and fixed f ( x ) ≡ 1 .
Revised: (1 + 12 cos( a k π ( x 1 + x 2 ))) (1 + 1 2 sin( a k π ( x 2 − 3 x 1 ))) , with a k = uniform(2 k − 1 , 1 . 5 × 2 k − 1 ) , and fixed f ( x ) ≡ 1 .

Original: The reference solutions are obtained using P 1 FEM on a 1023 × 1023 grid.
Revised: The reference solutions are obtained using P 1 FEM on a 1023 × 1023 grid.

Original: Datasets of lower resolution are sampled from the higher resolution dataset by linear interpolation.
Revised: Datasets of lower resolution are sampled from the higher resolution dataset by linear interpolation.

Original: The experiment results for the multiscale trigonometric case for different resolutions are shown in Table 4.
Revised: The experiment results for the multiscale trigonometric case with different resolutions are shown in Table 4.

Original: HT-Net obtains the best relative L 2 error compared to other neural operators at various resolutions at 600 epochs.
Revised: HT-Net obtains the best relative L 2 error compared to other neural operators at various resolutions at 600 epochs.

Original: Multiwavelet neural operator MWT has the second best performance as it also possess a multiresolution structure, but it does not adapt to the PDE.
Revised: Multiwavelet neural operator MWT has the second best performance as it also possesses a multiresolution structure, but it does not adapt to the PDE.

Original: It is not surprising that FNO, an excellent smoother which filters higher frequency modes, fails to capture the high frequency oscillations of the solution.
Revised: It is not surprising that FNO, an excellent smoother which filters higher frequency modes, fails to capture the high-frequency oscillations of the solution.

Original: In contrast, our method has a better performance in this respect.
Revised: In contrast, our method has a better performance in this respect.

Original: See Figure F.1 for illustrations of the coefficient and comparison of the solutions at the slice x = 0 .
Revised: See Figure F.1 for illustrations of the coefficient and comparison of the solutions at the slice x = 0 .

Original: [Table]
Revised: [Table]

Original: F. H ELMHOLTZ EQUATION
Revised: F. H ELMHOLTZ EQUATION

Original: We test the performance of HT-Net for the acoustic Helmholtz equation in highly heterogeneous media as an example for multiscale wave phenomena, whose solution is considerably expensive for complicated and large geological models.
Revised: We test the performance of HT-Net for the acoustic Helmholtz equation in highly heterogeneous media as an example of multiscale wave phenomena, whose solution is considerably expensive for complicated and large geological models.

Original: We adapt the setup from Hauck & Peterseim (2022), 
Revised: We adopt the setup from De Hoop et al. (2022), for the Helmholtz equation on the domain D = [0 , 1] 2 .

Revised: Given frequency ω = 10 3 and wavespeed field c : Ω → R , the excitation field u : Ω → R solves the equation

Original: [Equation]
Revised: [Equation]

Revised: where ∂ Ω 3 is the top side of the boundary, and ∂ Ω 1 , 2 , 4 are other sides.

Original: where the coefficient a ( x ) takes{+= 
 + tanh(˜ c ( x )) , where ˜ c is sampled from+} the value 1 or ε as shown in Figure F.2 with ε − 1 ∈ rand (128 , 256) , κ = 9 , and 
Revised: The wave speed field is c ( x ) {+= 
 + tanh(˜ c ( x )) , where ˜ c is sampled from+} the Gaussian field ˜ c ∼ N (0 , (cid:0) − ∆ + τ 2 (cid:1) − d ) , where τ = 3 and d = 2 are chosen to control the roughness. The Helmholtz equation is solved on a 100 × 100 grid by finite element methods. We aim to learn the mapping from c ∈ R 100 × 100 to u ∈ R 100 × 100 as shown in Fig F.2.

Original: [Equation]
Revised: [Figure]

Original: In this example, MWT outperforms HT-net at the intermediate training stages and at the highest resolution ( s = 512 ), it may attribute to the wave propagation nature of the problem or the regular arrangement of the coefficients a ( x ) .
Revised: The Helmholtz equation is notorious for solving. One reason is the so-called resonance phenomenon as the frequency ω is close to an eigenfrequency of the Helmholtz operator for some particular c . We found that it is necessary to use a large training dataset of size 8000 .

Original: We will investigate this example further in our future studies.
Revised: The test dataset is of size 800 . All models are trained within 500 epochs.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: We also compare the evaluation time of the trained models in 5. Compared to HT-Net, FNO has both a larger error and a longer evaluation time. UNet, as a CNN based method, has much faster evaluation time (30 times faster than HT- ) but the worst error (60 times as much as HT-

Original: [Table] [Figure] [Table]
Revised: F. C OMPARISON WITH CLASSICAL METHODS (FDM) AND MULTISCALE METHODS (GRPS)

Original: [Figure]
Revised: We compare the accuracy and efficiency of HT-Net with two conventional solvers - finite difference method (FDM) as a typical classical method, and generalized rough polyharmonic splines (GRPS) (Liu et al., 2021a) as a typical multiscale method, for the Darcy rough benchmark. The results are listed in Table 6.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: F.

Revised: S TUDY FOR HYPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: We conduct studies for hyperparameters such as number of hierarchical levels (depth), window size

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: and feature dimension. We list the results for the Darcy rough benchmark in Table

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: where the notation HTNet [3, 8, 80] represents HT-Net with 3 levels, window size

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: and feature dimension 80. The experiments are run for 100 epochs. The results show that larger values of hierarchical level, window size

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: and feature dimension might be helpful to reduce errors, though with a higher computational cost. To balance the model size, computational cost

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: and performance, we choose the hyperparameters as [3, 8, 80]. F. M EMORY USAGE

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: The CUDA mem(GB) is the sum of self cuda memory usuage from the PyTorch autograd profiler for 1 backpropagation. The mem(GB) is recorded from nvidia − smi of the memory allocated for the active Python process during profiling. We report the memory usage of different models for the Darcy smooth (with resolution 211 × 211) and Darcy rough (with resolution 256 × 256) benchmarks in Table F.5.

Original: [Table]
Revised: The table shows that the memory usage of HTNet is stable.

Original: [Figure] [Table]
Revised: For the resolution of 256 × 256, both MWT and GT consume larger or comparable CUDA memory compared with HT-Net, albeit HT-Net has much better accuracy.