Original: T RANSFORMER M EETS B OUNDARY V ALUE I NVERSE P ROBLEMS
Revised: T RANSFORMER M EETS B OUNDARY V ALUE I NVERSE P ROBLEMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A Transformer-based deep direct sampling method is proposed forsolving a class of boundary value inverse problem.
Revised: A Transformer-based deep direct sampling method is proposed for a class of boundary value inverse problems.

Original: A real-time reconstruction is achieved by evaluating the learned inverse operator between carefully designed data and the reconstructed images.
Revised: A real-time reconstruction is achieved by evaluating the learned inverse operator between carefully designed data and the reconstructed images.

Original: An effort is made to give a specific example to a fundamentalbut critical question: whether and how one can benefit from the theoretical structure of a mathematical problem to develop task-oriented and structure-conforming deep neural network?
Revised: An effort is made to give a specific example to a fundamental question: whether and how one can benefit from the theoretical structure of a mathematical problem to develop task-oriented and structure-conforming deep neural networks?

Original: Specifically, inspired by direct sampling methods for inverse problems, the 1D boundary data  are preprocessed by a partial differential equationbased feature map to yield 2D harmonic extensionsin different frequencies as different input channels.
Revised: Specifically, inspired by direct sampling methods for inverse problems, the 1D boundary data in different frequencies are preprocessed by a partial differential equation-based feature map to yield 2D harmonic extensions as different input channels.

Original: Then, by introducing learnable non-local kernel, theapproximation of direct sampling is recast to a modified attention mechanism.
Revised: Then, by introducing learnable non-local kernels, the direct sampling is recast to a modified attention mechanism.

Original: The proposed method is then applied to electrical impedance tomography, a well-known severely ill-posed nonlinear inverse problem.
Revised: The proposed method is then applied to electrical impedance tomography, a well-known severely ill-posed nonlinear inverse problem.

Original: The new method achieves superior accuracy over its predecessors and contemporary operator learners, as well as shows robustness with respect to noise.
Revised: The new method achieves superior accuracy over its predecessors and contemporary operator learners, as well as shows robustness with respect to noise.

Original: This research shall strengthen the insights that the attention mechanism, despite being invented for natural language processing tasks, offers great flexibility to be modified in conformity with the a priori mathematical knowledge, which ultimately leads to the design of more physics-compatible neural architectures.
Revised: This research shall strengthen the insights that the attention mechanism, despite being invented for natural language processing tasks, offers great flexibility to be modified in conformity with the a priori mathematical knowledge, which ultimately leads to the design of more physics-compatible neural architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Boundary value inverse problems aim to recover the internal structure or distribution of multiple media inside an object (2D reconstruction) based on only the data available on the boundary (1D signal input), which arise from many imaging techniques, e.g., electrical impedance tomography (EIT) (Holder, 2004), diffuse optical tomography (DOT)
Revised: Boundary value inverse problems aim to recover the internal structure or distribution of multiple media inside an object (2D reconstruction) based on only the data available on the boundary (1D signal input), which arise from many imaging techniques, e.g., electrical impedance tomography (EIT) (Holder, 2004), diffuse optical tomography (DOT)

Original: (Culver et al., 2003), magnetic induction tomography (MIT)
Revised: (Culver et al., 2003), magnetic induction tomography (MIT)

Original: (Griffiths et al., 1999).
Revised: (Griffiths et al., 1999).

Original: Not needing any internal data renders these techniques generally non-invasive, safe, cheap, and thus quite suitable for monitoring applications.
Revised: Not needing any internal data renders these techniques generally non-invasive, safe, cheap, and thus quite suitable for monitoring applications.

Original: Let F : P → M be a nonlinear operator associated a physical model governed by certain partial differential equations (PDE) on a  bounded  Ωin R 2 . For the measurement m ∈ M on the boundary ∂ Ω , and the coefficient p ∈ P of the underlying PDEs to be recovered, the forwardPDE model is 
Revised: In this work, we shall take EIT as an example to illustrate how a more structure-conforming neural network architecture leads to better results in certain physics-based tasks. Given a 2D bounded domain Ω and an inclusion D , the forward model is the following partial differential equation (PDE)

Original: [Equation]
Revised: [Equation]

Original: where ξ is  a noise following certain distribution.
Revised: where σ 0 and σ 1 are two (approximately) known constants, e.g., σ 0 = 1 and σ 1 = 10 , but the shape of the inclusion D buried in Ω is unknown. The goal is to recover the shape of D using only the boundary data on ∂ Ω (Figure 1). Specifically, by exerting a current g on the boundary, one solves (1) with the Neumann boundary condition σ ∇ u · n | ∂ Ω = g to get a unique u on the whole domain Ω . However, what can be measured is only the Dirichlet boundary value representing the voltages u | ∂ Ω on the boundary.

Revised: This procedure is called Neumann-to-Dirichlet (NtD) mapping (for various notations and the Sobolev space formalism we refer readers to Appendix A):

Revised: [Equation]

Revised: The NtD map in (2) can be expressed as

Revised: [Equation]

Revised: where g and f are (infinite dimensional) vector representations of functions g and f relative to a chosen basis, and A σ is the matrix representation of Λ σ (see Appendix B for an example).

Revised: [Figure]

Revised: The original mathematical setup of EIT is to use the NtD map Λ σ in (2) to recover σ , referred to as the case of full measurement (Calderón, 2006).

Revised: In this case, the forward and inverse operators associated with EIT can be formulated as

Revised: [Equation]

Revised: Fix a set of basis { g l } ∞ l =1 of the corresponding Hilbert space containing all admissible currents.

Revised: Then, mathematically speaking, “knowing the operator Λ σ ” means that one can measure all the current-to-voltage pairs { g l , f l := Λ σ g l } ∞ l =1 and construct the infinite matrix A σ .

Revised: However, as infinitely many boundary data pairs is not attainable in practice, the problem of more practical interests is to use only a few data pairs { ( g l , f l ) } Ll =1 for reconstruction.

Revised: In this case, the forward and inverse problems can be formulated as

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: For limited data pairs, the inverse operator above is extremely ill-posed or even not well-defined (Isakov & Powell, 1990; Barceló et al., 1994; Kang & Seo, 2001; Lionheart, 2004); namely, the same boundary measurements may correspond to different σ .

Revised: In view of the matrix representation A σ , L being small (very limited data pairs) is not enough to yield any reasonably good approximation of

Revised: A σ in (3).

Revised: For example, when g l = e l , l = 1 , ..., L , with e l being unit vectors of a chosen basis, ( f 1 , ..., f L ) only gives the first L columns of A σ .

Revised: It is possible that two matrices

Revised: A σ and A ˜ σ have similar first L columns but ∥ σ − ˜ σ ∥ is large.

Revised: Hence, seeking an effective and efficient approximation of the operator F − 1 L is by nature a very challenging problem and is the target of this work.

Revised: How to deal with this “ill-posedness” is a central theme in theories for boundary value inverse problems.

Revised: The operator learning approach has the potential to tame the ill-posedness by approximating

Revised: F − 1 L only at a set of sampled data D := { σ ( k ) } Nk =1 , where the problem becomes to approximate

Revised: [Equation]

Revised: The fundamental assumption here is that this map is a “well-defined” enough to be regarded as a high-dimensional interpolation (learning) problem on a compact data submanifold (Ghattas &

Revised: Willcox, 2021), and the learned approximate mapping can be evaluated at newly incoming σ ’s.

Revised: We refer readers to Appendix B for a brief review of the theoretical background of EIT.

Original: Comparing with theclassic linear inverse problems y
Revised: Here we emphasize that, compared with the linear inverse problems y

Original: =
Revised: =

Original: A x + ϵ in computer vision and signal processing, e.g. (Marroquin et al., 1987), to recover a signal x from measurement y with noise ϵ , the fundamental difference is that F  is highly nonlinear, and the Hilbert spaces M and P are usually infinite dimensional.
Revised: A x + ϵ in computer vision and signal processing, e.g., Marroquin et al. (1987), to recover a signal x from measurement y with noise ϵ , the fundamental difference is that F ( · ) itself is highly nonlinear and involves boundary value PDEs.

Original: In many situations, even though the inverse operator F − 1 : M → P is theoretically well-defined, as the infinite dimensional M is practically approximated by a finite dimensional (sub)space M L , the problem of seeking the approximated operator F − 1 L : M L → P is usually highly ill-posed (not having a well-defined unique output) and poses great challenges to the reconstruction algorithms.

Original:  B ACKGROUND , RELATED WORK , AND CONTRIBUTIONS
Revised:  B ACKGROUND , RELATED WORK , AND CONTRIBUTIONS

Original: Classical iterative methods.
Revised: Classical iterative methods.

Original: There are in general two types of methodology to solve inverse problems.
Revised: There are in general two types of methodology to solve inverse problems.

Original: The first one is a large family of iterative or optimization-based methods (Dobson &
Revised: The first one is a large family of iterative or optimization-based methods (Dobson &

Original: Santosa, 1994; Martin & Idier, 1997; Chan & Tai, 2003; Vauhkonen et al., 1999; Guo et al., 2018;
Revised: Santosa, 1994; Martin & Idier, 1997; Chan & Tai, 2003; Vauhkonen et al., 1999; Guo et al., 2018;

Original: Rondi & Santosa, 2001; Chen et al., 2020).
Revised: Rondi & Santosa, 2001; Chen et al., 2020).

Original: One usually looks for the desired p by solving a minimization problem 
Revised: One usually looks for an approximated σ by solving a minimization problem with a regularization R ( σ ) to alleviate the ill-posedness, say

Original: [Equation]
Revised: [Equation]

Original: where R ( p )is a regularization term to alleviate the ill-posedness, and its design plays a critical role for a successful reconstruction (Tarvainen et al., 2008; Tehrani et al., 2012; Wang et al., 2012).
Revised: The design of regularization R ( σ ) plays a critical role in a successful reconstruction (Tarvainen et al., 2008; Tehrani et al., 2012; Wang et al., 2012).

Original: For almost allthese iterative methods, due to the ill-posedness, the computation generally takes numerous iterations to converge, and the reconstruction is highly sensitive to noise.
Revised: For almost all iterative methods, due to the ill-posedness, the computation generally takes numerous iterations to converge, and the reconstruction is highly sensitive to noise.

Original: Besides,  F ( p ) needs to be evaluated at each iteration, which is itself expensive as it requires solving forward PDE models.
Revised: Besides, the forward operator F ( · ) needs to be evaluated at each iteration, which is itself expensive as it requires solving forward PDE models.

Original: Direct methods.
Revised: Classical direct methods.

Original: The second methodology is to develop a well-defined mapping G θ parametrized with θ , empirically constructed to approximate the inverse map itself: [Equation]
Revised: The second methodology is to develop a well-defined mapping G θ parametrized by θ , empirically constructed to approximate the inverse map itself, say G θ ≈ F − 1 .

Original: This method is referred to as non-iterative or direct methods in the literature.
Revised: These methods are referred to as non-iterative or direct methods in the literature.

Original: Distinguished from iterative approaches, direct methods are in general highly problem-specific, as they are designedon an ad hoc basis based on the mathematical structure of the various inverse operators.
Revised: Distinguished from iterative approaches, direct methods are in general highly problem-specific, as they are designed based on specific mathematical structures of their respective inverse operators.

Original: For instances, methods in EIT and DOT include factorization methods (Kirsch & Grinberg, 2007; Azzouz et al., 2007; Brühl, 2001; Hanke & Brühl, 2003), MUSIC-type algorithms (Cheney, 2001; Ammari & 
Revised: For instance, methods in EIT and DOT include factorization methods (Kirsch & Grinberg, 2007; Azzouz et al., 2007; Brühl, 2001; Hanke & Brühl, 2003), MUSIC-type algorithms (Cheney, 2001; Ammari & Kang, 2004; 2007;

Original: Kang, 2004; 2007; Lee et al., 2011), and the D-bar methods (Knudsen et al., 2007; 2009) based on a Fredholm integral equation (Nachman, 1996), among which are the direct sampling methods (DSM)  (Chow et al., 2014; 2015; Kang et al., 2018;
Revised: Lee et al., 2011), and the D-bar methods (Knudsen et al., 2007; 2009) based on a Fredholm integral equation (Nachman, 1996), among which are the direct sampling methods (DSM) being our focus in this work (Chow et al., 2014; 2015; Kang et al., 2018;

Original: Ito et al., 2013; Ji et al., 2019; Harris & Kleefeld, 2019; Ahn et al., 2020) being our focus in this work.
Revised: Ito et al., 2013; Ji et al., 2019; Harris & Kleefeld, 2019; Ahn et al., 2020).

Original: These methods generally have a closed-formexpression of G θ to approximate F − 1 , and the parameters θ have clear mathematical meaning.
Revised: These methods generally have a closed-form G θ for approximation, and the parameters θ represent model-specific mathematical objects.

Original: For each fixed θ , this procedure is usually much more stable than iterative approaches with respect to the input data.
Revised: For each fixed θ , this procedure is usually much more stable than iterative approaches with respect to the input data.

Original: Furthermore, the evaluation for each given m is distinctly fast as notraining optimization is needed anymore.
Revised: Furthermore, the evaluation for each given boundary data pair is distinctly fast as no optimization is needed.

Original: However,such a simple closed form of G θ admitting efficient execution may not be available in practice since some mathematical assumptions and derivation may not hold. 
Revised: However, a simple closed-form G θ admitting efficient execution may not be available in practice since some mathematical assumptions and derivation may not hold. For instance, MUSIC-type and

Revised: D-bar methods generally require an accurate approximation to Λ σ , while DSM poses restrictions on the boundary data, domain geometry, etc., see Appendix D for details.

Original: Boundary value inverse problems.
Revised: Boundary value inverse problems.

Original: For most cases of boundary value inverse problems in 2D, one key difference, e.g., with an image inverse problem, is that data are only available on 1D manifolds, which are used to reconstruct 2D targets.
Revised: For most cases of boundary value inverse problems in 2D, one key difference, e.g., with an image inverse problem, is that data are only available on 1D manifolds, which are used to reconstruct 2D targets.

Original: The boundary data themselves generally involve certain input-output structure, which adds more complexity.
Revised: The boundary data themselves generally involve certain input-output structures (NtD maps), which adds more complexity.

Original: In Adler & Guardo (1994); FernándezFuentes et al.
Revised: In Adler & Guardo (1994); Fernández-Fuentes et al.

Original: ; Feng et al.
Revised: ; Feng et al.

Original: (2018) boundary measurements are collected and directly input into feedforward fully connected networks.
Revised: (2018) boundary measurements are collected and directly input into feedforward fully connected networks.

Original: As the data reside on different manifolds, special treatments are made to the input data, such as employing pre-reconstruction stages to generate rough 2D input to CNNs (Ben Yedder et al., 2018; Ren et al., 2020).
Revised: As the data reside on different manifolds, special treatments are made to the input data, such as employing pre-reconstruction stages to generate rough 2D input to CNNs (Ben Yedder et al., 2018; Ren et al., 2020).

Original: Operator learning and inverse problems.
Revised: Deep neural network and inverse problems.

Original: Solving an inverse problem is essentially to approximate the inverse operator F − 1 but based on finitely many data.
Revised: Solving an inverse problem is essentially to give a satisfactory approximation to F − 1 but based on finitely many measurements.

Original: The emerging deep learning (DL) technique makes it possible todirectly emulate operators. The concept resembles greatly the aforementioned classical direct methodology.
Revised: The emerging deep learning (DL) based on Deep Neural Networks (DNN) to emulate operators directly resembles greatly the aforementioned classical direct methods.

Original: But operator learners by DNNs are generally considered as black boxes.
Revised: However, operator learners by DNNs are generally considered as black boxes.

Original: A natural question to ask iswhether and howexisting innovative neural architectures, usually designed for other purposes, can be modified conforming with the  mathematical nature of the underlying problem, which ultimately leads to structure-conforming DNNs.
Revised: A natural question to ask is how the a priori mathematical knowledge can be exploited to design more physics-compatible DNN architectures.

Original: Inspired by classical direct methods, we try to improve the DNN reconstruction pipeline from the architectural perspective. In this regard, theproposed study provides a positive example to a hopefully definitive answer to this question, which bridges deep learning  and conventional tasks in physical sciences.
Revised: In pursuing the answer to this question, we aim to provide a supportive example that bridges deep learning techniques and classical direct methods improving the reconstruction.

Original: Operator learners.
Revised: Operator learners.

Original: Operator learning has become an active research field for inverse problems in recent years, especially related to image reconstruction where CNN plays a central role, see e.g,
Revised: Operator learning has become an active research field for inverse problems in recent years, especially related to image reconstruction where CNN plays a central role, see e.g,

Original: Kłosowski & Rymarczyk (2017); Nguyen et al.
Revised: Kłosowski & Rymarczyk (2017); Nguyen et al.

Original: Tan et al.
Revised: Tan et al.

Original: ; Jin et al.
Revised: ; Jin et al.

Original: ; Kang et al.
Revised: ; Kang et al.

Original: ; Barbastathis et al.
Revised: ; Barbastathis et al.

Original: ; Latif et al.
Revised: ; Latif et al.

Original: Zhu et al.
Revised: Zhu et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: Coxson et al. (2022).
Revised: Coxson et al. (2022).

Original: Some effortshave been made to couple classical reconstruction methods and
Revised: Notable examples of efforts to couple classical reconstruction methods and

Original: CNN. Notable examples include Hamilton et al.
Revised: CNN include Hamilton et al.

Original: ; Hamilton & Hauptmann (2018), where a CNN post-processes images obtained by the classical D-bar methods; Fan et al. 
Revised: ; Hamilton & Hauptmann (2018), where a CNN post-processes images obtained by the classical D-bar methods, and Fan et al. ; Fan & Ying (2020) where

Original: ; Fan & Ying (2020) develop BCR-Net  to mimic pseudo-differential operators appearing in many inverse problems; a deep direct sampling method is proposed in Guo & Jiang (2020); Jiang et al.
Revised: BCR-Net is developed to mimic pseudo-differential operators appearing in many inverse problems. A deep direct sampling method is proposed in Guo & Jiang (2020); Jiang et al.

Original: (2021) that learns local convolutional kernels mimicking the gradient operator of DSM.
Revised: (2021) that learns local convolutional kernels mimicking the gradient operator of DSM.

Original: Another example is radial basis function neural networks seen in Hrabuska et al.
Revised: Another example is radial basis function neural networks seen in Hrabuska et al.

Original: ; Michalikova et al.
Revised: ; Michalikova et al.

Original: Wang et al.
Revised: Wang et al.

Original: (2021a).
Revised: (2021a).

Original: Nevertheless, convolutions in CNNs use locally supported kernel whose receptive field involves only a small neighborhood of a pixel, thus layer-wise speaking, CNN does not align well with the non-local nature of inverse problems.
Revised: Nevertheless, convolutions in CNNs use kernels whose receptive fields involve only a small neighborhood of a pixel. Thus, layer-wise speaking, CNN does not align well with the non-local nature of inverse problems.

Original: More recently, the learning of PDE-related forward problems using global kernel has gained attraction, most notably the Fourier Neural Operator (FNO) (Li et al., 2021).
Revised: More recently, the learning of PDE-related forward problems using global kernel has gained attraction, most notably the Fourier Neural Operator (FNO) (Li et al., 2021a).

Original: FNO takes the unique advantage of the low-rank nature of certain problems, and proposes to train a local kernel in the frequency domain yet global in the spatial-temporal domain, mimicking the solution’s kernel integral form.
Revised: FNO takes the unique advantage of the low-rank nature of certain problems, and proposes to train a local kernel in the frequency domain yet global in the spatial-temporal domain, mimicking the solution’s kernel integral form.

Original: Others include DeepONets (Lu et al., 2021;
Revised: Others include DeepONets (Lu et al., 2021;

Original: Wang et al., 2021b; Jin et al., 2022) and Transformers (Kissas et al., 2022;
Revised: Wang et al., 2021b; Jin et al., 2022) and Transformers (Kissas et al., 2022;

Original: Li et al., 2022; Cao, 2021).
Revised: Li et al., 2022; Cao, 2021).

Original: Related studies on Transformers.
Revised: Related studies on Transformers.

Original: The attention mechanism-based models have become the state of the art in many areas since Vaswani et al.
Revised: The attention mechanism-based models have become the state of the art in many areas since Vaswani et al.

Original: (2017), such as language tasks and computer vision e.g.,

Original: Ma et al. ; Dosovitskiy et al. One of the most important and attractive aspects of the attention mechanism is its unparalleled capability to model  long-range interactions (Tay et al., 2021) through many efficient variants see e.g., Katharopoulos et al. Choromanski et al.
Revised: One of the most important and attractive aspects of the attention mechanism is its unparalleled capability to model non-local long-range interactions (Tay et al., 2021) through many efficient variants (Katharopoulos et al., 2020; Choromanski et al., 2021; Nguyen et al., 2021).

Original: Nguyen et al. The relation of the attention with kernel learning is firstly studied in (Tsai et al., 2019) and later connected with random feature (Peng et al., 2021).
Revised: The relation of the attention with kernel learning is first studied in Tsai et al. (2019) and later connected with random feature (Peng et al., 2021).

Original: Connecting the non-PDE-based integral kernels and the attention mechanism has been seen in Hutchinson et al.
Revised: Connecting the non-PDE-based integral kernels and the attention mechanism has been seen in Hutchinson et al.

Original: ; Guibas et al.
Revised: (2021); Guibas et al.

Original: Among inverse problems, Transformers have been applied in medical imaging applications including segmentationproblems (Zhou et al., 2021;Hatamizadeh et al., 2022; Petit et al., 2021),
Revised: Among inverse problems, Transformers have been applied in medical imaging applications including segmentation (Zhou et al., 2021;

Original:  X-Ray (Tanzi et al., 2022), magnetic resonance imaging (MRI)
Revised: Hatamizadeh et al., 2022; Petit et al., 2021), X-Ray (Tanzi et al., 2022), magnetic resonance imaging (MRI)

Original: (He et al., 2022), ultrasound (Perera et al., 2021), optical coherence tomography (OCT) (Song et al., 2021), etc.
Revised: (He et al., 2022), ultrasound (Perera et al., 2021), optical coherence tomography (OCT) (Song et al., 2021), etc.

Original: To our best knowledge, there is no work in the literature that connects the attention mechanism in Transformer and the mathematical structure of PDE-based inverse problems.
Revised: To our best knowledge, there is no work in the literature that establishes an architectural connection between the attention mechanism in Transformer and the mathematical structure of PDE-based inverse problems.

Original: 2. C ONTRIBUTIONS
Revised: 2. C ONTRIBUTIONS

Original: • A structure-conforming network architecture.
Revised: • A structure-conforming network architecture.

Original: Inspired by the classical DSM, we decompose the  inverse operator into a harmonic extension and an integral operator with learnable non-local kernels that has an attention-like structure.
Revised: Inspired by the EIT theory and classic DSM, we decompose the approximation of the inverse operator into a harmonic extension and an integral operator with learnable non-local kernels that has an attention-like structure.

Original: Additionally, the attention architecture is reinterpreted through a Fredholm integral operator to rationalize the application of the Transformer to the boundary value inverse problem.
Revised: Additionally, the attention architecture is reinterpreted through a Fredholm integral operator to rationalize the application of the Transformer to the boundary value inverse problem.

Original: • Theoretical and experimental justification for the advantage of Transformer.
Revised: • Theoretical and experimental justification for the advantage of Transformer.

Original: We have proved that, in Transformeris, a modified attention is able to represent target functions exhibiting higher frequency natures from lower frequency input features.
Revised: We have proved that, in Transformers, a modified attention is able to represent target functions exhibiting higher frequency natures from lower frequency input features.

Original: In the experiments, a comparative study further demonstrates a favorable match between the Transformer and the problem structure.
Revised: In the experiments, a comparative study further demonstrates a favorable match between the Transformer and the benchmark problem.

Original:  I NTERPLAY BETWEEN MATHEMATICS AND NEURAL ARCHITECTURES
Revised:  I NTERPLAY BETWEEN MATHEMATICS AND NEURAL ARCHITECTURES

Original: In this section, we use EIT, a classical boundary value inverse problem, as a prominent example to walk through that the triple tensor product in the attention mechanism matches extremely well with representing a solution in the inverse operator theory of EIT.
Revised: In this section, we try to articulate that the triple tensor product in the attention mechanism matches extremely well with representing a solution in the inverse operator theory of EIT.

Original: The proposed method is motivated by and in pursuit to provide examples to the answers of the following questions
Revised: In pursuing this end goal, this study tries to answer the following motivating questions:

Original: ( Q1 )
Revised: ( Q1 )

Original: What is an appropriate finite dimensional data formatof p as inputs to the neural network?
Revised: What is an appropriate finite dimensional data format as inputs to the neural network?

Original: ( Q2 )
Revised: ( Q2 )

Original: Is there a suitable neural network G θ in (3) taking advantage of the mathematical structure?
Revised: Is there a suitable neural network G θ approximating (6) matching the mathematical structure?

Original: 3. F ROM EIT TO O PERATOR L EARNING
Revised: 3. F ROM EIT TO O PERATOR L EARNING

Original: The forward model of EIT is given by the following second order elliptic partial differential equation

Original: [Equation]

Original: Given the domain Ω ⊆ R 2 , the goal is to identify the unknown inclusion D buried in Ω .

Original: The value σ 0 and σ 1 are two (approximately) known constants discontinuous across the boundary ∂D .

Original: Without loss of generality, we assume σ 1 > σ 0 .

Original: Then, the coefficient of (4) to be recovered can be described by a characteristic function p =

Original: I D ( x ) defined for each point x

Original: = ( x 1 , x 2 ) ∈ Ω , and I D ( x ) is 1 for x ∈ D and 0 for x ∈ Ω \ D , or equivalently p =

Original: σ 1 I D + σ 0 (1 − I D ) .

Original: We refer readers to Figure 2 for the discretization procedure, where p h is a vector of values at the grid points approximating I D .

Original: In the application of EIT, the boundary measurement space M contains information of how the electric potential u behaves on the boundary.

Original: By exerting a current g on the boundary, solving (4) for a unique u with this specific boundary information σ ∇ u · n | ∂ Ω = g (Neumann boundary condition) reveals u ’s behavior on the whole domain Ω .

Original: However, what can be measured is only the voltages u | ∂ Ω on the boundary.

Original: This procedure is called Neumann-to-Dirichlet (NtD) mapping:

Original: [Equation] The H s - H − s duality for Sobolev spaces here is for formality, and we refer the readers to Appendix A for more detailed descriptions of function spaces. One measurement is a single data pair ( g, f ) , and all the measurements form this infinite-dimensional space:
Revised: In the case of full measurement, the operator F − 1 can be well approximated through a large number of ( σ, A σ ) data pairs.

Original: [Equation]

Original: If the NtD mapping Λ σ is known, the theoretical uniqueness of I D can be established (Brühl, 2001;

Original: Hanke & Brühl, 2003; Astala & Päivärinta, 2006; Nachman, 1996).

Original: Mathematically, it means if one is able to measure all the current-to-voltage pairs { g l , f l = Λ σ g l } ∞ l =1 in M , where { g l } ∞ l =1 form a set of basis functions of the Hilbert space H − 1 / 2 ( ∂ Ω) , then F − 1 : M → P , which is essentially Λ σ (cid:55)→ σ , is well-defined admitting explicit format (Brühl, 2001).

Original: We shall illustrate how the mathematical setup above is realized in a discrete level.

Original: For each fixed σ , the NtD mapping in (5) can be expressed as

Original: [Equation]

Original: where g and f are tensors containing boundary data at grid points on the boundary.

Original: If σ is known, then (6) is obtained through a typical PDE solver.

Original: For unknown σ , using a significantly large number of pairs of g - f can obtain a huge matrix A σ approximating to Λ σ on a reasonably fine grid.

Original: The mechanismhere essentially results in a tensor2tensor mapping/operator from A σ to the images of σ .
Revised: This mechanism essentially results in a tensor2tensor mapping/operator from A σ to the imagery data representing σ .

Original: Inthis case, the operator F − 1 can be learned through a large number of σ - A σ data pairs. In particular, the BCR-Net (Fan & Ying, 2020) is aDNN approximation falls into this category. However, as infinity or large number of boundary data pairs is not attainable in practice, the problem of more practical interests is to use only a few data pairs { ( g l , f l ) } Ll =1 for reconstruction with L being very small not enough to get any reasonably good approximation of A σ in (6). Let M L be the subspace M spanned by { ( g l , f l ) } Ll =1 .
Revised: In particular, the BCR-Net (Fan & Ying, 2020) is a

Original: Here, the highly nonlinear forward operator for finitely many pairs F = F L : σ (cid:55)→ ( g l , f l ) Ll =1 is still trivially well-defined.

Original: However, it becomes a longstanding open theoretical problem whether its inverse F − 1 L is well-defined.

Original: Namely, it is unclear if the inclusion D can be exactly identified by finite measurement on boundary.

Original: As we assume σ 1 > σ 0 , the operator Λ σ − Λ σ 0 is positive definite, and it has eigenvalues { λ l } ∞ l =1 with λ 1 > λ 2 > · · · > 0 (Cheng et al., 1989).

Original: The following theorem justifies the practice of approximating I D when L is large enough. Roughly speaking, as F − 1 ∞ is well defined, when L ≫ 1 , we can define an F − 1 L to approximate F − 1 ∞ .
Revised: DNN approximation falling into this category.

Original: Theorem 1.

Original: Suppose that the 1D boundary data g l is the eigenfunction of Λ σ − Λ σ 0 corresponding to the l -th eigenvalue λ l , and let the 2D data functions ϕ l be obtained by solving − ∆ ϕ l = 0 in Ω ,

Original: [Equation]

Original: where l = 1 , 2 , . Define S L = Span { ∂ x 1 ϕ l ∂ x 2 ϕ l : l = 1 , . . . , L } . Then, for any ϵ > 0 , there exists a sufficiently large L such that sup
Revised: However, when there are very limited boundary data pairs accessible, the task of learning the full matrix A σ becomes obscure, which complicates the development of a tensor2tensor pipeline for operator learning.

Original: [Equation]

Original: The proof of Theorem 1 can be found in Appendix C. Nevertheless, the constructional proof of the existence of I DL still relies on the entire NtD mapping Λ σ , which again resorts to infiniteness, thus inaccessible in real applications.

Original: Another valuable corollary of Theorem 1 is to justify using the data functions ϕ l as the input to neural networks, see Section 3.3 for more details.

Original: Operator learning problems for EIT.
Revised: Operator learning problems for EIT.

Original: Many works have made efforts to derive a well-defined operator that estimates σ instead of an exact or accurate reconstruction (Brühl, 2001;

Original: Chow et al., 2014; Dunlop & Stuart, 2015) for even L = 1 .

Original: Such effort could benefit modern operator learning methods.

Original: To this end, we first introduce several attainable approximations of infinite dimensional spaces by finite dimensional counterparts for the proposed method.
Revised: We first introduce several attainable approximations of infinite dimensional spaces by finite dimensional counterparts for the proposed method.

Original: (1) Spatial discretization.
Revised: (1) Spatial discretization.

Original: Let Ω h be a mesh of Ω with the mesh spacing h and let { z j } Mj =1 :=
Revised: Let Ω h be a mesh of Ω with the mesh spacing h and let { z j } Mj =1 :=

Original: M be the set of grid points to represent the 2D discretizations for continuous signals.
Revised: M be the set of grid points to represent the 2D discretization of continuous signals.

Original: Then a function u defined almost everywherein Ω can be approximated by a vector u h ∈ R M .
Revised: Then a function u defined almost everywhere

Original: (2) Sampling ofcoefficient σ or equivalently D .
Revised: (2) Sampling of D .

Original:  N samples of D with different shapes and locations following certain distribution are used approximate P .
Revised: We generate N samples of D with different shapes and locations following certain distribution.

Revised: For example, elliptical inclusions with random semi-axes and centers are generated as a benchmark (see Appendix C.1 for details).

Revised: With the known σ 0 and σ 1 , set the corresponding data set D = { σ (1) , σ (2) , ..., σ ( N ) } .

Original: N is usually large enough to represent field applications of interest.
Revised: N is usually large enough to represent field applications of interest.

Original: For the k -th sample of D , we let p ( k ) h be a discretization of I D on Ω h .

Original: Accordingly, m ( k ) h is a data function associated with F L ( p ( k ) h ) .

Original: (3) Sampling of NtD mapping.
Revised: (3) Sampling of NtD maps.

Original: The most highlighted approximation is, for each sample of D , there are Linput-output data pairs to sample the NtD mapping Λ σ ,  which are used to generate L channels in an input image.
Revised: For the k -th sample of D , we generate L pairs of boundary data { ( g ( k ) l , f ( k ) l ) } Ll =1 by solving PDE (1), which can be thought of as sampling of columns of the infinite matrix A σ representing the NtD map.

Original: Despite a “sufficiently large” L needed in Theorem 1, in practice, L can be chosen to be very small ( ≤ 3 ) to yield satisfactory results.
Revised: By the proposed method, L can be chosen to be very small ( ≤ 3 ) to yield satisfactory results.

Original: Our task is to find a parameterized mapping G θ to approximate F − 1  by minimizing
Revised: Our task is to find a parameterized mapping G θ to approximate F − 1 L, D (6) by minimizing

Original: [Equation]
Revised: [Equation]

Original: The parameters N, h, L will affect the  approximation  in the following way: h determines the resolution to approximate D ; N determines the representativity of the training data set; L determines the known spectral information of Λ σ .
Revised: for a suitable norm ∥ · ∥ . Hyper-parameters N, h, L will affect the finite dimensional approximation to the infiniteness in the following way: h determines the resolution to approximate D ; N affects the representativity of the training data set; L decides how much of a finite portion of the infinite spectral information of Λ σ can be accessed.

Revised: The incomplete information of Λ σ due to small L for one single σ is compensated by a large N ≫ 1 sampling of different σ ’s.

Original: 3. F ROM CHANNELS IN ATTENTION TO BASIS IN INTEGRAL TRANSFORM
Revised: 3. F ROM HARMONIC EXTENSION TO TENSOR - TO - TENSOR

Revised: Many mathematical works have made efforts to derive a well-defined operator that estimates σ instead of an exact or accurate reconstruction (Brühl, 2001;

Revised: Chow et al., 2014; Dunlop & Stuart, 2015) even for a single measurement, i.e., L = 1 .

Revised: Such effort could benefit modern operator learning methods.

Revised: To establish the connection between the problem of interest with the attention used in the Transformers, we first construct higher-dimensional tensors from the 1D boundary data.

Revised: The key is a harmonic extension of the boundary data that can be viewed as a PDE-based feature map.

Revised: We begin with a theorem to motivate it.

Revised: Let I D be the characteristic function of D , i.e., I D ( x )

Original: {+= 
+}
Revised: {+= 
+}

Revised: if x ∈ D

Original: {+and I D ( x ) = 
+}
Revised: {+and I D ( x ) = 
+}

Revised: if x / ∈ D .

Revised: Thus, σ can be directly identified by the shape of D through the formula σ = σ 1 I D + σ 0 (1 − I D ) .

Revised: In this setup, to reconstruct σ is equivalent to reconstructing I D .

Revised: Without loss of generality, we let σ 1 > σ 0 .

Revised: The NtD map Λ σ 0 for the background medium σ 0 is considered as known, and Λ σ 0 g can be readily computed.

Revised: Then f − Λ σ 0 g =

Revised: (Λ σ − Λ σ 0 )

Revised: g measures the difference between the NtD mappings, and encodes the information of σ .

Revised: The operator Λ σ − Λ σ 0 is positive definite, and it has eigenvalues { λ l } ∞ l =1 with λ 1 > λ 2 > · · · > 0 (Cheng et al., 1989).

Revised: Theorem 1.

Revised: Suppose that the 1D boundary data g l is the eigenfunction of Λ σ − Λ σ 0 corresponding to the l -th eigenvalue λ l , and let the 2D data functions ϕ l be obtained by solving

Revised: [Equation]

Revised: for l = 1 , 2 , . . . .

Revised: Define the space (cid:101)

Revised: S L = Span { ∂ x 1 ϕ l ∂ x 2 ϕ l : l = 1 , . . .

Revised: , L } , and the dictionary

Revised: S L = { a 1 + a 2 arctan( a 3 v ) : v ∈ (cid:101) S L , a 1 , a 2 , a 3 ∈ R } .

Revised: Then, for any ϵ > 0 , there exists a sufficiently large L such that sup

Revised: [Equation]

Revised: The full proof of Theorem 1 can be found in Appendix E.

Revised: This theorem gives a constructive approach for approximating F − 1 and justifies the practice of approximating I D when L is large enough.

Revised: The function ϕ l is called the harmonic extension of f l − Λ σ 0 g l .

Revised: On the other hand, it relies on “knowing” the entire NtD map Λ σ to construct I DL explicitly; namely the coefficients of ∂ x 1 ϕ l ∂ x

Revised: 2 ϕ l depend on a big chunk of spectral information (eigenvalues and eigenfunctions) of Λ σ , which may not be available in practice.

Revised: Thus, the mathematics itself in this theorem does not provide us with an architectural hint on how to build a structure-conforming DNN.

Revised: To further dig out the hidden structure, we focus on the case of a single measurement, i.e., L = 1 .

Revised: With this setting, it is possible to derive an explicit and simple formula to approximate I D which is achieved by the classical direct sampling methods (DSM) (Chow et al., 2014; 2015; Kang et al., 2018;

Revised: Ito et al., 2013; Ji et al., 2019; Harris & Kleefeld, 2019; Ahn et al., 2020).

Revised: For EIT,

Revised: [Equation]

Original: {+• ϕ is the harmonic extension of f − Λ σ 
+}
Revised: {+• ϕ is the harmonic extension of f − Λ σ 
+}

Revised: g with certain noise f − Λ σ 0 g + ξ ; • d ( x ) is called a probing direction and can be chosen empirically as d ( x ) =

Revised: ∇ ϕ

Revised: ( x ) / ∥∇ ϕ

Revised: ( x ) ∥ ; • R ( x )

Revised: =

Revised: ( ∥ f − Λ σ 0 g

Revised: ∥ ∂

Revised: Ω | η x | Y ) − 1 , where η x is a function of d ( x ) and measured in | · | Y semi-norm on boundary ∂ Ω .

Revised: Both ϕ and η x can be computed effectively by traditional fast PDE solvers, such as finite difference or finite element methods based on Ω h in Section 3.1.

Revised: However, the reconstruction accuracy is much limited by a single measurement, the nonparametric ansatz, and empirical choices of d ( x ) and | · | Y .

Revised: These restrictions leave room for DL methodology.

Revised: See Appendix D for a detailed discussion.

Revised: Constructing harmonic extension (2D features) from boundary data (1D signal input with limited depth) can contribute to the desired high-quality reconstruction.

Revised: First, harmonic functions are highly smooth away from the boundary, of which the solution automatically smooths out the noise on the boundary due to PDE theory (Gilbarg & Trudinger, 2001, Chapter 8), and thus make the reconstruction highly robust with respect to the noise (e.g., see Figure 3 in Appendix C.1).

Revised: Second, in terms of using certain backbone networks to generate features for downstream tasks, harmonic extensions can be understood as a problem-specific way to design higher dimensional feature maps (Álvarez et al., 2012), which renders samples more separable in a higher dimensional data manifold than the one with merely boundary data.

Revised: See Figure 1 for the illustration of this procedure.

Revised: The information of σ is deeply hidden in ϕ .

Revised: As shown in Figure 1 (see also Appendix C), one cannot observe any pattern of σ directly from ϕ .

Revised: It is different from and much more difficult than the inverse problems studied in (Bhattacharya et al., 2021; Khoo et al., 2021) that aim to reconstruct 2D targets from the much more informative 2D internal data of u .

Revised: In summary, both Theorem 1 and the formula of DSM (10) offer inspiration to give a potential answer to (Q1): the harmonic extension ϕ (2D features) of f − Λ σ 0 g (1D measurements) naturally encodes the information of the true characteristic function I D (2D targets).

Revised: As there is a pointwise correspondence between the harmonic extensions and the targets at 2D grids, a tensor representation of ∇ ϕ l at these grid points can then be used as the input to a tensor2tensor -type DNN to learn I D , and naturally the grids are explicitly set as the positional embedding.

Revised: In comparison, the positional information are buried more deeply in 1D measurements.

Revised: As shown in Figure 1, G θ can be nicely decoupled into a composition of a learnable neural network operator T θ and a non-learnable PDE-based feature map

Revised: H , i.e., G θ

Revised: =

Original:  In this subsection, a modified attention mechanism is proposed as the basic block in the tensor2tensortype mapping introduced in the next two subsections.
Revised: T θ ◦ H . The architecture of T θ shall be our interest henceforth. 3. F ROM CHANNELS IN ATTENTION TO BASIS IN INTEGRAL TRANSFORM In this subsection, a modified attention mechanism is proposed as the basic block in the tensor2tensortype mapping introduced in the next two subsections.

Original: Its reformulation conforms with one of the most used tools in applied mathematics: the integral transform.
Revised: Its reformulation conforms with one of the most used tools in applied mathematics: the integral transform.

Original: Solutions to many physical problems have this form, as it aggregates the interactions of a class of functions at different locations to characterize how the operator mapping behaves between infinite-dimensional spaces.

Original: More interestingly, in most applications, the interaction (kernel) does not have any explicit form, which meshes well with DL methodology philosophically.
Revised: In many applications such as inverse problems, the interaction (kernel) does not have any explicit form, which meshes well with DL methodology philosophically.

Original: In fact, this is precisely the situation of the considered problem.
Revised: In fact, this is precisely the situation of the EIT problem considered.

Original: In a single head, let the input ofthe attention block in an encoder layer be s h ∈ R M × N , where
Revised: Let the input of an encoder attention block be x h ∈ R M × n with n channels, then the query Q , key

Original: N ≫ L is the number of channels for the latent representations, then the query Q , key K , value

Original:  V are generated by three learnable projection matrices  W Q , W K , W V ∈ R N × N : Q = s h W Q ,
Revised: K , value V are generated by three learnable projection matrices θ := { W Q , W K , W V } ⊂ R n × n : ⋄ = x

Original: K = s h W K ,  V = s h W V .
Revised: h W ⋄ , ⋄ ∈ { Q, K, V } .

Revised: Here n ≫ L is the number of expanded channels for the latent representations.

Original: A modified dot-product attention is proposed as follows:
Revised: A modified dot-product attention is proposed as follows:

Original: [Equation]
Revised: [Equation]

Original: where N Q ( · ) and N K ( · ) are two learnable normalizations.
Revised: where nl Q ( · ) and nl K ( · ) are two learnable normalizations.

Original: Different from Nguyen & Salazar (2019)
Revised: Different from Nguyen & Salazar (2019)

Original: ; Xiong et al.
Revised: ; Xiong et al.

Original: (2020), this pre-inner-product normalization is applied right before the matrix multiplication of query and key, which takes inspiration from the normalization in the index map kernel integral (14) and (23), see also Boyd (2001) where the normalization for orthogonal bases essentially uses the (pseudo)inverse of the Gram matrices.
Revised: (2020), this pre-inner-product normalization is applied right before the matrix multiplication of query and key, which takes inspiration from the normalization in the index map kernel integral (10) and (20), see also Boyd (2001) where the normalization for orthogonal bases essentially uses the (pseudo)inverse of the Gram matrices.

Original: In practice, its cheap alternative is chosen to be layer normalization (Ba et al., 2016) or batch normalization (Ioffe & Szegedy, 2015).
Revised: In practice, its cheap alternative is chosen to be layer normalization (Ba et al., 2016) or batch normalization (Ioffe & Szegedy, 2015).

Original: α = h 2 is a mesh-based weight such that the summation becomes an approximation to an integral quadrature.
Revised: α = h 2 is a mesh-based weight such that the summation becomes an approximation to an integral.

Original: To elaborate these rationales, the j -th column of the i -th row
Revised: To elaborate these rationales, the j -th column of the i -th row

Original: U i of U and the i -th row
Revised: U i of U is ( U i )

Original:  A i • are [Equation] j , where A i 
Revised: j = α A i • · V j , in which the i -th row A i • =

Original: [Equation]
Revised: ( (cid:101) Q (cid:101) K ⊤ )

Original:  V [Equation]
Revised: i • and V j :=

Original: [Equation]
Revised: V • j .

Original: Thus, we can derive the dot-product attention in a basis expansion representation for U
Revised: Thus, applying this to every column 1 ≤ j ≤ n , attention (11) becomes a basis expansion representation for U

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, αA i • contains the coefficients for the linear combination of { V m } Mm =1 .
Revised: Here, αA i • contains the coefficients for the linear combination of { V m } Mm =1 .

Original: This set { V m } Mm =1 form the V ’s row space, and it further forms each row of the output U by multiplying with A .
Revised: This set { V m } Mm =1 form the V ’s row space, and it further forms each row of the output U by multiplying with A .

Original: A ( · , · ) in (11) stands for the attention kernel, which aggregates the pixel-wise feature maps to measure how the input latent representations interact.
Revised: A ( · , · ) in (12) stands for the attention kernel, which aggregates the pixel-wise feature maps to measure how the projected latent representations interact.

Original: Moreover, the latent representation U i in an encoder layer is spanned by the row space of V , and is being nonlinearly updated layer-wise.
Revised: Moreover, the latent representation U i in an encoder layer is spanned by the row space of V , and is being nonlinearly updated layer-wise.

Original: If we further assume that there exist a set of feature maps  for query, key, and value, e.g., see Choromanski et al. For U , the feature map u ( · )that maps R 2 → R 1 × N , i.e., U i =
Revised: For x h , U, Q, K, V , a set of feature maps are assumed to exist: for example u ( · ) maps R 2 → R 1 × n , i.e., U i =

Original: u ( z i ) =
Revised: u ( z i ) =

Original: [ u 1 ( z i ) , · · · , u N ( z i )] ,  then a kernel κ  ( · , · ) : R 2 × R 2 → R can be defined as follows,
Revised: [ u 1 ( z i ) , · · · , u n ( z i )] , e.g., see Choromanski et al. (2021), then an instancedependent kernel κ θ ( · , · ) : R 2 × R 2 → R can be defined by

Original: [Equation]
Revised: [Equation]

Original: Now the discrete kernel A ( · , · ) with vectorial input is rewritten to an integral kernel κ  ( · , · ) , thus the dot-product attention is expressed as a nonlinear integral transform for the l -th channel:
Revised: Now the discrete kernel A ( · , · ) with tensorial input is rewritten to this kernel κ θ ( · , · ) , thus the dot-product attention is expressed as a nonlinear integral transform for the l -th channel:

Original: [Equation]
Revised: [Equation]

Original: After plugging u ( · ) in certain minimization such as (8), the backpropagation updates weights, which further lead a new set of latent representations.
Revised: Through certain minimization such as (8), the backpropagation updates θ , which further lead a new set of latent representations.

Original: This procedure can be viewed as an iterative method to update the basis residing in each channel by solving the Fredholm integral equation of the first kind in (13).
Revised: This procedure can be viewed as an iterative method to update the basis residing in each channel by solving the Fredholm integral equation of the first kind in (14).

Original: To connect attention with inverse problems, the multiplicative structure in a kernel integral form for attention (13) is particularly useful.
Revised: To connect attention with inverse problems, the multiplicative structure in a kernel integral form for attention (14) is particularly useful.

Original: This falls into the category of Pincherle-Goursat (degenerate) kernels (Kress, 1999, Chapter 11), and its approximability depends on the  number of the expanded channels N .
Revised: (14) is a type of Pincherle-Goursat (degenerate) kernels (Kress, 1999, Chapter 11), and approximates the full kernel using only a finite number of bases.

Original: N decides number of learned basis functions in expansion (11), a subset of which eventually is used to form a set of basis.
Revised: The number of learned basis functions in expansion (12) depends on the number of channels n .

Original: Here we show the following theorem, heuristically it says that: given enough but finite channels of latent representations, the attention kernel integral is able to “bootstrap” in frequency domain, that is, generating an output representation with higher frequencies than the input.
Revised: Here we show the following theorem, heuristically it says that: given enough but finite channels of latent representations, the attention kernel integral is able to “bootstrap” in frequency domain, that is, generating an output representation with higher frequencies than the input.

Original: Similar approximation result is impossible for  CNN if opting for the usual framelet/wavelet interpretation (Ye et al., 2018).
Revised: Similar approximation result is impossible for a layer-wise propagation in CNN if opting for the usual framelet/wavelet interpretation (Ye et al., 2018), for example, a single layer of CNN filters without nonlinearity cannot learn weights to extract edges if there are no edge-like local features in the input (see for an empirical evidence in Figureand Figure 10).

Original: The full proof with a more rigorous setting is in Appendix D.
Revised: The full proof with a more rigorous setting is in Appendix F.

Original: Theorem 2 (Frequency bootstrapping) .
Revised: Theorem 2 (Frequency bootstrapping) .

Original: Suppose there exists a channel l in V such that ( V i ) l = sin( az i ) for some a ∈ Z + , the current finite-channel sum kernel A ( · , · ) approximates a non-separable kernel (e.g., Fourier-type sin( xz ) ) to an error of O ( ϵ ) under certain norm ∥ · ∥ X .
Revised: Suppose there exists a channel l in V such that ( V i ) l = sin( az i ) for some a ∈ Z + , the current finite-channel sum kernel A ( · , · ) approximates a non-separable kernel (e.g., Fourier-type sin( xz ) ) to an error of O ( ϵ ) under certain norm ∥ · ∥ X .

Original: Then, there exists a set of { W Q , W K , W V } such that certain channel k ′ in the output of (11) approximates sin( a ′ z ) , Z + ∋ a ′ > a with an error of O ( ϵ ) under the same norm.
Revised: Then, there exists a set of weights such that certain channel k ′ in the output of (12) approximates sin( a ′ z ) , Z + ∋ a ′ > a with an error of O ( ϵ ) under the same norm.

Original: The considered inverse problem is essentially to recover high-frequency eigenpairs of Λ σ based on the low-frequency data, see e.g. Figure 1.
Revised: The considered inverse problem is essentially to recover higher-frequency eigenpairs of Λ σ based on lower-frequency data, see e.g. Figure 1.

Original: Λ σ together with all its spectral information can be determined by the recovered inclusion shape, thus the existence result in Theorem 2  justifies the advantages of the attention mechanism for the considered problem.
Revised: Λ σ together with all its spectral information can be determined by the recovered inclusion shape, thus the existence result in Theorem 2 partially justifies the advantages to adopt the attention mechanism for the considered problem.

Original: 3. F ROM HARMONIC EXTENSION TO TENSOR - TO - TENSOR

Original: To establish the connection between the problem of interest with the attention used in the Transformers, we begin with the case of a single measurement, i.e., L = 1 .

Original: With this setting, it is possible to derive an explicit and simple formula to approximate I D inspired by direct sampling methods (DSM) (Chow et al., 2014; 2015; Kang et al., 2018;

Original: Ito et al., 2013; Ji et al., 2019; Harris & Kleefeld, 2019; Ahn et al., 2020) for a larger family of inverse problems.

Original: For instance, an approximation modified from the one in Chow et al.

Original: (2014) reads as

Original: [Equation]

Original: [Equation]

Original: where R ( x ) = (cid:0)

Original: [Equation]

Original: g

Original: [Equation]

Original: [Equation]

Original: ) |

Original: [Equation]

Original: x | Y (cid:1) − 1 , and f − Λ σ 0 g = (Λ σ − Λ σ 0 )

Original: g is the boundary data measuring the difference of NtD mappings.

Original: In particular, ϕ is the solution to (7) with l = L = 1 but with certain noise ξ on the boundary, i.e., f − Λ σ 0 g is replaced by f − Λ σ 0 g + ξ , which is called the harmonic extension .

Original: The vector function d ( x ) =

Original: [ d 1 ( x ) , d 2 ( x )] is called probing direction and can be chosen empirically as d ( x )

Original: = ∇ ϕ ( x ) / ∥∇ ϕ

Original: ( x ) ∥ .

Original: Lastly, η x is the solutions to:

Original: [Equation]

Original: where { η x } restricted on boundary

Original: ∂ Ω is equipped with | · | Y semi-norm, and δ x is δ function associated with the point x .

Original: Both ϕ and η x can be computed effectively by traditional fast PDE solvers, such as finite difference or finite element methods based on Ω h in Section 3.1.

Original: Essentially, (14) suggests that the derivative of ϕ along a direction d ( x ) scaled by a quantity R ( x ) approximates

Original: I

Original: D ( x ) , where R ( x ) and d ( x ) are based on empirical choices.

Original: Indeed, the reconstruction accuracy is much limited by a single measurement, the nonparametric ansatz, and empirical choices.

Original: These restrictions give room for DL methodology.

Original: Despite being restrictive, the formulation in (14) offers inspiration to give a potential answer to (Q1): the harmonic extension ϕ of ( f − Λ σ 0 g ) can be used as the input to a tensor2tensor -type DNN.

Original: Constructing harmonic extension (2D features) from boundary data (1D signal input with limited embedding dimensions) can contribute to the desired high-quality reconstruction.

Original: First, harmonic functions are highly smooth away from the boundary, of which the solution automatically smooths out the noise on the boundary due to PDE theory (Gilbarg & Trudinger, 2001, Chapter 8), and thus enable the reconstruction being highly robust with respect to the noise (e.g., see Figure 1).

Original: Second, in terms of using certain backbone network to generate features for downstream tasks, harmonic extensions can be understood as a problem-specific way to design higher dimensional feature maps (Álvarez et al., 2012), which renders samples more separable in a higher dimensional data manifold than the one with merely boundary data.

Original: [Figure]

Original: The information of σ is deeply hidden in ϕ .

Original: As shown in Figure 1, one cannot observe any pattern of σ directly from ϕ , for more examples see

Original: Appendix B. It is different from and much more difficult than the inverse problems studied in (Bhattacharya et al., 2021; Khoo et al., 2021) that aim to reconstruct 2D targets from the much more informative 2D internal data of u .

Original: As shown in Figure 1, G θ can be nicely decoupled into a composition of a parametrized neural network operator T θ and a non-learnable harmonic extension feature map H , i.e., G θ =

Original: T θ ◦ H .

Original: 3. F ROM INDEX MAP INTEGRAL TO T RANSFORMER
Revised: 3. F ROM INDEX MAP INTEGRAL TO T RANSFORMER

Original: In this last subsection, the probing direction d ( x ) , the inner product d ( x ) · ∇ ϕ ( x ) , and the norm | · | Yin (14) are used as ingredients to form certain non-local  learnable kernel integration.
Revised: In (10), the probing direction d ( x ) , the inner product d ( x ) · ∇ ϕ ( x ) , and the norm | · | Y are used as ingredients to form certain non-local instance-based learnable kernel integration.

Original: This non- localness is a fundamental trait for many inverse problems, in that I D ( x ) depends on the entire data function.
Revised: This non-localness is a fundamental trait for many inverse problems, in that I D ( x ) depends on the entire data function.

Original: Then, the discretization of the modified index function is shown to match the multiplicative structure of the modified attention mechanism in (9).
Revised: Then, the discretization of the modified index function is shown to match the multiplicative structure of the modified attention mechanism in (11).

Original: In the forthcoming derivations, K ( x, y ) , Q ( x, y )  and a self-adjoint positive definite linear operator
Revised: In the forthcoming derivations, K ( x, y ) , Q ( x, y ) , and a self-adjoint positive definite linear operator

Original: V : L 2 ( ∂ Ω) →
Revised: V : L 2 ( ∂ Ω) →

Original: L 2 ( ∂ Ω) , are shown to yield the emblematic Q - K - V structure of attention.
Revised: L 2 ( ∂ Ω) , are shown to yield the emblematic Q - K - V structure of attention.

Original: To this end, we make the following modifications and assumptions to the original index map in (14).
Revised: To this end, we make the following modifications and assumptions to the original index map in (10).

Original: • The reformulation of the index function is motivated by the heuristics that the  global information of ϕ should be used  to locate a point x .
Revised: • The reformulation of the index function is motivated by the heuristics that the agglomerated global information of ϕ could be used as “keys” to locate a point x .

Original: [Equation]
Revised: [Equation]

Original: If  K( x, y ) =
Revised: If an ansatz K x, y ) =

Original: δ x ( y ) , then (16) reverts to the original one in (14).
Revised: δ x ( y ) is adopted, then (15) reverts to the original one in (10).

Original: • Similarly, the probing direction d ( x )  is reasonably assumed to have a global dependence on ϕ
Revised: • The probing direction d ( x ) as “query” is reasonably assumed to have a global dependence on ϕ

Original: [Equation]
Revised: [Equation]

Revised: If Q ( x, y )

Revised: =

Revised: δ x ( y ) / ∥∇ ϕ

Revised: ( x ) ∥ , then d ( x ) =

Revised: ∇ ϕ

Revised: ( x ) / ∥∇ ϕ

Revised: ( x ) ∥ which is the choice of the probing direction in (Ikehata, 2000; Ikehata & Siltanen,

Original:  • In the quantigy R ( x ) in (14), the key is | · | Y which is assumed to have the following form:
Revised: [Equation] Ikehata, 2007). • In the quantity R ( x ) in (10), the key is | · | Y which is assumed to have the following form:

Original: [Equation]
Revised: [Equation]

Original: In Chow et al.
Revised: In Chow et al.

Original: (2014), it is shown that if V induces a kernel with sharply peaked Gaussian like distribution, 
Revised: (2014), it is shown that if V induces a kernel with sharply peaked Gaussian-like distribution, the index function in (10) can achieve maximum values for points inside D .

Original: the index

Original: function

Original: in (14) can

Original: achieve

Original: maximum values for points inside D Based on the assumptions from (16) to (18), we derive a matrix representation approximating the new index function on a grid, which accord well with an attention-like architecture.
Revised: Based on the assumptions from (15) to (17), we derive a matrix representation approximating the new index function on a grid, which accords well with an attention-like architecture.

Original: Denote by
Revised: Denote by

Original: • ϕ n : the vector that interpolates ∂ x n ϕ at the grid points { z j } , n = 1 , 2 .
Revised: • ϕ n : the vector that interpolates ∂ x n ϕ at the grid points { z j } , n = 1 , 2 .

Original: • B : thediscrete Laplacian on Ω h of a finite element/finite difference discretization of − ∆  coupled with the Neumann boundary condition and the zero integral normalization condition.
Revised: • B : the finite element/finite difference discretization of − ∆ on Ω h (a discrete Laplacian) coupled with the Neumann boundary condition and the zero integral normalization condition in (9).

Original: • R : the matrix that projects a vector defined at { z i } to  one definedat the nodes onthe boundary ∂ Ω .
Revised: • R : the matrix that projects a vector defined at interior grids to the one defined on ∂ Ω .

Revised: Here, we sketch the outline of the derivation, and present the detailed derivation in Appendix D.

Original: We shall discretize the variable x by grid points
Revised: We shall discretize the variable x by grid points

Original: z i in (16) and obtain an approximation to the integral:
Revised: z i in (15) and obtain an approximation to the integral:

Original: [Equation]
Revised: [Equation]

Revised: where { ω j } are some integration quadrature weights.

Original: We then consider (17) and focus on one component d n ( x ) of d ( x ) .
Revised: We then consider (16) and focus on one component d n ( x ) of d ( x ) .

Original: With a suitable quadrature rule to compute the integral, it can be rewritten as
Revised: With a suitable approximated integral, it can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Next, we proceed to express ∥ η z i ∥ Y by discretizing the variational form in (15) by a linear finite element method. Let { ψ ( j ) } Mj =1 be the set of the basis functions, and ψ n,i be the vector approximating [ ∂ x n ψ ( j ) ( z i )] M j =1 for each fixed
Revised: Note that the self-adjoint positive definite operator V in (17) can be parameterized by a symmetric positive definite (SPD) matrix denoted by V .

Original: z i . Denote η i as the vector approximating { η z i ( z j ) } M j =1 and ¯ η i as its restriction on the boundary nodes. Then, the finite element discretization yields the linear systems: [Equation]
Revised: There exist vectors v n,i depending on the matrices B ,

Original: Note that the self-adjoint positive definite operator V can be parameterized by a symmetric positive definite (SPD) matrix denoted by V .

Original: We can approximate | η z i | 2 Y as [Equation] where c n,i = ψ Tn,i B − 1 R T V RB − 1 ψ n,i ≥ 0 as V is SPD.
Revised: R and V such that | η z i | 2 Y ≈ (cid:80) n ϕ Tn v n,i v Tn,i ϕ n .

Original: Define v n,i = √ c n,i q i which can be considered asanother learnable vector since c n,i comes from the learnable matrix V . Putting (19), (20) and (18) into (16), we obtain
Revised: Then, with the modified indicator function can be written as

Original: [Equation]
Revised: [Equation]

Original: Now, using the notation from Section 3.2, we denote the learnable kernel matrices and an input vector: for ⋄ ∈ { Q, K, V } , and u ∈ { q , k , v }
Revised: Now, using the notation from Section 3.3, we denote the learnable kernel matrices and an input vector: for ⋄ ∈ { Q, K, V } , and u ∈ { q , k , v }

Original: [Equation]
Revised: [Equation]

Original: Then, we are able to rewrite (23) as
Revised: Then, we are able to rewrite (20) as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where C f,g = ∥
Revised: (22) where C f,g = ∥

Original: f − Λ σ 0 g ∥ − 1L 2 ( ∂ Ω) is a constant, and both ∗ and / aretaken element-wise.
Revised: f − Λ σ 0 g ∥ − 1 ∂ Ω is a normalization weight, and both ∗ and / are element-wise.

Original: Here, we may define Q = p h W Q , K = p h W K , and V = p h W V as the values, keys and query.
Revised: Here, we may define Q = x h W Q , K = x h W K , and V = x h W V as the query, keys, and values.

Original: We can see that, the right matrix multiplications  in (9) are low-rank approximations to the ones above in the attention mechanism.
Revised: We can see that, the right matrix multiplications (11) in the attention mechanism are low-rank approximations to the ones above.

Original: Hence, based on (25), essentially we need to find a function I resulting in a vector approximation to the true characteristic function {I D ( z j ) }
Revised: Hence, based on (22), essentially we need to find a function I resulting in a vector approximation to the true characteristic function {I D ( z j ) }

Original: [Equation]
Revised: [Equation]

Original: Thus, the expression in (25) and (26) reveal that a Transformer is able to capture the classical formula in (14) equipped with non-local learnable kernels.

Original: Moreover, when there are L data pairs, the data functions ϕ l are generated by computing their harmonic extensions as in (7).
Revised: Moreover, when there are L data pairs, the data functions ϕ l are generated by computing their harmonic extensions as in (9).

Original: Then, each harmonic extension is then treated as a channel of the input p h .
Revised: Then, each ϕ l is then treated as a channel of the input data x h .

Original: The derivation above motivates that the attention mechanism nicely fits the underlying mathematical structure. In summary, we propose to use a Transformer-based deep direct sampling method, and subsequently show that it has significantly better performance than the widely used CNN-based U-Net in linear inverse problems and modern operator learner such as Fourier Neural Operator.
Revised: In summary, the expressions in (22) and (23) reveal that a Transformer may be able to further generalize the classical non-parametrized DSM formula in (10) to non-local learnable kernels, and has an intrinsic architectural advantage to take multiple data pairs.

Original: In this regard, we provide a potential answer to the question (Q2); namely, the attention-based Transformer is better suited  as it conforms more with the underlying mathematical structure, and both enjoy a global kernel formulation using the input data, which match the long range dependence nature of inverse problems.
Revised: In the subsequent experiments with regard to the actual EIT benchmark, we provide a potential answer to the question (Q2); namely, the attention architecture is better suited for the reconstruction task as it conforms more with the underlying mathematical structure. Its non-local kernel interpretation to learn global interactions matches the long range dependence nature of inverse problems.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we present some experimental results to show the quality of the reconstruction using a single channel ( L = 1 ) of the 2D harmonic extension feature from the 1D signal input.
Revised: In this section we present some experimental results to show the quality of the reconstruction using a single channel ( L = 1 ) of the 2D harmonic extension feature from the 1D signal input.

Original: There are two baseline models to compare:one is the CNN-based U-Net (Ronneberger et al., 2015), the other is the state-of-the-art operator learner Fourier Neural Operator (FNO) (Li et al.,  2021).
Revised: There are several baseline models to compare: the CNN-based U-Nets (Ronneberger et al., 2015; Guo & Jiang, 2020); the state-of-the-art operator learner Fourier Neural Operator (FNO) (Li et al., 2021a) and its variant with a token-mixing layer (Guibas et al., 2022); MultiWavelet Neural Operator (MWO) (Gupta et al., 2021); Deep Operator Network (DeepONet) (Lu et al., 2021).

Original: The Transformer model of interest is a drop-in replacement of the baseline U-Net, and it is named by U-Integral Transformer (UIT).
Revised: The Transformer model of interest is a drop-in replacement of the baseline U-Net, and it is named by U-Integral Transformer (UIT).

Original: UIT uses the kernel integral inspired attention (9), and we also compare UIT with the linear attention-based Hybrid U-Transformer in Gao et al.
Revised: UIT uses the kernel integral inspired attention (11), and we also compare UIT with the linear attention-based Hybrid U-Transformer in Gao et al.

Original: (2021), as well as a Hadamard product-based cross-attention U-Transformer in Wang et al.
Revised: (2021), as well as a Hadamard product-based cross-attention U-Transformer in Wang et al.

Revised: An ablation study is also conducted by replacing the convolution layers in the U-Net with attention (11) on the coarsest level.

Original: For more details of the data setup, training, evaluation in the all experiments please refer to Appendix B.
Revised: For more details of the data setup, training, evaluation in the all experiments please refer to Appendix C.

Original: The comparison result can be found in Table 1.
Revised: The comparison result can be found in Table 1.

Original: Due to the fact that FNO  keeping only the  modes inthe lower end of the spectra, it performs relatively poor in this EIT benchmark that needs to recover traits that consist of higher modes (sharp boundary edges of inclusion) from lower modes (smooth harmonic extension).
Revised: Due to the fact that FNO (AFNO, MWO) keeping only the lower modes in spectra, it performs relatively poor in this EIT benchmark where one needs to recover traits that consist of higher modes (sharp boundary edges of inclusion) from lower modes (smooth harmonic extension).

Original: Thanks to Theorem 2, attention-based models are capable to recover “high frequency target from low frequency data”, and in general outperform the CNN-based U-Net despite having only 1 / 3 of the parameters.
Revised: Attention-based models are capable to recover “high frequency target from low frequency data”, and in general outperform the CNN-based U-Nets despite having only 1 /of the parameters.

Original: Another highlight is that, thanks to the unique PDE-based feature map through harmonic extension, the proposed models are extremely robust to noise.
Revised: Another highlight is that the proposed models are extremely robust to noise thanks to the unique PDE-based feature map through harmonic extension.

Original: The proposed models can recover the buried domain under a moderately large noise (5%) and an extreme amount of noise (20%) which can be disastrous for many classical methods.
Revised: The proposed models can recover the buried domain under a moderately large noise (5%) and an extreme amount of noise (20%) which can be disastrous for many classical methods.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: For a boundary value inverse problem [Table] we propose a novel operator learner based on the mathematical structure of the inverse operator and Transformer.
Revised: For a boundary value inverse problem, we propose a novel operator learner based on the mathematical structure of the inverse operator and Transformer.

Original: The proposed architecture consists of two components: the first one is a harmonic extension of boundary data (a PDE-based feature map), and the second one is a modified attention mechanism derived from the classical DSM by introducing some learnable non-local integral kernels.
Revised: The proposed architecture consists of two components: the first one is a harmonic extension of boundary data (a PDE-based feature map), and the second one is a modified attention mechanism derived from the classical DSM by introducing some learnable non-local integral kernels.

Original: The evaluation accuracy on the benchmark problems surpasses the current widely-used CNN-based U-Net and the best operator learner FNO.
Revised: The evaluation accuracy on the benchmark problems surpasses the current widely-used CNN-based U-Net and the best operator learner FNO.

Original: This research strengthens the insights that the attention is an adaptable neural architecture that can incorporate a priori mathematical knowledge to design more physics-compatible DNN architectures.
Revised: This research strengthens the insights that the attention is an adaptable neural architecture that can incorporate a priori mathematical knowledge to design more physics-compatible DNN architectures.

Original: However, we acknowledge some limitations: similar to other operator learners, the data manifold on which the operator is learned is assumed to exhibit certain low-dimensional attributes that can be reasonably well approximated by a fixed number of bases.
Revised: However, we acknowledge some limitations: similar to other operator learners, the data manifold on which the operator is learned is assumed to exhibit certain low-dimensional attributes that can be reasonably well approximated by a fixed number of bases.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: This paper is reproducible.
Revised: This paper is reproducible.

Original: Experimental details about all empirical results described in this paper are provided in Appendix B. Additionally, we provide the PyTorch (Paszke et al., 2019) code for reproducing in the supplemental material.
Revised: Experimental details about all empirical results described in this paper are provided in Appendix C. Additionally, we provide the PyTorch (Paszke et al., 2019) code for reproducing in the supplemental material.

Original: The dataset used in this paper is available at https://www.
Revised: The dataset used in this paper is available at https://www.

Original: kaggle.com/datasets/anonymousauthor25/eit-transformer .
Revised: kaggle.com/datasets/anonymousauthor25/eit-transformer .

Original: Formal proofs under a rigorous setting of all our theoretical results are provided in Appendices C-D.
Revised: Formal proofs under a rigorous setting of all our theoretical results are provided in Appendices E-F.

Original: A T ABLE OFC OMMON - USED N OTATIONS
Revised: A T ABLE OF N OTATIONS

Original: [Table]
Revised: [Table]

Original: problems

Original: [Table] are P [Table] L
Revised: normal derivative of u , measures the rate of change along the direction of n Λ σ

Original: [Table] Ω) [Table] for EIT with L data pairs ∇ u  the gradient vector of a function, ∇ u ( x ) =
Revised: NtD map from Neumann data g := ∇ u · n (how fast the solution changes toward the outward normal direction) to

Original: ( ∂ x 1 u ( x ) , ∂x 2 u ( x )) δ x the delta function such that (cid:82) Ω f ( y ) δ x ( y
Revised: Dirichlet data f := u | ∂ Ω (the solution’s value along the tangential direction)

Original: [Table]
Revised: [Table]

Original: d y = f ( x ) , ∀ f . [Table] normal derivative of u , measures the rate of change along the direction of n
Revised: the Sobolev space of functions

Original: [Table]
Revised: [Table]

Original: NtD mapping from Neumann data
Revised: the bounded linear functional defined on H s

Original: [Table]
Revised: [Table]

Original: u · n (how fast the solution changes toward the outward normal direction) to Dirichlet data
Revised: Ω)

Original: [Table]
Revised: [Table]

Original: the solution’s value [Table] the tangential direction)
Revised: all u

Original: [Table]
Revised: [Table]

Original: the bounded linear functional defined on H s (Ω)

Original:  [Table]
Revised: H s [Table]

Original: u ∈ H s (Ω) such that u ’s integral on Ω vanishes | · |
Revised: Ω) such that u ’s integral on Ω vanishes [Table] the seminorm [Table]

Revised: for

Original: [Table]
Revised: [Table]

Original: the seminorm defined for functions in Y 
Revised: in Y B B ACKGROUND OF EIT For EIT

Revised: [Table]

Original: B E XPERIMENT S ET - UP B. D ATA GENERATION AND TRAINING
Revised: an immediate question is whether F − 1 and F − 1 L in (4) and (5) are well-defined, namely whether σ can be uniquely determined. In fact, for the case of full measurements ( L = ∞ ), the uniqueness for F − 1 has been well established, (Brühl, 2001; Hanke & Brühl, 2003; Astala & Päivärinta, 2006a;b; Nachman, 1996; Kohn & Vogelius, 1984; Sylvester & Uhlmann, 1987). It is worthwhile to point out that, in this case, σ is not necessarily a piecewise constant function.

Revised: In (1), we present a simplified case for purposes of illustrating as well as benchmarking.

Revised: In general, with the full spectral information of the NtD map, σ can be uniquely determined as a general positive function.

Revised: In contrast to the “full measurement” (infinitely many eigen-pairs known), a more challenging and practical problem is to recover σ from only finitely many boundary data pairs.

Revised: A common practice for the theoretical study for finite measurements is the assumption of σ being a piecewise constant function, and the task is usually set to recover the shape and location of the inclusion D .

Revised: Otherwise, the problem is too ill-posed.

Revised: With finite measurements, the uniqueness of the inclusion remains a long-standing theoretical open problem, and it can be only established for several special classes of the inclusion shape, such as the convex cylinders in Isakov & Powell (1990) or convex polyhedrons in Barceló et al.

Revised: We refer readers some counter-examples in Kang & Seo (2001) where a twoor three-dimensional ball may not be identified uniquely by one single measurement if the values of σ 0 and σ 1 are unknown.

Revised: Furthermore, here we provide one example to illustrate the difficulty in the reconstruction procedure (Pidcock et al., 1995).

Revised: Let Ω be a unit circle, let D be a circle with the radius ρ < 1 , and define

Revised: [Equation]

Revised: with σ 1 < 1 being an arbitrary constant.

Revised: In this case, the eigenpairs of Λ σ can be explicitly calculated

Revised: [Equation]

Revised: with µ = (1 − √ σ 1 ) / (1 + √ σ 1 ) , which are exactly the Fourier modes in a unit circle.

Original: In  the numerical examples
Revised: In this case, if the set of basis { g l } ∞ l =1 is just chosen as { cos( θ ) , sin( θ ) , cos(2 θ ) , sin(2 θ ) , ... } , the matrix representation A σ in (3) can be written as an infinite diagonal matrix [Equation] Thanks to this special geometry, eigenvalues in (25) can clearly determine ρ and σ 1 as follows: [Equation]

Original: [Table]
Revised: However, in practice A σ does not have such a simple structure. Approximating A σ itself requires a large number of data pairs which are not available in the considered case. Besides, computing the eigenvalues of A σ accurately is also very expensive. Furthermore, for complex inclusion shapes, two eigenvalues are not sufficient to exactly recover the shape and conductivity values.

Original:  the data generation mainly follows what were featured in (Fan & Ying, 2020; Guo & Jiang, 2020)
Revised: In the numerical examples, the data generation mainly follows a standard practice in theoretical prototyping for solving EIT problems, see e.g.,Chow et al.

Original: [Table]
Revised: ; Michalikova et al. ; Hamilton & Hauptmann (2018); Guo & Jiang (2020); Fan & Ying (2020).

Original: For an example please refer to Figure 2.
Revised: For examples please refer to Figure 2.

Original: The computational domain is set to be Ω :=
Revised: The computational domain is set to be Ω :=

Original: ( − 1 , 1) [Table] and the two media with the different conductivities are with σ 1 = 10 (inclusion) and σ 0 = 1 (background).
Revised: ( − 1 , 1) 2 , and the two media with the different conductivities are with σ 1 = 10 (inclusion) and σ 0 = 1 (background).

Original: The inclusions are generated by four random ellipses with the length of the semi-major axis and semi-minor axis sampled from U (0 . 1 , 0 . 2) and U (0 . 2 , 0 . 4) , respectively.
Revised: The inclusions are generated by four random ellipses with the length of the semi-major axis and semi-minor axis sampled from U (0 . 1 , 0 . 2) and U (0 . 2 , 0 . 4) , respectively.

Original: The rotation angle sampled from U (0 , 2 π ) .
Revised: The rotation angle sampled from U (0 , 2 π ) .

Original: There are 10800 samples in the training set,  and 2000 in the testingset. The batch size is set to be 10.
Revised: There are 10800 samples in the training set, of which 20% are reserved as validation, and 2000 in the testing set for evaluation.

Original: All models are trained using the 1cycle scheduler (Smith & Topin, 2019) for 50 epochs with a 10-epoch warm-up.

Original: The noise ξ = ξ ( x ) below (14) is assumed to be
Revised: The noise ξ = ξ ( x ) below (10) is assumed to be

Original: [Equation]
Revised: [Equation]

Original: (27) where τ specifies the percentage of noise, and G ( x ) is a normal Gaussian distribution independent with respect to x .
Revised: (28) where τ specifies the percentage of noise, and G ( x ) is a normal Gaussian distribution independent with respect to x .

Original: As ξ ( x ) is merely pointwise imposed, the boundary data can be highly rough even though the true data is  smooth.
Revised: As ξ ( x ) is merely pointwise imposed, the boundary data can be highly rough even though the true data is chosen to be smooth.

Revised: Nevertheless, the harmonic extension makes the noise from boundary data to have impact on the overall reconstruction thanks for the smoothing property of the inverse of the Laplacian operator ( − ∆) − 1 , for an example please see Figure 3.

Original: Thanks to the position-wise binary nature of I D , another choice of the loss function during training can be the binary cross entropy L ( · , · ) , applied for a function in P , to measure the distance between 
Revised: Thanks to the position-wise binary nature of I D , another choice of the loss function during training can be the binary cross entropy L ( · , · ) , applied for a function in P , to measure the distance between the ground truth and the network’s prediction

Original: [Equation]
Revised: [Equation]

Original: Thanks for the Pinsker inequality (e.g., see (Cover, 1999, Section 11.6)), L ( p h , u h ) serves as a good upper bound for the square of the total variation, which can be further bounded below by the L 2 -error given the boundedness of the position-wise value. 
Revised: Thanks for the Pinsker inequality (e.g., see (Cover, 1999, Section 11.6)), L ( p h , u h ) serves as a good upper bound for the square of the total variation, which can be further bounded below by the L 2 -error given the boundedness of the position-wise value. −0.

Original:  The training uses 1cycle (Smith & Topin, 2019) learning rate strategy with a warm-up phase.
Revised: −0. −0. −0. The training uses 1cycle (Smith & Topin, 2019) learning rate strategy with a warm-up phase.

Original: A mini-batch ADAM iterations are run for a total number of 50 epochs with no extra regularization such as weight decay.
Revised: A mini-batch ADAM iterations are run for a total number of 50 epochs with no extra regularization such as weight decay.

Revised: The evaluated model is taken from the epoch that has the best validation metric.

Original: The learning rate starts and ends with 10 − 3 · lr max , and reaches the maximum of lr max at the end of the 10 -th epoch.
Revised: The learning rate starts and ends with 10 − 3 · lr max , and reaches the maximum of lr max at the end of the 10 -th epoch.

Original: The lr max = 10 − 3 .
Revised: The lr max = 10 − 3 .

Original: The result demonstrated is obtained from fixing the random number generator seed.
Revised: The result demonstrated is obtained from fixing the random number generator seed.

Original: Some testing results  can be seen in Figure  All models are trained on an RTX 3090 or an A4000.
Revised: The testing results for a randomly chosen sample can be seen in Figure 6. All models are trained on an RTX 3090 or an A4000.

Original: The codes to replicate the experiments is open-source and publicly available 1 .
Revised: The codes to replicate the experiments are open-source and publicly available.

Original: B. N ETWORK ARCHITECTURE
Revised: C. N ETWORK ARCHITECTURE

Original:  The U-Transformer architecture is a drop-in replacement of the standard CNN-based U-Net. For a high level encoder-decoder schematics please see Figure 7.
Revised: [Figure] The difference in architectural hyperparameters, together with training and evaluation costs comparison for all the models compared in the task of EIT reconstruction can be found in Table 3. U-Integral Transformer

Original: Positional embedding.
Revised: • Overall architecture. The U-Integral-Transformer architecture is a drop-in replacement of the standard CNN-based U-Net baseline model (7.7m) in Table 1. Same with the baseline U-Net, the UIT has total 3 downsampling layers.

Original: At each resolution, The 2D Euclidean coordinates of a √ M × √ M regular grid are applied a channel expansion through a learnable linear layer, and then are added to each latent representation.
Revised: The downsampling layers map m × m latent representations to m/ 2 × m/ 2 , and expand the number of channels from C to 2 C . UIT has 3 upsampling layers, which map m/ 2 × m/ 2 latent representations to m × m , and shrink the number of channels from 2 C to C . An attention block is applied the coarsest level, as well as in upsampling layers to compute the interaction between the latent representations on both coarse and find grids (see below). For a high level encoder-decoder schematics please see Figure 8.

Original:  Double convolution block.
Revised: • Double convolution block.

Original: The double convolution block is modified from that commonly seen in Computer Vision (CV) models, such as ResNet (He et al., 2016).
Revised: The double convolution block is modified from that commonly seen in Computer Vision (CV) models, such as ResNet (He et al., 2016).

Original: We modify this block such that upon being used in an attention block, the batch normalization (Ioffe & Szegedy, 2015) shall be replaced by the layer normalization (Ba et al., 2016). LN =
Revised: We modify this block such that upon being used in an attention block, the batch normalization (Ioffe & Szegedy, 2015) shall be replaced by the layer normalization (Ba et al., 2016) which can be understood as a learnable approximation to the Gram matrices’ inverse by a diagonal matrix.

Original: = =
Revised: • Positional embedding. At each resolution, The 2D Euclidean coordinates of an m × m regular Cartesian grid are applied a channel expansion through a learnable linear layer, and then are added to each latent representation.

Original: LN
Revised: This choice of positional embedding enables a bilinear interpolation between the coarse and fine grids or vice versa (see below). • Mesh-normalized attention. The scaled dot-product attention in the network is chosen to be a softmax-free integral kernel attention in (11) with a mesh-based normalization. For a diagram in a single attention head please refer to Figure 7.

Original: LN
Revised: • Interpolation. Instead of max pooling used in the standard CNN-based U-Net, we opt for a bilinear interpolation on the Cartesian grid to map a latent representation from the fine grid to the coarse grid or vice versa. Note that, in the upsampling layers, the interpolation’s outputs are directly fed into an attention block that computes the interaction of latent representations between coarse and fine grids (see below).

Original: Conv
Revised: [Figure]

Original: Conv Conv = 
Revised: = = = Norm Norm

Original: [Figure]
Revised: [Figure]

Original:  Coarse-fine interaction in up blocks.
Revised: • Coarse-fine attention in up blocks.

Original: The convolution layer on the coarsest level is replaced by the attention with pre-inner-product normalization in Section 3.2.
Revised: The convolution layer on the coarsest level is replaced by the attention with pre-inner-product normalization in Section 3.3.

Original: The skip connection from the encoder latent representations to the ones in the decoder are generated using an architecture similar to the cross attention used in (Petit et al., 2021).
Revised: The skip connection from the encoder latent representations to the ones in the decoder are generated using an architecture similar to the cross attention used in (Petit et al., 2021).

Original: Q and K are generated from the latent representation functions on the same coarser grids, such that the attention kernel to measure the interaction between different channels is built from on the coarse grid.
Revised: Q and K are generated from the latent representation functions on the same coarser grids, such that the attention kernel to measure the interaction between different channels is built from on the coarse grid.

Original: V is associated with a finer grid.
Revised: V is associated with a finer grid.

Original: The major differences of ours with the one in Petit et al.
Revised: The major differences of ours with the one in Petit et al.

Original: (2021) are that ours are inspired by the kernel integral for a PDE problem, thus the modified attention in our method has (1) no softmax normalization or (2) no Hadamard product-type skip connection. 
Revised: (2021) are that ours are inspired by the kernel integral for a PDE problem, thus the modified attention in our method has (1) no softmax normalization or (2) no Hadamard product-type skip connection. Other operator learners compared • Fourier Neural Operator (FNO) and variants.

Original: C P ROOF OF T HEOREM
Revised: Fourier Neural Operator (FNO2d) learns convolutional filters in the frequency domain for some pre-selected modes, which efficiently captures globally-supported spatial interactions for these modes. The weight filter in the frequency domain is multiplied to the lowest modes in the latent representations (four corners in the FFT). The Adaptive Fourier Neural Operator (AFNO2d) adds a token-mixing layer, as seen in Figure 2 in Guibas et al. , appending every spectral convolution layer in the baseline FNO2d model.

Revised: • Multi-Wavelet Neural Operator (MWO).

Revised: The MultiWavelet Neural Operator (MWO) is proposed in Gupta et al.

Revised: (2021), which introduces a multilevel structure into the FNO architecture.

Revised: On each level, MWO still follows FNO’s practice by pre-selecting the lowest modes.

Revised: Deep

Revised: Operator Network (DeepONet).

Revised: The Deep Operator Network (DeepONet) is proposed in

Revised: Lu et al.

Revised: DeepONet’s architecture leverages a learnable basis expansion to approximate nonlinear operators (Chen & Chen, 1995), where the bases and the coefficients are learned in two individual fully-connected neural networks (branch and trunk) which have no cross-layer communications.

Revised: However, we do acknowledge that the original DeepONet is not a tensor2tensor neural network, as the number of parameters depend on the number of “sensors” (discretization grid points).

Revised: In the comparison, the branch DNN is changed from a fully-connected NN to a

Revised: DCNN.

Revised: D F ROM DSM TO T RANSFORMER

Revised: In this section, we present a more detailed presentation of the attention-like operator being derived from (15) with learnable kernels.

Revised: We begin with recalling the original indicator function from (10):

Revised: [Equation]

Revised: where C f,g = ∥

Revised: f − Λ σ 0 g ∥ − 1 L 2 ( ∂ Ω) is a constant, and the equations of the functions ϕ and η x are :

Revised: [Equation]

Revised: [Table]

Revised: [Figure]

Revised: : 3 × 3 convolution + ReLU; : layer normalization or batch normalization; : bilinear interpolation from the fine grid to the coarse grid; : cross attention from the coarse grid to the fine grid; : input and output discretized functions in certain Hilbert spaces.

Revised: In Chow et al.

Revised: (2014), it has been shown that I D ( x ) can be written as a sum of Gaussian-like distributions which attain maximal values for x ∈ D or close to D .

Revised: However, the accuracy is much limited by some empirical choices of quantities in (30), for example d ( x ) =

Revised: ∇ ϕ

Revised: ( x ) / ∥∇ ϕ

Revised: ( x ) ∥ and s = 3 / 2 .

Revised: In addition, the frequency of the boundary data plays an important role in reconstruction, see the derivation in (Chow et al., 2014, Section 5) for circular Ω .

Revised: What’s more, only for a single data pair, such a simple formula can be derived.

Revised: Henceforth, these “empirical chosen quantities” are made to be learnable from data by introducing two undetermined kernels K ( x, y ) , Q ( x, y ) , and a self-adjoint positive definite linear operator V , the modified indicator function is written as

Revised: [Equation]

Revised: 33) with

Revised: [Equation]

Revised: [Equation]

Revised: Applying certain quadrature rule to (33) with the quadrature points z i , i.e., the grid points of Ω h , and weights { ω j } , we obtain an approximation to the integral:

Revised: [Equation]

Revised: i.e., q Ti is the vector of [ ω j K ( z i , z j )]

Revised: j .

Revised: For (34), we consider one component d n ( x ) of d ( x ) .

Revised: With the same rule to compute the integral, (16) can be written as

Revised: [Equation]

Revised: i.e., q Ti is the vector of [ ω j Q ( z i , z j )] j .

Revised: Next, we proceed to express | η

Revised: z i | Y by discretizing the variational form in (32) using a linear finite element method (FEM).

Revised: Applying integration by parts, the weak form of (32) at x =

Revised: z i is

Revised: [Equation]

Revised: for any test function ψ ∈ H 10 (Ω) .

Revised: Here, we let { ψ ( j ) } Mj =1 be the collection of the finite element basis functions, and for the fixed z i we further let ψ n,i be the vector approximating [ ∂ x n ψ ( j ) ( z i )]

Revised: Mj =1 .

Revised: Denote η i as the vector approximating { η z i (

Revised: z j ) } Mj =1 .

Revised: Then, the finite element discretization of (38) yields the linear system:

Revised: [Equation]

Revised: where we have used (37).

Revised: Then, the trace of η z i on ∂ Ω admits the following approximation

Revised: [Equation]

Revised: [Equation]

Revised: Now, we can discretize (35).

Revised: Note that the trace of a linear finite element space on ∂ Ω is still a continuous piecewise linear space which is defined as S h ( ∂ Ω) .

Revised: Then, the self-adjoint positive definite operator V can be parameterized by a symmetric positive definite (SPD) matrix denoted by

Revised: V operating on the space S h ( ∂ Ω) .

Revised: We can approximate | η

Revised: z i | 2 Y as

Revised: [Equation]

Revised: ) where ψ Tn,i B − 1 R T V RB − 1 ψ n,i ≥ 0 as V is SPD.

Revised: Define

Revised: [Equation]

Revised: which can be considered as another learnable vector since the coefficient of q i comes from the learnable matrix V .

Revised: Then, (41) reduces to

Original: Theorem 1 (A finite dimensional approximation of the index map) .
Revised: [Equation] Putting (36), (37) and (43) into (15), we have [Equation] Now, using the notation in (21), we get the desired representation (22).

Original:  Suppose the boundary data g l is the eigenfunction of Λ σ − Λ σ 0 corresponding to the l -th eigenvalue λ l , and let ϕ l be the data functions generated by harmonic extensions 
Revised: Lemma 1. Suppose the boundary data g l is the eigenfunction of Λ σ − Λ σ 0 corresponding to the l -th eigenvalue λ l , and let ϕ l be the data functions generated by harmonic extensions − ∆ ϕ l = 0 in Ω ,

Original: [Equation]
Revised: [Equation]

Original: where l = 1 , 2 ,·
Revised: where l = 1 , 2 ,

Original: Define the space on R N(the spatial dimension N = 2 , 3 ):
Revised: Let d be an arbitrary direction in R N , define a function [Equation] Then, there holds

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: Proof. See Theorem 4.1 in (Guo & Jiang, 2020), and also see (Brühl, 2001; Hanke & Brühl, 2003). Theorem 1 (A finite dimensional approximation of the index map) .

Original: Then, for any ϵ > 0  , there exists a sufficiently large L such that
Revised: Suppose the boundary data g l is the eigenfunction of Λ σ − Λ σ 0 corresponding to the l -th eigenvalue λ l , and let ϕ l be the data functions generated by harmonic extensions given in (9) . Define the space:

Original: [Equation]
Revised: [Equation]

Original: Proof. Let d be any probing direction in R N (Brühl, 2001; Hanke & Brühl, 2003). Define a function
Revised: and the dictionary:

Original: [Equation]
Revised: [Equation]

Original: By Theorem 4.1 in (Guo & Jiang, 2020), we can show that
Revised: Then, for any ϵ > 0 , there exists a sufficiently large L such that

Original: [Equation]
Revised: [Equation]

Original:  As Θ L ( x ) > 0 , it is increasing with respect to L .
Revised: Proof. Consider the function Θ L ( x ) from Lemma 1. As Θ L ( x ) > 0 , it is increasing with respect to L .

Original: Then, there is a constant ρ such that ρ > Θ L ( x ) , ∀ x ∈ D .
Revised: Then, there is a constant ρ such that ρ > Θ L ( x ) , ∀ x ∈ D .

Original: Given any ϵ > 0 , there is an interger L such that Θ L ( x ) > 4 ρϵ − 2 /π 2 , ∀ x / ∈ D .
Revised: Given any ϵ > 0 , there is an integer L such that Θ L ( x ) > 4 ρϵ − 2 /π 2 , ∀ x / ∈ D .

Original: Define
Revised: Define

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note the fundamental inequality z > arctan( z ) ≥ π 2 − z − 1 , ∀ z > 0 .
Revised: Note the fundamental inequality z > arctan( z ) ≥ π 2 − z − 1 , ∀ z > 0 .

Original: Then, if x ∈ D , there holds|I
Revised: Then, if x ∈ D , there holds

Original: [Equation]
Revised: [Equation]

Original: if x / ∈ D , there holds
Revised: if x / ∈ D , there holds

Original: [Equation]
Revised: [Equation]

Original: Therefore, the function in (34) fulfills (31).
Revised: Therefore, the function in (51) fulfills (50).

Original: D P ROOF OF T HEOREM
Revised: F P ROOF OF T HEOREM

Original: In presenting Theorem 2 in Section 3.2, we use the term “multiplicative” to describe the fact that two latent representations are multiplied in the attention mechanism, and, in contrast, there is no such operation in, e.g., a pointwise FFN or a convolution layer.
Revised: In presenting Theorem 2 in Section 3.3, we use the term “multiplicative” to describe the fact that two latent representations are multiplied in the attention mechanism, and, in contrast, there is no such operation in, e.g., a pointwise FFN or a convolution layer.

Original: Heuristically speaking, the main result in
Revised: Heuristically speaking, the main result in

Original: Theorem 2 states that the output latent representations can be of a higher “frequency” than the input if the neural network architecture has “multiplicative” layers in it.
Revised: Theorem 2 states that the output latent representations can be of a higher “frequency” than the input if the neural network architecture has “multiplicative” layers in it.

Original: The input latent representations are discretizations of certain functions, and they are combined using matrix dot product as the one used in attention.
Revised: The input latent representations are discretizations of certain functions, and they are combined using matrix dot product as the one used in attention.

Original: If it is assumed that this discretization can represent functions of such a frequency with certain approximation error, then the resulting matrix/tensor can be an approximation of a function with higher frequency than the existing latent representation under the same discretization.
Revised: If it is assumed that this discretization can represent functions of such a frequency with certain approximation error, then the resulting matrix/tensor can be an approximation of a function with higher frequency than the existing latent representation under the same discretization.

Revised: For an empirical evidence of this phenomenon for the latent representation please see Figure 9 and Figure 10: with completely smooth input (harmonic extensions), attention-based learner is able to generate latent representations with multiple peaks and valleys.

Original: Theorem 2 (Frequency-bootstrapping for multiplicative neural architectures) .
Revised: Theorem 2 (Frequency-bootstrapping for multiplicative neural architectures) .

Original: Consider Ω = (0 , π ) which has a uniform discretization of { z i } Mi =1 of size h , and N ≥ 1 is the number of channels in the attention layer of interest, assume that (i) the current latent representation p h ∈ R M × N consists the discretization of the first N Legendre polynomials { p j ( · ) } Nj =1 such that ( p h ) ij =
Revised: Consider Ω = (0 , π ) which has a uniform discretization of { z i } Mi =1 of size h , and N ≥ 1 is the number of channels in the attention layer of interest, assume that (i) the current latent representation p h ∈ R M × N consists the discretization of the first N Legendre polynomials { p j ( · ) } Nj =1 such that ( p h ) ij =

Original: p j ( z i ) , (ii) p h is normalized and the normalization weights α ≡ 1 in (13) , (iii) the discretization satisfies that | 
Revised: p j ( z i ) , (ii) p h is normalized and the normalization weights α ≡ 1 in (14) , (iii) the discretization satisfies that | (cid:80) M i

Original: [Equation] x ) d x | ≤ C h .
Revised: =1 hf ( z i ) − (cid:82) Ω f ( x ) d x | ≤ C h .

Original: Then, for v ( x ) = sin( ax ) for some a ∈ R + , there exists a set of attention weights { W Q , W K , W V } such that for u ( x ) = sin( a ′ x ) with a ′ >
Revised: Then, for v ( x ) = sin( ax ) for some a ∈ R + , there exists a set of attention weights { W Q , W K , W V } such that for u ( x ) = sin( a ′ x ) with a ′ >

Original: a
Revised: a

Original: [Equation] 
Revised: [Equation] [Equation]

Original: 35) where ˜ u and ˜ κ ( · , · ) are defined as the output of and the kernel of the attention formulation in (13) , respectively; ε ( x ) :
Revised: where ˜ u and ˜ κ ( · , · ) are defined as the output of and the kernel of the attention formulation in (14) , respectively; ε ( x ) :

Original: = ∥ κ ( x, · ) − ˜ κ ( x, · ) ∥
Revised: = ∥ κ ( x, · ) − ˜ κ ( x, · ) ∥

Original: L 2 (Ω) is the error function for the kernel approximation.
Revised: L 2 (Ω) is the error function for the kernel approximation.

Original: Proof.
Revised: Proof.

Original: Without loss of generality, it is assumed that a = 1 and a ′ = 2 .
Revised: Without loss of generality, it is assumed that a = 1 and a ′ = 2 .

Original: The essential technical tools used suggest the validity for any a ′ > a > 0 ( a ′ , a ∈ Z + ).
Revised: The essential technical tools used suggest the validity for any a ′ > a > 0 ( a ′ , a ∈ Z + ).

Original: Consider a simple non-separable smooth kernel function
Revised: Consider a simple non-separable smooth kernel function

Original: [Equation]
Revised: [Equation]

Original: it is straightforward to verify that for v ( x
Revised: it is straightforward to verify that for v ( x

Original: [Equation]
Revised: [Equation]

Original: = sin[Equation] [Equation] As is shown
Revised: sin

Original: [Equation]
Revised: [Equation]

Original:  it suffices to show that the matrix multiplication (a separable kernel) in the attention mechanism approximates this non-separable kernel with an error related to the number of channels.
Revised: As is shown, it suffices to show that the matrix multiplication (a separable kernel) in the attention mechanism approximates this non-separable kernel with an error related to the number of channels.

Original: To this end, taking the Taylor expansion, centered at a z 0 ∈ Ω of κ ( x, · ) with respect to the second variable at each x ∈ Ω , we have[Equation]
Revised: To this end, taking the Taylor expansion, centered at a z 0 ∈ Ω of κ ( x, · ) with respect to the second variable at each x ∈ Ω , we have

Original: [Equation]
Revised: [Equation]

Original: It is straightforward to check that
Revised: It is straightforward to check that

Original: [Equation]
Revised: [Equation]

Original: By the assumptions on κ ( · , · ) [Equation] and a simple calculus we have
Revised: By the assumptions on κ ( · , · ) , and a simple calculus we have

Original: [Equation]
Revised: [Equation]

Original: Next, denote q l ( z ) :=
Revised: Next, denote q l ( z ) :=

Original: ( z − z 0 ) l − 1 / ( l − 1)!
Revised: ( z − z 0 ) l − 1 / ( l − 1)!

Original: and k l ( x ) :
Revised: and k l ( x ) :

Original: = ∂ l − 1 κ/∂z l − 1 ( x, z 0 ) for 1 ≤ l ≤ N , i.e., 
Revised: = ∂ l − 1 κ/∂z l − 1 ( x, z 0 ) for 1 ≤ l ≤ N , i.e., κ N ( x, z ) =

Original: [Equation]
Revised: N (cid:88) l = q l ( x ) k l ( z ) .

Original: By the choice of the latent representation space, q l ∈ Y = span { p j } , thus there exists a set of weights { w Ql ∈ R L } Nl =1 such that q l ( · ) =
Revised: By the choice of the latent representation space, q l ∈ Y = span { p j } , thus there exists a set of weights { w Ql ∈ R L } Nl =1 such that q l ( · ) =

Original: (cid:80) lj
Revised: (cid:80) lj

Original: =
Revised: =

Original: 1 w Ql,j p j ( · ) =:
Revised: 1 w Ql,j p j ( · ) =:

Original: ˜ q l ( · ) .
Revised: ˜ q l ( · ) .

Original: This is to say, Q = p h W Q ∈ R M × N with the l -th column of Q being the discretization of ˜ q l ( · ) .
Revised: This is to say, Q = p h W Q ∈ R M × N with the l -th column of Q being the discretization of ˜ q l ( · ) .

Original: For the key matrix, by standard polynomial approximation estimate, there exists a set of weights {
Revised: For the key matrix, by standard polynomial approximation estimate, there exists a set of weights {

Original: w Kl ∈ R N } Nl =1 , such that k l ( · ) ≈ (cid:80) Nj =
Revised: w Kl ∈ R N } Nl =1 , such that k l ( · ) ≈ (cid:80) Nj =

Original: 1 w Kl,j p j ( · )
Revised: 1 w Kl,j p j ( · )

Original: =:
Revised: =:

Original: ˜ k l ( · ) , i.e., K = p h W K ∈ R M × N with the l -th column of K being the discretization of ˜ k l ( · ) .
Revised: ˜ k l ( · ) , i.e., K = p h W K ∈ R M × N with the l -th column of K being the discretization of ˜ k l ( · ) .

Original: Moreover, it can approximate k l ( · )
Revised: Moreover, it can approximate k l ( · )

Original: [Equation]
Revised: [Equation]

Original: Similarly, for v ( · ) := v 1 ( · ) which can be chosen as the 1st channel of V defined earlier, we have 
Revised: Similarly, for v ( · ) := v 1 ( · ) which can be chosen as the 1st channel of V defined earlier, we have { w Vl ∈ R N } such that v ( · ) ≈ (cid:80) Nj =

Original: [Equation]
Revised: 1 w K 1 ,j p j ( · ) =:

Original: such that v 
Revised: ˜ v ( · )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now, to approximate the frequency bootstrapping in (37)
Revised: Now, to approximate the frequency bootstrapping in (54), define

Original: [Equation]
Revised: [Equation]

Original: define
Revised: with ˜

Original: [Equation]
Revised: [Equation]

Original: with ˜ κ N 
Revised: N [Equation] x

Original: [Equation]
Revised: [Equation]

Revised: z

Original: [Equation]
Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: Then, we have for any z ∈ Ω
Revised: Then [Equation] we have for any z ∈ Ω

Original: [Equation]
Revised: [Equation]

Original: Thus we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: Now, by triangle inequality, q l = ˜ q l , the definitions above (36) implying | k l ( · ) | H N (Ω) ≤ c 2 N and ∥
Revised: Now, by triangle inequality, q l = ˜ q l , the definitions above (53) implying | k l ( · ) | H N (Ω) ≤ c 2 N and ∥

Original: q l ∥ L 2 (Ω) ≤ cπ N / ( √ NN !) , and the estimate in (42)
Revised: q l ∥ L 2 (Ω) ≤ cπ N / ( √ NN !) [Equation] and the estimate in (59)

Original: [Equation]
Revised: [Equation]

Original: Notice this is the same order with the estimate of ∥ κ ( x, · ) − κ N ( x, · )
Revised: Notice this is the same order with the estimate of ∥ κ ( x, · ) − κ N ( x, · )

Original: ∥
Revised: ∥

Original: L 2 (Ω) .
Revised: L 2 (Ω) .

Original: For the term ∥ ˜ κ N ( x, · ) ∥
Revised: For the term ∥ ˜ κ N ( x, · ) ∥

Original: L 2 (Ω) , a simple triangle inequality trick can be used:
Revised: L 2 (Ω) , a simple triangle inequality trick can be used:

Original: [Equation]
Revised: [Equation]

Original: which can be further estimate reusing the argument in (47).
Revised: which can be further estimate reusing the argument in (64).

Original: Lastly, using the following argument and the estimate for ∥ u − ˜ u
Revised: Lastly, using the following argument and the estimate for ∥ u − ˜ u

Original: ∥ L ∞ (Ω) yield the desired result:
Revised: ∥ L ∞ (Ω) yield the desired result:

Original: [Equation]
Revised: [Equation]

Revised: G L IMITATIONS , E XTENSIONS , AND F UTURE WORK

Revised: In this study, the σ to be recovered relies on a piecewise constant assumption.

Revised: This assumption is commonly seen in the theoretical study of the original DSM.

Revised: For many EIT applications in medical imaging and industrial monitoring, σ may involve non-sharp transitions or even contain highly anisotropic/multiscale behaviors making it merely an L ∞ function.

Revised: If the boundary data pairs are still quite limited, i.e., only a few electric modes are placed on the boundary ∂ Ω , the proposed model alone is not expected to perform as well as in benchmark problems.

Revised: Nevertheless, it can still contribute in achieving reconstruction with relatively higher accuracy, if certain a priori knowledge of the problem is accessible.

Revised: End2end-wise, the modification can be to employ a suitable data set, in which the sampling of { σ ( k ) } represent the true σ ’s distribution a posteriori to a certain degree.

Revised: However, to reconstruct non-piecewise constant conductivities, more boundary data pairs or even the entire NtD map is demanded from a theoretical perspective (Astala & Päivärinta, 2006a;b; Nachman, 1996; Kohn & Vogelius, 1984; Sylvester & Uhlmann, 1987).

Revised: In the case considered, a Fredholm integral equation needs to be solved for reconstruction, see Nachman (1996, Theorem 4.1), which itself is strongly related to the modified attention mechanism (14) of the proposed Transformer.

Revised: The architectural resemblance may lead to future explorations in this direction.

Revised: From the perspective of improving the reconstruction for a single instance, inspired by the classical iterative methods, as well as a novel adaption by the authors of FNO (Li et al., 2021b), an optimization with regularization can be applied for the instance of interest (fine-tune), while the initial guess is the reconstruction by the operator learner trained in the end2end pipeline (pre-train).

Revised: [Figure]

Revised: ) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l)

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]