Original: R ANDOM W EIGHT F ACTORIZATION
Revised: R ANDOM W EIGHT F ACTORIZATION

Original: IMPROVES
Revised: IMPROVES

Original: THE TRAINING OF C ONTINUOUS N EURAL R EPRESENTATIONS
Revised: THE TRAINING OF C ONTINUOUS N EURAL R EPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Continuous neural representations have recently emerged as a powerful and flex ible alternative to classical discretized representations of signals.
Revised: Continuous neural representations have recently emerged as a powerful and flex ible alternative to classical discretized representations of signals.

Original: However, train
Revised: However, train

Original: - ing them to capture fine details in multi-scale signals is difficult and computationally expensive.
Revised: - ing them to capture fine details in multi-scale signals is difficult and computationally expensive.

Original: Here we propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers in coordinate-based multi-layer perceptrons (MLPs) that significantly accelerates and improves their training.
Revised: Here we propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers in coordinate-based multi-layer perceptrons (MLPs) that significantly accelerates and improves their training.

Original: We show how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate.
Revised: We show how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate.

Original: This not only helps with mitigating spectral bias, but also allows networks to quickly recover from poor initializations and reach better local minima.
Revised: This not only helps with mitigating spectral bias, but also allows networks to quickly recover from poor initializations and reach better local minima.

Original: We demonstrate how random weight factorization can be leveraged to improve the training of neural representations on a variety of tasks, including image regression, shape representation, computed tomography, inverse rendering, solving partial differential equations, and learning operators between function spaces.
Revised: We demonstrate how random weight factorization can be leveraged to improve the training of neural representations on a variety of tasks, including image regression, shape representation, computed tomography, inverse rendering, solving partial differential equations, and learning operators between function spaces.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Some of the recent advances in machine learning can be attributed to new developments in the design of continuous neural representations , which employ coordinate-based multi-layer perceptrons (MLPs) to parameterize discrete signals (e.g. images, videos, point clouds) across space and time.
Revised: Some of the recent advances in machine learning can be attributed to new developments in the design of continuous neural representations , which employ coordinate-based multi-layer perceptrons (MLPs) to parameterize discrete signals (e.g. images, videos, point clouds) across space and time.

Original: Such parameterizations are appealing because they are differentiable and much more memory efficient than grid-sampled representations, naturally allowing smooth interpolations to unseen input coordinates.
Revised: Such parameterizations are appealing because they are differentiable and much more memory efficient than grid-sampled representations, naturally allowing smooth interpolations to unseen input coordinates.

Original: As such, they have achieved widespread success in a variety of computer vision and graphics tasks, including image representation (Stanley, 2007;
Revised: As such, they have achieved widespread success in a variety of computer vision and graphics tasks, including image representation (Stanley, 2007;

Original: Nguyen et al., 2015), shape representation (Chen & Zhang, 2019; Park et al., 2019; Genova et al., 2019; 2020), view synthesis (Sitzmann et al., 2019;
Revised: Nguyen et al., 2015), shape representation (Chen & Zhang, 2019; Park et al., 2019; Genova et al., 2019; 2020), view synthesis (Sitzmann et al., 2019;

Original: Saito et al., 2019; Mildenhall et al., 2020; Niemeyer et al., 2020), texture generation (Oechsle et al., 2019; Henzler et al., 2020), etc.
Revised: Saito et al., 2019; Mildenhall et al., 2020; Niemeyer et al., 2020), texture generation (Oechsle et al., 2019; Henzler et al., 2020), etc.

Original: Coordinate-based MLPs have also been applied to scientific computing applications such as physics-informed neural networks (PINNs) for solving forward and inverse partial differential equations (PDEs)
Revised: Coordinate-based MLPs have also been applied to scientific computing applications such as physics-informed neural networks (PINNs) for solving forward and inverse partial differential equations (PDEs)

Original: Raissi et al.
Revised: Raissi et al.

Original: Karniadakis et al.
Revised: Karniadakis et al.

Original: (2021), and Deep Operator networks (DeepONets) for learning operators between infinitedimensional function spaces Lu et al.
Revised: (2021), and Deep Operator networks (DeepONets) for learning operators between infinitedimensional function spaces Lu et al.

Original: ; Wang et al. (2021e).
Revised: ; Wang et al. (2021e).

Original: Despite their flexibility, it has been shown both empirically and theoretically that coordinate-based
Revised: Despite their flexibility, it has been shown both empirically and theoretically that coordinate-based

Original: MLPs suffer from “spectral bias” (Rahaman et al., 2019; Cao et al., 2019; Xu et al., 2019).
Revised: MLPs suffer from “spectral bias” (Rahaman et al., 2019; Cao et al., 2019; Xu et al., 2019).

Original: This manifests as a difficulty in learning the high frequency components and fine details of a target function.
Revised: This manifests as a difficulty in learning the high frequency components and fine details of a target function.

Original: A popular method to resolve this issue is to embed input coordinates into a higher dimensional space, for example by using Fourier features before the MLP (Mildenhall et al., 2020; Tancik et al., 2020).
Revised: A popular method to resolve this issue is to embed input coordinates into a higher dimensional space, for example by using Fourier features before the MLP (Mildenhall et al., 2020; Tancik et al., 2020).

Original: Another widely used approach is the use of SIREN networks (Sitzmann et al., 2020), which employs MLPs with periodic activations to represent complex natural signals and their derivatives.
Revised: Another widely used approach is the use of SIREN networks (Sitzmann et al., 2020), which employs MLPs with periodic activations to represent complex natural signals and their derivatives.

Original: One main limitation of these methods is that a number of associated hyper-parameters (e.g. scale factors) need to be carefully tuned in order to avoid catastrophic generalization/interpolation errors.
Revised: One main limitation of these methods is that a number of associated hyper-parameters (e.g. scale factors) need to be carefully tuned in order to avoid catastrophic generalization/interpolation errors.

Original: Unfortunately, the selection of appropriate hyper-parameters typically requires some prior knowledge about the target signals, which may not be available in some applications.
Revised: Unfortunately, the selection of appropriate hyper-parameters typically requires some prior knowledge about the target signals, which may not be available in some applications.

Original: More general approaches to improve the training and performance of MLPs involve different types of normalizations, such as Batch Normalization (Ioffe & Szegedy, 2015), Layer Normalization (Ba et al., 2016) and Weight Normalization (Salimans & Kingma, 2016).
Revised: More general approaches to improve the training and performance of MLPs involve different types of normalizations, such as Batch Normalization (Ioffe & Szegedy, 2015), Layer Normalization (Ba et al., 2016) and Weight Normalization (Salimans & Kingma, 2016).

Original: However, despite their remarkable success in deep learning benchmarks, these techniques are not widely used in MLP-based neural representations.
Revised: However, despite their remarkable success in deep learning benchmarks, these techniques are not widely used in MLP-based neural representations.

Original: Here we draw motivation from the work of (Salimans & Kingma, 2016;
Revised: Here we draw motivation from the work of (Salimans & Kingma, 2016;

Original: Wang et al., 2021a)
Revised: Wang et al., 2021a)

Original: and investigate a simple yet remarkably effective re-parameterization of weight vectors in MLP networks, coined as random weight factorization , which provides a generalization of Weight Normalization and demonstrates significant performance gains.
Revised: and investigate a simple yet remarkably effective re-parameterization of weight vectors in MLP networks, coined as random weight factorization , which provides a generalization of Weight Normalization and demonstrates significant performance gains.

Original: Our main contributions are summarized as
Revised: Our main contributions are summarized as

Original: • We show that random weight factorization alters the loss landscape of a neural representation in a way that can drastically reduce the distance between different parameter configurations, and effectively assigns a self-adaptive learning rate to each neuron in the network.
Revised: • We show that random weight factorization alters the loss landscape of a neural representation in a way that can drastically reduce the distance between different parameter configurations, and effectively assigns a self-adaptive learning rate to each neuron in the network.

Original: • We empirically illustrate that random weight factorization can effectively mitigate spectral bias, as well as enable coordinate-based MLP networks to escape from poor intializations and find better local minima.
Revised: • We empirically illustrate that random weight factorization can effectively mitigate spectral bias, as well as enable coordinate-based MLP networks to escape from poor intializations and find better local minima.

Original: • We demonstrate that random weight factorization can be used as a simple drop-in enhancement to conventional linear layers, and yield consistent and robust improvements across a wide range of tasks in computer vision, graphics and scientific computing.
Revised: • We demonstrate that random weight factorization can be used as a simple drop-in enhancement to conventional linear layers, and yield consistent and robust improvements across a wide range of tasks in computer vision, graphics and scientific computing.

Original:  W EIGHT F ACTORIZATION
Revised:  W EIGHT F ACTORIZATION

Original: Let x ∈ R
Revised: Let x ∈ R

Original: d be the input, g (0) ( x ) = x and d 0 = d .
Revised: d be the input, g (0) ( x ) = x and d 0 = d .

Original: We consider a standard multi-layer perceptron (MLP)
Revised: We consider a standard multi-layer perceptron (MLP)

Original: f θ ( x ) recursively defined by
Revised: f θ ( x ) recursively defined by

Original: [Equation]
Revised: [Equation]

Original: with a final layer
Revised: with a final layer

Original: [Equation]
Revised: [Equation]

Original: where W ( l ) ∈ R d l × d l − 1 is the weight matrix in l -th layer and σ is an element-wise activation function.
Revised: where W ( l ) ∈ R d l × d l − 1 is the weight matrix in l -th layer and σ is an element-wise activation function.

Original: Here, θ = (cid:0) W (1) , b (1) , . . .
Revised: Here, θ = (cid:0) W (1) , b (1) , . . .

Original: , W ( L +1) , b ( L +1) (cid:1) represents all trainable parameters in the network.
Revised: , W ( L +1) , b ( L +1) (cid:1) represents all trainable parameters in the network.

Original: MLPs are commonly trained by minimizing an appropriate loss function L ( θ ) via gradient descent.
Revised: MLPs are commonly trained by minimizing an appropriate loss function L ( θ ) via gradient descent.

Original: To improve convergence, we propose to factorize the weight parameters associated with each neuron in the network as follows
Revised: To improve convergence, we propose to factorize the weight parameters associated with each neuron in the network as follows

Original: [Equation]
Revised: [Equation]

Original: where w ( k,l ) ∈ R d l − 1 is a weight vector representing the k -th row of the weight matrix W ( l ) , s ( k,l ) ∈ R is a trainable scale factor assigned to each individual neuron, and v ( k,l ) ∈ R
Revised: where w ( k,l ) ∈ R d l − 1 is a weight vector representing the k -th row of the weight matrix W ( l ) , s ( k,l ) ∈ R is a trainable scale factor assigned to each individual neuron, and v ( k,l ) ∈ R

Original: d l − 1 .
Revised: d l − 1 .

Original: Consequently, the proposed weight factorization can be written by
Revised: Consequently, the proposed weight factorization can be written by

Original: [Equation]
Revised: [Equation]

Original: with s ∈ R d l .
Revised: with s ∈ R d l .

Original: 2. A G EOMETRIC P ERSPECTIVE
Revised: 2. A G EOMETRIC P ERSPECTIVE

Original: In this section, we provide a geometric motivation for the proposed weight factorization.
Revised: In this section, we provide a geometric motivation for the proposed weight factorization.

Original: To this end, we consider the simplest setting of a one-parameter loss function ℓ ( w ) .
Revised: To this end, we consider the simplest setting of a one-parameter loss function ℓ ( w ) .

Original: For this case, the weight factorization is reduced to w = s · v with two scalars s, v .
Revised: For this case, the weight factorization is reduced to w = s · v with two scalars s, v .

Original: Note that for a given w ̸
Revised: Note that for a given w ̸

Original: =there are infinitely many pairs ( s, v ) such that w = s · v .
Revised: =there are infinitely many pairs ( s, v ) such that w = s · v .

Original: The set of such pairs forms a family of hyperbolas in the sv -plane (one for each choice of signs for both s and v ).
Revised: The set of such pairs forms a family of hyperbolas in the sv -plane (one for each choice of signs for both s and v ).

Original: As such, the loss function in the sv -plane is constant along these hyperbolas.
Revised: As such, the loss function in the sv -plane is constant along these hyperbolas.

Original: Figure 1 gives a visual illustration of the difference between the original loss landscape as a function of w versus the loss landscape in the factorized sv -plane.
Revised: Figure 1 gives a visual illustration of the difference between the original loss landscape as a function of w versus the loss landscape in the factorized sv -plane.

Original: In the left panel, we plot the original loss function as well as an initial parameter point, the local minimum, and the global minimum.
Revised: In the left panel, we plot the original loss function as well as an initial parameter point, the local minimum, and the global minimum.

Original: The right panel shows how in the factorized parameter space, each of these three points corresponds to two hyperbolas in the sv -plane.
Revised: The right panel shows how in the factorized parameter space, each of these three points corresponds to two hyperbolas in the sv -plane.

Original: Note how the distance between the initialization and the global minima is reduced from the top to the bottom panel upon an appropriate choice of factorization.
Revised: Note how the distance between the initialization and the global minima is reduced from the top to the bottom panel upon an appropriate choice of factorization.

Original: The key observation is that the distance between factorizations representing the initial parameter and the global minimum become arbitrarily small in the sv -plane for larger values of s .
Revised: The key observation is that the distance between factorizations representing the initial parameter and the global minimum become arbitrarily small in the sv -plane for larger values of s .

Original: Indeed, we can prove that this holds for any general loss function in arbitrary parameter dimensions (the proof is provided in Appendix A.1).
Revised: Indeed, we can prove that this holds for any general loss function in arbitrary parameter dimensions (the proof is provided in Appendix A.1).

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose that L ( θ ) is the associated loss function of a neural network defined in equation 2.1 and equation 2.2.
Revised: Suppose that L ( θ ) is the associated loss function of a neural network defined in equation 2.1 and equation 2.2.

Original: For a given θ , we define U θ as the set containing all possible weight factorizations
Revised: For a given θ , we define U θ as the set containing all possible weight factorizations

Original: [Equation]
Revised: [Equation]

Original: Then for any θ , θ ′ , we have dist
Revised: Then for any θ , θ ′ , we have dist

Original: [Equation]
Revised: [Equation]

Original: 2. S ELF - ADAPTIVE LEARNING RATE FOR EACH NEURON
Revised: 2. S ELF - ADAPTIVE LEARNING RATE FOR EACH NEURON

Original: A different way to examine the effect of the proposed weight factorization is by studying its associated gradient updates.
Revised: A different way to examine the effect of the proposed weight factorization is by studying its associated gradient updates.

Original: Recall that a standard gradient descent update with a learning rate η takes the form
Revised: Recall that a standard gradient descent update with a learning rate η takes the form

Original: [Equation]
Revised: [Equation]

Original: The following theorem derives the corresponding gradient descent update expressed in the original parameter space for models using the proposed weight factorization.
Revised: The following theorem derives the corresponding gradient descent update expressed in the original parameter space for models using the proposed weight factorization.

Original: Theorem 2.
Revised: Theorem 2.

Original: Under the weight factorization of equation 2.3, the gradient descent update is given by
Revised: Under the weight factorization of equation 2.3, the gradient descent update is given by

Original: [Equation]
Revised: [Equation]

Original: for l = 1 , 2 , . . .
Revised: for l = 1 , 2 , . . .

Original: , L + 1 and k = 1 , 2 , . . .
Revised: , L + 1 and k = 1 , 2 , . . .

Original: , d l .
Revised: , d l .

Original: The proof is provided in Appendix A.2.
Revised: The proof is provided in Appendix A.2.

Original: By comparing equation 2.7 and equation 2.8, we observe that the weight factorization w = s · v re-scales the learning rate of w by a factor of ( s 2 + ∥ v ∥ 22 ) .
Revised: By comparing equation 2.7 and equation 2.8, we observe that the weight factorization w = s · v re-scales the learning rate of w by a factor of ( s 2 + ∥ v ∥ 22 ) .

Original: Since s , v are trainable parameters, this analysis suggests that this weight factorization effectively assigns a self-adaptive learning rate to each neuron in the network.
Revised: Since s , v are trainable parameters, this analysis suggests that this weight factorization effectively assigns a self-adaptive learning rate to each neuron in the network.

Revised: Similar analysis has been explored in the context of deep linear networks , suggesting that the weight factorization can significantly accelerate the optimization .

Original: In the following sections, we will demonstrate that the proposed weight factorization (under an appropriate initialization of the scale factors), not only helps with mitigating spectral bias (Rahaman et al., 2019; Bietti & Mairal, 2019; Tancik et al., 2020; Wang et al., 2021c), but also allows networks to quickly move away from a poor initialization and reach better local minima faster.
Revised: In the following sections, we will demonstrate that the proposed weight factorization (under an appropriate initialization of the scale factors), not only helps with mitigating spectral bias (Rahaman et al., 2019; Bietti & Mairal, 2019; Tancik et al., 2020; Wang et al., 2021c), but also allows networks to quickly move away from a poor initialization and reach better local minima faster.

Original: 2. R ELATION TO EXISTING WORKS
Revised: 2. R ELATION TO EXISTING WORKS

Original: The proposed weight factorization is largely motivated by weight normalization (Salimans &
Revised: The proposed weight factorization is largely motivated by weight normalization (Salimans &

Original: Kingma, 2016), which decouples the norm and the directions of the weights associated with each neuron as
Revised: Kingma, 2016), which decouples the norm and the directions of the weights associated with each neuron as

Original: [Equation]
Revised: [Equation]

Original: where g = ∥ w ∥ , and gradient descent updates are applied directly to the new parameters v , g .
Revised: where g = ∥ w ∥ , and gradient descent updates are applied directly to the new parameters v , g .

Original: Indeed, this can be viewed as a special case of the proposed weight factorization by setting s =
Revised: Indeed, this can be viewed as a special case of the proposed weight factorization by setting s =

Original: ∥ w ∥ in equation 2.3.
Revised: ∥ w ∥ in equation 2.3.

Original: In contrast to weight normalization, our weight factorization scheme allows for more flexibility in the choice of the scale factors s .
Revised: In contrast to weight normalization, our weight factorization scheme allows for more flexibility in the choice of the scale factors s .

Original: We note that SIREN networks (Sitzmann et al., 2020) also employ a special weight factorization for each hidden layer weight matrix,
Revised: We note that SIREN networks (Sitzmann et al., 2020) also employ a special weight factorization for each hidden layer weight matrix,

Original: [Equation]
Revised: [Equation]

Original: where the scale factor ω 0 ∈ R is a user-defined hyper-parameter.
Revised: where the scale factor ω 0 ∈ R is a user-defined hyper-parameter.

Original: Although the authors attribute the success of SIREN to the periodic activation functions in conjunction with a tailored initialization scheme, here we will demonstrate that the specific choice of ω 0 is the most crucial element in
Revised: Although the authors attribute the success of SIREN to the periodic activation functions in conjunction with a tailored initialization scheme, here we will demonstrate that the specific choice of ω 0 is the most crucial element in

Original: SIREN’s performance, see Appendix G, H for more details.
Revised: SIREN’s performance, see Appendix G, H for more details.

Original: It is worth pointing out that the proposed weight factorization also bears some resemblance to the adaptive activation functions introduced in (Jagtap et al., 2020), which modifies the activation of each neuron by introducing an additional trainable parameter a as
Revised: It is worth pointing out that the proposed weight factorization also bears some resemblance to the adaptive activation functions introduced in (Jagtap et al., 2020), which modifies the activation of each neuron by introducing an additional trainable parameter a as

Original: [Equation]
Revised: [Equation]

Original: These adaptive activations aim to help networks learn sharp gradients and transitions of the target functions.

Original: In practice, the scale factor is generally initialized as a = 1 , yielding a trivial weight factorization.
Revised: In practice, the scale factor is generally initialized as a = 1 , yielding a trivial weight factorization.

Original: As illustrated in the next section, this is fundamentally different from our approach as we initialize the scale factors s by a random distribution and re-parameterize the weight matrix accordingly.
Revised: As illustrated in the next section, this is fundamentally different from our approach as we initialize the scale factors s by a random distribution and re-parameterize the weight matrix accordingly.

Original: In Section 4 we demonstrate that, by initializing s using an appropriate distribution, we can consistently outperform both weight normalization, SIREN, and adaptive activations across a broad range of supervised and self-supervised learning tasks.
Revised: In Section 4 we demonstrate that, by initializing s using an appropriate distribution, we can consistently outperform both weight normalization, SIREN, and adaptive activations across a broad range of supervised and self-supervised learning tasks.

Original:  R ANDOM WEIGHT F ACTORIZATION IN PRACTICE
Revised:  R ANDOM WEIGHT F ACTORIZATION IN PRACTICE

Original: Here we illustrate the use of weight factorization through the lens of a simple regression task.
Revised: Here we illustrate the use of weight factorization through the lens of a simple regression task.

Original: Specifically, we consider a smooth scalar-valued function f sampled from a Gaussian random field using a square exponential kernel with a length scale of l = 0 .
Revised: Specifically, we consider a smooth scalar-valued function f sampled from a Gaussian random field using a square exponential kernel with a length scale of l = 0 .

Original: This generates a data-set of N = 256 observation pairs { x i , f ( x ) i } Ni =1 , where { x i } Ni =1 lie on a uniform grid in [0 , 1] .
Revised: This generates a data-set of N = 256 observation pairs { x i , f ( x ) i } Ni =1 , where { x i } Ni =1 lie on a uniform grid in [0 , 1] .

Original: The goal is to train a network f θ to learn f by minimizing the mean square error loss
Revised: The goal is to train a network f θ to learn f by minimizing the mean square error loss

Original: L ( θ ) = 1 /N (cid:80) Ni =1 | f θ ( x i ) − f ( x i ) | 2 .
Revised: L ( θ ) = 1 /N (cid:80) N i =1 | f θ ( x i ) − f ( x i ) | 2 .

Original: The proposed random weight factorization is applied as follows.
Revised: The proposed random weight factorization is applied as follows.

Original: We first initialize the parameters of an MLP network via the Glorot scheme (Glorot & Bengio, 2010).
Revised: We first initialize the parameters of an MLP network via the Glorot scheme (Glorot & Bengio, 2010).

Original: Then, for every weight matrix
Revised: Then, for every weight matrix

Original: W , we proceed by initializing a scale vector exp( s ) where s is sampled from a multivariate normal distribution N ( µ , σ I) .
Revised: W , we proceed by initializing a scale vector exp( s ) where s is sampled from a multivariate normal distribution N ( µ , σ I) .

Original: Finally, every weight matrix is factorized by the associated scale factor as W = diag(exp( s )) · V at initialization.
Revised: Then every weight matrix is factorized by the associated scale factor as W = diag(exp( s )) · V at initialization.

Original:  We train this network by gradient descent on the new parameters s , V directly.
Revised: Finally, We train this network by gradient descent on the new parameters s , V directly.

Original: This procedure is summarized in Appendix B, along with a simple JAX
Revised: This procedure is summarized in Appendix B, along with a simple JAX

Original: Flax implementation (Heek et al., 2020) in Appendix C.
Revised: Flax implementation (Heek et al., 2020) in Appendix C.

Revised: The use of exponential parameterization is motivated by Weight Normalization (Salimans & Kingma, 2016) to strictly avoid zeros or very small values in the scale factors and allow them to span a wide range of different magnitudes.

Revised: An ablation study in Appendix D further validates its benefit.

Original: In Figure 2, we train networks (3 layers, 128 neurons per layer, ReLU activations) to learn the target function using: (a) a conventional MLP, (b) an MLP with adaptive activations (AA) (Jagtap et al., 2020), (c) an MLP with weight normalization (WN) (Salimans & Kingma, 2016), and (d) an MLP with the proposed random weight factorization scheme (RWF).
Revised: In Figure 2, we train networks (3 layers, 128 neurons per layer, ReLU activations) to learn the target function using: (a) a conventional MLP, (b) an MLP with adaptive activations (AA) (Jagtap et al., 2020), (c) an MLP with weight normalization (WN) (Salimans & Kingma, 2016), and (d) an MLP with the proposed random weight factorization scheme (RWF).

Original: Evidently, RWF yields the best predictive accuracy and loss convergence.
Revised: Evidently, RWF yields the best predictive accuracy and loss convergence.

Original: Moreover, we plot the relative change of the weights in the original (unfactorized) parameter space during training in the bottom middle panel. 
Revised: Moreover, we plot the relative change of the weights in the original (unfactorized) parameter space during training in the bottom middle panel. We observe that 0 − Plain 0 .0 .1 .

Revised: −

Original: {+AA 0 .0 .1 . 
+}
Revised: {+AA 0 .0 .1 . 
+}

Original: We observe that{+− WN 0 .0 .1 . 
 −+} RWFleads to the largest weight change during training, thereby enabling the network to find better local minima further away from its initialization.
Revised: {+− WN 0 .0 .1 . 
 −+} RWF

Revised: [Figure]

Revised: RWF leads to the largest weight change during training, thereby enabling the network to find better local minima further away from its initialization.

Original: To further emphasize the benefit of weight factorization, we compute the eigenvalues of the resulting empirical Neural Tangent Kernel (NTK)
Revised: To further emphasize the benefit of weight factorization, we compute the eigenvalues of the resulting empirical Neural Tangent Kernel (NTK)

Original: (Jacot et al., 2018)
Revised: (Jacot et al., 2018)

Original: [Equation]
Revised: K θ =

Original: [Figure] AA [Figure] at the last step of training and visualize them in the bottom right panel.
Revised: (cid:68) ∂f θ ∂ θ ( x i ) , ∂f θ ∂ θ ( x j ) (cid:69) ij , at the last step of training and visualize them in the bottom right panel.

Original: Notice how RWF exhibits a flatter NTK spectrum and slower eigenvalue decay than the other methods, indicating betterconditioned training dynamics and less severe spectral bias, see (Rahaman et al., 2019; Bietti &
Revised: As demonstrated in Tancik et al. (2020) and Wang et al. (2021c), the eigenvalues of the Neural Tangent Kernel (NTK) characterize the convergence rate of the corresponding kernel eigenvectors. Specifically, components of the target function that are aligned with kernel eigenvectors corresponding to larger eigenvalues will be learned faster.

Original: Mairal, 2019; Tancik et al., 2020; Wang et al., 2021c) for more details.
Revised: Therefore, the rapid decay of the NTK eigenvalues implies extremely slow convergence to high frequency components of the target function, a pathology commonly referred to as “spectral bias”. Notice how RWF exhibits a flatter NTK spectrum and slower eigenvalue decay than the other methods, indicating better-conditioned training dynamics and less severe spectral bias.

Original: To explore the robustness of the proposed RWF, we conduct a systematic study on the effect of µ and σ in the initialization of the scale factor s .
Revised: To explore the robustness of the proposed RWF, we conduct a systematic study on the effect of µ and σ in the initialization of the scale factor s .

Original: The results suggest that the choice of µ, σ plays an important role.
Revised: The results suggest that the choice of µ, σ plays an important role.

Original: Specifically, too small µ, σ values may lead to performance that is similar to a conventional MLP, while setting µ, σ too large can result in an unstable training process.
Revised: Specifically, too small µ, σ values may lead to performance that is similar to a conventional MLP, while setting µ, σ too large can result in an unstable training process.

Original: We empirically find that µ = 1 , σ = 0 .
Revised: We empirically find that µ = 1 , σ = 0 .

Original: 1 consistently improves the loss convergence and model accuracy for the vast majority of tasks considered in this work.
Revised: 1 consistently improves the loss convergence and model accuracy for the vast majority of tasks considered in this work.

Original: Additional details are presented in Appendix D.
Revised: Additional details are presented in Appendix D.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we demonstrate the effectiveness and robustness of random weight factorization for training continuous neural representations across a range of tasks in computer vision, graphics, and scientific computing.
Revised: In this section, we demonstrate the effectiveness and robustness of random weight factorization for training continuous neural representations across a range of tasks in computer vision, graphics, and scientific computing.

Original: More precisely, we compare the performance of plain MLPs, MLPs with adaptive activations (AA) (Jagtap et al., 2020), weight normalization (WN) (Salimans & Kingma, 2016), and the proposed random weight factorization (RWF).
Revised: More precisely, we compare the performance of plain MLPs, MLPs with adaptive activations (AA) (Jagtap et al., 2020), weight normalization (WN) (Salimans & Kingma, 2016), and the proposed random weight factorization (RWF).

Original: The comparison is performed over a collection of MLP architectures, including conventional MLPs, SIREN (Sitzmann et al., 2020), modified MLPs (Wang et al., 2021b), as well as MLPs with positional encodings (Mildenhall et al., 2020) and Gaussian Fourier features (Tancik et al., 2020).
Revised: The comparison is performed over a collection of MLP architectures, including conventional MLPs, SIREN (Sitzmann et al., 2020), modified MLPs (Wang et al., 2021b), as well as MLPs with positional encodings (Mildenhall et al., 2020) and Gaussian Fourier features (Tancik et al., 2020).

Revised: In addition, for the Image Regression and Computed Tomography benchmarks, we perform a comprehensive hyper-parameter sweep over different random seeds, learning rates, activation functions, scale hyper-parameters, as well as MLPs with different depths and widths.

Original: The hyper-parameters of our experimentsalong with the computational cost associated with each experiment are presented in Appendix E and Appendix F, respectively.
Revised: The hyper-parameters of our experiments are presented in Appendix E. Table 1 summarizes the results obtained for each benchmark, corresponding to the optimal input mapping and hyper-parameter setup.

Original: Notice that the computational overhead of our method is marginal, and RWF can be therefore considered as a drop-in enhancement to any architecture that uses linear layers.
Revised: Overall, the proposed random weight factorization yields the best performance, significantly outperforming the other competing approaches. It is also worth noting that the proposed moderately increases the model sizes by O ( LD ) , where L denotes the number of layers and D the width of the network. Besides, the computational overhead

Original:  Table 1 summarizes the results obtained for each benchmark, corresponding to the optimal input mapping and network architecture.
Revised: [Table] of our method is marginal, which is further clarified by Table 7 where we report the computational cost of each benchmark.

Original: Overall, RWF consistently achieves the best performance across tasks and architectures.
Revised: Therefore, RWF can be therefore considered as a drop-in enhancement to any architecture that uses linear layers.

Original: All code and data will be made publicly available.
Revised: All code and data will be made publicly available.

Original: A summary of each benchmark study is presented below, with more details provided in Appendix.
Revised: A summary of each benchmark study is presented below, with more details provided in Appendix.

Original: 4. 2D IMAGE REGRESSION
Revised: 4. 2D IMAGE REGRESSION

Original: We train coordinate-based MLPs to learn a map from 2D input pixel coordinates to the corresponding
Revised: We train coordinate-based MLPs to learn a map from 2D input pixel coordinates to the corresponding

Original: RGB values of an image, using the benchmarks put forth in (Tancik et al., 2020).
Revised: RGB values of an image, using the benchmarks put forth in (Tancik et al., 2020).

Original: We conduct experiments using two data-sets: Natural and Text , each containing 16 images.
Revised: We conduct experiments using two data-sets: Natural and Text , each containing 16 images.

Original: The Natural data- [Table] set is constructed by taking center crops of randomly sampled images from the Div2K data-set
Revised: The Natural dataset is constructed by taking center crops of randomly sampled images from the Div2K data-set

Original: (Agustsson & Timofte
Revised: (Agustsson & Timofte, 2017).

Original: The Text set constructed by placing random text with random font sizes and colors on a white background.
Revised: The Text data-set is constructed by placing random text with random font sizes and colors on a white background.

Original: The training data is obtained by downsampling  test image by a factor of 2.
Revised: The training data is obtained by downsampling each test image by a factor of 2.

Original: We compare the resulting peak signal-to-noise ratio (PSNR) in the full resolution test images, obtained with different MLP architectures using different input mappings and weight parametrizations.
Revised: We compare the resulting peak signal-to-noise ratio (PSNR) in the full resolution test images, obtained with different MLP architectures using different input mappings and weight parametrizations.

Original: 4. 3D SHAPE REPRESENTATION
Revised: 4. 3D SHAPE REPRESENTATION

Original: This task follows the original problem setup in (Tancik et al., 2020).
Revised: This task follows the original problem setup in (Tancik et al., 2020).

Original: The goal is to learn an implicit representation of a 3D shape using Occupancy networks (Mescheder et al., 2019), which take spatial coordinates as inputs and predict 0 for points outside a given shape and 1 for points inside the shape.
Revised: The goal is to learn an implicit representation of a 3D shape using Occupancy networks (Mescheder et al., 2019), which take spatial coordinates as inputs and predict 0 for points outside a given shape and 1 for points inside the shape.

Original: We use two complex triangle meshes commonly used in computer graphics: Dragon and Armadillo .
Revised: We use two complex triangle meshes commonly used in computer graphics: Dragon and Armadillo .

Original: The training data is generated by randomly sampling points inside a bounding box and calculating their labels using the ground truth mesh.
Revised: The training data is generated by randomly sampling points inside a bounding box and calculating their labels using the ground truth mesh.

Original: We evaluate the trained model performance using the
Revised: We evaluate the trained model performance using the

Original: Intersection over Union (IoU) metric on a set of points randomly sampled near the mesh surface to better highlight the different mappings’ abilities to resolve fine details.
Revised: Intersection over Union (IoU) metric on a set of points randomly sampled near the mesh surface to better highlight the different mappings’ abilities to resolve fine details.

Original: 4. 2D COMPUTED TOMOGRAPHY (CT)
Revised: 4. 2D COMPUTED TOMOGRAPHY (CT)

Original: This task follows the original problem setup in (Tancik et al., 2020).
Revised: This task follows the original problem setup in (Tancik et al., 2020).

Original: We train an MLP to learn a map from 2D pixel coordinates to a corresponding volume density at those locations.
Revised: We train an MLP to learn a map from 2D pixel coordinates to a corresponding volume density at those locations.

Original: Two datasets are considered: procedurally-generated Shepp-Logan phantoms (Shepp & Logan, 1974) and 2D brain images from the ATLAS data-set (Liew et al., 2018).
Revised: Two datasets are considered: procedurally-generated Shepp-Logan phantoms (Shepp & Logan, 1974) and 2D brain images from the ATLAS data-set (Liew et al., 2018).

Original: Different from the previous tasks, we observe integral projections of a density field instead of direct measurements.
Revised: Different from the previous tasks, we observe integral projections of a density field instead of direct measurements.

Original: The network is trained in an indirect supervised fashion by minimizing a loss between a sparse set of ground-truth integral projections and integral projections computed from the network’s output.
Revised: The network is trained in an indirect supervised fashion by minimizing a loss between a sparse set of ground-truth integral projections and integral projections computed from the network’s output.

Original: We use PSNR to quantify the performance of the trained MLPs with different input mappings and different weight parametrizations.
Revised: We use PSNR to quantify the performance of the trained MLPs with different input mappings and different weight parametrizations.

Original: 4. 3D INVERSE RENDERING FOR VIEW SYNTHESIS
Revised: 4. 3D INVERSE RENDERING FOR VIEW SYNTHESIS

Original: This task follows the original problem setup in (Tancik et al., 2020).
Revised: This task follows the original problem setup in (Tancik et al., 2020).

Original: We aim to learn an implicit representation of a 3D scene from 2D photographs using Neural Radiance Field (NeRF) (Mildenhall et al., 2020), which is a coordinate-based MLP that takes a 3D location as input and outputs a color and volume density.
Revised: We aim to learn an implicit representation of a 3D scene from 2D photographs using Neural Radiance Field (NeRF) (Mildenhall et al., 2020), which is a coordinate-based MLP that takes a 3D location as input and outputs a color and volume density.

Original: The network is trained by minimizing a rendering loss between the set of 2D image observations and the same rendered views from the predicted scene representation.
Revised: The network is trained by minimizing a rendering loss between the set of 2D image observations and the same rendered views from the predicted scene representation.

Original: In our experiments, we consider a down-sampled NeRF Lego data-set and use a simplified version of the method described in (Mildenhall et al., 2020), where we remove hierarchical sampling and view dependence.
Revised: In our experiments, we consider a down-sampled NeRF Lego data-set and use a simplified version of the method described in (Mildenhall et al., 2020), where we remove hierarchical sampling and view dependence.

Original: We compare the PSNR of the trained MLPs with different weight parametrizations.
Revised: We compare the PSNR of the trained MLPs with different weight parametrizations over 10 random seeds.

Original: 4. S OLVING PARTIAL DIFFERENTIAL EQUATIONS (PDE S )
Revised: 4. S OLVING PARTIAL DIFFERENTIAL EQUATIONS (PDE S )

Original: Our goal is to solve partial differential equations (PDEs) using physics-informed neural network (PINNs)
Revised: Our goal is to solve partial differential equations (PDEs) using physics-informed neural network (PINNs)

Original: (Raissi et al., 2019), which take the coordinates of a spatio-temporal domain as inputs and predict the corresponding target solution function.
Revised: (Raissi et al., 2019), which take the coordinates of a spatio-temporal domain as inputs and predict the corresponding target solution function.

Original: PINNs are trained in a self-supervised fashion by minimizing a composite loss function for fitting given initial and boundary conditions, as well as satisfying the underlying PDE constraints.
Revised: PINNs are trained in a self-supervised fashion by minimizing a composite loss function for fitting given initial and boundary conditions, as well as satisfying the underlying PDE constraints.

Original: We consider two benchmarks, an advection equation modeling the transport of a scalar field, and the Navier-Stokes equation modeling the motion of an incompressible fluid in a square cavity.
Revised: We consider two benchmarks, an advection equation modeling the transport of a scalar field, and the Navier-Stokes equation modeling the motion of an incompressible fluid in a square cavity.

Revised: We compare the resulting relative L 2 errors of each parameterizations over 10 random seeds.

Original: Detailed descriptions and implementations of each problem are provided below and in the Appendix K.
Revised: Detailed descriptions and implementations of each problem are provided below and in the Appendix K.

Original: Figure 3 and Figure 4 present the ground truth against the predicted solutions obtained by training
Revised: Figure 3 and Figure 4 present the ground truth against the predicted solutions obtained by training

Original: PINNs with different weight parameterizations.
Revised: PINNs with different weight parameterizations.

Original: It can be observed that the predictions obtained by
Revised: It can be observed that the predictions obtained by

Original: RWF are in excellent agreement with the ground truth, while the other three parameterizations result in poor or even non-physical approximations.
Revised: RWF are in excellent agreement with the ground truth, while the other three parameterizations result in poor or even non-physical approximations.

Original: The rapid decrease in the test error further validates the benefit of our method.
Revised: The rapid decrease in the test error further validates the benefit of our method.

Original: We attribute these significant performance improvements to the fact that
Revised: We attribute these significant performance improvements to the fact that

Original: PINN models, due to their self-supervised nature, often suffer from poor initializations.
Revised: PINN models, due to their self-supervised nature, often suffer from poor initializations.

Original: Evidently,
Revised: Evidently,

Original: RWF can precisely mitigate this by being able to reach better local minima that are located further away from the model initialization neighborhood.
Revised: RWF can precisely mitigate this by being able to reach better local minima that are located further away from the model initialization neighborhood.

Original: Advection equation: The first example is 1D advection equation, a linear hyperbolic equation commonly used to model transport phenomena
Revised: Advection equation: The first example is 1D advection equation, a linear hyperbolic equation commonly used to model transport phenomena

Original: [Equation]
Revised: [Equation]

Original: with periodic boundary conditions.
Revised: with periodic boundary conditions.

Original: This example has been studied in (Krishnapriyan et al., 2021;
Revised: This example has been studied in (Krishnapriyan et al., 2021;

Original: Daw et al., 2022), exposing some of the limitations that PINNs suffer from as the transport velocity c is increased.
Revised: Daw et al., 2022), exposing some of the limitations that PINNs suffer from as the transport velocity c is increased.

Original: In our experiments, we consider c = 50 and an initial condition g ( x ) = sin( x ) .
Revised: In our experiments, we consider c = 50 and an initial condition g ( x ) = sin( x ) .

Original: Navier-Stokes equation: The second example is a classical benchmark problem in computational fluid dynamics, describing the motion of an incompressible fluid in a two-dimensional lid-driven cavity.
Revised: Navier-Stokes equation: The second example is a classical benchmark problem in computational fluid dynamics, describing the motion of an incompressible fluid in a two-dimensional lid-driven cavity.

Original: The system is governed by the Navier–Stokes equations written in a non-dimensional form
Revised: The system is governed by the Navier–Stokes equations written in a non-dimensional form

Original: [Equation]
Revised: [Equation]

Original: where u = ( u, v ) denotes the velocity in x and y directions, respectively, and p is the scalar pressure field.
Revised: where u = ( u, v ) denotes the velocity in x and y directions, respectively, and p is the scalar pressure field.

Original: We assume u
Revised: We assume u

Original: = (1 , 0) on the top lid of the cavity, and a non-slip boundary condition on the other three walls.
Revised: = (1 , 0) on the top lid of the cavity, and a non-slip boundary condition on the other three walls.

Original: All experiments are performed with a Reynolds number of Re = 1 , 000 .
Revised: All experiments are performed with a Reynolds number of Re = 1 , 000 .

Original: 4. L EARNING O PERATORS
Revised: 4. L EARNING O PERATORS

Original: In this task, we focus on learning the solution operators of parametric PDEs.
Revised: In this task, we focus on learning the solution operators of parametric PDEs.

Original: To describe the problem setup in general, consider a parametric PDE of the following form
Revised: To describe the problem setup in general, consider a parametric PDE of the following form

Original: [Equation]
Revised: [Equation]

Original: where N : A × U → V is a linear or nonlinear differential operator between infinite-dimensional function spaces.
Revised: where N : A × U → V is a linear or nonlinear differential operator between infinite-dimensional function spaces.

Original: Moreover, a ∈ A denotes the PDE parameters, and u ∈ U is the corresponding unknown solutions of the PDE system.
Revised: Moreover, a ∈ A denotes the PDE parameters, and u ∈ U is the corresponding unknown solutions of the PDE system.

Original: The solution operator G :
Revised: The solution operator G :

Original: A → U is given by
Revised: A → U is given by

Original: [Equation]
Revised: [Equation]

Original: In our experiments, we consider three benchmarks: Diffusion-reaction, Darcy flow and the Burgers’ equation.
Revised: In our experiments, we consider three benchmarks: Diffusion-reaction, Darcy flow and the Burgers’ equation.

Original: Detailed descriptions of each problem setup are shown below.
Revised: Detailed descriptions of each problem setup are shown below.

Original: As shown in Figure 5, we plot the training losses of each model with different weight parameterizations.
Revised: As shown in Figure 5, we plot the training losses of each model with different weight parameterizations.

Original: One can see that random weight factorization yields the best loss convergence for every example, indicating the capability of the proposed method to accelerate the convergence of stochastic gradient descent and achieve better local minima.
Revised: One can see that random weight factorization yields the best loss convergence for every example, indicating the capability of the proposed method to accelerate the convergence of stochastic gradient descent and achieve better local minima.

Original:  Diffusion-reaction: Our first example involves a nonlinear diffusion-reaction PDE with a source term a : (0 , 1) → R ,
Revised: [Figure] Diffusion-reaction: Our first example involves a nonlinear diffusion-reaction PDE with a source term a : (

Original: [Equation]
Revised: [Equation]

Original: with the zero initial and boundary conditions, where D = 0 .
Revised: with the zero initial and boundary conditions, where D = 0 .

Original: 01 is the diffusion coefficient and k = 0 .
Revised: 01 is the diffusion coefficient and k = 0 .

Original: 01 is the reaction rate.
Revised: 01 is the reaction rate.

Original: We train a Deep Operator Network (DeepONet)
Revised: We train a Deep Operator Network (DeepONet)

Original: Lu et al.
Revised: Lu et al.

Original: (2021) to learn the solution operator for mapping source terms a ( x ) to the corresponding PDE solutions u ( x ) .
Revised: (2021) to learn the solution operator for mapping source terms a ( x ) to the corresponding PDE solutions u ( x ) .

Original: This network takes a PDE parameter and a spatial-temporal coordinate as inputs, and predicts the associated PDE solution evaluated at that location.
Revised: This network takes a PDE parameter and a spatial-temporal coordinate as inputs, and predicts the associated PDE solution evaluated at that location.

Original: The model is trained in a supervised manner by minimizing a loss between the predicted PDE solutions and the available solution measurements.[Figure]
Revised: The model is trained in a supervised manner by minimizing a loss between the predicted PDE solutions and the available solution measurements.

Original: Darcy flow: The Darcy equation describes steady-state flow through a porous medium, taking the following form in two spatial dimensions
Revised: Darcy flow: The Darcy equation describes steady-state flow through a porous medium, taking the following form in two spatial dimensions

Original: [Equation]
Revised: [Equation]

Original: where a : (0 , 1) → R + is the diffusion coefficient and f : (0 , 1) → R is a forcing term.
Revised: where a : (0 , 1) → R + is the diffusion coefficient and f : (0 , 1) → R is a forcing term.

Original: This is a linear second-order elliptic PDE with numerous applications in modeling subsurface flow, porous media, elastic materials, etc.

Original: We fix f ( x, y )
Revised: We fix f ( x, y )

Original: = 1 and aim to learn a continuous representation of the solution operator G : a ( x, y )
Revised: = 1 and aim to learn a continuous representation of the solution operator G : a ( x, y )

Original: → u ( x, y ) with a DeepONet.
Revised: → u ( x, y ) with a DeepONet.

Original: Burgers’ equation: As the last example, we consider a fundamental nonlinear PDE, the onedimensional viscous Burgers’ equation.
Revised: Burgers’ equation: As the last example, we consider a fundamental nonlinear PDE, the onedimensional viscous Burgers’ equation.

Original: This equation arises in various areas of applied mathematics, such as gas dynamics, nonlinear acoustics, and fluid mechanics.

Original: It takes the form
Revised: It takes the form

Original: [Equation]
Revised: [Equation]

Original: with periodic boundary conditions and ν = 0 . 001 .
Revised: with periodic boundary conditions and ν = 0 . 001 .

Original: Our goal is to learn the solution operator from the initial condition to the associated PDE solution with a physics-informed DeepONet (Wang et al., 2021e).
Revised: Our goal is to learn the solution operator from the initial condition to the associated PDE solution with a physics-informed DeepONet (Wang et al., 2021e).

Original: Different from the first two examples, the physics-informed DeepONet is trained in a selfsupervised manner, i.e. without any paired input-output observations, except for a set of given initial or boundary conditions (see Wang et al. (2021e) for more details).
Revised: Different from the first two examples, a physics-informed DeepONet is trained in a selfsupervised manner, i.e. without any paired input-output observations, except for a set of given initial or boundary conditions (see Wang et al. (2021e) for more details).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we proposed random weight factorization , a simple and remarkably effective reparameterization of the weight matrices in neural networks.
Revised: In this work, we proposed random weight factorization , a simple and remarkably effective reparameterization of the weight matrices in neural networks.

Original: Theoretically, we show how this factorization alters the geometry of a loss landscape by assigning a self-adaptive learning rate to each neuron.
Revised: Theoretically, we show how this factorization alters the geometry of a loss landscape by assigning a self-adaptive learning rate to each neuron.

Original: Empirically, we show that our method can mitigate spectral bias in MLPs and enable networks to search for good local optima further away from their initialization.
Revised: Empirically, we show that our method can mitigate spectral bias in MLPs and enable networks to search for good local optima further away from their initialization.

Original: We validate random weight factorization using six different benchmarks ranging from image regression to learning operators, showcasing a consistent and robust improvements across various tasks in computer vision, graphics, and scientific computing.
Revised: We validate random weight factorization using six different benchmarks ranging from image regression to learning operators, showcasing a consistent and robust improvements across various tasks in computer vision, graphics, and scientific computing.

Original: These findings provide new insights into the training of continuous neural representations and open several exciting avenues for future work, including the application of our method to deep learning models beyond coordinate-based MLPs, such as convolutional networks (LeCun et al., 1998), graph networks (Scarselli et al., 2008), and Transformers (Vaswani et al., 2017).
Revised: These findings provide new insights into the training of continuous neural representations and open several exciting avenues for future work, including the application of our method to deep learning models beyond coordinate-based MLPs, such as convolutional networks (LeCun et al., 1998), graph networks (Scarselli et al., 2008), and Transformers (Vaswani et al., 2017).

Original: A P ROOFS A. P ROOF OF T HEOREM
Revised: A P ROOFS A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Note that for any pair ( s ( l ) , V ( l ) ) with diag ( s ( l ) ) · V ( l ) = W ( l ) , we have
Revised: Note that for any pair ( s ( l ) , V ( l ) ) with diag ( s ( l ) ) · V ( l ) = W ( l ) , we have

Original: [Equation]
Revised: [Equation]

Original: , as
Revised: , as

Original: [Equation]
Revised: [Equation]

Original: for l = 1 , 2 , . . .
Revised: for l = 1 , 2 , . . .

Original: , L + 1 .
Revised: , L + 1 .

Original: Then for any ϵ > 0 , there exists M > 0 such that for any ( V ( l ) , s ( l ) )
Revised: Then for any ϵ > 0 , there exists M > 0 such that for any ( V ( l ) , s ( l ) )

Original: L +1 l =1 ∈
Revised: L +1 l =1 ∈

Original: U θ ,and ∥ s ( l ) ∥ > M , we obtain ∥ V ( l ) ∥ <
Revised: U θ ,and ∥ s ( l ) ∥ > M , we obtain ∥ V ( l ) ∥ <

Original: ϵ , for l = 1 , 2 , . . .
Revised: ϵ , for l = 1 , 2 , . . .

Original: , L + 1 .
Revised: , L + 1 .

Original: We define U ∗ by
Revised: We define U ∗ by

Original: [Equation]
Revised: [Equation]

Original: Now we can choose s ( l ) such that ∥ s ( l ) ∥ > M .
Revised: Now we can choose s ( l ) such that ∥ s ( l ) ∥ > M .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can show that
Revised: Similarly, we can show that

Original: [Equation]
Revised: [Equation]

Original: Therefore, dist
Revised: Therefore, dist

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: Since ϵ is arbitrary, letting ϵ → 0 gives
Revised: Since ϵ is arbitrary, letting ϵ → 0 gives

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Suppose that f ( k,l ) denotes k -th component of f ( l ) ∈ R d l .
Revised: Suppose that f ( k,l ) denotes k -th component of f ( l ) ∈ R d l .

Original: Under the proposed weight factorization in equation 2.3, differentiating the loss function L with respect to w k,l and s ( k,l ) , respectively, yields
Revised: Under the proposed weight factorization in equation 2.3, differentiating the loss function L with respect to w k,l and s ( k,l ) , respectively, yields

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: and the update rule of v ( k,l ) and s ( k,l ) can be re-written as
Revised: and the update rule of v ( k,l ) and s ( k,l ) can be re-written as

Original: [Equation]
Revised: [Equation]

Original: Since w ( k,l ) =
Revised: Since w ( k,l ) =

Original: s ( k,l ) · v ( k,l ) , the update rule of w ( k,l ) is given by
Revised: s ( k,l ) · v ( k,l ) , the update rule of w ( k,l ) is given by

Original: [Equation]
Revised: [Equation]

Original: B A LGORITHM OF R ANDOM WEIGHT FACTORIZATION
Revised: B A LGORITHM OF R ANDOM WEIGHT FACTORIZATION

Original: recommended hyper-parameters are µ =
Revised: recommended hyper-parameters are µ =

Original: , σ
Revised: , σ

Original: C A DROP - IN ENHANCEMENT FOR LINEAR LAYERS
Revised: C A DROP - IN ENHANCEMENT FOR LINEAR LAYERS

Original: D 1D R EGRESSION A BLATION S TUDY
Revised: D 1D R EGRESSION A BLATION S TUDY

Original: We perform a systematic study on the effect of µ and σ used for initializing the scale factor s in random weight factorization .
Revised: We perform a systematic study on the effect of µ and σ used for initializing the scale factor s in random weight factorization .

Original: To this end, we train MLPs with random weight factorization initialized by different µ and σ .
Revised: To this end, we train MLPs with random weight factorization initialized by different µ and σ .

Original: Each model (3 layers, 128 channels, ReLU activations) is trained via a full-batch gradient descent for 10 5 iterations using the Adam optimizer (Kingma & Ba, 2014) with a starting learning rate of 10 − 3 followed by an exponential decay by a factor of 0.9 in every 5 , 000 steps.
Revised: Each model (3 layers, 128 channels, ReLU activations) is trained via a full-batch gradient descent for 10 5 iterations using the Adam optimizer (Kingma & Ba, 2014) with a starting learning rate of 10 − 3 followed by an exponential decay by a factor of 0.9 in every 5 , 000 steps.

Original: The resulting relative L 2 errors  are visualized in Figure 6.
Revised: The resulting relative L 2 errors over 10 random seeds are visualized in Figure 6.

Original: We observe that models initialized with a small µ achieve similar performance to our baseline, while large µ and σ values can lead to an unstable training process and poor predictive accuracy. In particular, µ = 1 and σ = 0 .
Revised: We observe that all the curves are roughly basin-shaped, indicating a ”sweet spot” for the random weight factorization, i.e., µ ∈ [1 , 2] .

Original: 1 yield the best results, and thus we will  use this as the default hyper-parameter of random weight factorization in the majority of the benchmarks presented here (see Table 4).
Revised: Thus we will mainly use these as the default hyper-parameter of random weight factorization in the majority of the benchmarks presented here (see Table 5).

Revised: Furthermore, we perform an ablation study on the different distributions (Normal, LogNormal,Uniform, LogUniform) initializing the scale factors and report the average and standard deviation of mean square errors over 10 random seeds.

Revised: For fair comparison, we manually tune each distribution such that the sampled values are roughly in the same range.

Revised: We see that LogNormal is slightly better than the other distributions while overall there is no significant discrepancies in terms of the accuracy for using different distributions.

Original: [Figure]
Revised: [Figure]

Original: Baseline σ =0.

Original: σ =0. σ =0. σ =1.
Revised: [Figure] [Table]

Original: E H YPER - PARAMETERS
Revised: E H YPER - PARAMETERS

Original: The following tables summarizes the hyper-parameters of the different networks architectures employed in each benchmark (Table 2), their associated learning rate schedules (Table 3)
Revised: The following tables summarizes the hyper-parameters of the different networks architectures employed in each benchmark (Table 3), their associated learning rate schedules (Table 4)

Original: [Table]
Revised: [Table]

Original: and the corresponding random weight factorization settings (Table 4).
Revised: and the corresponding random weight factorization settings (Table 5).

Original: [Table]
Revised: [Table]

Original: GELU

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: F C OMPUTATIONAL C OSTS
Revised: F C OMPUTATIONAL C OSTS

Original: Table 5 presents the computational cost in terms of training iterations per second for the networksnetworks employed in each benchmark.
Revised: Table 7 presents the computational cost in terms of training iterations per second for the networks employed in each benchmark.

Original: All timings are reported on a single NVIDIA RTX 
Revised: All timings are reported on a single NVIDIA RTX A6000 GPU. Task [Table]

Original: A
Revised: Shape Representation Dragon 12. [Table] [Table]

Original: GPU.
Revised: [Table] [Table] [Table] Shepp 30.

Original: [Table]
Revised: [Table]

Original: training
Revised: ATLAS 29. [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: G 2D I MAGE REGRESSION
Revised: G 2D I MAGE REGRESSION

Original: As mentioned in Section 4.1, we use two image data-sets: Natural and Natural .
Revised: As mentioned in Section 4.1, we use two image data-sets: Natural and Natural .

Original: All the test images have a 512 × 512 resolution while the training data has a 256 × 256 resolution.
Revised: All the test images have a 512 × 512 resolution while the training data has a 256 × 256 resolution.

Revised: For each data-set, we train models with different weight parametrizations (Plain, Adaptive Activation, Weight Normalization, Random Weight Factorization) over different random seeds, learning rates, activation functions, scale hyper-parameters, as well as MLP architectures with different depths and widths.

Revised: The range of hyper-parameters is summarized in Table 6.

Revised: For random weight factorization, we initialize the scale factor s ∼ N (2 , 0 . 01) and s ∼ N (1 , 0 . 01) for the Natural and Text data-set, respectively.

Revised: Each model is trained via a full-batch gradient descent for 2,000 iterations using the

Original: For each data-set, we compare the performance of MLPs with different parameterizations (see 2.3) and the following input embeddings:
Revised: Adam optimizer (Kingma & Ba, 2014) with default settings and 200 warm-up steps. We compute the mean and standard deviations of the resulting PSNRs and visualize our results in Figure 7 and Figure 8, respectively. We reach the conclusion that the random weight factorization (RWF) consistently yields the best optimal performance than the alternatives. Furthermore, under the best hyper-parameter setup, we compare the performance of MLPs with different parameterizations (see 2.3) and the following input embeddings:

Original: No mapping: MLP with no input feature mapping.
Revised: No mapping: MLP with no input feature mapping.

Original: Positional encoding (Tancik et al., 2020): γ ( x ) = (cid:2) . .
Revised: Positional encoding (Tancik et al., 2020): γ ( x ) = (cid:2) . .

Original: , cos (cid:0) 2 πσ j/m x (cid:1) , sin (cid:0) 2 πσ j/m x (cid:1) , . . .
Revised: , cos (cid:0) 2 πσ j/m x (cid:1) , sin (cid:0) 2 πσ j/m x (cid:1) , . . .

Original: (cid:3) T for j = 0 , 1 , . . .
Revised: (cid:3) T for j = 0 , 1 , . . .

Original: , m − 1 .
Revised: , m − 1 .

Original: where the frequencies are log-linear spaced and the scale σ > 0 is a user-specified hyper-parameter.
Revised: where the frequencies are log-linear spaced and the scale σ > 0 is a user-specified hyper-parameter.

Original: Gaussian (Tancik et al., 2020): γ ( x ) =
Revised: Gaussian (Tancik et al., 2020): γ ( x ) =

Original: [cos(2 π B x ) , sin(2 π B x )] T , where B ∈ R m × d is sampled from a Gaussian distribution N (0 , σ 2 ) .
Revised: [cos(2 π B x ) , sin(2 π B x )] T , where B ∈ R m × d is sampled from a Gaussian distribution N (0 , σ 2 ) .

Original: The scale σ > 0 is a user-specified hyper-parameter.
Revised: The scale σ > 0 is a user-specified hyper-parameter.

Original: Each model (4 layers, 256 channels, ReLU activations) is trained via a full-batch gradient descent for 2,000 iterations using the Adam optimizer (Kingma & Ba, 2014) with default settings and 200 warmup steps.

Original: Particularly, the mapping scales of positional encoding and Gaussian Fourier features are the same as in (Tancik et al., 2020).

Original: For random weight factorization, we initialize the scale factor s ∼ N (2 , 0 . 01) and s ∼ N (1 , 0 . 01) for the Natural and Text data-set, respectively.

Original: The resulting test PSNR is reported in Table 6 and Table 7.
Revised: The resulting test PSNR is reported in Table 8 and Table 9.

Original: We can see that the weight factorization with Gaussian Fourier features achieves the best PSNR among all the cases.
Revised: We can see that the weight factorization with Gaussian Fourier features achieves the best PSNR among all the cases.

Original: Some visualizations are shown in Figure 7 and Figure 8.
Revised: Some visualizations are shown in Figure 9 and Figure 10.

Original: Natural data-set

Original: Plain

Original: AA

Original: WN

Original: RWF (ours)

Original: No

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Comparison with SIREN (Sitzmann et al., 2020):
Revised: Comparison with SIREN (Sitzmann et al., 2020):

Original: We find that SIREN also factorizes the weight matrix of every hidden layer as W = ω 0 × (cid:99) W with some scale factor ω 0 .
Revised: We find that SIREN also factorizes the weight matrix of every hidden layer as W = ω 0 × (cid:99) W with some scale factor ω 0 .

Original: It is indeed a special case of our approach.
Revised: It is indeed a special case of our approach.

Original: To examine its performance, we vary the scale factor w 0 and train SIREN networks under the same hyper-parameter setting, and present the test error over the Natural and Text dataset in Figure 9.
Revised: To examine its performance, we vary the scale factor w 0 and train SIREN networks under the same hyper-parameter setting, and present the test error over the Natural and Text data-set in Figure 11.

Original: It can be observed that the scale factor w 0 plays a fundamental role in the  SIREN performance.
Revised: If we take w 0 = 1 , the main difference between SIREN and our baseline is the activation function (sine vs. ReLU).

Original: If we take w 0 = 1 , then SIREN just performs similarly to our baseline (Plain MLP with no input mapping).
Revised: However, SIREN just performs similarly to our baseline (Plain MLP with no input mapping).

Original:  Therefore, we may argue that the success ofSIREN can be attributed to that simple weight factorization instead of the sine activations with the associated initialization scheme
Revised: This suggests that the periodic activation function does not play an crucial role in improving the model performance. In the same figure, we can see that the SIREN performance is mostly affected by the value of w 0 . Therefore, we may argue that the success of

Original:  W ∼ U ( − (cid:112) 6 /d, (cid:112) 6 /d ) .
Revised: SIREN can be attributed to that simple weight factorization instead of the sine activations with the associated initialization scheme W ∼ U ( − (cid:112) 6 /d, (cid:112) 6 /d ) .

Original: Nevertheless, the best PSNR that SIREN achieves is still significantly lower than the proposed random weight factorization with positional encodings or Gaussian Fourier features. 
Revised: Nevertheless, the best PSNR that SIREN achieves is still significantly lower than the proposed random weight factorization with positional encodings or Gaussian Fourier features. Scale ω P N S R

Original: {+Fitting Natural image baseline 
+}
Revised: {+Fitting Natural image baseline 
+}

Original: [Figure]
Revised: Scale ω P N S R Fitting Text image baseline

Original: H 2D C OMPUTED T OMOGRAPHY
Revised: H 2D C OMPUTED T OMOGRAPHY

Original: For this task, we use two data-sets: procedurally-generated Shepp-Logan phantoms (Shepp & Logan, 1974) and 2D brain images from the ATLAS data-set (Liew et al., 2018).
Revised: For this task, we use two data-sets: procedurally-generated Shepp-Logan phantoms (Shepp & Logan, 1974) and 2D brain images from the ATLAS data-set (Liew et al., 2018).

Original: Each data-set consist of 20 images of 512 × 512 resolution.
Revised: Each data-set consist of 20 images of 512 × 512 resolution.

Original: To generate the training data, we compute 20 and 40 synthetic integral projections at evenly-spaced angles for every image of Shepp and ATLAS data-set, respectively.
Revised: To generate the training data, we compute 20 and 40 synthetic integral projections at evenly-spaced angles for every image of Shepp and ATLAS data-set, respectively.

Original: Similar to the previous tasks, we compare the performance of MLPs with different input mappings and weight parameterizations.
Revised: Similar to the previous tasks, we first perform a hyper-parameter sweep over different random seeds, learning rates, activation functions, scale hyper-parameters, as well as MLP architectures with different depths and widths.

Original: In experiments, wetake a scale factor σ = 3 for both positional encoding and Gaussian Fourier features and initialize s ∼ N (1 , 0 . 1) for using random weight factorization.
Revised: Particularly, we initialize s ∼ N (1 , 0 . 1) for using random weight factorization.

Original: Each model(5 layers, 256 channels, ReLU activations) is trained via a full-batch gradient 
Revised: Each model is trained via a full-batch gradient descent for 2,000 iteration using the the Adam optimizer (Kingma & Ba, 2014) (Kingma & Ba, 2014) with default settings and 200 warm-up steps. Figure 12 and Figure 13 visualizes the the performance of MLPs with each weight parameterizations under different hyper-parameter settings. Furthermore, under the best hyper-parameter setup, we compare the performance of MLPs with different input mappings.

Original:  Table 8 summarizes the test PSRN over Shepp and ATLAS data-set, respectively.
Revised: In experiments, Table 10 summarizes the test PSRN over Shepp and ATLAS data-set, respectively.

Original: For different input mappings, random weight factorization yields the best PSNR, consistently outperforming other parameterizations.
Revised: For different input mappings, random weight factorization yields the best PSNR, consistently outperforming other parameterizations.

Original: Besides, we plot some model predictions corresponding to Gaussian input mapping in Figure 10 and 11.
Revised: Besides, we plot some model predictions corresponding to Gaussian input mapping in Figure 14 and 15. Shepp data-set

Original: [Table]
Revised: [Table]

Original: Comparison with SIREN (Sitzmann et al., 2020):
Revised: Comparison with SIREN (Sitzmann et al., 2020):

Original: We also test the performance of SIREN for this example.
Revised: We also test the performance of SIREN for this example.

Original: Specifically, we train SIREN network with different scale factor ω 

Revised: Specifically, we train SIREN network with different scale factor ω

Original: under the same hyper-parameter settings.
Revised: under the same hyper-parameter settings.

Original: As shown in Figure 12, SIREN using the optimal scale factor is just slightly better than our baseline (plain MLP with no input mapping), but still worse than using positional encodings or random Fourier features.
Revised: As shown in Figure 16, SIREN using the optimal scale factor is just slightly better than our baseline (plain MLP with no input mapping), but still worse than using positional encodings or random Fourier features.

Original: [Figure]
Revised: [Figure]

Original: I 3D S HAPE R EPRESENTATION
Revised: I 3D S HAPE R EPRESENTATION

Original: For this example, we consider two complex triangle meshes Dragon and Armadillo , both of which contain hundreds of thousands of vertices.
Revised: For this example, we consider two complex triangle meshes Dragon and Armadillo , both of which contain hundreds of thousands of vertices.

Original: In our experiments, every mesh is rescaled to fit inside the unit cube [0 , 1] 3 such that the centroid of the mesh is (0 . 5 , 0 . 5 , 0 . 5) .
Revised: In our experiments, every mesh is rescaled to fit inside the unit cube [0 , 1] 3 such that the centroid of the mesh is (0 . 5 , 0 . 5 , 0 . 5) .

Original: We represent each shape by MLPs with different input mappings and weight parameterizations.
Revised: We represent each shape by MLPs with different input mappings and weight parameterizations.

Original: For models with input mappings, we use the same hyper-parameters as in (Tancik et al., 2020).
Revised: For models with input mappings, we use the same hyper-parameters as in (Tancik et al., 2020).

Original: For models using random weight factorization , we initialize the scale factor using the recommended settings s ∼ N (1 , 0 . 1) .
Revised: For models using random weight factorization , we initialize the scale factor using the recommended settings s ∼ N (1 , 0 . 1) .

Original: All networks are trained by minimizing a cross-entropy loss to match the corresponding classification labels (0 for points outside the mesh, 1 for points inside).
Revised: All networks are trained by minimizing a cross-entropy loss to match the corresponding classification labels (0 for points outside the mesh, 1 for points inside).

Original: We train each model (8 layers, 128 channels, ReLU activations) via a mini-batch gradient descent for 10 4 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate 5 × 10 −and an exponential decay by a factor of 0 .
Revised: We train each model (8 layers, 128 channels, ReLU activations) via a mini-batch gradient descent for 10 4 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate 5 × 10 −and an exponential decay by a factor of 0 .

Original: 1 for every 5 , 000 steps.
Revised: 1 for every 5 , 000 steps.

Original: The batch size we use is 8192.
Revised: The batch size we use is 8192.

Original: To emphasize the learning of fine surface details, we calculate the test error on a set close to the mesh surface, which is generated by randomly choosing mesh vertices that have been perturbed by a random Gaussian vector with a standard deviation of 0.01.
Revised: To emphasize the learning of fine surface details, we calculate the test error on a set close to the mesh surface, which is generated by randomly choosing mesh vertices that have been perturbed by a random Gaussian vector with a standard deviation of 0.01.

Original: The resulting IoU scores of each model is reported in Table 9.
Revised: The resulting IoU scores of each model is reported in Table 11.

Original: One can observe consistent improvements of RWF across different input mappings and data-sets, outperforming the other parameterizations.
Revised: One can observe consistent improvements of RWF across different input mappings and data-sets, outperforming the other parameterizations.

Original: Moreover, the learned shape representations are depicted in Figure 13.
Revised: Moreover, the learned shape representations are depicted in Figure 17. Dragon data-set Plain AA

Revised: WN

Original: [Table]
Revised: RWF (ours) No mapping 0. Positional encoding 0. Gaussian 0.

Revised: Armadillo data-set

Revised: Plain

Revised: [Table]

Original: J 3D INVERSE RENDERING FOR VIEW SYNTHESIS
Revised: J 3D INVERSE RENDERING FOR VIEW SYNTHESIS

Original: For this task, we use the NeRF Lego data-set of 120 images downsampled to 400 × 400 pixel resolution.
Revised: For this task, we use the NeRF Lego data-set of 120 images downsampled to 400 × 400 pixel resolution.

Original: The data-set is split into 100 training images, 7 validation images, and 13 test images.
Revised: The data-set is split into 100 training images, 7 validation images [Table] and 13 test images.

Original: In our experiments, we only use Gaussian Fourier features of a scale σ = 10 , as it has been empirically validated to be the best input mapping in the previous tasks.
Revised: In our experiments, we only use Gaussian Fourier features of a scale σ = 10 , as it has been empirically validated to be the best input mapping in the previous tasks.

Original: We train MLPs (5 layers, 256 channels, ReLU activations) with different parameterizations  for 5 × 10 4 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and a warmup exponential decay by a factor of 0 .
Revised: We train MLPs (5 layers, 256 channels, ReLU activations) with different parameterizations andrandom seeds for 5 × 10 4 iterations over 10 random seeds. We use the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and a warmup exponential decay by a factor of 0 .

Original: 9 for every 1 , 000 steps.
Revised: 9 for every 1 , 000 steps.

Original: The batch size is 2048.
Revised: The batch size is 2048.

Original: In Figure 14, we visualize the test PSNR of each model during training.
Revised: In Figure 18, we visualize the test PSNR of each model during training.

Original: In comparison with other three parameterizations, the MLP with RWF achieves the best PSRN.
Revised: In comparison with other three parameterizations, the MLP with RWF achieves the best PSRN.

Original: Some visualizations are provided in Figure
Revised: Some visualizations are provided in Figure

Original: [Figure]
Revised: [Figure]

Original: K S OLVING PDE S
Revised: K S OLVING PDE S

Original: In this section, we present the implementation details of PINNs for solving advection and NaiverStokes equation, respectively.
Revised: In this section, we present the implementation details of PINNs for solving advection and NavierStokes equation, respectively.

Original: K. A DVECTION EQUATION 
Revised: K. A DVECTION EQUATION Recall

Original: Recall

Original: [Equation]
Revised: [Equation]

Original: with periodic boundary conditions and c = 50 .
Revised: with periodic boundary conditions and c = 50 .

Original: We represent the unknown solution u by an MLP u θ .
Revised: We represent the unknown solution u by an MLP u θ .

Original: In particular, we impose the exact periodic boundary conditions by constructing a special Fourier feature embedding of the form (Dong & Ni, 2021)
Revised: In particular, we impose the exact periodic boundary conditions by constructing a special Fourier feature embedding of the form (Dong & Ni, 2021)

Original: [Equation]
Revised: [Equation]

Original: The network can be trained by minimizing the composite loss below
Revised: The network can be trained by minimizing the composite loss below

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: L ic

Original: [Equation]
Revised: [Equation]

Original: Here we set N ic = 128 and N r = 1024 , and { x ic } N ic i =1 , { ( x r , t r ) } N r i =1 are randomly sampled from the computational domain, respectively, at each iteration of gradient descent.
Revised: Here we set N ic = 128 and N r = 1024 , and { x ic } N ic i =1 , { ( x r , t r ) } N r i =1 are randomly sampled from the computational domain, respectively, at each iteration of gradient descent.

Original: In addition, we take λ ic = 100 , λ r = 1 for better enforcing the initial condition.
Revised: In addition, we take λ ic = 100 , λ r = 1 for better enforcing the initial condition.

Original: It is worth pointing out that all the network derivatives are computed via automatic differentiation.
Revised: It is worth pointing out that all the network derivatives are computed via automatic differentiation.

Original: To enhance the model performance, we introduce the curriculum training (Krishnapriyan et al., 2021) and causal training (Wang etal., 2022) in the training process.
Revised: To enhance the model performance, we introduce the curriculum training (Krishnapriyan et al., 2021) and causal training (Wang et

Original: Curriculum training starts with a simple PDE system and progressively solves the target PDE system.
Revised: Curriculum training starts with a simple PDE system and progressively solves the target PDE system.

Original: For this example, it is accomplished by minimizing the above PINN loss with a lower advection coefficient c = 10 first and then gradually increasing c to the target value (i.e. c = 50 ) during training.
Revised: For this example, it is accomplished by minimizing the above PINN loss with a lower advection coefficient c = 10 first and then gradually increasing c to the target value (i.e. c = 50 ) during training.

Original: Causal training aims to impose temporal causality during the training of a PINNs model by appropriately re-weighting the PDE residual loss at each iteration of gradient descent.
Revised: Causal training aims to impose temporal causality during the training of a PINNs model by appropriately re-weighting the PDE residual loss at each iteration of gradient descent.

Original: Specifically, we split the temporal domain into M chunks [0 , ∆ t ] , [∆ t, 2∆ t ] , . . .
Revised: Specifically, we split the temporal domain into M chunks [0 , ∆ t ] , [∆ t, 2∆ t ] , . . .

Original: , and assign a weight to the corresponding temporal residuals losses L ir ( θ ) as
Revised: , and assign a weight to the corresponding temporal residuals losses L ir ( θ ) as

Original: [Equation]
Revised: [Equation]

Original: with w 1 = 1 , and
Revised: with w 1 = 1 , and

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: M

Original: [Equation]
Revised: [Equation]

Original: Here ϵ is a so-called causal parameter, which is a user-specified hyper-parameter that determines the slope of the causal weights.
Revised: Here ϵ is a so-called causal parameter, which is a user-specified hyper-parameter that determines the slope of the causal weights.

Original: We take M = 16 and ϵ = 0 .
Revised: We take M = 16 and ϵ = 0 .

Original: 1 in this example.
Revised: 1 in this example.

Original: We initialize MLPs (5 layers, 256 channels, tanh activations) with different weight parameterizations, and  train each model with different strategies for 2 × 10 5 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and an exponential decay by a factor of 0 .for
Revised: We initialize MLPs (5 layers, 256 channels, tanh activations) with different weight parameterizations and 10 random seeds. We train each model with different strategies for 2 × 10 5 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and an exponential decay by a factor of 0 .

Original:  every 5 , 000 steps.
Revised: 9 for every 5 , 000 steps.

Original: The resulting relative L 2 errors are presented in Table 10.
Revised: The resulting relative L 2 errors are presented in Table 12.

Original: In contrast to the failure of the other three parameterizations, RWF is the only one that enables PINN models to solve the advection equation with a reasonable and stable predictive accuracy.
Revised: In contrast to the failure of the other three parameterizations, RWF is the only one that enables PINN models to solve the advection equation with a reasonable and stable predictive accuracy.

Original: Further improvements can be obtained by combining RWF with curriculum or causal training strategies.
Revised: Further improvements can be obtained by combining RWF with curriculum or causal training strategies.

Original: These conclusions are further clarified by the visualizations in Figures 3, 16 and 17, where the predicted solutions corresponding to RWF are in excellent agreement with the ground truth.
Revised: These conclusions are further clarified by the visualizations in Figures 3, 20 and 21, where the predicted solutions corresponding to RWF are in excellent agreement with the ground truth.

Original: [Table]
Revised: [Table]

Original: K. N AVIER -S TOKES EQUATION
Revised: K. N AVIER -S TOKES EQUATION

Original: The underlying PDE system for this benchmark takes the form
Revised: The underlying PDE system for this benchmark takes the form

Original: [Equation]
Revised: [Equation]

Original: Here Γ 1 is the top boundary of a square cavity, while Γ 2 denotes the other three sides of the cavity.
Revised: Here Γ 1 is the top boundary of a square cavity, while Γ 2 denotes the other three sides of the cavity.

Original: We represent the unknown solution u, v, p using a neural network u θ :
Revised: We represent the unknown solution u, v, p using a neural network u θ :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, the PDE residuals are defined by
Revised: Then, the PDE residuals are defined by

Original: [Equation]
Revised: [Equation]

Original: Given these residuals
Revised: Given these residuals

Original: [Equation]
Revised: [Equation]

Original: along with a set of appropriate boundary conditions
Revised: along with a set of appropriate boundary conditions

Original: [Equation]
Revised: [Equation]

Original: we can now formulate a loss function for training a physics-informed neural network as
Revised: we can now formulate a loss function for training a physics-informed neural network as

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: = (cid:2) R c θ (cid:0) x ir , y ir (cid:1)(cid:3) 2 , (K.22)
Revised: [Equation]

Original: where (cid:8)(cid:0) x ib , y ib (cid:1) , u ib (cid:9) N b
Revised: where (cid:8)(cid:0) x ib , y ib (cid:1) , u ib (cid:9) N b

Original: i =1 and (cid:8)(cid:0) x ib , y ib (cid:1) , v ib (cid:9) N b
Revised: i =1 and (cid:8)(cid:0) x ib , y ib (cid:1) , v ib (cid:9) N b

Original: i =1 denote the boundary data for the two velocity components at the domain boundaries Γ 1 and Γ 2 , respectively, while (cid:8)(cid:0) x
Revised: i =1 denote the boundary data for the two velocity components at the domain boundaries Γ 1 and Γ 2 , respectively, while (cid:8)(cid:0) x

Original: ir , y ir (cid:1)(cid:9) N r i =1 is a set of collocation points for enforcing the PDE constraints.
Revised: ir , y ir (cid:1)(cid:9) N r i =1 is a set of collocation points for enforcing the PDE constraints.

Original: All of them are sampled randomly at each iteration of gradient descent.
Revised: All of them are sampled randomly at each iteration of gradient descent.

Original: In experiments, we set N b = 256 , N r = 1024 and λ u = λ v = 100 , λ r u = λ r v = λ
Revised: In experiments, we set N b = 256 , N r = 1024 and λ u = λ v = 100 , λ r u = λ r v = λ

Original: r c = 1 .
Revised: r c = 1 .

Original: We employ an MLP (5 layers, 128 channels, tanh activations) to represent the latent variables of interest, and train the network with different parameterizations for 10 5 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and an exponential decay by a factor of 0.9 for every 2 , 000 training iterations.
Revised: We employ an MLP (5 layers, 128 channels, tanh activations) to represent the latent variables of interest, and train the network with different parameterizations for 10 5 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and an exponential decay by a factor of 0.9 for every 2 , 000 training iterations.

Original: Moreover, we use a modified MLP architecure (see definition below) and curriculum training to enhance the model stability and performance.
Revised: Moreover, we use a modified MLP architecure (see definition below) and curriculum training to enhance the model stability and performance.

Original: For the curriculum training, we minimize the loss with Re = 100 and Re = 500 for 2 × 10 4 iterations sequentially and change the Reynolds number to Re = 1 , 000 for the rest of the training.
Revised: For the curriculum training, we minimize the loss with Re = 100 and Re = 500 for 2 × 10 4 iterations sequentially and change the Reynolds number to Re = 1 , 000 for the rest of the training.

Original: The resulting relative L 2 errors  are reported in Table 11.
Revised: The resulting relative L 2 errors over 10 random seeds are reported in Table 13.

Original: We can see that RWF performs the best among all parameterizations by a large margin.
Revised: We can see that RWF performs the best among all parameterizations by a large margin.

Original: Some visualizations are shown in Figure 4 and Figure 18.
Revised: Some visualizations are shown in Figure 4 and Figure 22.

Original: We attribute this significant performance improvements to the fact that PINN models often suffer from poor initializations, and RWF precisely mitigates this by being able to reach better local minima that are located further away from the model initialization neighborhood.Navier-Stokes Plain AA
Revised: We attribute this significant performance improvements to the fact that PINN models often suffer from poor initializations, and RWF precisely mitigates this by being able to reach better local minima that are located further away from the model initialization neighborhood.

Original: WN

Original: RWF

Original: [Table]
Revised: [Table]

Original: Modified MLP:
Revised: Modified MLP:

Original: In (Wang et al., 2021b)
Revised: In (Wang et al., 2021b)

Original: Wang et al. proposed a novel architecture that was demonstrated to outperform conventional MLPs across a variety of PINNs benchmarks.
Revised: Wang et al. proposed a novel architecture that was demonstrated to outperform conventional MLPs across a variety of PINNs benchmarks.

Original: Here, we will refer to this architecture as ”modified MLP”.
Revised: Here, we will refer to this architecture as ”modified MLP”.

Original: The forward pass of a L -layer modified MLP is defined as follows
Revised: The forward pass of a L -layer modified MLP is defined as follows

Original: [Equation]
Revised: [Equation]

Original: where σ denotes a nonlinear activation function, ⊙ denotes a point-wise multiplication.
Revised: where σ denotes a nonlinear activation function, ⊙ denotes a point-wise multiplication.

Original: All trainable parameters are given by
Revised: All trainable parameters are given by

Original: [Equation]
Revised: [Equation]

Original: This architecture is almost the same as a standard MLP network, with the addition of two encoders and a minor modification in the forward pass.
Revised: This architecture is almost the same as a standard MLP network, with the addition of two encoders and a minor modification in the forward pass.

Original: Specifically, the inputs x are embedded into a feature space via two encoders U , V , respectively, and merged in each hidden layer of a standard MLP using a point-wise multiplication.
Revised: Specifically, the inputs x are embedded into a feature space via two encoders U , V , respectively, and merged in each hidden layer of a standard MLP using a point-wise multiplication.

Original: L L EARNING OPERATORS
Revised: L L EARNING OPERATORS

Original: Overview of DeepONets: DeepONet is supposed to approximate an operator G between functional spaces.
Revised: Overview of DeepONets: DeepONet is supposed to approximate an operator G between functional spaces.

Original: DeepONet consists of two separate neural networks referred to as the ”branch net” and ”trunk net”, respectively.
Revised: DeepONet consists of two separate neural networks referred to as the ”branch net” and ”trunk net”, respectively.

Original: The branch net takes a vector a as input and returns a features embedding [ b 1 , b 2 , . . .
Revised: The branch net takes a vector a as input and returns a features embedding [ b 1 , b 2 , . . .

Original: , b q ] T ∈ R q as output, where a = [ a ( x 1 ) , a ( x 2 ) , . .
Revised: , b q ] T ∈ R q as output, where a = [ a ( x 1 ) , a ( x 2 ) , . .

Original: , a ( x m )] represents a function a evaluated at a collection of fixed locations { x i } m i =1 .
Revised: , a ( x m )] represents a function a evaluated at a collection of fixed locations { x i } mi =1 .

Original: The trunk net takes the continuous coordinates y as inputs, and outputs a features embedding [ t 1 , t 2 , . .
Revised: The trunk net takes the continuous coordinates y as inputs, and outputs a features embedding [ t 1 , t 2 , . .

Original: . , t q ] T ∈ R q .
Revised: . , t q ] T ∈ R q .

Original: The DeepONet output is obtained by merging the outputs of the branch and trunk networks together via a dot product
Revised: The DeepONet output is obtained by merging the outputs of the branch and trunk networks together via a dot product

Original: [Equation]
Revised: [Equation]

Original: where θ denotes the collection of all trainable weight and bias parameters in the branch and trunk networks.
Revised: where θ denotes the collection of all trainable weight and bias parameters in the branch and trunk networks.

Original: These parameters can be optimized by minimizing the following mean square error loss
Revised: These parameters can be optimized by minimizing the following mean square error loss

Original: [Equation]
Revised: [Equation]

Revised: L.

Revised: [Equation]

Revised: q (cid:88) k

Original:  where { a ( i ) } Ni =1 denotes N separate input functions sampled from a function space U .
Revised: = b k ( a ( i ) ( x 1 ) , . . . , a ( i ) ( x m )) t k ( y ( i ) j ) − G ( a ( i ) ) ( y ( i ) j ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , (L.3) where { a ( i ) } Ni =1 denotes N separate input functions sampled from a function space U .

Original: For each a ( i ) , { y ( i ) j } Pj =1 are P locations in the domain of G ( a ( i ) ) , and G ( a ( i ) )
Revised: For each a ( i ) , { y ( i ) j } Pj =1 are P locations in the domain of G ( a ( i ) ) , and G ( a ( i ) )

Original: ( y ( i ) j ) is the corresponding output data evaluated at y ( i ) j .
Revised: ( y ( i ) j ) is the corresponding output data evaluated at y ( i ) j .

Original: Contrary to the fixed sensor locations of { x i } mi =1 , we remark that the locations of { y ( i ) }
Revised: Contrary to the fixed sensor locations of { x i } mi =1 , we remark that the locations of { y ( i ) }

Original: Pj =1 may vary for different i , thus allowing us to construct a continuous representation of the output function G ( a ) .
Revised: Pj =1 may vary for different i , thus allowing us to construct a continuous representation of the output function G ( a ) .

Original: Remark: All parameterizations (AA, WN, RWF) are applied to every dense layer of the DeepONet architecture (in the cases where such parametrizations are employed).Case Plain AA
Revised: Remark: All parameterizations (AA, WN, RWF) are applied to every dense layer of the DeepONet architecture (in the cases where such parametrizations are employed).

Original: WN

Original: RWF

Original: DR

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Table]

Original: L. D IFFUSION - REACTION
Revised: L. D IFFUSION - REACTION

Original: The underlying PDE for this benchmark takes the form
Revised: The underlying PDE for this benchmark takes the form

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: × (

Revised: [Equation]

Revised: [Equation]

Original: subject to zero initial and boundary conditions.
Revised: subject to zero initial and boundary conditions.

Original: Data Generation: We sample N = 5 , 000 input functions a ( x ) from a GRF with length scale l = 0 .
Revised: Data Generation: We sample N = 5 , 000 input functions a ( x ) from a GRF with length scale l = 0 .

Original: 2 and solve the diffusion-reaction system using a second-order implicit finite difference method on a 100 × 100 equispaced grid.
Revised: 2 and solve the diffusion-reaction system using a second-order implicit finite difference method on a 100 × 100 equispaced grid.

Original: To generate the training data, we randomly take P =measurements from each solution.
Revised: To generate the training data, we randomly take P =measurements from each solution.

Original: The test data-set contains another 100 solutions evaluated at the same mesh.
Revised: The test data-set contains another 100 solutions evaluated at the same mesh.

Original: We represent the solution operator by a DeepONet G θ , where the branch and trunk networks are two separate MLPs (5 layers, 64 channels, tanh activations).
Revised: We represent the solution operator by a DeepONet G θ , where the branch and trunk networks are two separate MLPs (5 layers, 64 channels, tanh activations).

Original: The model is trained for 5 × 10iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and an exponential decay by a factor of 0 .
Revised: The model is trained for 5 × 10iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and an exponential decay by a factor of 0 .

Original: 9 for every 1000 steps.
Revised: 9 for every 1000 steps.

Original: The mean and standard deviation of the relative L 2 errors over the test date-set are shown in Table 12.
Revised: The mean and standard deviation of the relative L 2 errors over the test date-set are shown in Table 14.

Original: Figure 19 provides several representative predictions using RWF.
Revised: Figure 23 provides several representative predictions using RWF.

Original: L. D ARCY FLOW
Revised: L. D ARCY FLOW

Original: The PDE system for this benchmark takes the form
Revised: The PDE system for this benchmark takes the form

Original: [Equation]
Revised: [Equation]

Original: Data Generation: We sample the coefficient function a from a Gaussian random field with a length scale l = − 0 .
Revised: Data Generation: We sample the coefficient function a from a Gaussian random field with a length scale l = − 0 .

Original: 5 and solve the associated Darcy flow using finite element method.
Revised: 5 and solve the associated Darcy flow using finite element method.

Original: The training data contains 2 , 000 solutions evaluated at a 64 × 64 uniform mesh while the test data containssolutions on the same mesh.
Revised: The training data contains 2 , 000 solutions evaluated at a 64 × 64 uniform mesh while the test data containssolutions on the same mesh.

Original: We represent the solution operator by a DeepONet G θ , where the branch network is a convolutional neural network (CNN) for extracting latent feature representation of the input coefficients and the trunk network is a 4-layer MLP with GELU activations and 128 neurons per hidden layer.
Revised: We represent the solution operator by a DeepONet G θ , where the branch network is a convolutional neural network (CNN) for extracting latent feature representation of the input coefficients and the trunk network is a 4-layer MLP with GELU activations and 128 neurons per hidden layer.

Original: We train each model with different parameterizations for 5 × 10 4 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and an exponential decay by a factor of 0 .
Revised: We train each model with different parameterizations for 5 × 10 4 iterations using the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 − 3 and an exponential decay by a factor of 0 .

Original: 9 for every 1000 steps.
Revised: 9 for every 1000 steps.

Original: The results are summarized in Table 12 and some predicted solutions are plotted in Figure 20.
Revised: The results are summarized in Table 14 and some predicted solutions are plotted in Figure 24.

Original: L. B URGERS ’ EQUATION
Revised: L. B URGERS ’ EQUATION

Original: Recall the one-dimensional Burgers’ equation is given by
Revised: Recall the one-dimensional Burgers’ equation is given by

Original: [Equation]
Revised: [Equation]

Original: subject to initial and the periodic boundary conditions
Revised: subject to initial and the periodic boundary conditions

Original: [Equation]
Revised: [Equation]

Original: Data Generation: The training data only consists of 1 , 000 input functions a sampled from a
Revised: Data Generation: The training data only consists of 1 , 000 input functions a sampled from a

Original: Gaussian random field ∼ N (cid:0) 0 , 25 2 ( − ∆ + 5 2 I ) − 4 (cid:1) .
Revised: Gaussian random field ∼ N (cid:0) 0 , 25 2 ( − ∆ + 5 2 I ) − 4 (cid:1) .

Original: To generate the test data-set, we sample another 100 input functions from the same Gaussian random field and solve the Burgers’ equation using the Chebfun package Driscoll et al.
Revised: To generate the test data-set, we sample another 100 input functions from the same Gaussian random field and solve the Burgers’ equation using the Chebfun package Driscoll et al.

Original: (2014) with a spectral Fourier discretization and a fourthorder stiff time-stepping scheme (ETDRK4) Cox & Matthews (2002) with a time-step size of 10 − 4 .
Revised: (2014) with a spectral Fourier discretization and a fourthorder stiff time-stepping scheme (ETDRK4) Cox & Matthews (2002) with a time-step size of 10 − 4 .

Original: Temporal snapshots of the solution are saved every ∆ t = 0 .
Revised: Temporal snapshots of the solution are saved every ∆ t = 0 .

Original: 01 to give us 101 snapshots in total.
Revised: 01 to give us 101 snapshots in total.

Original: Consequently, the test data-set contains 500 realizations evaluated at a 100 × 101 spatio-temporal grid.
Revised: Consequently, the test data-set contains 500 realizations evaluated at a 100 × 101 spatio-temporal grid.

Original: Our objective here is to learn the solution operator mapping initial conditions a ( x ) to the associated full spatio-temporal solution u ( x, t ) .
Revised: Our objective here is to learn the solution operator mapping initial conditions a ( x ) to the associated full spatio-temporal solution u ( x, t ) .

Original: Here proceed by representing the solution operator by a modified DeepONet architecture Wang et al.
Revised: Here proceed by representing the solution operator by a modified DeepONet architecture Wang et al.

Original: (2021d) outlined below.
Revised: (2021d) outlined below.

Original: To impose the periodic the exact boundary condition, we apply a Fourier feature mapping to the input coordinates before passing them through the trunk network
Revised: To impose the periodic the exact boundary condition, we apply a Fourier feature mapping to the input coordinates before passing them through the trunk network

Original: [Equation]
Revised: [Equation]

Original: Then the PDE residual is then defined by
Revised: Then the PDE residual is then defined by

Original: [Equation]
Revised: [Equation]

Original: Consequently, a physics-informed DeepONet can be trained by minimizing the following weighted loss function
Revised: Consequently, a physics-informed DeepONet can be trained by minimizing the following weighted loss function

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: For this example, we take N = 64 , P = 100 and Q = 512 , which means that we randomly sample N = 64 input functions from the training data-set and Q = 512 collocation points inside the computational domain.
Revised: For this example, we take N = 64 , P = 100 and Q = 512 , which means that we randomly sample N = 64 input functions from the training data-set and Q = 512 collocation points inside the computational domain.

Original: In particular, we set λ ic = 100 , λ r = 1 for better enforcing the initial condition across different input samples.
Revised: In particular, we set λ ic = 100 , λ r = 1 for better enforcing the initial condition across different input samples.

Original: The model with different parameterizations is trained for 10 5 iterations using the the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 −and an exponential decay by a factor of 0 .
Revised: The model with different parameterizations is trained for 10 5 iterations using the the Adam optimizer (Kingma & Ba, 2014) with a start learning rate of 10 −and an exponential decay by a factor of 0 .

Original: 9 for every 2 , 000 steps.
Revised: 9 for every 2 , 000 steps.

Original: We report the test errors in Table 12 and visualize some predicted solutions in Figure 21.
Revised: We report the test errors in Table 14 and visualize some predicted solutions in Figure 25.

Original: Modified DeepONet: Wang et al.
Revised: Modified DeepONet: Wang et al.

Original: (2021d) modify the forward pass of an L-layer DeepONet as follows
Revised: (2021d) modify the forward pass of an L-layer DeepONet as follows

Original: [Equation]
Revised: [Equation]

Original: where ⊙ denotes point-wise multiplication, ϕ denotes a activation function, and θ represents all trainable parameters of the DeepONet model.
Revised: where ⊙ denotes point-wise multiplication, ϕ denotes a activation function, and θ represents all trainable parameters of the DeepONet model.

Original: In particular, { W ( l ) a , b ( l +1) a }
Revised: In particular, { W ( l ) a , b ( l +1) a }

Original: L +1 l =1 and { W ( l ) y , b ( l +1) y }
Revised: L +1 l =1 and { W ( l ) y , b ( l +1) y }

Original: L +1 l =1 are the weights and biases of the branch and trunk networks, respectively.
Revised: L +1 l =1 are the weights and biases of the branch and trunk networks, respectively.

Original: we embed the DeepONet inputs a and y into a high-dimensional feature space via two encoders U , V , respectively.
Revised: we embed the DeepONet inputs a and y into a high-dimensional feature space via two encoders U , V , respectively.

Original: Instead of just merging the propagated information in the output layer of the branch and trunk networks, we merge the embeddings U , V in each hidden layer of these two sub-networks using a point-wise multiplication (equation (L.17) - (L.18)).
Revised: Instead of just merging the propagated information in the output layer of the branch and trunk networks, we merge the embeddings U , V in each hidden layer of these two sub-networks using a point-wise multiplication (equation (L.17) - (L.18)).

Original: Heuristically, this design may not only help input signals propagate through the DeepONet, but also enhance its capability of representing non-linearity due to the extensive use of point-wise multiplications.
Revised: Heuristically, this design may not only help input signals propagate through the DeepONet, but also enhance its capability of representing non-linearity due to the extensive use of point-wise multiplications.