Original: L EARNING S PECIALIZED A CTIVATION F UNCTIONS FOR P HYSICS - INFORMED N EURAL N ETWORKS
Revised: L EARNING S PECIALIZED A CTIVATION F UNCTIONS FOR P HYSICS - INFORMED N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: At the heart of network architectures lie the non-linear activation functions, the choice of which affects the model optimization and task performance.
Revised: At the heart of network architectures lie the non-linear activation functions, the choice of which affects the model optimization and task performance.

Original: In computer vision and natural language processing, the Rectified Linear Unit is widely adopted across different tasks.
Revised: In computer vision and natural language processing, the Rectified Linear Unit is widely adopted across different tasks.

Original: However, there is no such default choice of activation functions in the context of physics-informed neural networks (PINNs).
Revised: However, there is no such default choice of activation functions in the context of physics-informed neural networks (PINNs).

Original: It is observed that PINNs exhibit high sensitivity to activation functions due to the various characteristics of each physics system, which makes the choice of the suitable activation function for PINNs a critical issue.
Revised: It is observed that PINNs exhibit high sensitivity to activation functions due to the various characteristics of each physics system, which makes the choice of the suitable activation function for PINNs a critical issue.

Original: Existing works usually choose activation functions in an inefficient trial-and-error manner.
Revised: Existing works usually choose activation functions in an inefficient trial-and-error manner.

Original: To address this problem, we propose to search automatically for the optimal activation function when solving different PDEs.
Revised: To address this problem, we propose to search automatically for the optimal activation function when solving different PDEs.

Original: This is achieved by learning an adaptive activation function as linear combinations of a set of candidate functions, whose coefficients can be directly optimized by gradient descent.
Revised: This is achieved by learning an adaptive activation function as linear combinations of a set of candidate functions, whose coefficients can be directly optimized by gradient descent.

Original: In addition to its efficient optimization, the proposed method enables the discovery of novel activation function and the incorporation with prior knowledge about the PDE system.
Revised: In addition to its efficient optimization, the proposed method enables the discovery of novel activation function and the incorporation with prior knowledge about the PDE system.

Original: We can further enhance its search space with adaptive slope.
Revised: We can further enhance its search space with adaptive slope.

Original: While being surprisingly simple, the effectiveness of the proposed adaptive activation function is demonstrated on a series of benchmarks, including the Poisson’s equation, convection equation, Burgers’ equation, Allen-Cahn equation, Korteweg–de Vries equation, and Cahn-Hilliard equation.
Revised: While being surprisingly simple, the effectiveness of the proposed adaptive activation function is demonstrated on a series of benchmarks, including the Poisson’s equation, convection equation, Burgers’ equation, Allen-Cahn equation, Korteweg–de Vries equation, and Cahn-Hilliard equation.

Original: The performance gain of the proposed method is further interpreted from the neural tangent kernel perspective.
Revised: The performance gain of the proposed method is further interpreted from the neural tangent kernel perspective.

Original: Code will be released.
Revised: Code will be released.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent years have witnessed the remarkable progress of physics-informed neural networks (PINNs) on simulating the dynamics of physical systems, in which the activation functions play a significant role in the expressiveness and optimization of models.
Revised: Recent years have witnessed the remarkable progress of physics-informed neural networks (PINNs) on simulating the dynamics of physical systems, in which the activation functions play a significant role in the expressiveness and optimization of models.

Original: While the Rectified Linear Unit (Hahnloser et al., 2000; Jarrett et al., 2009; Nair & Hinton, 2010) is widely adopted in most computer vision and natural language processing tasks (Ramachandran et al., 2017), there is no such default choice of activation functions in the context of PINNs.
Revised: While the Rectified Linear Unit (Hahnloser et al., 2000; Jarrett et al., 2009; Nair & Hinton, 2010) is widely adopted in most computer vision and natural language processing tasks (Ramachandran et al., 2017), there is no such default choice of activation functions in the context of PINNs.

Original: In fact, PINNs show great sensitivity to activation functions when applied to different physical systems, since each system has its own characteristic.
Revised: In fact, PINNs show great sensitivity to activation functions when applied to different physical systems, since each system has its own characteristic.

Original: On one hand, the utilization of unsuitable activation functions would cause over-parameterizaiton and overfitting.
Revised: On one hand, the utilization of unsuitable activation functions would cause over-parameterization and overfitting.

Original: One the other hand, accurate simulations can be achieved with fast convergence and high precision by choosing proper activation functions.
Revised: One the other hand, accurate simulations can be achieved with fast convergence and high precision by choosing proper activation functions.

Original: For example, the hyperbolic tangent function is shown to suffer from numerical instability when simulating vortex induced vibrations, while a PINN with sinusoidal function can be optimized smoothly (Raissi et al., 2019b).
Revised: For example, the hyperbolic tangent function is shown to suffer from numerical instability when simulating vortex induced vibrations, while a PINN with sinusoidal function can be optimized smoothly (Raissi et al., 2019b).

Original: The various characteristics of different PDE systems make it critical to select proper activation functions in PINNs.
Revised: The various characteristics of different PDE systems make it critical to select proper activation functions in PINNs.

Original: The common practice to find the optimal activation functions is by trial-and-error, which requires extensive computational resources and human knowledge.
Revised: The common practice to find the optimal activation functions is by trial-and-error, which requires extensive computational resources and human knowledge.

Original: This method is inefficient especially when solving complex problems, where searching for a set of activation functions is necessary to make accurate predictions.
Revised: This method is inefficient especially when solving complex problems, where searching for a set of activation functions is necessary to make accurate predictions.

Original: For instance, a combination of the sinusoidal and the exponential function is demonstrated effective to solve the heat transfer equation, whose solution is periodic in space and exponentially decaying in time (Zobeiry & Humfeld, 2021).
Revised: For instance, a combination of the sinusoidal and the exponential function is demonstrated effective to solve the heat transfer equation, whose solution is periodic in space and exponentially decaying in time (Zobeiry & Humfeld, 2021).

Original: In this case, the trial-and-error strategy leads to the combinatorial search problem, which becomes infeasible when the candidate activation function set is large.
Revised: In this case, the trial-and-error strategy leads to the combinatorial search problem, which becomes infeasible when the candidate activation function set is large.

Original: In this work, we propose a simple and effective physics-informed activation function (PIAC), aiming at the automatic design of activation functions for solving PDE systems with various characteristics.
Revised: In this work, we propose a simple and effective physics-informed activation function (PIAC), aiming at the automatic design of activation functions for solving PDE systems with various characteristics.

Original: Sharing the same spirits with differentiable neural architecture search (Liu et al., 2018), we parameterize the categorical selection of one particular activation function into a continuous search space, leading to an end-to-end differentiable problem which can be integrated into the training procedure of PINNs.
Revised: Sharing the same spirits with differentiable neural architecture search (Liu et al., 2018), we parameterize the categorical selection of one particular activation function into a continuous search space, leading to an end-to-end differentiable problem which can be integrated into the training procedure of PINNs.

Original: Specifically, we first define a set of candidate activation functions, which is conceptually composed of simple elementary functions or commonly-used activation functions.
Revised: Specifically, we first define a set of candidate activation functions, which is conceptually composed of simple elementary functions or commonly-used activation functions.

Original: Then, the proposed PIAC is learned as the linear combination of candidate functions with adaptive coefficients.
Revised: Then, the proposed PIAC is learned as the linear combination of candidate functions with adaptive coefficients.

Original: Besides its efficient optimization, this continuous parameterization also enables the discovery of novel activation functions, whose capacity can be further enhanced by cascading these learnable functions in a layer-wise or neuron-wise manner.
Revised: Besides its efficient optimization, this continuous parameterization also enables the discovery of novel activation functions, whose capacity can be further enhanced by cascading these learnable functions in a layer-wise or neuron-wise manner.

Original: Our method can be regraded as adaptive activation functions built on a predefined candidate function set.
Revised: Our method can be regraded as adaptive activation functions built on a predefined candidate function set.

Original: Although adaptive activation functions have been explored for PINNs, previous works mainly focus on accelerating the convergence of PINNs by introducing the adaptive slope (Jagtap et al.
Revised: Although adaptive activation functions have been explored for PINNs, previous works mainly focus on accelerating the convergence of PINNs by introducing the adaptive slope (Jagtap et al.

Original: , 2020b;a), while leaving the inefficient selection of activation function for different PDE systems unexplored.
Revised: , 2020b;a), while leaving the inefficient selection of activation function for different PDE systems unexplored.

Original: In fact, our method is orthogonal to these methods and can be extended by incorporating the adaptive slope into our search space.
Revised: In fact, our method is orthogonal to these methods and can be extended by incorporating the adaptive slope into our search space.

Original: Moreover, the introduction of candidate function set can be leveraged to embed prior knowledge about the PDE system into the neural networks.
Revised: Moreover, the introduction of candidate function set can be leveraged to embed prior knowledge about the PDE system into the neural networks.

Original: For example, the sinusoidal function can be added into this set to assist the modeling of periodicity.
Revised: For example, the sinusoidal function can be added into this set to assist the modeling of periodicity.

Original: We evaluate the proposed PIAC on a series of PDEs, including the Poisson’s equation, convection equation, Burgers’ equation, Allen-Cahn equation, Korteweg–de Vries equation, and Cahn-Hilliard equation.
Revised: We evaluate the proposed PIAC on a series of PDEs, including the Poisson’s equation, convection equation, Burgers’ equation, Allen-Cahn equation, Korteweg–de Vries equation, and Cahn-Hilliard equation.

Original: Extensive experiments show that the proposed PIAC can consistently outperform the standard activation functions.
Revised: Extensive experiments show that the proposed PIAC can consistently outperform the standard activation functions.

Original: We further explain the performance gain from the neural tangent kernel (NTK) perspective (Jacot et al., 2018; Wang et al., 2022).
Revised: We further explain the performance gain from the neural tangent kernel (NTK) perspective (Jacot et al., 2018; Wang et al., 2022).

Original: The main contribution of this paper can be summarized as:
Revised: The main contribution of this paper can be summarized as:

Original: • We investigate the influence of activation functions for PINNs to solve different problems and reveal the high sensitivity of PINNs to the choice of activation functions, which can be related to various characteristics of the underlying PDE system; • We explore the automatic design of activation functions for PINNs.
Revised: • We investigate the influence of activation functions for PINNs to solve different problems and reveal the high sensitivity of PINNs to the choice of activation functions, which can be related to various characteristics of the underlying PDE system; • We explore the automatic design of activation functions for PINNs.

Original: The proposed method can be efficiently optimized and adapted to the PDE system, while enabling the utilization of prior knowledge about the problem; • While being simple, the effectiveness of the proposed PIAC is demonstrated on extensive experiments and interpreted from the perspective of neural tangent kernel.
Revised: The proposed method can be efficiently optimized and adapted to the PDE system, while enabling the utilization of prior knowledge about the problem; • While being simple, the effectiveness of the proposed PIAC is demonstrated on extensive experiments and interpreted from the perspective of neural tangent kernel.

Original:  P RELIMINARIES AND RELATED WORK 2. P HYSICS - INFORMED NEURAL NETWORKS
Revised:  P RELIMINARIES AND RELATED WORK 2. P HYSICS - INFORMED NEURAL NETWORKS

Original: Physics-informed neural networks have emerged as a promising method for solving forward and inverse problems of PDEs (Raissi et al., 2019a;
Revised: Physics-informed neural networks have emerged as a promising method for solving forward and inverse problems of PDEs (Raissi et al., 2019a;

Original: Chen et al., 2020; Lu et al., 2021b; Karniadakis et al., 2021), fractional PDEs (Pang et al., 2019), and stochastic PDEs (Zhang et al., 2019).
Revised: Chen et al., 2020; Lu et al., 2021b; Karniadakis et al., 2021), fractional PDEs (Pang et al., 2019), and stochastic PDEs (Zhang et al., 2019).

Original: In this work, we focus on solving the forward problems of PDEs as described in (Raissi et al., 2019a).
Revised: In this work, we focus on solving the forward problems of PDEs as described in (Raissi et al., 2019a).

Original: Specifically, we consider PDEs of the general form
Revised: Specifically, we consider PDEs of the general form

Original: [Equation]
Revised: [Equation]

Original: subject to the initial and boundary conditions
Revised: subject to the initial and boundary conditions

Original: [Equation]
Revised: [Equation]

Original: where u ( t, x ) denotes the solution, N [ · ; λ ] is a differential operator parameterized by λ , B [ · ] is a boundary operator, and subscripts denote the partial differentiation.
Revised: where u ( t, x ) denotes the solution, N [ · ; λ ] is a differential operator parameterized by λ , B [ · ] is a boundary operator, and subscripts denote the partial differentiation.

Original: A physics-informed neural network (PINN) u ′ ( t, x ; θ ) is optimized to approximate the solution u ( t, x ) by minimize the following objective function 
Revised: A physics-informed neural network (PINN) u ′ ( t, x ; θ ) is optimized to approximate the solution u ( t, x ) by minimizing the following objective function [Equation]

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here θ denotes the parameters of neural networks.
Revised: Here θ denotes the parameters of neural networks.

Original: The L ic and L bc measure the prediction error on initial training data { x i ic }
Revised: The L ic and L bc measure the prediction error on initial training data { x i ic }

Original: N ic i and boundary training data { t i bc , x i bc } N bc
Revised: N ic i and boundary training data { t i bc , x i bc } N bc

Original: i .
Revised: i .

Original: The residual loss L r is imposed to make the neural network satisfy the PDE constraint on a set of collocation points { t i r , x i r } N r i .
Revised: The residual loss L r is imposed to make the neural network satisfy the PDE constraint on a set of collocation points { t i r , x i r } N r i .

Original: To compute the residual loss L r , partial derivatives of the neural network output with respect to t and x can be obtained via automatic differentiation techniques (Baydin et al., 2018).
Revised: To compute the residual loss L r , partial derivatives of the neural network output with respect to t and x can be obtained via automatic differentiation techniques (Baydin et al., 2018).

Original: Despite the effectiveness, the introduction of physics-based loss function makes the optimization more ill-conditioned (Wang et al., 2021; Krishnapriyan et al., 2021).
Revised: Despite the effectiveness, the introduction of physics-based loss function makes the optimization more ill-conditioned (Wang et al., 2021; Krishnapriyan et al., 2021).

Original: Efforts have been made to alleviate this problem from the aspects of loss weight balancing (Wang et al., 2021; 2022), loss function design (Psaros et al., 2022; Yu et al., 2022), adaptive collocation point sampling (Wight &
Revised: Efforts have been made to alleviate this problem from the aspects of loss weight balancing (Wang et al., 2021; 2022), loss function design (Psaros et al., 2022; Yu et al., 2022), adaptive collocation point sampling (Wight &

Original: Zhao, 2020; McClenny & Braga-Neto, 2020; Lu et al., 2021a; Wu et al., 2022), domain decomposition (Jagtap et al., 2020c; Jagtap & Karniadakis, 2020), and curriculum learning (Krishnapriyan et al., 2021).
Revised: Zhao, 2020; McClenny & Braga-Neto, 2020; Lu et al., 2021a; Wu et al., 2022), domain decomposition (Jagtap et al., 2020c; Jagtap & Karniadakis, 2020), and curriculum learning (Krishnapriyan et al., 2021).

Original: Different from previous works, we shed light on the relationship between the optimization difficulty of PDE constraint and activation functions.
Revised: Different from previous works, we shed light on the relationship between the optimization difficulty of PDE constraint and activation functions.

Original: Our work reveals the high sensitivity of PINNs to the choice of activation function and proposes to reduce the optimization difficulty of PINNs by learning specialized activation functions for different PDEs.
Revised: Our work reveals the high sensitivity of PINNs to the choice of activation function and proposes to reduce the optimization difficulty of PINNs by learning specialized activation functions for different PDEs.

Original: We hope our work could inspire further study on the convergence issue of PINNs from the perspective of activation functions.
Revised: We hope our work could inspire further study on the convergence issue of PINNs from the perspective of activation functions.

Original: 2. A CTIVATION FUNCTIONS
Revised: 2. A CTIVATION FUNCTIONS

Original: Fixed-shape activation function.
Revised: Fixed-shape activation function.

Original: Activation functions are crucial for the optimization and performance of deep neural networks (Glorot & Bengio, 2010).
Revised: Activation functions are crucial for the optimization and performance of deep neural networks (Glorot & Bengio, 2010).

Original: The Rectified Linear Units (ReLU) (Hahnloser et al., 2000; Jarrett et al., 2009; Nair & Hinton, 2010) outperform the logistic and hyperbolic tangent activation functions and have become the default choice for most neural networks.
Revised: The Rectified Linear Units (ReLU) (Hahnloser et al., 2000; Jarrett et al., 2009; Nair & Hinton, 2010) outperform the logistic and hyperbolic tangent activation functions and have become the default choice for most neural networks.

Original: Since then, designing activation functions has been an active research direction.
Revised: Since then, designing activation functions has been an active research direction.

Original: Variants with improved learning characteristics are proposed, such as Softplus (Dugas et al., 2000; Glorot et al., 2011),
Revised: Variants with improved learning characteristics are proposed, such as Softplus (Dugas et al., 2000; Glorot et al., 2011),

Original: Leaky ReLU (Maas et al., 2013), ELU (Clevert et al., 2015) and GELU (Hendrycks & Gimpel, 2016).
Revised: Leaky ReLU (Maas et al., 2013), ELU (Clevert et al., 2015) and GELU (Hendrycks & Gimpel, 2016).

Original: Besides these hand-designed activation functions, Ramachandran et al.
Revised: Besides these hand-designed activation functions, Ramachandran et al.

Original: (2017) leverages neural architecture search techniques to discover the novel Swish function.
Revised: (2017) leverages neural architecture search techniques to discover the novel Swish function.

Original: In this work, we focus on the selection of activation functions for PINNs when solving different problems.
Revised: In this work, we focus on the selection of activation functions for PINNs when solving different problems.

Original: While the common practice to find the optimal activation function is by trial-and-error, we formulate this manual selection as a learning problem, which can be optimized efficiently.
Revised: While the common practice to find the optimal activation function is by trial-and-error, we formulate this manual selection as a learning problem, which can be optimized efficiently.

Original: Adaptive activation functions are explored to find specialized activation functions for different architectures and tasks.
Revised: Adaptive activation functions are explored to find specialized activation functions for different architectures and tasks.

Original: The major difference of existing methods lies in the search space.
Revised: The major difference of existing methods lies in the search space.

Original: The piece-wise linear function is adopted as the universal function approximator in some methods, such as APL (Agostinelli et al., 2014), RePLU (Li et al., 2016) and PWLU (Zhou et al., 2021).
Revised: The piece-wise linear function is adopted as the universal function approximator in some methods, such as APL (Agostinelli et al., 2014), RePLU (Li et al., 2016) and PWLU (Zhou et al., 2021).

Original: The formulation of SLAF (Goyal et al., 2019) is based on Taylor approximation with polynomial basis.
Revised: The formulation of SLAF (Goyal et al., 2019) is based on Taylor approximation with polynomial basis.

Original: PAU (Molina et al., 2019) leverages Pad´e approximation to form its search space.
Revised: PAU (Molina et al., 2019) leverages Pad´e approximation to form its search space.

Original: Motivated by the connection between Swish and ReLU, ACON (Ma et al., 2021) is proposed as an smooth approximator to the general Maxout family activation functions (Goodfellow et al., 2013).
Revised: Motivated by the connection between Swish and ReLU, ACON (Ma et al., 2021) is proposed as a smooth approximator to the general Maxout family activation functions (Goodfellow et al., 2013).

Original: Our work proposes to learn  adaptive activation function as a weighted sum of candidate functions, whose weights can be adapted to the underlying physics laws when modelling different PDE systems.
Revised: Our work proposes to learn an adaptive activation function as a weighted sum of candidate functions, whose weights can be adapted to the underlying physics laws when modelling different PDE systems.

Original: While learning combinations of activation functions has been studied for convolutional neural networks on image classification (Dushkoff & Ptucha, 2016; Qian et al., 2018; Manessi & Rozza, 2018), we would like to argue that it is non-trivial to explore this idea in the context of PINNs.
Revised: While similar ideas have been studied for convolutional neural networks in image classification (Dushkoff & Ptucha, 2016; Qian et al., 2018; Manessi & Rozza, 2018; S¨utfeld et al., 2020), some technical challenges remain unexplored in the context of PINNs, which have a higher demand for the smoothness and diversity of the candidate functions.

Original: First, different PDE systems have various characteristics, which are difficult to model accurately by a single activation function.
Revised: First, the optimization of PDE-based constraints needs the activation function to provide higher-order derivatives, which causes the failure of widely-used

Original: In contrast, learning a combination of candidate functions makes it possible to embed prior knowledge about the physics system into neural networks by including activation functions with suitable properties.
Revised: ReLUs in PINNs.

Original: In addition, while previous methods (Dushkoff & Ptucha, 2016; Qian et al., 2018; Manessi & Rozza, 2018) experiment with limit choice of activation functions, we add most of commonly-used activation functions into the candidate function set  to ensure its diversity and to avoid the repetitive evaluations of each candidate activation function for different PDEs.
Revised: Second, unlike the image classification tasks, different PDE systems could have various characteristics, such as periodicity and rapid decay. This leads to a higher requirement for the diversity of the candidate functions. To overcome these challenges, we propose to build the candidate function set with simple elementary functions to embed the prior knowledge of physics systems, as well as commonly-used activation functions to ensure the diversity.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we  investigate the influence of activation functions in PINNs for solving PDE/ODE systems.
Revised: In this section, we first investigate the influence of activation functions in PINNs for solving simple

Original: We evaluate and compare the effectiveness of several common activation functions on some simple problems with analytical solutions.
Revised: ODE systems with analytical solutions.

Original: The results show that the choice of activation functions is crucial for PINNs and depends on the problem.
Revised: The results show that the choice of activation functions is crucial for PINNs and depends on the problem.

Original: Motivated by this observation, we propose to learn specialized activation function for different PDE systems.
Revised: Motivated by this observation, we propose to learn specialized activation functions for different PDE systems.

Original: 3. A CTIVATION FUNCTIONS IN PINN S
Revised: 3. A CTIVATION FUNCTIONS IN PINN S

Original: Across the deep learning community, the Rectified Linear Unit (ReLU) enjoys widespread adoption and becomes the default activation function.
Revised: Across the deep learning community, the ReLU enjoys widespread adoption and becomes the default activation function.

Original: However, there is no such default choice of activation functions in the context of PINNs, regardless of the importance of activation functions in the training dynamics and expressive power of neural networks.
Revised: However, there is no such default choice of activation functions for PINNs, regardless of the importance of activation functions in the optimization and expressivity of neural networks.

Original: One of the reasons lies in the failure of ReLUs in PINNs, whose second-order derivative is zero everywhere.
Revised: One of the reasons lies in the failure of ReLUs in PINNs, whose second-order derivative is zero everywhere.

Original: More importantly, PINNs show great sensitivity to activation functions due to the various characteristics of the underlying PDE system.
Revised: More importantly, PINNs show great sensitivity to activation functions due to the various characteristics of the underlying PDE system.

Original: To give a further illustration, we compare the performance of PINNs with different activation functions on the following toy examples.
Revised: To give a further illustration, we compare the performance of PINNs with different activation functions on the following toy examples.

Original: Problem formulation.
Revised: Problem formulation.

Original: Here we consider the one-dimensional Poisson’s equation as
Revised: Here we consider the one-dimensional Poisson’s equation as

Original: [Equation]
Revised: [Equation]

Original: with boundary condition u (0) = 0 and u ( π ) = π , where ∆ is the Laplace operator and f ( x ) denotes the source term.
Revised: with boundary condition u (0) = 0 and u ( π ) = π , where ∆ is the Laplace operator and f ( x ) denotes the source term.

Original: As shown in Table 1, we use different source terms to construct analytical solutions with various characteristics.
Revised: As shown in Table 1, we use different source terms to construct analytical solutions with various characteristics.

Original: For the problem P1 , the solution is composed of trigonometric functions with different periods; for the problem P2 , its solution is dominant by the exponential function.
Revised: For the problem P1 , the solution is composed of trigonometric functions with different periods; for the problem P2 , its solution is dominant by the exponential function.

Original: Following the setting described in Yu et al.
Revised: Following the setting described in Yu et al.

Original: (2022), we impose the boundary conditions as hard-constraints by choosing the surrogate of solution as
Revised: (2022), we impose the boundary conditions as hard-constraints by choosing the surrogate of solution as

Original: [Equation]
Revised: [Equation]

Original: where u ′ ( x ; θ ) is the output of PINN.
Revised: where u ′ ( x ; θ ) is the output of PINN.

Original: Hence, the training objective is reduced to the residual loss L r .
Revised: Hence, the training objective is reduced to the residual loss L r .

Original: Note that we consider a extreme case of insufficient collocation points to highlight the difference between activation functions.
Revised: Note that we consider an extreme case of insufficient collocation points to highlight the difference between activation functions.

Original: The details of activation functions and experiment setups can refer to
Revised: The details of activation functions and experiment setups can refer to

Original: Appendix A and Appendix B, respectively.
Revised: Appendix A and Appendix C, respectively.

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: PINNs exhibit high sensitivity to the choice of activation functions.
Revised: PINNs exhibit high sensitivity to the choice of activation functions.

Original: Table 2 shows the results of different activation functions on P1 and P2 .
Revised: Table 2 shows the results of different activation functions on P1 and P2 .

Original: As we can see, the choice of activation functions makes a big difference on the prediction accuracy.
Revised: As we can see, the choice of activation functions makes a big difference on the prediction accuracy.

Original: For example, there exists a large performance gap between the exp and tanh functions on P2 ( 0 .
Revised: For example, there exists a large performance gap between the exp and tanh functions on P2 ( 0 .

Original: 73 ± 2 . 

Revised: 73 ± 2 .

Original: vs. 75 .
Revised: vs. 75 .

Original: 77 ± 7 . 93 ).
Revised: 77 ± 7 . 93 ).

Original: Moreover, the performance of the same activation function can vary significantly across different problems.
Revised: Moreover, the performance of the same activation function can vary significantly across different problems.

Original: One can find that while sin achieves the lowest error on P1 , it produces poor results on P2 .
Revised: One can find that while sin achieves the lowest error on P1 , it produces poor results on P2 .

Original: A similar phenomenon is observed on the exp function.
Revised: A similar phenomenon is observed on the exp function.

Original: Note that none of these activation functions can achieve the best performance simultaneously on these two problems.
Revised: Note that none of these activation functions can achieve the best performance simultaneously on these two problems.

Original: We argue that this sensitivity arises from the various characteristics of each problem.
Revised: We argue that this sensitivity arises from the various characteristics of each problem.

Original: In
Revised: In

Original: Figure 1, we visualize the second-order derivative of PINNs to analyse the influence of activation functions on the optimization of residual loss L r .
Revised: Figure 1, we visualize the second-order derivative of PINNs to analyse the influence of activation functions on the optimization of residual loss L r .

Original: We find that the commonly-used tanh function fails to model the periodic nature of P1 and the rapid decay of P2 , exhibiting severe overfitting with insufficient collocation points.
Revised: We find that the commonly-used tanh function fails to model the periodic nature of P1 and the rapid decay of P2 , exhibiting severe overfitting with insufficient collocation points.

Original: In contrast, accurate approximation can be achieved under the same conditions by using activation functions with suitable properties, such as the sin function for P1 and the exp function for P2 .
Revised: In contrast, accurate approximation can be achieved under the same conditions by using activation functions with suitable properties, such as the sin function for P1 and the exp function for P2 .

Original: However, there dose not exist a generic activation function which could model different characteristics simultaneously.
Revised: However, there dose not exist a generic activation function which could model different characteristics simultaneously.

Original: This finding demonstrates the necessity of careful selection of activation functions when solving different problems.
Revised: This finding demonstrates the necessity of careful selection of activation functions when solving different problems.

Original: [Table]
Revised: [Table]

Original: c
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: n r r r
Revised: [Figure] r

Original: i v a t i v

Original: [Table]
Revised: [Figure]

Original: [Table]
Revised: i

Original: [Figure]
Revised: [Figure]

Original: 3. L EARNING SPECIALIZED ACTIVATION FUNCTIONS FOR SOLVING PDE S
Revised: a t i [Figure] f u [Figure]

Revised: 1D Poisson’s equation, the residual loss

Revised: L r penalizes the deviation of the second-order derivative of PINNs to the exact values on sampled collocation points (black dots in figures).

Revised: We plot the results of sin , exp and tanh .

Revised: 3. L EARNING SPECIALIZED ACTIVATION FUNCTIONS FOR SOLVING PDE S

Original: In light of the diversity and complexity of PDEs, it is critical to select proper activation functions in PINNs.
Revised: In light of the diversity and complexity of PDEs, it is critical to select proper activation functions in PINNs.

Original: Existing works usually make the choice though trial-and-error.
Revised: Existing works usually make the choice though trial-and-error.

Original: However, this strategy is inefficient and leads to the impractical combinatorial search problem in some cases where choosing a set of suitable activation functions is necessary for solving complex problems.
Revised: However, this strategy is inefficient and leads to the impractical combinatorial search problem in some cases where choosing a set of suitable activation functions is necessary for solving complex problems.

Original: To address this problem, we propose a simple and effective method to search for the optimal activation functions for different problems automatically.
Revised: To address this problem, we propose a simple and effective method to search for the optimal activation functions for different problems automatically.

Original: To be specific, we formulate the manual selection of activation functions as a learning problem, following the spirits of neural architecture search.
Revised: To be specific, we formulate the manual selection of activation functions as a learning problem, following the spirits of neural architecture search.

Original: Our basic idea is to construct the search space with a set of candidate activation functions and learn to predict the optimal activation function for different PDEs.
Revised: Our basic idea is to construct the search space with a set of candidate activation functions and learn to predict the optimal activation function for different PDEs.

Original: Instead of categorically selecting one particular activation function, we relax the search space to be a linear combination of all candidate activation functions with learnable coefficients.
Revised: Instead of categorically selecting one particular activation function, we relax the search space to be a linear combination of all candidate activation functions with learnable coefficients.

Original: This continuous search space allows efficient optimization through backpropagation and enables the discovery of novel activation functions.
Revised: This continuous search space allows efficient optimization through backpropagation and enables the discovery of novel activation functions.

Original: Formally, we define the physics-informed activation function (PIAC) as
Revised: Formally, we define the physics-informed activation function (PIAC) as

Original: [Equation]
Revised: [Equation]

Original: 10) where σ i ( · ) and α i denote a candidate activation function and a learnable parameter, respectively.
Revised: where σ i ( · ) and α i denote a candidate activation function and a learnable parameter, respectively.

Original: Together with a gate function G ( · ) , the parameter α i determines the weight (or coefficient) of its corresponding candidate activation function.
Revised: Together with a gate function G ( · ) , the parameter α i determines the weight (or coefficient) of its corresponding candidate activation function.

Original: We use the softmax as the gate function by default.
Revised: We use the softmax as the gate function by default.

Original: In this case, the search space is the convex hull of the set of candidate activation functions.
Revised: In this case, the search space is the convex hull of the set of candidate activation functions.

Original: When the weights are fixed as one-hot vectors, the proposed PIAC is reduced to its candidate function.
Revised: When the weights are fixed as one-hot vectors, the proposed PIAC is reduced to its candidate function.

Original: When the weights are learnable, a specialized novel activation function can be efficiently searched for different PDE systems thanks to the continuous parameterization.
Revised: When the weights are learnable, a specialized novel activation function can be efficiently searched for different PDE systems thanks to the continuous parameterization.

Original: We employ the PIAC by replacing all the activation functions in PINNs.
Revised: We employ the PIAC by replacing all the activation functions in PINNs.

Original: It can be applied in a neuron-wise manner where learnable parameters are allowed to be vary across neurons, or in a layer-wise manner where the parameters are shared for all neurons in one layer.
Revised: It can be applied in a neuron-wise manner where learnable parameters are allowed to be vary across neurons, or in a layer-wise manner where the parameters are shared for all neurons in one layer.

Original: The search space of PIAC is built upon the set of candidate functions F = { σ 1 , σ 2 , ..., σ N } .
Revised: The search space of PIAC is built upon the set of candidate functions F = { σ 1 , σ 2 , ..., σ N } .

Original: Conceptually, we build this candidate function set with simple elementary functions and commonly-used activation functions.
Revised: Conceptually, we build this candidate function set with simple elementary functions and commonly-used activation functions.

Original: First, we can embed prior knowledge about the PDE system into the networks by including appropriate elementary functions.
Revised: First, we can embed prior knowledge about the PDE system into the networks by including appropriate elementary functions.

Original: For example, the periodicity and exponential decay are commonly seen in physics equations but rarely observed in image data.
Revised: For example, the periodicity and exponential decay are commonly seen in physics equations but rarely observed in image data.

Original: As shown in Section 3.1, they are difficult to model by neural networks with normal activation functions such as tanh .
Revised: As shown in Section 3.1, they are difficult to model by neural networks with normal activation functions such as tanh .

Original: We can alleviate the modeling difficulty by taking advantage of sinusoidal and exponential functions in our search space.
Revised: We can alleviate the modeling difficulty by taking advantage of sinusoidal and exponential functions in our search space.

Original: We can also add those elementary functions observed in the initial condition (see the convection equation) or the force term (see Poisson’s equation) to help the optimization of corresponding loss function.
Revised: We can also add those elementary functions observed in the initial condition (see the convection equation) or the force term (see Poisson’s equation) to help the optimization of corresponding loss function.

Original: Second, we add most of commonly-used activation functions into this function set.
Revised: Second, we add most of commonly-used activation functions into this function set.

Original: By doing this, we ensure the diversity of our search space while avoiding the repetitive evaluations of each candidate activation function for different problems.
Revised: By doing this, we ensure the diversity of our search space while avoiding the repetitive evaluations of each candidate activation function for different problems.

Original: Moreover, the search space can be further enhanced with the adaptive slope, which is proposed to improve the convergence rate of standard activation functions for PINNs (Jagtap et al., 2020b;a).
Revised: Moreover, the search space can be further enhanced with the adaptive slope, which is proposed to improve the convergence rate of standard activation functions for PINNs (Jagtap et al., 2020b;a).

Original: This can be achieved by simply introducing a learnable scaling factor β
Revised: This can be achieved by simply introducing a learnable scaling factor β

Original: i for each candidate function σ i , which can be represented as
Revised: i for each candidate function σ i , which can be represented as

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, We evaluate the performance of our method on several benchmarks of time-dependent
Revised: In this section, We evaluate the performance of our method on several benchmarks of time-dependent

Original: PDEs.
Revised: PDEs.

Original: Then, we provide detailed ablation results to analyse the proposed PIAC.
Revised: Then, we provide detailed ablation results to analyse the proposed PIAC.

Original: Finally, we present an intuitive interpretation of the performance gain of our method from the neural tangent kernel perspective.
Revised: Finally, we present an intuitive interpretation of the performance gain of our method from the neural tangent kernel perspective.

Original: More results on the 1D Poisson’s equation can be found in Appendix B.
Revised: More results on the 1D Poisson’s equation can be found in Appendix C.

Original: 4. E XPERIMENTS ON TIME - DEPENDENT PDE S
Revised: 4. E XPERIMENTS ON TIME - DEPENDENT PDE S

Original: We demonstrate the generalization ability of PIAC to solve various time-dependent PDEs, ranging from first-order linear PDE to fourth-order nonlinear PDE, including the convection equation, the
Revised: We demonstrate the generalization ability of PIAC to solve various time-dependent PDEs, ranging from first-order linear PDE to fourth-order nonlinear PDE, including the convection equation, the

Original: Burgers’ equation, the Allen-Cahn equation, the Korteweg–de Vries equation, and the Cahn-Hilliard equation.
Revised: Burgers’ equation, the Allen-Cahn equation, the Korteweg–de Vries equation, and the Cahn-Hilliard equation

Original: Note that for the linear convection problem, we consider the case of a high convection coefficient ( β =64), which is shown as a difficult problem for vanilla PINNs (Krishnapriyan et al., 2021).
Revised: Note that for the linear convection problem , we consider the case of a high convection coefficient ( β = , which is shown as a difficult problem for vanilla PINNs (Krishnapriyan et al

Original: The details of PDEs and experimental configurations can refer to Appendix C.
Revised: The details of PDEs and experimental configurations can refer to Appendix D

Original: Baselines.
Revised: Baselines and PIAC setups.

Original: We comparethe proposed PIAC with several commonly-used activation functions, including the sinusoidal functions ( sin ), the hyperbolic tangent function ( tanh ), the logistic function ( sigmoid ), the Softplus function ( Softplus ) (Dugas et al., 2000; Glorot et al., 2011), the Exponential 
Revised: We compare PIAC with several commonly-used activation functions, including the sinusoidal functions ( sin ), the hyperbolic tangent function ( tanh ), the logistic function ( sigmoid ), the Softplus function ( Softplus ), the Exponential Linear Unit ( ELU ), the Gaussian Error

Original: Linear Unit ( ELU ) (Clevert et al., 2015), the Gaussian Error Linear Unit ( GELU ) (Hendrycks &
Revised: Linear Unit ( GELU ) and the Swish function ( Swish ).

Original: Gimpel, 2016) and the Swish function ( Swish ) (Ramachandran et al., 2017).
Revised: We compare PIAC to other adaptive activation functions which could provide higher-order derivatives, including SLAF Goyal et al. PAU Molina et al.

Original:  The details of each activation function can refer to Appendix A. We also compare PIAC with standard activation functions with the layer-wise adaptive slopes (Jagtap et al., 2020a).PIAC setups.
Revised: and ACON Ma et al. The details of each activation function can refer to Appendix A and Appendix B. We also compare PIAC with standard activation functions with the layer-wise adaptive slopes (Jagtap et al., 2020a).

Original: We employ the PIAC in a layer-wise manner by default.
Revised: We employ the PIAC in a layer-wise manner by default.

Original: We set the candidate function set F as { sin , tanh , GELU , Swish , Softplus } .
Revised: We set the candidate function set F as { sin , tanh , GELU , Swish , Softplus } .

Original: The learnable parameters { α i } Ni =1 are initialized as zeros and optimized jointly with the weights and biases of neural networks.
Revised: The learnable parameters { α i } Ni =1 are initialized as zeros and optimized jointly with the weights and biases of PINNs.

Original: The scaling factors { β i } Ni =1 are initialized as ones and can be fixed or learnable.
Revised: The scaling factors { β i } Ni =1 are initialized as ones and can be fixed or learnable.

Original: 4.1. M AIN RESULTS
Revised: 4.1. M AIN RESULTS

Original: Comparisons with fixed activation functions.
Revised: Comparisons with fixed activation functions.

Original: Table 3 presents the L 2 relative error of PIAC and seven standard activation functions on five time-dependent PDEs.
Revised: Table 3 presents the L 2 relative error of PIAC and seven standard activation functions on five time-dependent PDEs.

Original: One can observe that PIAC outperforms standard activation functions on all PDEs.
Revised: One can observe that PIAC outperforms standard activation functions on all PDEs.

Original: For example, PIAC reduces the error rate by 83% on the convetion equation ( 0 .
Revised: For example, PIAC reduces the error rate by 83% on the convetion equation ( 0 .

Original: 06 ± 0 .
Revised: 06 ± 0 .

Original: 03% vs. 0 .
Revised: 03% vs. 0 .

Original: 36 ± 0 .
Revised: 36 ± 0 .

Original: 15% ), by 46% on the KdV equation 
Revised: 15% ), by 46% on the KdV equation ( 0 .

Original: [Table] [Table] [Table]
Revised: 34 ± 0 .

Original: 05% vs. 0 .
Revised: 05% vs. 0 .

Original: 64 ± 0 .
Revised: 64 ± 0 .

Original: 34% ), and by 56% on the Cahn-Hilliard equation ( 0 .
Revised: 34% ), and by 56% on the Cahn-Hilliard equation ( 0 .

Original: 50 ± 0 .
Revised: 50 ± 0 .

Original: 15% vs. 1 . 01 ± 1 . 27% ), compared with optimal standard activation function of each problem.
Revised: 15% vs. 1 . 01 ± 1 . 27% ), compared with optimal standard activation function of each problem.

Original: More importantly, PIAC performs consistently over all these problems, while standard activation functions suffer from high variance in the performance of different PDEs. 
Revised: More importantly, PIAC performs consistently over all these problems, while standard activation functions suffer from high variance in the performance of different PDEs. [Table] With AS [Table]

Original: Comparisons with activation functions with adaptive slopes.
Revised: Comparisons with other adaptive activation functions.

Revised: The results of different adaptive activation functions are shown in Table 3.

Revised: SLAF achieves performance comparable to the best standard activation function on the first-order convection equation

Revised: [Table]

Revised: but does not produce accurate predictions for other higher-order PDEs.

Revised: Although SLAF shares a similar formulation with PIAC, its polynomial bases can cause vanishing or exploding gradients due to the high order powers in its derivatives.

Revised: This problem might hinder the optimization of higher-order PDE-based constraints.

Revised: PAU suffers from instability of training (see Appendix Figure 2) and performs poorly on all PDEs.

Revised: We attribute the training instability of PAU to its discontinuous derivatives, which arise from the absolute value in the denominator of its formulation.

Revised: ACON achieves stable performance but does not surpass the best standard activation function in each problem due to the limited flexibility of its formulation.

Revised: The proposed PIAC outperforms these methods thanks to the diverse and smooth candidate functions, which can be used to incorporate the prior knowledge of different physics systems.

Revised: Comparisons with activation functions with adaptive slopes.

Original: We also compare PIAC with three standard activation function enhanced with adaptive slopes (AS).
Revised: We also compare PIAC with three standard activation function enhanced with adaptive slopes (AS).

Original: We find PIAC without AS achieves competitive or even better results.
Revised: We find PIAC without AS achieves competitive or even better results.

Original: For example, PIAC without AS outperforms the tanh with AS on the Allen-Cahn equation and outperforms the sin with AS on the convection equation.
Revised: For example, PIAC without AS outperforms the tanh with AS on the Allen-Cahn equation and outperforms the sin with AS on the convection equation.

Original: The performance of PIAC can be further improved by incorporating adaptive slopes into the search space.
Revised: The performance of PIAC can be further improved by incorporating adaptive slopes into the search space.

Original: As a showcase, PIAC with AS obtains a performance gain by 64% compared to the GELU with AS on the Cahn-Hilliard equation and by 82% compared to the sin with AS on the convection equation.
Revised: As a showcase, PIAC with AS obtains a performance gain by 64% compared to the GELU with AS on the Cahn-Hilliard equation and by 82% compared to the sin with AS on the convection equation.

Original: Furthermore, our method has a better generalization ability and achieves the lowest average error rate over all problems (0.29% with AS and 0.37% without AS), while the adaptive slope technique cannot alleviate the need of the manual selection of activation functions.
Revised: Furthermore, our method has a better generalization ability and achieves the lowest average error rate over all problems (0.29% with AS and 0.37% without AS), while the adaptive slope technique cannot alleviate the need of the manual selection of activation functions.

Original: Visualization of the learned PIAC.
Revised: Visualization of the learned PIAC.

Original: We visualize the learned PIAC in  Figure 2.
Revised: We show the learned PIAC in Appendix Figure 3.

Original: One can observethat the learned activation functions are different from candidate functions.
Revised: One can observe the learned activation functions are different from candidate functions.

Original: We also find differences in learned functions of different layers.
Revised: We also find differences in learned functions of different layers.

Original: For example, a deeper layer tends to has a larger weight of sin function in the cases of convection equation.
Revised: For example, a deeper layer tends to has a larger weight of sin function in the cases of convection equation.

Original: Moreover, the learned activation functions vary across problems, which conforms to our motivation to learn specialized activation function for different PDE systems.
Revised: Moreover, the learned activation functions vary across problems, which conforms to our motivation to learn specialized activation functions for different PDE systems.

Original: The prediction results of PIAC for all problems can refer to Appendix Figure 3.
Revised: The prediction results of PIAC for all problems can refer to Appendix Figure 4.

Original: 4. A BLATION STUDY
Revised: 4. A BLATION STUDY

Original: We ablate different design choices to provide a better understanding of the proposed method.
Revised: We ablate different design choices to provide a better understanding of the proposed method.

Original: For all ablation experiments, we deploy layer-wise PIAC to solve the convection equation.
Revised: For all ablation experiments, we deploy layer-wise PIAC to solve the convection equation.

Original: The adaptive slope is deactivated for a clear comparison.
Revised: The adaptive slope is deactivated for a clear comparison.

Revised: [Table]

Revised: [Table]

Original: Comparing different candidate function set.
Revised: Comparing different candidate function set.

Original: Here we study the influence of candidate function set by gradually increasing its size as shown in Table 4a.
Revised: Here we study the influence of candidate function set by gradually increasing its size as shown in Table 4a.

Original: We begin with the two most commonlyused activation functions, the sin and tanh functions.
Revised: We begin with the two most commonlyused activation functions, the sin and tanh functions.

Original: In this case, PIAC is shown to achieve competitive result compared with its two candidate functions.
Revised: In this case, PIAC is shown to achieve competitive result compared with its two candidate functions.

Original: Based on that, we observe that adding new activation functions could lead to a better performance.
Revised: Based on that, we observe that adding new activation functions could lead to a better performance.

Original: Surprisingly, the introduction of GELU can still improve the performance despite its poor results on this problem.
Revised: Surprisingly, the introduction of GELU can still improve the performance despite its poor results on this problem.

Original: We argue that this performance gain arises from the enlarged search space.
Revised: We argue that this performance gain arises from the enlarged search space.

Original: We notice that the performance tends to saturate as the number of candidate functions increases.
Revised: We notice that the performance tends to saturate as the number of candidate functions increases.

Original: This indicates that PIAC has a good robustness to the choice of candidate function set if this set contains activation functions with sufficient diversity.
Revised: This indicates that PIAC has a good robustness to the choice of candidate function set if this set contains activation functions with sufficient diversity.

Original: Notably, the additional parameters of PIAC is negligible compared with total number of weights.[Table] [Table]
Revised: Notably, the additional parameters of PIAC is negligible compared with total number of weights.

Original: Comparing different gate functions.
Revised: Comparing different gate functions.

Original: We also compare the softmax function with other gate functions in Table 4b.
Revised: We also compare the softmax function with other gate functions in Table 4b.

Original: The identity function G ( α i ) =
Revised: The identity function G ( α i ) =

Original: α i includes all linear combinations of candidate activation functions into the search space; while the sigmoid function G ( α i )
Revised: α i includes all linear combinations of candidate activation functions into the search space; while the sigmoid function G ( α i )

Original: = 1 / (1 + exp( − α i )) restricts the coefficients to be between 0 and 1.
Revised: = 1 / (1 + exp( − α i )) restricts the coefficients to be between 0 and 1.

Original: We also consider the L 1 -normalization G ( α i )
Revised: We also consider the L 1 -normalization G ( α i )

Original: =
Revised: =

Original: α i / (cid:80) N j =1 | α j | , which allows for negative coefficients and keeps the competition between candidate functions.
Revised: α i / (cid:80) Nj =1 | α j | , which allows for negative coefficients and keeps the competition between candidate functions.

Original: The trainable { α i } Ni =1 are initialized as zeros for sigmoid and softmax functions and as 1 /N for identity and L 1 -normalization functions, where N denotes the number of candidate functions.
Revised: The trainable { α i } Ni =1 are initialized as zeros for sigmoid and softmax functions and as 1 /N for identity and L 1 -normalization functions, where N denotes the number of candidate functions.

Original: One can observe that the results of softmax function are more stable and more accurate.
Revised: One can observe that the results of softmax function are more stable and more accurate.

Original: This implies proper restrictions of the search space can work as a regularization to the learning of PIAC.
Revised: This implies proper restrictions of the search space can work as a regularization to the learning of PIAC.

Original: Effectiveness of learnable coefficients of PIAC.
Revised: Effectiveness of learnable coefficients of PIAC.

Original: To demonstrate that the learnable coefficients of
Revised: To demonstrate that the learnable coefficients of

Original: PIAC is effective and necessary, we compare PIAC to two variants with fixed coefficients.
Revised: PIAC is effective and necessary, we compare PIAC to two variants with fixed coefficients.

Original: The first one initializes its parameters as zeros and the second one initializes its parameters from a standard normal distribution.
Revised: The first one initializes its parameters as zeros and the second one initializes its parameters from a standard normal distribution.

Original: As shown in Table 4b, the automatically learned coefficients outperforms evenly distributed coefficients and random sampled coefficients by a large margin.
Revised: As shown in Table 4b, the automatically learned coefficients outperforms evenly distributed coefficients and random sampled coefficients by a large margin.

Original: Furthermore, one can observe that the performance gap between the random and constant initialization is eliminated if these parameters are trainable, which also demonstrates the effectiveness of PIAC’s optimization.
Revised: Furthermore, one can observe that the performance gap between the random and constant initialization is eliminated if these parameters are trainable, which also demonstrates the effectiveness of PIAC’s optimization.

Original: 4. U NDERSTANDING PIAC THROUGH THE LENS OF NEURAL TANGENT KERNEL
Revised: 4. U NDERSTANDING PIAC THROUGH THE LENS OF NEURAL TANGENT KERNEL

Original: The neural tangent kernel is proposed to describe the evolution of neural networks (Jacot et al., 2018).
Revised: The neural tangent kernel is proposed to describe the evolution of neural networks (Jacot et al., 2018).

Original: Its eigenvalues can be leveraged to analyse the rate of convergence as shown in previous works (Jacot et al., 2018; Tancik et al., 2020; Wang et al., 2022).
Revised: Its eigenvalues can be leveraged to analyse the rate of convergence as shown in previous works (Jacot et al., 2018; Tancik et al., 2020; Wang et al., 2022).

Original: We show that the introduction of PIAC makes the NTK learnable.
Revised: We show that the introduction of PIAC makes the NTK learnable.

Original: Empirically, we observe the optimization of PIAC adapts the
Revised: Empirically, we observe the optimization of PIAC adapts the

Original: NTK’s eigenvalue spectrum to the underlying PDE system and leads to an improvement on the average eigenvalue, which partially explains the stable and fast convergence of PIAC.
Revised: NTK’s eigenvalue spectrum to the underlying PDE system and leads to an improvement on the average eigenvalue, which partially explains the stable and fast convergence of PIAC.

Original: Brief introduction of NTK.
Revised: Brief introduction of NTK.

Original: Specially, the neural tangent kernel of a L -layer fully-connected network u ( L ) with parameters θ is defined as
Revised: Specially, the neural tangent kernel of a L -layer fully-connected network u ( L ) with parameters θ is defined as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where x and x ′ are two inputs.
Revised: where x and x ′ are two inputs.

Original: As demonstrated in Jacot et al.
Revised: As demonstrated in Jacot et al.

Original: (2018), in the infinite-width limit and under suitable conditions, the NTK Θ ( L ) converges to a deterministic kernel Θ ( L ) ∞ at initialization and stays constant during the training.
Revised: (2018), in the infinite-width limit and under suitable conditions, the NTK Θ ( L ) converges to a deterministic kernel Θ ( L ) ∞ at initialization and stays constant during the training.

Original: This limiting kernel can be represented as
Revised: This limiting kernel can be represented as

Original: [Equation]
Revised: [Equation]

Original: Note Σ (1) ( x , x ′ )
Revised: Note Σ (1) ( x , x ′ )

Original: = 1 n 0 x T x ′ + β 2 .
Revised: = 1 n 0 x T x ′ + β 2 .

Original: The expectation in Eq.(13)is taken with respect to a centered
Revised: The expectation in Eq.(13)is taken with respect to a centered

Original: Gaussian process f with covariance Σ ( l ) ; n 0 is the dimension of input x and β is a scaling factor; σ and ˙ σ denote the activation function and its derivative, respectively.
Revised: Gaussian process f with covariance Σ ( l ) ; n 0 is the dimension of input x and β is a scaling factor; σ and ˙ σ denote the activation function and its derivative, respectively.

Original: Note that the covariance Σ ( l ) depends on the choice of activation functions and so dose the NTK Θ ( L ) ∞ .
Revised: Note that the covariance Σ ( l ) depends on the choice of activation functions and so dose the NTK Θ ( L ) ∞ .

Original: PIAC and the learnable NTK.
Revised: PIAC and the learnable NTK.

Original: For simplicity, we derive the NTK of standard neural networks with PIAC.
Revised: For simplicity, we derive the NTK of standard neural networks with PIAC.

Original: The NTK of PINNs with PIAC can be derived in a similar way.
Revised: The NTK of PINNs with PIAC can be derived in a similar way.

Original: Firstly, we assume that the optimization of neural networks with PIAC can be decomposed into two phases, where we learn the coefficients of PIAC in the first phase and then train the parameters of neural network in the second phase.
Revised: Firstly, we assume that the optimization of neural networks with PIAC can be decomposed into two phases, where we learn the coefficients of PIAC in the first phase and then train the parameters of neural network in the second phase.

Original: This assumption is reasonable as the number of parameters of PIAC is far less than those of networks and they quickly converge at the early stage of training.
Revised: This assumption is reasonable as the number of parameters of PIAC is far less than those of networks and they quickly converge at the early stage of training.

Original: Empirically, we observe that a short warmup (2.5% of the whole schedule) is sufficient for PIAC to learn suitable activation functions and to achieve competitive performance compared with the counterpart whose coefficients are updated during the whole schedule (0.58% vs. 0.60% on the Allen-Cahn equation).
Revised: Empirically, we observe that a short warmup (2.5% of the whole schedule) is sufficient for PIAC to learn suitable activation functions and to achieve competitive performance compared with the counterpart whose coefficients are updated during the whole schedule (0.58% vs. 0.60% on the Allen-Cahn equation).

Original: By decoupling the updates of PIAC and networks, we find the second optimization phase is equivalent to the training of a standard network with a learned activation function, whose NTK is derived as 
Revised: By decoupling the updates of PIAC and networks, we find the second optimization phase is equivalent to the training of a standard network with a learned activation function, whose NTK is derived as ¯Θ

Original: [Equation]
Revised: [Equation]

Original: We omit the factor β for convenience.
Revised: We omit the factor β for convenience.

Original: One can observe that the covariance Σ ( l ) in Eq(13) is replaced with a weighted sum of Σ ( l ) σ i σ j , which is calculated with different combinations of candidate activation functions.
Revised: One can observe that the covariance Σ ( l ) in Eq(13) is replaced with a weighted sum of Σ ( l ) σ i σ j , which is calculated with different combinations of candidate activation functions.

Original: Note that the weights are learned in the first training phase.
Revised: Note that the weights are learned in the first training phase.

Original: To conclude, the introduction of PIAC leads to a learnable NTK which could be adapted to the underlying PDE.
Revised: To conclude, the introduction of PIAC leads to a learnable NTK which could be adapted to the underlying PDE.

Original: The eigenvalue spectrum of NTK with PIAC.
Revised: The eigenvalue spectrum of NTK with PIAC.

Original: Through the optimization of PIAC, we can modify the limiting NTK ¯Θ ( L ) ∞ , which corresponds to a change in the convergence behavior of the neural network.
Revised: Through the optimization of PIAC, we can modify the limiting NTK ¯Θ ( L ) ∞ , which corresponds to a change in the convergence behavior of the neural network.

Original: We show empirically that the optimization of PIAC has an effect on the NTK’s eigenvalue spectrum.
Revised: We show empirically that the optimization of PIAC has an effect on the NTK’s eigenvalue spectrum.

Original: As shown in  Figure 2d, PIAC leads to a larger average eigenvalue compared with the best standard activation function on the Allen-Cahn equation (5859 vs. 2407), which implies a larger convergence rate (Wang et al., 2022).[Figure]
Revised: As shown in Appendix Figure 3d, PIAC leads to a larger average eigenvalue compared with the best standard activation function on the Allen-Cahn equation (5859 vs. 2407), which implies a larger convergence rate (Wang et al., 2022).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we reveal the high sensitivity of PINNs to the choice of activation functions and relate it to various characteristics of the underlying PDE system.
Revised: In this paper, we reveal the high sensitivity of PINNs to the choice of activation functions and relate it to various characteristics of the underlying PDE system.

Original: To avoid the inefficient manual selection of activation functions when solving different PDEs, we sought to learn specialized activation functions automatically for PINNs.
Revised: We sought to learn specialized activation functions automatically for PINNs to avoid the inefficient manual selection of activation functions and to alleviate the optimization difficulty of PINNs.

Original: The proposed physics-informed activation function is presented as learnable combinations of a set of candidate functions, whose coefficients can be adapted to the governing PDEs.
Revised: The proposed physics-informed activation function is presented as learnable combinations of a set of candidate functions, whose coefficients can be adapted to the governing PDEs.

Original: Intuitively, we show that PIAC makes the neural tangent kernel of PINNs learnable, which partially explains the performance improvement of PIAC.
Revised: Intuitively, we show that PIAC makes the neural tangent kernel of PINNs learnable, which partially explains the performance improvement of PIAC.

Original: Extensive experiments on a series of challenging benchmarks demonstrate the effectiveness and generalization ability of PIAC.
Revised: Extensive experiments on a series of challenging benchmarks demonstrate the effectiveness and generalization ability of PIAC.

Original: A S TANDARD ACTIVATION FUNCTIONS
Revised: A S TANDARD ACTIVATION FUNCTIONS

Original: We compare the proposed PIAC with several standard activation functions for solving a series of
Revised: We compare the proposed PIAC with several standard activation functions for solving a series of

Original: PDEs.
Revised: PDEs.

Original: The details are shown as follows.
Revised: The details are shown as follows.

Original: • sin :
Revised: • sin :

Original: [Equation]
Revised: [Equation]

Original: where scaling factor β is set to 1 as default.
Revised: where scaling factor β is set to 1 as default.

Original: • exp :
Revised: • exp :

Original: [Equation]
Revised: [Equation]

Original: where β is set to 1 as default.
Revised: where β is set to 1 as default.

Original: For the P1 setting of 1D Poisson’s equation, β is set to 0.
Revised: For the P1 setting of 1D Poisson’s equation, β is set to 0.

Original: for better performance.
Revised: for better performance.

Original: • tanh :
Revised: • tanh :

Original: [Equation]
Revised: [Equation]

Original: • sigmoid :
Revised: • sigmoid :

Original: [Equation]
Revised: [Equation]

Original: • Softplus (Dugas et al., 2000; Glorot et al., 2011):
Revised: • Softplus (Dugas et al., 2000; Glorot et al., 2011):

Original: [Equation]
Revised: [Equation]

Original: where β
Revised: where β

Original: = 1 .
Revised: = 1 .

Original: Softplus can be regard as a smooth version of ReLU.
Revised: Softplus can be regard as a smooth version of ReLU.

Original: • the Exponential Linear Unit ( ELU ) (Clevert et al
Revised: • the Exponential Linear Unit ( ELU ) (Clevert et al.

Original: [Equation]
Revised: [Equation]

Original: where α = 1 .
Revised: [Equation]

Original: • the Gaussian Error Linear Unit ( GELU ) (Hendrycks & Gimpel, 2016):
Revised: • the Gaussian Error Linear Unit ( GELU ) (Hendrycks & Gimpel, 2016):

Original: [Equation]
Revised: [Equation]

Original: where Φ( x ) is the cumulative distribution function of the standard Gaussian distribution.
Revised: where Φ( x ) is the cumulative distribution function of the standard Gaussian distribution.

Original: • Swish (Hendrycks & Gimpel, 2016;
Revised: • Swish (Hendrycks & Gimpel, 2016;

Original: Ramachandran et al., 2017;
Revised: Ramachandran et al.

Original:  Elfwing et al., 2018):
Revised: ; Elfwing et al [Equation]

Original: [Equation]
Revised: [Equation]

Original: B E XPERIMENTS ON 1D P OISSON ’ S EQUATION
Revised: B A DAPTIVE ACTIVATION FUNCTIONS

Revised: We further compare the proposed PIAC with other adaptive activation functions which could provide higher-order derivatives. The details are shown as follows. • The SLAF Goyal et al.

Revised: (2019) proposes to learn activation functions with Taylor polynomial bases, which is presented as:

Revised: [Equation]

Revised: where m is set to 5 in our experiments.

Revised: The learnable parameters { a i } mi =0 are initialized from a normal distribution.

Revised: • The PAU Molina et al.

Revised: (2019) is presented as the rational functions of the form

Revised: [Equation]

Revised: where m is set to 5 and n is set to 4 following the default setting in Molina et al.

Revised: The absolute value is to avoid a zero-valued denominator.

Revised: The learnable parameters { a i } mi =0 and { b i } ni =1 are initialized from a normal distribution.

Revised: • The ACON Ma et al.

Revised: (2021) is a smooth approximator to the general Maxout family activation functions, which is presented as

Revised: [Equation]

Revised: where p 1 , p 2 and β are learnable parameters and initialized from a normal distribution.

Revised: C E XPERIMENTS ON 1D P OISSON ’ S EQUATION

Original: Experiment setups.
Revised: Experiment setups.

Original: For each problem, we use a 4-layer multilayer perceptron (MLP) with the hidden dimension set to 16.
Revised: For each problem, we use a 4-layer multilayer perceptron (MLP) with the hidden dimension set to 16.

Original: The model is initialized by the Xavier initialization (Glorot & Bengio, 2010) and trained for 20000 iterations using the Adam optimizer (Kingma & Ba, 2014).
Revised: The model is initialized by the Xavier initialization (Glorot & Bengio, 2010) and trained for 20000 iterations using the Adam optimizer (Kingma & Ba, 2014).

Original: We initialize the learning rate as 1e-3 and adapt it with a half-cycle cosine decay schedule.
Revised: We initialize the learning rate as 1e-3 and adapt it with a half-cycle cosine decay schedule.

Original: We use 32 collocation points for both P1 and P2 .
Revised: We use 32 collocation points for both P1 and P2 .

Original: For each problem, we experiment with 8 standard activation functions as described in Appendix A.
Revised: For each problem, we experiment with 8 standard activation functions as described in Appendix A.

Original: We repeat each experiment 10 times and report the mean and standard deviation of the L 

Revised: We repeat each experiment 10 times and report the mean and standard deviation of the L

Original: relative error.
Revised: relative error.

Original: Comparisons between PIAC and standard activation functions.
Revised: Comparisons between PIAC and standard activation functions.

Original: To demonstrate the effectiveness of PIAC, we use it to learn combinations of the sin and exp functions for solving the 1D
Revised: To demonstrate the effectiveness of PIAC, we use it to learn combinations of the sin and exp functions for solving the 1D

Original: Poisson’s equation as described in Section 3.1.
Revised: Poisson’s equation as described in Section 3.1.

Original: As shown in Table 2, although the sin and exp functions can achieve the lowest L 2 relative error on P1 and P2 respectively, they perform poorly on the other problem.
Revised: As shown in Table 2, although the sin and exp functions can achieve the lowest L 2 relative error on P1 and P2 respectively, they perform poorly on the other problem.

Original: We expect the PIAC could adapt its attention over these two candidate functions according to the problem.
Revised: We expect the PIAC could adapt its attention over these two candidate functions according to the problem.

Original: We define F = { sin , exp } .
Revised: We define F = { sin , exp } .

Original: The trainable parameters { α i } Ni =1 are initialized as zeros, which implies equal attention over all candidate functions.
Revised: The trainable parameters { α i } Ni =1 are initialized as zeros, which implies equal attention over all candidate functions.

Original: The results of PIAC and its two candidate functions are presented in Table 5a.
Revised: The results of PIAC and its two candidate functions are presented in Table 5a.

Original: It can be observed that
Revised: It can be observed that

Original: PIAC can achieve comparable accuracy with the sin on P1 and outperform the exp on P2 , despite the large performance gap between its two candidate functions.
Revised: PIAC can achieve comparable accuracy with the sin on P1 and outperform the exp on P2 , despite the large performance gap between its two candidate functions.

Original: We also report the values of learned coefficients in Table 5b.
Revised: We also report the values of learned coefficients in Table 5b.

Original: For P1 , PIAC pays more attention on the sin function as expected.
Revised: For P1 , PIAC pays more attention on the sin function as expected.

Original: For P2 , we observe that the learned activation functions are different across layers.
Revised: For P2 , we observe that the learned activation functions are different across layers.

Original: The exp is preferred only on the third layer, which still leads a performance gain compared with standard activation functions.
Revised: The exp is preferred only on the third layer, which still leads a performance gain compared with standard activation functions.

Original: Overall, the results indicate that the proposed method can learn useful combination coefficients by considering the effectiveness of candidate functions for different problems.
Revised: Overall, the results indicate that the proposed method can learn useful combination coefficients by considering the effectiveness of candidate functions for different problems.

Revised: on the 1D Poisson’s equation.

Revised: L 2 relative error (%) is reported.

Revised: We also report the learned coefficients of sin , α sin .

Revised: Note that the coefficients of exp can be obtained by α exp = 1 − α sin , since the softmax gate function is used.

Original: [Table]
Revised: [Table]

Original: C E XPERIMENTS ON TIME - DEPENDENT PDE S C. P ROBLEM FORMULATIONS
Revised: D E XPERIMENTS ON TIME - DEPENDENT PDE S D. P ROBLEM FORMULATIONS

Original: Convection equation.
Revised: Convection equation.

Original: Consider a one-dimensional linear convection problem as
Revised: Consider a one-dimensional linear convection problem as

Original: [Equation]
Revised: [Equation]

Original: The solution of this problem is periodic over time, whose period is inversely proportional to the convection coefficient β .
Revised: The solution of this problem is periodic over time, whose period is inversely proportional to the convection coefficient β .

Original: Previous work (Krishnapriyan et al., 2021) finds it difficult for vanilla PINNs to learn the solution with a large β and proposes to train the PINNs with curriculum learning.
Revised: Previous work (Krishnapriyan et al., 2021) finds it difficult for vanilla PINNs to learn the solution with a large β and proposes to train the PINNs with curriculum learning.

Original: To be specific, the β is initialized as a small value and gradually increases during the training.
Revised: To be specific, the β is initialized as a small value and gradually increases during the training.

Original: Despite its effectiveness, the training cost is increased significantly.
Revised: Despite its effectiveness, the training cost is increased significantly.

Original: We show that accurate prediction can be achieved with a normal training strategy by simply using a suitable activation function.
Revised: We show that accurate prediction can be achieved with a normal training strategy by simply using a suitable activation function.

Original: Moreover, the performance can be further improved by our method.
Revised: Moreover, the performance can be further improved by our method.

Original: Here we set the convection coefficient β to 64.
Revised: Here we set the convection coefficient β to 64.

Original: We use N ic = 512 training points for the initial condition and N bc = 200 for the boundary condition.
Revised: We use N ic = 512 training points for the initial condition and N bc = 200 for the boundary condition.

Original: The residual loss L r is computed on N r = 6400 randomly sampled collocation points.
Revised: The residual loss L r is computed on N r = 6400 randomly sampled collocation points.

Original: Burgers’ equation.
Revised: Burgers’ equation.

Original: We consider the one-dimensional Burgers’ equation:
Revised: We consider the one-dimensional Burgers’ equation:

Original: [Equation]
Revised: [Equation]

Original: where ν = 0 .
Revised: where ν = 0 .

Original: 01 /π .
Revised: 01 /π .

Original: Following (Lu et al., 2021a), we uniformly sample N ic = 

Revised: Following (Lu et al., 2021a), we uniformly sample N ic =

Original: and N bc =points for initial and boundary training data, respectively.
Revised: and N bc =points for initial and boundary training data, respectively.

Original: We compute the residual loss L r on
Revised: We compute the residual loss L r on

Original: N r = 4800 collocation points, which are randomly selected from the spatial-temporal domain.
Revised: N r = 4800 collocation points, which are randomly selected from the spatial-temporal domain.

Original: Allen-Cahn equation.
Revised: Allen-Cahn equation.

Original: We next consider the Allen-Cahn equation as
Revised: We next consider the Allen-Cahn equation as

Original: [Equation]
Revised: [Equation]

Original: where D = 0 .
Revised: where D = 0 .

Original: 001 following the setting in (Yu et al., 2022).
Revised: 001 following the setting in (Yu et al., 2022).

Original: We choose the following surrogate of solution to enforce the initial and boundary conditions:
Revised: We choose the following surrogate of solution to enforce the initial and boundary conditions:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where u ′ ( t, x ; θ ) is the output of neural networks.
Revised: where u ′ ( t, x ; θ ) is the output of neural networks.

Original: We 
Revised: We use N r = 8000 collocation points.

Original: Korteweg–de Vries equation.
Revised: Korteweg–de Vries equation.

Original: We consider the Korteweg–de Vries (KdV) equation as
Revised: We consider the Korteweg–de Vries (KdV) equation as

Original: [Equation]
Revised: [Equation]

Original: 27) where λ 1 = 1 and λ 2 = 0 . {+
+}
Revised: where λ 1 = 1 and λ 2 = 0 . {+
+}

Original: 0025 following the setting in (Raissi et al., 2019a).
Revised: following the setting in (Raissi et al., 2019a).

Original: We choose the following surrogate of solution to enforce the initial conditions:
Revised: We choose the following surrogate of solution to enforce the initial conditions:

Original: [Equation]
Revised: [Equation]

Original: where u ′ ( t, x ; θ ) is the output of neural networks.
Revised: where u ′ ( t, x ; θ ) is the output of neural networks.

Original: We use N bc = 200 training points for the boundary condition and N r = 8000 collocation points for the residual loss L r .
Revised: We use N bc = 200 training points for the boundary condition and N r = 8000 collocation points for the residual loss L r .

Original: Cahn-Hilliard equation.
Revised: Cahn-Hilliard equation.

Original: We consider the phase space representation of Cahn-Hilliard equation as
Revised: We consider the phase space representation of Cahn-Hilliard equation as

Original: [Equation]
Revised: [Equation]

Original: where λ 1 = 0 .
Revised: where λ 1 = 0 .

Original: 02 and λ 2 = 1 following the setting in (Mattey & Ghosh, 2022).
Revised: 02 and λ 2 = 1 following the setting in (Mattey & Ghosh, 2022).

Original: Previous work proposes a novel sequential training method to solve this strongly non-linear and high-order problem (Mattey & Ghosh, 2022).
Revised: Previous work proposes a novel sequential training method to solve this strongly non-linear and high-order problem (Mattey & Ghosh, 2022).

Original: We find that a normal training strategy can lead to a comparable result if we add more collocation points around t = 0 .
Revised: We find that a normal training strategy can lead to a comparable result if we add more collocation points around t = 0 .

Original: To be specific, we sample 4000 collocation points from time interval [0 , 0 . 05) and 8000 points from [0 . 05 , 1] .
Revised: To be specific, we sample 4000 collocation points from time interval [0 , 0 . 05) and 8000 points from [0 . 05 , 1] .

Original: We set N ic = 256 and N bc = 100 .
Revised: We set N ic = 256 and N bc = 100 .

Original: We show that PINNs under this training setting can work as a strong baseline and can be further improved by PIAC.
Revised: We show that PINNs under this training setting can work as a strong baseline and can be further improved by PIAC.

Original: C. E XPERIMENTAL CONFIGURATIONS
Revised: D. E XPERIMENTAL CONFIGURATIONS

Original: Experiment setups.
Revised: Experiment setups.

Original: We train the PINNs by a two-stage optimization except the convection equation.
Revised: We train the PINNs by a two-stage optimization except the convection equation.

Original: At the first stage, the model is trained for a certain number of iterations by Adam (Kingma &
Revised: At the first stage, the model is trained for a certain number of iterations by Adam (Kingma &

Original: Ba, 2014).
Revised: Ba, 2014).

Original: Then, the L-BFGS (Byrd et al., 1995) is used to train the network until convergence.
Revised: Then, the L-BFGS (Byrd et al., 1995) is used to train the network until convergence.

Original: The first stage is to provide a good initialization for the L-BFGS optimizer.
Revised: The first stage is to provide a good initialization for the L-BFGS optimizer.

Original: In the case of convection equation, only the Adam optimizer is used.
Revised: In the case of convection equation, only the Adam optimizer is used.

Original: The detailed experimental configurations can be found in Table 6.
Revised: The detailed experimental configurations can be found in Table 6.

Original: The model is initialized by the Xavier initialization.
Revised: The model is initialized by the Xavier initialization.

Original: The learning rate is adapted with a half-cycle cosine decay schedule in the first stage with Adam optimizer.
Revised: The learning rate is adapted with a half-cycle cosine decay schedule in the first stage with Adam optimizer.

Original: We repeat each experiment 5 times and report the mean and standard deviation of the L 

Revised: We repeat each experiment 5 times and report the mean and standard deviation of the L

Original: relative error.
Revised: relative error.

Original: We run all experiments on one NVIDIA GeForce RTX 2080Ti GPU.
Revised: We run all experiments on one NVIDIA GeForce RTX 2080Ti GPU.

Original: [Table]
Revised: [Table]

Original: C. V ISUALIZATIONOF PIAC’ S PREDICTIONS
Revised: D. V ISUALIZATION

Revised: We show the training losses of different activation functions in Figure 2. One can observe SLAF seems to get stuck in local minima when solving the Burgers’ equation and the Allen-Cahn equation. We find PAU suffers from instability of training as its loss oscillates severely.

Revised: ACON achieves better performance than SLAF and PAU, but does not surpass tanh on both PDEs.

Revised: Compared to these learnable activation functions, the training loss of PIAC is more stable and converges faster.

Original: We visualize the exact solution, the prediction  of PIAC, and the absolute error between them for each PDE in Figure 
Revised: We present the learned activation functions of PIAC in Figure 3 and the prediction results of PIAC in Figure 4. We also visualize the eigenvalue spectrum of NTK with PIAC in Figure 3d, which shows a larger average eigenvalue compared with the best standard activation function on the Allen-Cahn equation. [Figure]

Original: [Figure]
Revised: [Figure]

Revised: t p u t

Revised: [Figure]

Revised: [Figure]

Original: equation
Revised: equation

Original: [Figure]
Revised: [Figure]

Original: The Allen-Cahn equation
Revised: The Allen-Cahn equation

Original: [Figure]
Revised: [Figure]