Original: Real-Valued Backpropagation is Unsuitable for Complex-Valued Neural Networks
Revised: Real-Valued Backpropagation is Unsuitable for Complex-Valued Neural Networks

Original: Abstract
Revised: Abstract

Original: Recently complex-valued neural networks have received increasing attention dueto successful applications in various tasks and the potential advantages of bettertheoretical properties and richer representational capacity.
Revised: Recently complex-valued neural networks have received increasing attention due to successful applications in various tasks and the potential advantages of better theoretical properties and richer representational capacity.

Original: However, the trainingdynamics of complex networks compared to real networks remains an open problem.
Revised: However, the training dynamics of complex networks compared to real networks remains an open problem.

Original: In this paper, we investigate the dynamics of deep complex networks duringreal-valued backpropagation in the infinite-width limit via neural tangent kernel(NTK).
Revised: In this paper, we investigate the dynamics of deep complex networks during real-valued backpropagation in the infinite-width limit via neural tangent kernel (NTK).

Original: We first extend the Tensor Program to the complex domain, to show thatthe dynamics of any basic complex network architecture is governed by its NTKunder real-valued backpropagation.
Revised: We first extend the Tensor Program to the complex domain, to show that the dynamics of any basic complex network architecture is governed by its NTK under real-valued backpropagation.

Original: Then we propose a way to investigate thecomparison of training dynamics between complex and real networks by studyingtheir NTKs.
Revised: Then we propose a way to investigate the comparison of training dynamics between complex and real networks by studying their NTKs.

Original: As a result, we surprisingly prove that for most complex activationfunctions, the commonly used real-valued backpropagation reduces the trainingdynamics of complex networks to that of ordinary real networks, thus eliminatingthe characteristics of complex-valued neural networks.
Revised: As a result, we surprisingly prove that for most complex activation functions, the commonly used real-valued backpropagation reduces the training dynamics of complex networks to that of ordinary real networks, thus eliminating the characteristics of complex-valued neural networks.

Original: Finally, we study the resultsnumerically and the experiments verify our theoretical findings.
Revised: Finally, we study the results numerically and the experiments verify our theoretical findings.

Original:  Introduction
Revised:  Introduction

Original: Recently complex-valued neural networks have been successfully applied to various tasks, suchas time-series prediction [Wisdom et al., 2016], computer vision [Trabelsi et al., 2018], signalprocessing [Yao et al., 2020].
Revised: Recently complex-valued neural networks have been successfully applied to various tasks, such as time-series prediction [Wisdom et al., 2016], computer vision [Trabelsi et al., 2018], signal processing [Yao et al., 2020].

Original: Compared to real-valued neural networks, it is shown that complexnetworks have the potential to provide richer representational capacity [Arjovsky et al., 2016], fasterlearning [Danihelka et al., 2016], better motivation and generalization for signal-related tasks [Hiroseand Yoshida, 2012, Tygert et al., 2016].
Revised: Compared to real-valued neural networks, it is shown that complex networks have the potential to provide richer representational capacity [Arjovsky et al., 2016], faster learning [Danihelka et al., 2016], better motivation and generalization for signal-related tasks [Hirose and Yoshida, 2012, Tygert et al., 2016].

Original: Theoretically, there have been significant advances forcomplex networks regarding the universal approximation property [Voigtlaender, 2020], criticalpoints [Nitta, 2002], local minima [Nitta, 2013] and separation results [Zhang et al., 2021].
Revised: Theoretically, there have been significant advances for complex networks regarding the universal approximation property [Voigtlaender, 2020], critical points [Nitta, 2002], local minima [Nitta, 2013] and separation results [Zhang et al., 2022].

Original: However, training deep complex networks has been challenging because of several non-intuitiveanalytical properties of complex algebra.
Revised: However, training deep complex networks has been challenging because of several non-intuitive analytical properties of complex algebra.

Original: Firstly, in practice, we often deal with a real-valued costfunction, which is non-analytic with respect to complex-valued parameters.
Revised: Firstly, in practice, we often deal with a real-valued cost function, which is non-analytic with respect to complex-valued parameters.

Original: Secondly, Liouville’stheorem asserts that every bounded and complex-differentiable function is a constant.
Revised: Secondly, Liouville’s theorem asserts that every bounded and complex-differentiable function is a constant.

Original: Thus almostall activation functions are non-analytic due to the preference for boundedness before the popularityof ReLU [Scardapane et al., 2018].
Revised: Thus almost all activation functions are non-analytic due to the preference for boundedness before the popularity of ReLU [Scardapane et al., 2018].

Original: Moreover, Voigtlaender [2020] has proved that the universalapproximation property holds only when using non-holomorphic activation functions.
Revised: Moreover, Voigtlaender [2020] has proved that the universal approximation property holds only when using non-holomorphic activation functions.

Original: Due to these reasons, different backpropagation algorithms in the complex domain were independentlyproposed for non-holomorphic networks [Bassey et al., 2021], mostly by optimizing the real andimaginary components separately.
Revised: Due to these reasons, different backpropagation algorithms in the complex domain were independently proposed for non-holomorphic networks [Bassey et al., 2021], mostly by optimizing the real and imaginary components separately.

Original: Recently, real-valued backpropagation [Nitta, 1997] is widelyused due to the convenience of utilizing the real-valued deep learning library [Arjovsky et al., 2016,
Revised: Recently, real-valued backpropagation [Nitta, 1997] is widely used due to the convenience of utilizing the real-valued deep learning library [Arjovsky et al., 2016,

Original: Trabelsi et al., 2018, Tan et al., 2020].
Revised: Trabelsi et al., 2018, Tan et al., 2020].

Original: It optimized the complex network just like a real network by computing partial derivatives of the cost with respect to the real and imaginary parts separately andachieved state-of-the-art performance.
Revised: It optimized the complex network just like a real network by computing partial derivatives of the cost with respect to the real and imaginary parts separately and achieved state-of-the-art performance.

Original: As a result, a natural and fundamental problem in complex-valued neural networks arises: Do complexnetworks have a different inductive bias from real networks during training?
Revised: As a result, a natural and fundamental problem in complex-valued neural networks arises: Do complex networks have a different inductive bias from real networks during training?

Original: Do complex networkstrained by gradient descent tend to learn different hypotheses from real networks?
Revised: Do complex networks trained by gradient descent tend to learn different hypotheses from real networks?

Original: However, to thebest of the authors’ knowledge, the training dynamics of backpropagation for complex networkscompared to real networks remains open.
Revised: However, to the best of the authors’ knowledge, the training dynamics of backpropagation for complex networks compared to real networks remains open.

Original: In this paper, we investigate the training dynamics of deep complex networks under real-valuedbackpropagation from neural tangent kernel (NTK) perspective
Revised: In this paper, we investigate the training dynamics of deep complex networks under real-valued backpropagation from neural tangent kernel (NTK) perspective

Original: [Jacot et al., 2018], which capturesthe optimization behavior of neural networks in the infinite-width limit.
Revised: [Jacot et al., 2018], which captures the optimization behavior of neural networks in the infinite-width limit.

Original: Then we provide a way toinvestigate the comparison of training dynamics between complex and real networks via their NTKs.
Revised: Then we provide a way to investigate the comparison of training dynamics between complex and real networks via their NTKs.

Original: As a result, we obtain informative results which may guide the algorithm selection and complexnetwork design in practice if people want to take full advantage of complex networks.
Revised: As a result, we obtain informative results which may guide the algorithm selection for complex networks in practice if people want to take full advantage of complex networks.

Original: Our contributions.
Revised: Our contributions.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions can be summarized as follows:

Original: • First, we extend the Tensor Program
Revised: • First, we extend the Tensor Program

Original: [Yang, 2020] to the complex domain and show that fora complex-valued neural network of any basic architecture in the infinite-width limit, thetraining dynamics of real-valued backpropagation is determined by kernel gradient descentwith a deterministic NTK at initialization.
Revised: [Yang, 2020] to the complex domain and show that for a complex-valued neural network of any basic architecture in the infinite-width limit, the training dynamics of real-valued backpropagation is determined by kernel gradient descent with a deterministic NTK at initialization.

Original: • Second, we investigate the comparison of training dynamics between complex and real networks based on their NTKs.
Revised: Second, we investigate the comparison of training dynamics between complex and real networks based on their NTKs.

Original: We surprisingly prove that the commonly used real-valued backpropagation reduces the training dynamics of complex-valued multi-layer perceptrons(MLPs) to that of ordinary real MLPs, thus eliminating the characteristics of complex-valued neural networks.
Revised: We surprisingly prove that the commonly used real-valued backpropagation reduces the training dynamics of complex-valued multi-layer perceptrons (MLPs) to that of ordinary real MLPs, thus eliminating the characteristics of complex-valued neural networks.

Original: This result holds for most commonly used complex activation functions, including all split activation functions such as C ReLU, C Sigmoid, C tanh; and part of magnitude-based holomorphic activation functions.
Revised: This result holds for most commonly used complex activation functions, including all split activation functions such as C ReLU, C Sigmoid, C tanh; and part of magnitude-based holomorphic activation functions.

Original: • Finally, we study the results numerically, and the experiments verifies our findings.
Revised: • Finally, we study the results numerically, and the experiments verifies our findings.

Original: Specifically, in several settings with different depths and various activation functions, the NTKs of complex networks converges to the NTKs of real networks as the widths grow.
Revised: Specifically, in several settings with different depths and various activation functions, the NTKs of complex networks converge to the NTKs of real networks as the widths grow.

Original: Organization.
Revised: Organization.

Original: We start with some preliminaries and notations about complex networks and neuraltangent kernels in Section 2.
Revised: We start with some preliminaries and notations about complex networks and neural tangent kernels in Section 2.

Original: In Section 3, we investigate the NTK of complex-valued neural networksof any architecture during real-valued backpropagation in the infinite-width limit.
Revised: In Section 3, we investigate the NTK of complex-valued neural networks of any architecture during real-valued backpropagation in the infinite-width limit.

Original: Section 4 firstlypresents the NTK of complex MLPs in any depth and then investigates the conditions that trainingdynamics of complex-valued MLPs reduce to that of ordinary real MLPs.
Revised: Section 4 firstly presents the NTK of complex MLPs in any depth and then investigates the conditions that training dynamics of complex-valued MLPs reduce to that of ordinary real MLPs.

Original: We verified our resultsempirically in Section 5.
Revised: We verify our results empirically in Section 5.

Original: Finally we discuss the related works and conclude the paper.
Revised: Finally we discuss the related works and conclude the paper.

Original: Due to thelimited space, all proofs are placed in the appendices.
Revised: Due to the limited space, all proofs are placed in the appendices.

Original:  Preliminaries2.1 Complex-Valued Neural Networks
Revised:  Preliminaries 2.1 Complex-Valued Neural Networks

Original: Without loss of generality, we focus on complex-valued neural networks with real-valued outputf θ ( z ) ∈ R d out with parameter set θ ∈ C p , input z ∈ C d and z = x
Revised: Without loss of generality, we focus on complex-valued neural networks with real-valued output f θ ( z ) ∈ R d out with parameter set θ ∈ C p , input z ∈ C d and z = x

Original: + y i with x , y ∈ R d , and trainedby real-valued backpropagation algorithm (also called complex-BP or generalized complex BP, see
Revised: + y i with x , y ∈ R d , and trained by real-valued backpropagation algorithm (also called complex-BP or generalized complex BP, see

Original: Appendix B for more details), which is conventional in the literature [Arjovsky et al., 2016, Wisdomet al., 2016, Trabelsi et al., 2018, Zhang  et al., 2021, Wu et al., 2021].
Revised: Appendix A for more details), which is conventional in the literature [Arjovsky et al., 2016, Wisdom et al., 2016, Trabelsi et al., 2018, Zhang and Zhou, 2021, Wu et al., 2021, Zhang et al., 2022].

Original: Our analysis can be naturallyapplied to complex-valued output by decomposing the real and imaginary part of output into twofunctions, or applied to real-valued input by treating its imaginary part as zero.
Revised: Our analysis can be naturally applied to complex-valued output by decomposing the real and imaginary part of output into two functions, or applied to real-valued input by treating its imaginary part as zero.

Original: For an L -hidden layer complex network, we denote the output of last hidden layer as h L ∈ C n .
Revised: For an L -hidden layer complex network, we denote the output of last hidden layer as h L ∈ C n .

Original: Without loss of generality, we consider that the output of a complex network with a linear readout layer is achieved via
Revised: Without loss of generality, we consider that the output of a complex network with a linear readout layer is achieved via

Original: [Equation]
Revised: [Equation]

Original: where W L +1 ∈ C n × d out [Wisdom et al., 2016, Zhang et al., 2021].
Revised: where W L +1 ∈ C n × d out [Wisdom et al., 2016, Zhang et al., 2022]

Original: Note that there are other twocommon forms of linear readout layer to generate real-valued output, like f θ ( z )
Revised: Note that there are other two common forms of linear readout layer to generate real-valued output, like f θ ( z )

Original: =
Revised: =

Original: W L +1 ℜ{ h L } where the output weight W L +1 ∈ R n × d out [Wu et al., 2021], and f θ ( z ) =
Revised: W L +1 ℜ{ h L } where the output weight W L +1 ∈ R n × d out [Wu et al., 2021], and f θ ( z ) =

Original: W L + (cid:20) ℜ
Revised: W L + (cid:20) ℜ

Original: ( h L ) ℑ ( h L ) (cid:21) where W L +1 ∈ R 2 n × d out [Arjovsky et al., 2016].
Revised: ( h L ) ℑ ( h L ) (cid:21) where W L +1 ∈ R 2 n × d out [Arjovsky et al., 2016].

Original: They can be treated as special cases of our settings.
Revised: They can be treated as special cases of our settings.

Original: For a complex-valued neural network f θ ( z ) , we can always decompose all the complex operationsinto two-dimensional real-valued operations, denoted as f
Revised: For a complex-valued neural network f θ ( z ) , we can always decompose all the complex operations into two-dimensional real-valued operations, denoted as f

Original: [ θ R ,θ I ] ([ x , y ]) where θ R , θ I ∈
Revised: [ θ R ,θ I ] ([ x , y ]) where θ R , θ I ∈

Original: R p are thereal and imaginary parts of all complex parameters respectively.
Revised: R p are the real and imaginary parts of all complex parameters respectively.

Original: However, it would be erroneous toassume that a complex network is equivalent to an ordinary real-valued neural network, because theoperation of complex multiplication limits the degree of freedom [Hirose and Yoshida, 2012].
Revised: However, it would be erroneous to assume that a complex network is equivalent to an ordinary real-valued neural network, because the operation of complex multiplication limits the degree of freedom [Hirose and Yoshida, 2012].

Original: 2.2 Neural Tangent Kernel
Revised: 2.2 Neural Tangent Kernel

Original: For a real-valued deep neural network f θ r ( x ) ∈ R d out with parameter set θ r ∈ R p and input x ∈ R d ,its Neural Tangent Kernel (NTK) under gradient descent is defined as
Revised: For a real-valued deep neural network f θ r ( x ) ∈ R d out with parameter set θ r ∈ R p and input x ∈ R d , its Neural Tangent Kernel (NTK) under gradient descent is defined as

Original: [Equation]
Revised: [Equation]

Original:  which quantifies the functional gradient descent when taking an infinitely small gradient step ona new observation.
Revised: (1) which quantifies the functional gradient descent when taking an infinitely small gradient step on a new observation.

Original: In case f θ r corresponds to an infinite width MLP, Jacot et al.
Revised: In case f θ r corresponds to an infinite width MLP, Jacot et al.

Original: [2018] showedthat (cid:98) Θ r ( x , x ′ ) converges to a limiting kernel ˚Θ r ( x , x ′ ) at initialization and remains frozen duringtraining, i.e.,[Equation] ∀
Revised: [2018] showed that (cid:98) Θ r ( x , x ′ ) converges to a limiting kernel ˚Θ r ( x , x ′ ) at initialization and remains frozen during training, i.e.,

Original: [Equation]
Revised: [Equation]

Original: , which could give an accurate description of training dynamics with kernel gradient descent trajectory.
Revised: which could give an accurate description of training dynamics with kernel gradient descent trajectory.

Original: Thus a infinitely wide neural network is governed by a linear model based on its first order Taylorexpansion in the parameter space [Lee et al., 2019].
Revised: Thus a infinitely wide neural network is governed by a linear model based on its first order Taylor expansion in the parameter space [Lee et al., 2019].

Original: Tensor Programs.
Revised: Tensor Programs.

Original: After the original NTK was derived from multi-layer perceptrons, it is soonextended to a variety of network structures including convolution neural networks (CNTK)
Revised: After the original NTK was derived from multi-layer perceptrons, it is soon extended to a variety of network structures including convolution neural networks (CNTK)

Original: [Aroraet al., 2019], recurrent neural networks (RNTK)
Revised: [Arora et al., 2019], recurrent neural networks (RNTK)

Original: [Yang, 2019, Alemohammad et al., 2020], graphneural networks (GNTK)
Revised: [Yang, 2019, Alemohammad et al., 2020], graph neural networks (GNTK)

Original: [Du et al., 2019] and so on.
Revised: [Du et al., 2019] and so on.

Original: Importantly, Yang [2020], Yang and Littwin[2021] propose NETSOR ⊤ program, a basic form in Tensor Programs series, and prove that for a realvalued neural network of any architecture that can be represented by NETSOR ⊤ program language,its NTK converges to a deterministic limit and stays frozen during training in the infinite-width limit.
Revised: Importantly, Yang [2020], Yang and Littwin [2021] propose NETSOR ⊤ program, a basic form in Tensor Programs series, and prove that for a realvalued neural network of any architecture that can be represented by NETSOR ⊤ program language, its NTK converges to a deterministic limit and stays frozen during training in the infinite-width limit.

Original: Neural Tangent Kernel of complex networks.
Revised: Neural Tangent Kernel of complex networks.

Original: For a complex-valued neural network f θ ( z ) , alsodenoted as f [ θ R ,θ I ] ([ x , y ]) , when it is trained by real-valued backpropagation, the empirical neuraltangent kernel is as follows due to that the real and imaginary parts are optimized separately 
Revised: For a complex-valued neural network f θ ( z ) , also denoted as f [ θ R ,θ I ] ([ x , y ]) , when it is trained by real-valued backpropagation, the empirical neural tangent kernel is as follows due to that the real and imaginary parts are optimized separately (cid:98)

Original: [Equation]
Revised: [Equation]

Original: We can always rewrite the NTK as Θ([ x , y ] , [ x ′ , y ′ ]) .
Revised: We can always rewrite the NTK as Θ([ x , y ] , [ x ′ , y ′ ]) .

Original: Note that at each layer of complex networksin both feed-forward and backward procedure, complex matrix multiplication structure leads tonumerous interactions and weight sharing between the real and imaginary parts, which makes theanalysis of the complex NTK challenging.
Revised: Note that at each layer of complex networks in both feed-forward and backward procedure, complex matrix multiplication structure leads to numerous interactions and weight sharing between the real and imaginary parts, which makes the analysis of the complex NTK challenging.

Original:  Complex Tensor Program
Revised:  Complex Tensor Program

Original: In this section, we show that complex-valued neural networks of any basic architecture also have
Revised: In this section, we show that complex-valued neural networks of any basic architecture also have

Original: NTK behavior in the infinite-width limit: the training dynamics of real-valued backpropagation isdetermined by kernel gradient descent with its NTK at initialization.
Revised: NTK behavior in the infinite-width limit: the training dynamics of real-valued backpropagation is determined by kernel gradient descent with its NTK at initialization.

Original: Specifically, we extend thesimplified NETSOR ⊤ program
Revised: Specifically, we extend the simplified NETSOR ⊤ program

Original: [Yang, 2020] to the complex domain, and propose the basic Complex
Revised: [Yang, 2020] to the complex domain, and propose the basic Complex

Original: Tensor Program (CTP).
Revised: Tensor Program (CTP).

Original: Note that the complex networks are mostly non-holomorphic, thus we do notrequire the complex tensor program to represent the backward propagation of the complex networks.
Revised: Note that the complex networks are mostly non-holomorphic, thus we do not require the complex tensor program to represent the backward propagation of the complex networks.

Original: Definition 1 (Complex Tensor Program)
Revised: Definition 1 (Complex Tensor Program)

Original: Given an initial set V of random C n vectors and a set Wof random C n × n complex matrices, a sequence of C n vectors is called a complex tensor program ifthey are recursively generated through one of the following ways:
Revised: Given an initial set V of random C n vectors and a set W of random C n × n complex matrices, a sequence of C n vectors is called a complex tensor program if they are recursively generated through one of the following ways:

Original: ComNonlin Given ϕ : C k → C and z 1 , . . .
Revised: ComNonlin Given ϕ : C k → C and z 1 , . . .

Original: , z k ∈ C n , generate ϕ ( z 1 , . .
Revised: , z k ∈ C n , generate ϕ ( z 1 , . .

Original: . , z k ) ∈ C  ; ComMatMul Given W ∈ C n × n and z ∈ C n , generate W z ∈ C n .
Revised: z k ) ∈ C n ; ComMatMul Given W ∈ C n × n and z ∈ C n , generate W z ∈ C n .

Original: Obviously, the complex tensor program could represent the forward procedures of all basic complexnetwork architectures, such as the generic feed-forward full-connected complex networks [Nitta,2004], the complex-valued recurrent neural networks [Wisdom et al., 2016] and complex-valuedconvolutional neural networks [Trabelsi et al., 2018, Tan et al., 2020].
Revised: Obviously, the complex tensor program could represent the forward procedures of all basic complex network architectures, such as the generic feed-forward full-connected complex networks [Nitta, 2004], the complex-valued recurrent neural networks [Wisdom et al., 2016] and complex-valued convolutional neural networks [Trabelsi et al., 2018, Tan et al., 2020].

Original: 3.1 Complex network setup
Revised: 3.1 Complex network setup

Original: This subsection introduces the settings and assumptions of complex networks considered.
Revised: This subsection introduces the settings and assumptions of complex networks considered.

Original: Firstly weintroduce the required assumption for activation functions of complex networks, which generalizesthe assumption in Yang [2020] to the complex domain.
Revised: Firstly we introduce the required assumption for activation functions of complex networks, which generalizes the assumption in Yang [2020] to the complex domain.

Original: Assumption 2 We assume that the complex activation function ϕ : C k
Revised: Assumption 2 We assume that the complex activation function ϕ : C k

Original: → C used in the complexnetworks and its derivative are polynomially-bounded, i.e., ϕ ( z ) satisfies that | ϕ ( z ) | ≤ C ∥
Revised: → C used in the complex networks and its derivative are polynomially-bounded, i.e., ϕ ( z ) satisfies that | ϕ ( z ) | ≤ C ∥

Original: z ∥ p + cfor some C, p, c > 0 and z ∈ C k ; so is its derivative.
Revised: z ∥ p + c for some C, p, c > 0 and z ∈ C k ; so is its derivative.

Original: It is worth mentioning that numerous activation functions have been proposed to deal with complexvalued representations and basically they satisfy the assumption.
Revised: It is worth mentioning that numerous activation functions have been proposed to deal with complexvalued representations and basically they satisfy the assumption.

Original: For example, the most commonlyused complex sigmoid function C Sigmoid [Nitta, 1997, 2004] and complex hyperbolic tangentfunction C tanh [Nitta, 2002], which apply sigmoid and hyperbolic tangent function respectively tothe real part and imaginary part separately; Moreover, the recently proposed ReLU-based complexactivation functions like C ReLU [Trabelsi et al., 2018, Tan et al., 2020], zReLU [Guberman, 2016]and modReLU [Arjovsky et al., 2016] also satisfy the assumption.
Revised: For example, the most commonly used complex sigmoid function C Sigmoid [Nitta, 1997, 2004] and complex hyperbolic tangent function C tanh [Nitta, 2002], which apply sigmoid and hyperbolic tangent function respectively to the real part and imaginary part separately; Moreover, the recently proposed ReLU-based complex activation functions like C ReLU [Trabelsi et al., 2018, Tan et al., 2020], zReLU [Guberman, 2016] and modReLU [Arjovsky et al., 2016] also satisfy the assumption.

Original: Complex NTK parametrization.
Revised: Complex NTK parametrization.

Original: For the complex weights, we initialize each W ∈ W with
Revised: For the complex weights, we initialize each W ∈ W with

Original: W =
Revised: W =

Original: A + B i =
Revised: A + B i =

Original: σ A √ n A + σ B √ n Bi where A αβ , B αβ ∼ N (0 , 1) , which we refer to as complex NTKparametrization .
Revised: σ A √ n A + σ B √ n Bi where A αβ , B αβ ∼ N (0 , 1) , which we refer to as complex NTK parametrization .

Original: Without loss of generality, we set the variances of real and imaginary parts ofall layers as σ A and σ B respectively in the following paper.
Revised: Without loss of generality, we set the variances of real and imaginary parts of all layers as σ A and σ B respectively in the following paper.

Original: It naturally extends the real-valued
Revised: It naturally extends the real-valued

Original: NTK parametrization [Jacot et al., 2018, Lee et al., 2019] to complex parameters.
Revised: NTK parametrization [Jacot et al., 2018, Lee et al., 2019] to complex parameters.

Original: Note that thisparametrization is non-vacuous because many previous works [Nitta, 1997, 2004] choose to initializethe real and imaginary parts separately.
Revised: Note that this parametrization is non-vacuous because many previous works [Nitta, 1997, 2004] choose to initialize the real and imaginary parts separately.

Original: Setup.
Revised: Setup.

Original: Consider a complex-valued neural network f θ ( z ) with complex NTK parametrization, itsfeed-forward procedure can be represented by a complex tensor program and complex activationfunctions all satisfy Assumption 2.
Revised: Consider a complex-valued neural network f θ ( z ) with complex NTK parametrization, its feed-forward procedure can be represented by a complex tensor program and complex activation functions all satisfy Assumption 2.

Original: Suppose that there is a multivariate Gaussian N V defined on153 R 2 |V| such that the real and imaginary variables of the initial set of vectors V are sampled like{ℜ [ q ] α : q ∈ V} ∪ {ℑ [ q ] α : q ∈ V} ∼ N V i.i.d.
Revised: Suppose that there is a multivariate Gaussian N V defined on R 2 |V| such that the real and imaginary variables of the initial set of vectors V are sampled like {ℜ [ q ] α : q ∈ V} ∪ {ℑ [ q ] α : q ∈ V} ∼ N V i.i.d.

Original: for each coordinate α ∈ [ n ] .
Revised: for each coordinate α ∈ [ n ] .

Original: For the output readoutmatrix W L +1 , we also adopt complex NTK parametrization, and it is sampled independently fromall other parameters and is not used anywhere else in the interior of the network.
Revised: For the output readout matrix W L +1 , we also adopt complex NTK parametrization, and it is sampled independently from all other parameters and is not used anywhere else in the interior of the network.

Original: Without loss ofgenerality, the network is trained by SGD with batch-size 1 and learning rate 1.
Revised: Without loss of generality, the network is trained by SGD with batch-size 1 and learning rate 1.

Original: 3.2 NTK for any complex network
Revised: 3.2 NTK for any complex network

Original: Theorem 3 (Complex NTK at initialization) Consider a complex-valued neural network f θ ( z )with above setup, then as its widths go to infinity,
Revised: Theorem 3 (Complex NTK at initialization) Consider a complex-valued neural network f θ ( z ) with above setup, then as its widths go to infinity,

Original: its NTK (cid:98) Θ( z , z ′ ) at initialization converges almostsurely to a deterministic limiting kernel ˚Θ( z , z ′ ) over any finite set of inputs.
Revised: its NTK (cid:98) Θ( z , z ′ ) at initialization converges almost surely to a deterministic limiting kernel ˚Θ( z , z ′ ) over any finite set of inputs.

Original: Corollary 4 (Complex NTK during training) Consider training a complex-valued neural networkf θ ( z ) with above setup.
Revised: Corollary 4 (Complex NTK during training) Consider training a complex-valued neural network f θ ( z ) with above setup.

Original: At training time t , denote the input sample as z t and the loss function as
Revised: At training time t , denote the input sample as z t and the loss function as

Original: L t : R → R .
Revised: L t : R → R .

Original: Suppose L t is continuous for all t .
Revised: Suppose L t is continuous for all t .

Original: Then as widths approach infinity, for any z ∈ C dand training time t , f t ( z ) converges almost surely to a random variable ˚ f t ( z ) and˚ f t +1 ( z ) − ˚ f t ( z ) = − ˚Θ ( z , z t ) L ′ t (cid:16) ˚ f t ( z t ) (cid:17) ,
Revised: Then as widths approach infinity, for any z ∈ C d and training time t , f t ( z ) converges almost surely to a random variable ˚ f t ( z ) and

Original: [Equation]
Revised: [Equation]

Original: ) where ˚Θ ( z , z t ) is the limiting NTK of the complex network at initialization.
Revised: where ˚Θ ( z , z t ) is the limiting NTK of the complex network at initialization.

Original: The proofs of Theorem 3 and Corollary 4 are given in Appendix A. Theorem 3 and Corollary 4 showthat for a complex-valued neural network of any architecture trained by real-valued backpropagation,its NTK at initialization converges to a deterministic limiting kernel in the infinite-width limit, andthe training dynamics is determined by kernel gradient descent with the NTK at initialization.
Revised: The proofs of Theorem 3 and Corollary 4 are given in Appendix B. Theorem 3 and Corollary 4 show that for a complex-valued neural network of any architecture trained by real-valued backpropagation, its NTK at initialization converges to a deterministic limiting kernel in the infinite-width limit, and the training dynamics is determined by kernel gradient descent with the NTK at initialization.

Original:  Comparison of training dynamics between complex and real networks
Revised:  Comparison of training dynamics between complex and real networks

Original: In this section, we focus on the important problem: when will complex networks have differentinductive bias during training from real networks?
Revised: In this section, we focus on the important problem: when will complex networks have different inductive bias during training from real networks?

Original: The problem can not be solved unless thetraining dynamics of complex networks could be captured.
Revised: The problem can not be solved unless the training dynamics of complex networks could be captured.

Original: Based on the NTK theory obtained in
Revised: Based on the NTK theory obtained in

Original: Section 3, we could provide a way to compare the training dynamics of complex and real networksby comparing their NTKs.
Revised: Section 3, we could provide a way to compare the training dynamics of complex and real networks by comparing their NTKs.

Original: Specifically, we have derived the NTK formula of complex multi-layerperceptrons (MLPs) and investigated the conditions under which complex MLPs trained by realvalued backpropagation will reduce to ordinary real MLPs in the infinite-width limit.
Revised: Specifically, we have derived the NTK formula of complex multi-layer perceptrons (MLPs) and investigated the conditions under which complex MLPs trained by realvalued backpropagation will reduce to ordinary real MLPs in the infinite-width limit.

Original: 4.1 Neural Tangent Kernel of complex multi-layer perceptrons
Revised: 4.1 Neural Tangent Kernel of complex multi-layer perceptrons

Original: This subsection presents the NTK formula of the most generic complex network, i.e., the L -hiddenlayer complex MLPs.
Revised: This subsection presents the NTK formula of the most generic complex network, i.e., the L -hidden layer complex MLPs

Original: The network performs the following computation at layer l ∈ [1 , L ]
Revised: The network performs the following computation at layer l ∈ [1 , L ]

Original: [Equation]
Revised: [Equation]

Original: where W l =
Revised: where W l =

Original: A l + B l i is the complex weight matrix and h l = s l + r l i with h l ∈ C n is the output ofl -th layer.
Revised: A l + B l i is the complex weight matrix and h l = s l + r l i with h l ∈ C n is the output of l -th layer.

Original: For the first layer, we set s 0 = x and r 0 =
Revised: For the first layer, we set s 0 = x and r 0 =

Original: y where z = x
Revised: y where z = x

Original: + y i and z ∈ C d .
Revised: + y i and z ∈ C d .

Original: Besides, theoutput of the complex network with a linear read-out layer is achieved via f θ ( x ) =
Revised: Besides, the output of the complex network with a linear read-out layer is achieved via f θ ( x ) =

Original: Re { W L +1 h L }where W L +1 ∈ C n × d out .
Revised: Re { W L +1 h L } where W L +1 ∈ C n × d out .

Original: Suppose all activation functions ϕ satisfy the Assumption 2.
Revised: Suppose all activation functions ϕ satisfy the Assumption 2.

Original: Complex
Revised: Complex

Original: NTK parametrization is applied for all complex parameters W 1 ∈ C d × n , W L +1 ∈ C n × d out and
Revised: NTK parametrization is applied for all complex parameters W 1 ∈ C d × n , W L +1 ∈ C n × d out and

Original: W l ∈ C n × n for l ∈ [2 , L ] .
Revised: W l ∈ C n × n for l ∈ [2 , L ] .

Original: We denote the real part of the l -th hidden layer pre-activation as α l ( z ) and the imaginary part asβ l ( z ) .
Revised: We denote the real part of the l -th hidden layer pre-activation as α l ( z ) and the imaginary part as β l ( z ) .

Original: In the feed-forward procedure, we denote the covariance kernel functions between the realand imaginary part of the pre-activations respectively as 
Revised: In the feed-forward procedure, we denote the covariance kernel functions between the real and imaginary part of the pre-activations respectively as [Equation] In the backward procedure

Original: [Equation]
Revised: [Equation]

Revised: we denote the gradient vector of the real part of the l -th hidden layer pre-activation as δ lα (

Revised: [Equation]

Revised: ) :=

Original: In the backward procedure, we denote the gradient vector of the real part of the l -th hiddenlayer pre-activation as δ lα ( z ) := √ n (cid:0) ∇ α l ( z ) f θ ( z ) (cid:1) and that of the imaginary part as δ lβ ( z ) :=194 √ n(cid:0) ∇ β l ( z ) f θ ( z ) (cid:1) .
Revised: [Equation] n [Equation] ∇ α l ( z ) f θ ( z ) (cid:1) and that of the imaginary part as δ lβ ( z ) := √ n

Revised: [Equation]

Revised: l

Revised: [Equation]

Revised: z ) f θ ( z ) (cid:1) .

Original: Similarly, we denote the covariance kernel functions of the gradient vectorbetween the real and imaginary part of the pre-activations respectively as
Revised: Similarly [Equation] we denote the covariance kernel functions of the gradient vector between the real and imaginary part of the pre-activations respectively as

Original: [Equation]
Revised: [Equation]

Original: Theorem 5 For a L -hidden layer complex MLP, with all activation functions satisfying Assumption 2,in the limit as all widths n → ∞ , the empirical NTK at initialization converges to the followinglimiting kernel
Revised: Theorem 5 For a L -hidden layer complex MLP, with all activation functions satisfying Assumption 2, in the limit as all widths n → ∞ , the empirical NTK at initialization converges to the following limiting kernel

Original: [Equation]
Revised: [Equation]

Original: whereΘ ( z , z
Revised: where Θ ( z , z

Original: [Equation]

Original: ) =

Original: L

Original: [Equation]
Revised: [Equation]

Original: where the covariance functions Σ lα , Σ lβ , Π lα , Π lβ are defined in Eq.
Revised: where the covariance functions Σ lα , Σ lβ , Π lα , Π lβ are defined in Eq.

Original: The result is proved in the Appendix C, where the detailed recursions of intermediate kernels for the
Revised: The result is proved in the Appendix C, where the detailed recursions of intermediate kernels for the

Original: NTK calculation are also presented.
Revised: NTK calculation are also presented.

Original: Note that the NTK formula of a complex MLP looks very different from the NTK of a real MLPgiven by Jacot et al.
Revised: Note that the NTK formula of a complex MLP looks very different from the NTK of a real MLP given by Jacot et al.

Original: [2018] due to the existence of interaction between real and imaginary parts,which is caused by joint weight sharing in complex matrix multiplication.
Revised: [2018] due to the existence of interaction between real and imaginary parts, which is caused by joint weight sharing in complex matrix multiplication.

Original: However, if we go deeper,does there exist situations that the NTK of a complex MLP will reduce to that of a real MLP?
Revised: However, if we go deeper, does there exist situations that the NTK of a complex MLP will reduce to that of a real MLP?

Original: 4.2 Asymptotic equivalence of training dynamics
Revised: 4.2 Asymptotic equivalence of training dynamics

Original: In this subsection, we provide our main results.
Revised: In this subsection, we provide our main results.

Original: Surprisingly, we show that, for commonly usedcomplex activation functions, complex networks trained by real-valued backpropagation have thesame inductive bias as real networks during training in the infinite-width limit.
Revised: Surprisingly, we show that, for commonly used complex activation functions, complex networks trained by real-valued backpropagation have the same inductive bias as real networks during training in the infinite-width limit.

Original: We first define asymptotic equivalence between neural networks, which represents a perspective toinvestigate when will a complex network have different inductive bias from a real network duringtraining.
Revised: We first define asymptotic equivalence between neural networks, which represents a perspective to investigate when will a complex network have different inductive bias from a real network during training.

Original: It also helps if we change the network structure or backpropagation algorithms.
Revised: It also helps if we change the network structure or backpropagation algorithms.

Original: Definition 6 Two neural networks trained by gradient descent are asymptotic equivalent , if as allwidths go to infinity, their neural tangent kernels (cid:98) Θ converge to the same deterministic limit ˚Θ atinitialization and have the same optimization trajectory during training.
Revised: Definition 6 Two neural networks trained by gradient descent are asymptotic equivalent , if as all widths go to infinity, their neural tangent kernels (cid:98) Θ converge to the same deterministic limit ˚Θ at initialization and have the same optimization trajectory during training.

Original: The following theorem is our main result: if we train complex networks with real-valued backpropagation, under very common conditions, the complex MLPs are asymptotic equivalent with real
Revised: The following theorem is our main result: if we train complex networks with real-valued back propagation, under very common conditions, the complex MLPs are asymptotic equivalent with real

Original: MLPs, thus they have the same training dynamics.
Revised: MLPs, thus they have the same training dynamics.

Original: Theorem 7 Consider a complex MLP in Eq.
Revised: Theorem 7 Consider a complex MLP in Eq.

Original: (3) and a ordinary real MLP with L hidden layerstrained by real-valued backpropagation.
Revised: (3) and an ordinary real MLP with L hidden layers trained by real-valued backpropagation.

Original: Suppose σ A = σ B at initialization and the activationfunctions satisfy Assumption 2.
Revised: Suppose σ A = σ B at initialization and the activation functions satisfy Assumption 2.

Original: As the widths go to infinity, they are asymptotic equivalent if theactivation functions satisfy one of the following conditions
Revised: As the widths go to infinity, they are asymptotic equivalent if the activation functions satisfy one of the following conditions

Original: Condition 1 All split activation functions ϕ satisfying ϕ
Revised: Condition 1 All split activation functions ϕ satisfying ϕ

Original: ( α, β ) =
Revised: ( α, β ) =

Original: ϕ R ( α ) + ϕ R ( β ) i  Condition 2 A subset of holomorphic activation functions ϕ satisfying ϕ 2 ( α, β ) 
Revised: ϕ R ( α ) + ϕ R ( β ) i , where ϕ R is a real-valued activation function; Condition 2 A subset of holomorphic activation functions ϕ satisfying ϕ 2 ( α, β ) =

Original: =

Original: ϕ 1 ( β, − α ) and ∂ϕ 1 ( α,β ) ∂β = ∂ϕ 2 ( α,β) ∂α =
Revised: ϕ 1 ( β, − α ) and ∂ϕ 1 ( α,β ) ∂β [Equation] 2 ( α,β

Revised: [Equation]

Original:  where the general complex activation function is denoted as ϕ ( α, β ) =
Revised: ∂α = 0 , where the general complex activation function is denoted as ϕ ( α, β ) =

Original: ϕ 1 ( α, β ) + ϕ 2 ( α, β ) i withinput pre-activations α + βi .
Revised: ϕ 1 ( α, β ) + ϕ 2 ( α, β ) i with input pre-activations α + βi .

Original: In the Appendix D the result is proved and we also give the sufficient and necessary conditions.
Revised: In the Appendix D the result is proved and we also give the sufficient and necessary conditions.

Original: For simplicity, here we only show the most informative conditions.
Revised: For simplicity, here we only show the most informative conditions.

Original: The key idea of the proof istransforming asymptotic equivalence into four complex conditions and find the common solutionsbased on Rules.
Revised: The key idea of the proof is transforming asymptotic equivalence into four complex conditions and find the common solutions based on Rules.

Original: F.13 in Appendix F.
Revised: F.13 in Appendix F.

Original: Discussion about the Condition 1.
Revised: Discussion about the Condition 1.

Original: Note that most commonly used complex activation functionssatisfy the Condition 1:
Revised: Note that most commonly used complex activation functions satisfy the Condition 1:

Original: [Equation]
Revised: [Equation]

Original: like complex sigmoid function [Benvenuto and Piazza, 1992, Nitta, 1997], complex hyperbolictangent function [Hirose and Yoshida, 2012], etc.
Revised: like complex sigmoid function [Benvenuto and Piazza, 1992, Nitta, 1997], complex hyperbolic tangent function [Hirose and Yoshida, 2012], etc.

Original: It is also worth mentioning that the recentlyproposed ReLU-based complex activation function C ReLU also satisfies the Condition 1, which hasachieved the best performance in feed-forward complex networks in image processing tasks [Trabelsiet al., 2018, Tan et al., 2020] among all ReLU-based complex activation functions.
Revised: It is also worth mentioning that the recently proposed ReLU-based complex activation function C ReLU also satisfies the Condition 1, which has achieved the best performance in feed-forward complex networks in image processing tasks [Trabelsi et al., 2018, Tan et al., 2020] among all ReLU-based complex activation functions.

Original: Remark 8 Because of Liouville’s theorem, the only complex-valued functions that are bounded andanalytic everywhere are constants.
Revised: Remark 8 Because of Liouville’s theorem, the only complex-valued functions that are bounded and analytic everywhere are constants.

Original: Thus in practice, one must choose between boundedness andanalyticity for a complex activation function.
Revised: Thus in practice, one must choose between boundedness and analyticity for a complex activation function.

Original: Before the popularity of ReLU, almost all activationfunctions in the real case were bounded.
Revised: Before the popularity of ReLU, almost all activation functions in the real case were bounded.

Original: Consequently, previous works about complex networksalways preferred non-analytic functions to preserve boundedness.
Revised: Consequently, previous works about complex networks always preferred non-analytic functions to preserve boundedness.

Original: Most commonly they applied splitactivation functions separately to the real and imaginary parts, as investigated in Bassey et al.
Revised: Most commonly they applied split activation functions separately to the real and imaginary parts, as investigated in Bassey et al.

Original: [2021]and Scardapane et al.
Revised: [2021] and Scardapane et al.

Original: So the condition contains most complex networks in practice.
Revised: So the condition contains most complex networks in practice.

Original: As a result, the theorem demonstrates that for complex networks with all these common complexactivations, if they are trained by real-valued BP, then these complex networks reduce to real networksas widths grow, despite the joint interaction weight sharing caused by complex matrix multiplicationstructure.
Revised: As a result, the theorem demonstrates that for complex networks with all these common complex activations, if they are trained by real-valued BP, then these complex networks reduce to real networks as widths grow, despite the joint interaction weight sharing caused by complex matrix multiplication structure.

Original: Consequently, real-valued backpropagation  eliminates the characteristics of most complexnetworks.
Revised: Consequently, real-valued backpropagation totally eliminates the characteristics of complex networks at infinite width.

Original: This may guide the selection of training algorithm in practice if people want to take fulladvantage of complex networks, and encourage people to use backpropagation algorithms speciallydesigned for complex networks, like Wirtinger calculus [Adali et al., 2011].
Revised: This may guide the selection of training algorithm in practice if people want to take full advantage of complex networks, and encourage people to explore learning algorithms specially designed for complex networks.

Original: [Figure]
Revised: [Figure]

Original: Discussion about the Condition 2.
Revised: Discussion about the Condition 2.

Original: Condition 2 is also non-vacuous since it is a subset of CauchyRiemann condition.
Revised: Condition 2 is also non-vacuous since it is a subset of CauchyRiemann condition.

Original: It includes all the magnitude-based ReLU-type complex activation functions.
Revised: It includes all the magnitude-based ReLU-type complex activation functions.

Original: For example, we can easily obtain a modified modReLU [Arjovsky et al., 2016] and a modifiedphase-based ReLU satisfying Condition 2 as follows
Revised: For example, we can easily obtain a modified modReLU [Arjovsky et al., 2016] and a modified phase-based ReLU satisfying Condition 2 as follows

Original: [Equation]
Revised: [Equation]

Original: where g (cos θ z ) can be any function of phase cos θ z .
Revised: where g (cos θ z ) can be any function of phase cos θ z .

Original: Note that, these activation functions areanalytic almost everywhere, and according to previous theory, they enjoy better theoretical resultslike separation results [Zhang et al., 2021] and local minima [Wu et al., 2021].
Revised: Note that, these activation functions are analytic almost everywhere, and according to previous theory, they enjoy better theoretical results like separation results [Zhang et al., 2022] and local minima [Wu et al., 2021].

Original: However, our resultsindicate that real-valued backpropagation eliminates these advantages of complex networks in theinfinite-width limit, which further illustrates the inappropriateness of real-valued backpropagation.
Revised: However, our results indicate that real-valued backpropagation eliminates these advantages of complex networks in the infinite-width limit, which further illustrates the inappropriateness of real-valued backpropagation.

Original:  Empirical study
Revised:  Empirical study

Original: In this section, we empirically verify the relationship between NTKs of the complex networks andreal networks, and investigate the network widths required for the establishment of our results.
Revised: In this section, we empirically verify the relationship between NTKs of the complex networks and real networks, and investigate the network widths required for the establishment of our results.

Original: We consider complex-valued MLPs with one or two hidden layers and we use C ReLU, modifiedmodReLU, C Sigmoid, C tanh and zReLU as the activation functions.
Revised: We consider complex-valued MLPs with one or two hidden layers and we use C ReLU, modified modReLU, C Sigmoid, C tanh and zReLU as the activation functions.

Original: Note that all these activationfunctions satisfy the conditions of our theorem except zReLU.
Revised: Note that all these activation functions satisfy the conditions of our theorem except zReLU.

Original: Through the following experiments,we want to check whether the empirical complex NTKs (cid:98) Θ ( n ) t converge to the corresponding real
Revised: Through the following experiments, we want to check whether the empirical complex NTKs (cid:98) Θ ( n ) t converge to the corresponding real

Original: NTKs ˚Θ r with different activation functions as the widths n grow.
Revised: NTKs ˚Θ r with different activation functions as the widths n grow.

Original: For real networks, the input is the concatenated vector of the real and imaginary parts of the complexvalued input.
Revised: For real networks, the input is the concatenated vector of the real and imaginary parts of the complexvalued input.

Original: Corresponded to the complex-valued fully-connected layer, we use the commonly usedreal-valued fully-connected layer without complex matrix multiplication structure.
Revised: Corresponded to the complex-valued fully-connected layer, we use the commonly used real-valued fully-connected layer without complex matrix multiplication structure.

Original: To implementthe corresponding activation functions, complex-valued activation functions are transformed to realvalued ones in the following way: we divide the pre-activation vector into two half, treat the firsthalf as real parts and the second as imaginary parts, as the input of ϕ ( α, β ) .
Revised: To implement the corresponding activation functions, complex-valued activation functions are transformed to realvalued ones in the following way: we divide the pre-activation vector into two half, treat the first half as real parts and the second as imaginary parts, as the input of ϕ ( α, β ) .

Original: Then we concatenatethe real and imaginary parts after activation.
Revised: Then we concatenate the real and imaginary parts after activation.

Original: For split activation functions like C ReLU, it can justcorrespond to real ReLU activation.
Revised: For split activation functions like C ReLU, it can just correspond to real ReLU activation.

Original: In NTK initialization, the standard deviations are set as 1 forcomplex networks and scaled to √ 2 for real networks.
Revised: In NTK initialization, the standard deviations are set as 1 for complex networks and scaled to √ 2 for real networks.

Original: All empirical NTKs of complex networks arecalculated based on the Neural Tangents library [Novak et al., 2019].
Revised: All empirical NTKs of complex networks are calculated based on the Neural Tangents library [Novak et al., 2019].

Original: [Figure]
Revised: [Figure]

Original: Verifying asymptotic equivalence at initialization.
Revised: Verifying asymptotic equivalence at initialization.

Original: The first experiment shows the distributionof empirical NTKs of complex MLPs (cid:98) Θ ( n ) t ( z, z ′ ) and analytic NTKs of real MLPs ˚ Θ r( z, z ′ )
Revised: The first experiment shows the distribution of empirical NTKs of complex MLPs (cid:98) Θ ( n ) t ( z, z ′ ) and analytic NTKs of real MLPs ˚Θ r

Original: withdifferent z ′ at initialization on a synthetic dataset.
Revised: ( z, z ′ ) with different z ′ at initialization on a synthetic dataset.

Original: We define z = 1 − i and z ′ =
Revised: We define z = 1 − i and z ′ =

Original: cos α + sin αifor α ∈ [0 , 2 π ] .
Revised: cos α + sin αi for α ∈ [0 , 2 π ] .

Original: Then we can view (cid:98) Θ ( n ) t ( z, z ′ ) and ˚Θ r ( z, z ′ ) as functions of α .
Revised: Then we can view (cid:98) Θ ( n ) t ( z, z ′ ) and ˚Θ r ( z, z ′ ) as functions of α .

Original: For the complexnetworks, we calculate empirical NTKs for hidden layer width n = 100 , 1000 at initialization andhidden layer number l = 1 , 2 .
Revised: For the complex networks, we calculate empirical NTKs for hidden layer widths n = 100 , 1000 at initialization and hidden layer number l = 1 , 2 .

Original: In each case, we calculate (cid:98) Θ ( n ) t ( z, z ′ ) 100 times with different random
Revised: In each case, we calculate (cid:98) Θ ( n ) t ( z, z ′ ) 100 times with different random

Original: NTK initialization.
Revised: NTK initialization.

Original: We compare these empirical complex NTKs with corresponding real NTKs.
Revised: We compare these empirical complex NTKs with corresponding real NTKs.

Original: Inthe case of C ReLU, we calculate ˚Θ r by the analytic form solution of NTK [Cho and Saul, 2009];in the case of zReLU and modified modReLU, it’s hard to get the closed form solution, so we usethe average of a large mount of wide empirical NTKs to approximate ˚Θ r .
Revised: In the case of C ReLU, we calculate ˚Θ r by the analytic form solution of NTK [Cho and Saul, 2009]; in the case of zReLU and modified modReLU, it’s hard to get the closed form solution, so we use the average of a large mount of wide empirical NTKs to approximate ˚Θ r .

Original: The results are shownin Figure 1.
Revised: The results are shown in Figure 1.

Original: In the figure we observe that for C ReLU and modified modReLU, which satisfy ourconditions, their NTKs (cid:98) Θ ( n ) 0 concentrate to the NTKs of real MLPs ˚Θ r perfectly, and when n = 1000 ,the convergence is more concentrated and the complex NTKs almost equal to real NTKs; for zReLUwhich does not satisfy the conditions, there’s a gap between complex and real NTKs.
Revised: In the figure we observe that for C ReLU and modified modReLU, which satisfy our conditions, their NTKs (cid:98) Θ ( n ) 0 concentrate to the NTKs of real MLPs ˚Θ r perfectly, and when n = 1000 , the convergence is more concentrated and the complex NTKs almost equal to real NTKs; for zReLU which does not satisfy the conditions, there’s a gap between complex and real NTKs.

Original: Therefore, theresults verify our results perfectly and demonstrate our results are non-vacuous.
Revised: Therefore, the results verify our theorem perfectly and demonstrate our results are non-vacuous.

Original: Verifying asymptotic equivalence with more activation functions as widths grow much larger.
Revised: Verifying asymptotic equivalence with more activation functions as widths grow much larger.

Original: In the first experiment, although for zReLU there is a gap between complex and real networks, thetendency is still similar.
Revised: In the first experiment, although for zReLU there is a gap between complex and real networks, the tendency is still similar.

Original: For the second experiment, we do the similar experiment on the samesynthetic dataset, so that we can see what will happen when n becomes much larger.
Revised: For the second experiment, we do the similar experiment on the same synthetic dataset, to see what will happen when n becomes much larger, so that it can be more convincing to verify whether it converges.

Original: Besides,we consider more different activation functions which satisfy our conditions including C ReLU,301 C Sigmoid, C tanh and modified modReLU with different hyper-parameters ρ .
Revised: Besides, we consider more different activation functions which satisfy our conditions including C ReLU, C Sigmoid, C tanh and modified modReLU with different hyperparameters ρ .

Original: We calculate relative Frobenius norm ∥ (cid:98) Θ ( n ) 

Revised: We calculate relative Frobenius norm ∥ (cid:98) Θ ( n )

Original: ( X, X ) − ˚Θ r
Revised: ( X, X ) − ˚Θ r

Original: ( X, X )
Revised: ( X, X )

Original: ∥ F / ∥ ˚Θ r ( X, X ) ∥ F on set X with widths n ranging from2 5 to 2 13 , which measures the difference between complex NTKs (cid:98) Θ ( n )and real NTKs ˚ Θ r .
Revised: ∥ F / ∥ ˚Θ r ( X, X ) ∥ F on set X with widths n ranging from 2 5 to 2 13 , which measures the difference between complex NTKs (cid:98) Θ ( n )and real NTKs ˚Θ r .

Original: Figure 2shows the result.
Revised: Figure 2 shows the result.

Original: For all those split activation functions( C ReLU, C Sigmoid and C tanh), the tendencyof convergence remains unchanged even when the width n goes to quite a large number 2 13 .
Revised: For all those split activation functions( C ReLU, C Sigmoid and C tanh), the tendency of convergence remains unchanged even when the width n goes to quite a large number 2 13 .

Original: The twocurves of modified modReLU act similarly with that of split activation functions; However, the curveof zReLU does not converge at all at initialization.
Revised: The two curves of modified modReLU act similarly with that of split activation functions; However, the curve of zReLU does not converge at all at initialization.

Original: Verifying asymptotic equivalence during training.
Revised: Verifying asymptotic equivalence during training.

Original: The third experiment investigates the convergence of difference between complex NTKs (cid:98) Θ ( n ) t and real NTKs ˚Θ r during training as the widths goto infinity on MNIST [LeCun et al., 1998].
Revised: The third experiment investigates the convergence of difference between complex NTKs (cid:98) Θ ( n ) t and real NTKs ˚Θ r during training as the widths go to infinity on MNIST [LeCun et al., 1998].

Original: We randomly choose a subset of MNIST as training set
Revised: We randomly choose a subset of MNIST as training set

Original: D =
Revised: D =

Original: ( X, Y ) ( |D| = 128 ), and treat the first half of features as real parts, the second half as imaginaryparts.
Revised: ( X, Y ) ( |D| = 128 ), and treat the first half of features as real parts, the second half as imaginary parts.

Original: Then we calculate relative Frobenius norm between empirical NTKs of complex networks attime t and real NTKs at initialization with widths n ranging from 2 5 to 2 10 at initialization ( t = 0 )and during training ( t = 1000 ).
Revised: Then we calculate relative Frobenius norm between empirical NTKs of complex networks at time t and real NTKs at initialization with widths n ranging from 2 5 to 2 10 at initialization ( t = 0 ) and during training ( t = 1000 ).

Original: Due to the memory limitations, we cannot try larger n .
Revised: Due to the memory limitations, we cannot try larger n .

Original: The learningrate η is 0 .
Revised: The learning rate η is 0 .

Original: 5 for l = 1 and 0 .
Revised: 5 for l = 1 and 0 .

Original: 2 for l = 2 .
Revised: 2 for l = 2 .

Original: The results are shown in Figure 3.
Revised: The results are shown in Figure 3.

Original: We can see that in allthese cases, the relative Frobenius norm decreases as n goes up, regardless of the training steps andhidden layer numbers.
Revised: We can see that in all these cases, the relative Frobenius norm decreases as n goes up, regardless of the training steps and

Original: Therefore, our theoretical results hold during training.

Original: More experimentsverifying asymptotic equivalence can be found in Appendix E.

Original: 222222 10 n 2222222( n ) t r F / r F
Revised: 222222 10 n 2222222( n ) t r F / r F

Original: CReLU t=0, L=

Revised: CReLU t=0, L=

Original: t=1000, L= t=0, L=2 t=1000, L= 222222 10 n 222222( n ) t r F / r F modified modReLU, =0.
Revised: t=1000, L= t=0, L=2 t=1000, L= 222222 10 n 222222( n ) t r F / r F modified modReLU, =0.

Original: t=0, L=

Revised: t=0, L=

Original: t=1000, L= t=0, L=2 t=1000, L= 
Revised: t=1000, L= t=0, L=2 t=1000, L= hidden layer numbers. Therefore, our theoretical results hold during training. More experiments verifying asymptotic equivalence can be found in Appendix E.

Original: Empirical results.
Revised: Empirical results.

Original: Overall, in the case of complex activation functions satisfying our theorems,such as C ReLU and modified modReLU, complex NTKs (cid:98) Θ ( n ) t converges to real NTKs ˚Θ r quitewell even the widths are about 1000; in the case of complex activation function that does satisfy ourtheorems like zReLU, the convergence from the complex NTK (cid:98) Θ ( n )to the real NTK ˚Θ r does notoccur.
Revised: Overall, in the case of complex activation functions satisfying our theorems, such as C ReLU and modified modReLU, complex NTKs (cid:98) Θ ( n ) t converges to real NTKs ˚Θ r quite well even the widths are about 1000; in the case of complex activation function that does satisfy our theorems like zReLU, the convergence from the complex NTK (cid:98) Θ ( n )to the real NTK ˚Θ r does not occur.

Original: This validates our theory perfectly and demonstrates that our conditions are non-vacuous.
Revised: This validates our theory perfectly and demonstrates that our conditions are non-vacuous.

Original:  Related work
Revised:  Related work

Original: [Figure] Long before the popularity of deep learning, there have been many investigations on complexvalued neural networks [Hirose, 1992, Benvenuto and Piazza, 1992, Nitta, 1997].
Revised: Long before the popularity of deep learning, there have been many investigations on complexvalued neural networks [Hirose, 1992, Benvenuto and Piazza, 1992, Nitta, 1997].

Original: However, It isalways challenging to train complex networks due to  analytical properties. The most notable reasonis that almost all cost functions are real-valued and thus non-holomorphic.
Revised: However, It is always challenging to train complex networks due to some non-intuitive analytical properties, of which the most notable reason is that almost all cost functions are real-valued and thus nonholomorphic.

Original: In order to performbackpropagation for complex networks, the conventional approach to overcome the limitation is touse separate derivatives with respect to the real-imaginary parts of a non-analytic function [Nitta,2004], or split amplitude-phase parts [Hirose, 1992].
Revised: In order to perform backpropagation for complex networks, the conventional approach to overcome the limitation is to use separate derivatives with respect to the real-imaginary parts of a non-analytic function [Nitta, 2004], or split amplitude-phase parts [Hirose, 1992].

Original: Hirose and Yoshida [2012] has shown that splitbackpropagation for amplitude-phase parts could achieve better generalization than real networkson signal processing tasks.
Revised: Hirose and Yoshida [2012] has shown that split backpropagation for amplitude-phase parts could achieve better generalization than real networks on signal processing tasks.

Original: Backpropagation based on Wirtinger calculus [Wirtinger, 1927] hasrecently received increasing attention [Adali et al., 2011, Bassey et al., 2021], which presents anelegant alternative and allows keeping all computations in the complex domain.
Revised: Recently Wirtinger calculus [Wirtinger, 1927] has received increasing attention [Adali et al., 2011, Bassey et al., 2021], which presents an elegant formulation to derive complex-valued gradient, Jacobian, and Hessian.

Original: However, real-valuedbackpropagation [Nitta, 1997] is  widely used recently to train deep complex networks due to theconvenience of utilizing the real-valued deep learning library [Arjovsky et al., 2016, Trabelsi et al. ,2018, Tan et al., 2020] and achieved state-of-the-art performance.Thus we start investigating fromit.
Revised: The real-valued backpropagation [Nitta, 1997] is most widely used, and it becomes essential for training deep complex networks due to the convenience of utilizing the real-valued deep learning library [Arjovsky et al., 2016, Trabelsi et al., 2018, Tan et al., 2020] and achieves state-of-the-art performance.

Original: Theoretically, there have been important advances for complex networks regarding the universalapproximation property [Voigtlaender, 2020], local minima [Nitta, 2013, Wu et al., 2021], separationresults [Zhang et al., 2021].
Revised: Theoretically, there have been important advances for complex networks regarding the universal approximation property [Voigtlaender, 2020], local minima [Nitta, 2013, Wu et al., 2021], separation results [Zhang et al., 2022].

Original: However, to our best knowledge, there is still no theoretical analysis ofthe training dynamics of complex networks and the equivalence  between complex and real networks.
Revised: However, to our best knowledge, there is still no theoretical analysis of the training dynamics of complex networks and the equivalence after training between complex and real networks.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we first extend the NTK theory to the complex domain, and propose a way to comparethe training dynamics between complex and real networks based on their NTKs.
Revised: In this paper, we first extend the NTK theory to the complex domain, and propose a way to compare the training dynamics between complex and real networks based on their NTKs.

Original: Surprisingly, wefind that the commonly used real-valued backpropagation reduces the training dynamics of complexvalued MLPs to that of ordinary real MLPs, thus eliminating the characteristics of complex-valuedneural networks.
Revised: Surprisingly, we find that the commonly used real-valued backpropagation reduces the training dynamics of complexvalued MLPs to that of ordinary real MLPs, thus eliminating the characteristics of complex-valued neural networks.

Original: Empirical study verifies that our results are practical for commonly used complexactivation functions.
Revised: Empirical study verifies that our results are practical for commonly used complex activation functions.

Original: We hope the proposed method and results could help the design and analysis oftraining for complex networks.

Original: For future work, developing the analysis for more training algorithmsand complex network structures will be exciting directions. It is also significant to consider otherparametrizations in the future investigation. Checklist For all authors...
Revised: The results encourage the design of new training algorithms for complex networks in the future.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 3 for assumptions.(c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [N/A](d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3for assumptions. (b) Did you include complete proofs of all theoretical results?
Revised: Besides, the proposed asymptotic equivalence of training dynamics between networks provides a new perspective for theoretical analysis of neural networks, which may offer a possibility to divide various neural network architectures into equivalence classes.

Original: [Yes] See Appendix.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] (b) Did you specify all the training details (e
Revised: Acknowledgment

Original: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes](c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)? [N/A]
Revised: This research was supported by NSFC (62176116, 61921006) and Collaborative Innovation Center of

Original: The numerical experiments only aim to verify thetheoretical results.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]

Original: The numerical experiments only

Original: aim to verify the theoretical results.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] MNIST. (b) Did you mention the license of the assets?
Revised: Novel Software Technology and Industrialization and the Program B for Outstanding PhD candidate of Nanjing University.

Original: [N/A] MNIST.

Original: GNU General Public

Original: License v3.

Original: (c) Did you include any new assets either in the supplemental material or as a URL? No (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: The authors would like to thank Jin-Hui Wu and Peng Zhao for helpful discussions.

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
Revised: We are also grateful for the anonymous reviewers for their valuable comments.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]