Original: P RECONDITION L AYER AND I TS U SE FOR GAN S
Revised: P RECONDITION L AYER AND I TS U SE FOR GAN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the major challenges when training generative adversarial nets (GANs) is instability.
Revised: One of the major challenges when training generative adversarial nets (GANs) is instability.

Original: To address this instability spectral normalization (SN) is remarkably successful.
Revised: To address this instability spectral normalization (SN) is remarkably successful.

Original: However, SN-GAN still suffers from training instabilities, especially when working with higher-dimensional data.
Revised: However, SN-GAN still suffers from training instabilities, especially when working with higher-dimensional data.

Original: We ﬁnd that those instabilities are accompanied by large condition numbers of the discriminator weight matrices.
Revised: We ﬁnd that those instabilities are accompanied by large condition numbers of the discriminator weight matrices.

Original: To improve training stability we study common linear-algebra practice and employ preconditioning.
Revised: To improve training stability we study common linear-algebra practice and employ preconditioning.

Original: Speciﬁcally, we introduce a preconditioning layer (PC-layer) that performs a low-degree polynomial preconditioning.
Revised: Speciﬁcally, we introduce a preconditioning layer (PC-layer) that performs a low-degree polynomial preconditioning.

Original: We use this PC-layer in two ways: 1) ﬁxed preconditioning (FPC) adds a ﬁxed PC-layer to all layers; and 2) adaptive preconditioning (APC) adaptively controls the strength of preconditioning.
Revised: We use this PC-layer in two ways: 1) ﬁxed preconditioning (FPC) adds a ﬁxed PC-layer to all layers; and 2) adaptive preconditioning (APC) adaptively controls the strength of preconditioning.

Original: Empirically, we show that FPC and APC stabilize training of unconditional GANs using classical architectures.
Revised: Empirically, we show that FPC and APC stabilize training of unconditional GANs using classical architectures.

Original: On LSUN 256 ⇥ 256 data, APC improves FID scores by around 5 points over baselines.
Revised: On LSUN 256 ⇥ 256 data, APC improves FID scores by around 5 points over baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) successfully transform samples from one distribution to another.
Revised: Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) successfully transform samples from one distribution to another.

Original: Nevertheless, training GANs is known to be challenging, and its performance is often sensitive to hyper-parameters and datasets.
Revised: Nevertheless, training GANs is known to be challenging, and its performance is often sensitive to hyper-parameters and datasets.

Original: Understanding the training difﬁculties of
Revised: Understanding the training difﬁculties of

Original: GAN is thus an important problem.
Revised: GAN is thus an important problem.

Original: Recent studies in neural network theory (Pennington et al., 2017; Xiao et al., 2018; 2020) suggest that the spectrum of the input-output Jacobian or neural tangent kernel (NTK) is an important metric for understanding training performance.
Revised: Recent studies in neural network theory (Pennington et al., 2017; Xiao et al., 2018; 2020) suggest that the spectrum of the input-output Jacobian or neural tangent kernel (NTK) is an important metric for understanding training performance.

Original: While directly manipulating the spectrum of the Jacobian or NTK is not easy, a practical approach is to manipulate the spectrum of weight matrices, such as orthogonal initialization (Xiao et al., 2018).
Revised: While directly manipulating the spectrum of the Jacobian or NTK is not easy, a practical approach is to manipulate the spectrum of weight matrices, such as orthogonal initialization (Xiao et al., 2018).

Original: For a special neural net, Hu et al.
Revised: For a special neural net, Hu et al.

Original: (2020) showed that orthogonal initialization leads to better convergence result than Gaussian initialization, which provides early theoretical evidence for the importance of manipulating the weight matrix spectrum.
Revised: (2020) showed that orthogonal initialization leads to better convergence result than Gaussian initialization, which provides early theoretical evidence for the importance of manipulating the weight matrix spectrum.

Original: Motivated by these studies, we suspect that an ‘adequate’ weight matrix spectrum is also important for GAN training.
Revised: Motivated by these studies, we suspect that an ‘adequate’ weight matrix spectrum is also important for GAN training.

Original: Indeed, one of the most popular techniques for GAN training, spectral normalization (SN) (Miyato et al., 2018), manipulates the spectrum by scaling all singular values by a constant.
Revised: Indeed, one of the most popular techniques for GAN training, spectral normalization (SN) (Miyato et al., 2018), manipulates the spectrum by scaling all singular values by a constant.

Original: This ensures the spectral norm is upper bounded.
Revised: This ensures the spectral norm is upper bounded.

Original: However, we ﬁnd that for some hyperparameters and for high-resolution datasets, SN-GAN fails to generate good images.
Revised: However, we ﬁnd that for some hyperparameters and for high-resolution datasets, SN-GAN fails to generate good images.

Original: In a study we ﬁnd the condition numbers of weight matrices to become very large and the majority of the singular values are close to 0 during training.
Revised: In a study we ﬁnd the condition numbers of weight matrices to become very large and the majority of the singular values are close to 0 during training.

Original: See Fig. 1(a) and Fig. 2(a).
Revised: See Fig. 1(a) and Fig. 2(a).

Original: This can happen as SN does not promote a small condition number.
Revised: This can happen as SN does not promote a small condition number.

Original: This ﬁnding motivates to reduce the condition number of weights during GAN training.
Revised: This ﬁnding motivates to reduce the condition number of weights during GAN training.

Original: Recall that controlling the condition number is also a central problem in numerical linear algebra, known as preconditioning (see Chen (2005)).
Revised: Recall that controlling the condition number is also a central problem in numerical linear algebra, known as preconditioning (see Chen (2005)).

Original: We hence seek to develop a “plug-in” preconditioner for weights.
Revised: We hence seek to develop a “plug-in” preconditioner for weights.

Original: This requires the preconditioner to be differentiable.
Revised: This requires the preconditioner to be differentiable.

Original: Out of various preconditioners, we ﬁnd the polynomial preconditioner to be a suitable choice due to the simple differentiation and strong theoretical support from approximation theory.
Revised: Out of various preconditioners, we ﬁnd the polynomial preconditioner to be a suitable choice due to the simple differentiation and strong theoretical support from approximation theory.

Original: Further, we suggest to adaptively adjust the strength of the preconditioner during training so as to not overly restrict the expressivity.
Revised: Further, we suggest to adaptively adjust the strength of the preconditioner during training so as to not overly restrict the expressivity.

Original: We show the efﬁcacy of preconditioning on CIFAR10 (32 ⇥ 32), STL (48 ⇥ 48) and LSUN bedroom, tower and living room (256 ⇥ 256).
Revised: We show the efﬁcacy of preconditioning on CIFAR10 (32 ⇥ 32), STL (48 ⇥ 48) and LSUN bedroom, tower and living room (256 ⇥ 256).

Original: Summary of contributions.
Revised: Summary of contributions.

Original: For a deep linear network studied in (Hu et al., 2020), we prove that if all weight matrices have bounded spectrum, then gradient descent converges to global min-
Revised: For a deep linear network studied in (Hu et al., 2020), we prove that if all weight matrices have bounded spectrum, then gradient descent converges to global min-

Original: [Figure]
Revised: [Figure]

Original: around shown

Original: [Figure]
Revised: [Figure]

Original: imum at a geometric rate.
Revised: imum at a geometric rate.

Original: We then introduce a PC-layer (preconditioning layer) that consists of a low-degree polynomial preconditioner.
Revised: We then introduce a PC-layer (preconditioning layer) that consists of a low-degree polynomial preconditioner.

Original: We further study adaptive preconditioning (APC) which adaptively controls the strength of PC on different layers in different iterations.
Revised: We further study adaptive preconditioning (APC) which adaptively controls the strength of PC on different layers in different iterations.

Original: Applying PC and
Revised: Applying PC and

Original: APC to unconditional GAN training on LSUN data ( 256 ⇥ 256 ), permits to generate high-quality images when SN-GAN fails.
Revised: APC to unconditional GAN training on LSUN data ( 256 ⇥ 256 ), permits to generate high-quality images when SN-GAN fails.

Original: We also show that APC achieves better FID scores on CIFAR10, STL, and LSUN than a recently proposed method of Jiang et al
Revised: We also show that APC achieves better FID scores on CIFAR10, STL, and LSUN than a recently proposed method of Jiang et al.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Related to the proposed method is work by Jiang et al.
Revised: Related to the proposed method is work by Jiang et al.

Original: (2019), which also controls the spectrum in
Revised: (2019), which also controls the spectrum in

Original: GAN training.
Revised: GAN training.

Original: They re-parameterize a weight matrix W via W = USV T , add orthogonal regularization of U, V and certain regularizer of entries of the diagonal matrix S .
Revised: They re-parameterize a weight matrix W via W = USV T , add orthogonal regularization of U, V and certain regularizer of entries of the diagonal matrix S .

Original: This approach differs from ours in a few aspects.
Revised: This approach differs from ours in a few aspects.

Original: First, Jiang et al.
Revised: First, Jiang et al.

Original: (2019) essentially solves a constrained optimization problem with constraints U T U
Revised: (2019) essentially solves a constrained optimization problem with constraints U T U

Original: =
Revised: =

Original: I, V T V
Revised: I, V T V

Original: =
Revised: =

Original: I using a penalty method (Bertsekas, 1997).
Revised: I using a penalty method (Bertsekas, 1997).

Original: In contrast, our approach solves an unconstrained problem since we add one layer into the neural net, similar to batch normalization (BN) (Ioffe & Szegedy, 2015) and SN (Miyato et al., 2018).
Revised: In contrast, our approach solves an unconstrained problem since we add one layer into the neural net, similar to batch normalization (BN) (Ioffe & Szegedy, 2015) and SN (Miyato et al., 2018).

Original: Second, our PC layer is a direct generalization of SN as it includes SN-layer as a special case.
Revised: Second, our PC layer is a direct generalization of SN as it includes SN-layer as a special case.

Original: In contrast, the method of Jiang et al.
Revised: In contrast, the method of Jiang et al.

Original: (2019) differs from SN-layer in any case.
Revised: (2019) differs from SN-layer in any case.

Original: Our proposed method thus offers a smoother transition for existing users of SN.
Revised: Our proposed method thus offers a smoother transition for existing users of SN.

Original: In a broader context, a number of approaches have been proposed to stabilize and improve GAN training, such as modifying the loss function (Arjovsky et al., 2017; Arjovsky & Bottou, 2017; Mao et al., 2017; Li et al., 2017b; Deshpande et al., 2018), normalization and regularization (Gulrajani et al., 2017; Miyato et al., 2018), progressive growing techniques (Karras et al., 2018;
Revised: In a broader context, a number of approaches have been proposed to stabilize and improve GAN training, such as modifying the loss function (Arjovsky et al., 2017; Arjovsky & Bottou, 2017; Mao et al., 2017; Li et al., 2017b; Deshpande et al., 2018), normalization and regularization (Gulrajani et al., 2017; Miyato et al., 2018), progressive growing techniques (Karras et al., 2018;

Original: Huang et al., 2017), changing architecture (Zhang et al., 2019; Karnewar & Wang, 2019), utilizing side information like class labels (Mirza & Osindero, 2014; Odena et al., 2017; Miyato & Koyama, 2018).
Revised: Huang et al., 2017), changing architecture (Zhang et al., 2019; Karnewar & Wang, 2019), utilizing side information like class labels (Mirza & Osindero, 2014; Odena et al., 2017; Miyato & Koyama, 2018).

Original: Using this taxonomy, our approach ﬁts the “normalization and regularization” category (even though our method is not exactly normalization, the essence of “embedded control” is similar).
Revised: Using this taxonomy, our approach ﬁts the “normalization and regularization” category (even though our method is not exactly normalization, the essence of “embedded control” is similar).

Original: Note that these directions are relatively orthogonal, and our approach can be potentially combined with other techniques such as progressive growing.
Revised: Note that these directions are relatively orthogonal, and our approach can be potentially combined with other techniques such as progressive growing.

Original: However, due to limited computational resources, we focus on unconditional GANs using classical architectures, the setting studied by Miyato et al.
Revised: However, due to limited computational resources, we focus on unconditional GANs using classical architectures, the setting studied by Miyato et al.

Original: 1. N OTATION AND D EFINITION
Revised: 1. N OTATION AND D EFINITION

Original: We use eig ( A ) to denote the multiset (i.e., allow repetition) of all eigenvalues of A .
Revised: We use eig ( A ) to denote the multiset (i.e., allow repetition) of all eigenvalues of A .

Original: If all eigenvalues of A are non-negative real numbers, we say A is a positive semideﬁnite (PSD) matrix.
Revised: If all eigenvalues of A are non-negative real numbers, we say A is a positive semideﬁnite (PSD) matrix.

Original: The singular values of a matrix A 2 R n ⇥ m are the square root of the eigenvalues of
Revised: The singular values of a matrix A 2 R n ⇥ m are the square root of the eigenvalues of

Original: A T
Revised: A T

Original: A 2 R m ⇥ m .
Revised: A 2 R m ⇥ m .

Original: Let  max ( A ) and  min ( A ) denote the maximum and minimum singular values of A .
Revised: Let  max ( A ) and  min ( A ) denote the maximum and minimum singular values of A .

Original: Let k A k 2 denote the spectral norm of A , i.e., the largest singular value.
Revised: Let k A k 2 denote the spectral norm of A , i.e., the largest singular value.

Original: The condition number of a square matrix A is traditionally deﬁned as  ( A ) =
Revised: The condition number of a square matrix A is traditionally deﬁned as  ( A ) =

Original: k
Revised: k

Original: A 
Revised: A k 2 k

Original: [Equation] =
Revised: A  1 k 2 =

Original: [Equation] A ) .
Revised:  max ( A )  min ( A ) .

Original: We extend this deﬁnition to a rectangular matrix A 2 R n ⇥ m where n  m via  ( A ) =
Revised: We extend this deﬁnition to a rectangular matrix A 2 R n ⇥ m where n  m via  ( A ) =

Original:  max ( A )  min ( A ) .
Revised:  max ( A )  min ( A ) .

Original: Let deg ( p ) denote the degree of a polynomial p and let P k
Revised: Let deg ( p ) denote the degree of a polynomial p and let P k

Original: = { p | deg ( p )  k } be the set of polynomials with degree no more than k .
Revised: = { p | deg ( p )  k } be the set of polynomials with degree no more than k .

Original:  W HY C ONTROLLING THE S PECTRUM ?
Revised:  W HY C ONTROLLING THE S PECTRUM ?

Original: To understand why controlling the spectrum is helpful we leverage recent tools in neural network theory to prove the following result: if weight matrices have small condition numbers, then gradient descent for deep pyramid linear networks converges to the global-min fast.
Revised: To understand why controlling the spectrum is helpful we leverage recent tools in neural network theory to prove the following result: if weight matrices have small condition numbers, then gradient descent for deep pyramid linear networks converges to the global-min fast.

Original: This is inspired by Hu et al.
Revised: This is inspired by Hu et al.

Original: (2020) analyzing a deep linear network to justify orthogonal initialization.
Revised: (2020) analyzing a deep linear network to justify orthogonal initialization.

Original: Similar to Hu et al.
Revised: Similar to Hu et al.

Original: (2020), we consider a linear network that takes an input x 2 R d x ⇥ 1 and outputs
Revised: (2020), we consider a linear network that takes an input x 2 R d x ⇥ 1 and outputs

Original: [Equation]
Revised: [Equation]

Original: where ✓ = ( W 1 , . .
Revised: where ✓ = ( W 1 , . .

Original: , W L ) represents the collection of all parameters and W j is a matrix of dimension
Revised: , W L ) represents the collection of all parameters and W j is a matrix of dimension

Original: d j ⇥ d j  1 , j = 1 , . . .
Revised: d j ⇥ d j  1 , j = 1 , . . .

Original: , L .
Revised: , L .

Original: Here we deﬁne d 0 = d x and d L = d y .
Revised: Here we deﬁne d 0 = d x and d L = d y .

Original: Assume there exists r 2 { 1 , . . .
Revised: Assume there exists r 2 { 1 , . . .

Original: , L } , such that d y =
Revised: , L } , such that d y =

Original: d L  d L  1  · · ·  d r , and n  d 0  d 1  · · ·  d r .
Revised: d L  d L  1  · · ·  d r , and n  d 0  d 1  · · ·  d r .

Original: This means the network is a pyramid network, which generalizes the equal-width network of Hu et al.
Revised: This means the network is a pyramid network, which generalizes the equal-width network of Hu et al.

Original: Suppose y =
Revised: Suppose y =

Original: ( y 1 ; . . .
Revised: ( y 1 ; . . .

Original: ; y n ) 2 R nd y ⇥ 1 are the labels, and the predictions are F ( ✓ ; X ) =
Revised: ; y n ) 2 R nd y ⇥ 1 are the labels, and the predictions are F ( ✓ ; X ) =

Original: ( F ( ✓ ; x 1 ); . . .
Revised: ( F ( ✓ ; x 1 ); . . .

Original: , F ( ✓ ; x n ))
Revised: , F ( ✓ ; x n ))

Original: 2 R nd y ⇥ 1 .
Revised: 2 R nd y ⇥ 1 .

Original: We consider a quadratic loss L ( ✓ ) = 12 k y  F ( ✓ ; X ) k 2 .
Revised: We consider a quadratic loss L ( ✓ ) = 12 k y  F ( ✓ ; X ) k 2 .

Original: Starting from ✓ (0) , we generate ✓ ( k ) =
Revised: Starting from ✓ (0) , we generate ✓ ( k ) =

Original: ( W 1 ( k ) , . . .
Revised: ( W 1 ( k ) , . . .

Original: , W L ( k )) , k = 1 , 2 , . . .
Revised: , W L ( k )) , k = 1 , 2 , . . .

Original: via gradient descent:
Revised: via gradient descent:

Original: [Equation]
Revised: [Equation]

Original: Denote the residual e ( k ) =
Revised: Denote the residual e ( k ) =

Original: F ( ✓ ( k ); X )  y. For given ⌧ l  1 , µ l  0 , l = 1 , . . .
Revised: F ( ✓ ( k ); X )  y. For given ⌧ l  1 , µ l  0 , l = 1 , . . .

Original: , L , deﬁne
Revised: , L , deﬁne

Revised: R ,

Original: [Equation]
Revised: [Equation]

Original: The following result states that if ✓ ( k ) stay within region R (i.e., weight matrices have bounded spectrum) for k = 0 , 1 , . . .
Revised: The following result states that if ✓ ( k ) stay within region R (i.e., weight matrices have bounded spectrum) for k = 0 , 1 , . . .

Original: , K , then the loss decreases at a geometric rate until iteration K .
Revised: , K , then the loss decreases at a geometric rate until iteration K .

Original: The rate (1   µ ) depends on ( ⌧ L ... ⌧ 1 ) 2 ( µ L ...µ 1 ) 2 , which is related to the condition numbers of all weights.
Revised: The rate (1   µ ) depends on ( ⌧ L ... ⌧ 1 ) 2 ( µ L ...µ 1 ) 2 , which is related to the condition numbers of all weights.

Original: Theorem 1 Suppose ⌘ = 1  .
Revised: Theorem 1 Suppose ⌘ = 1  .

Original: Assume ✓ ( k ) 2 R, k = 0 , 1 , . . .
Revised: Assume ✓ ( k ) 2 R, k = 0 , 1 , . . .

Original: , K. Then we have
Revised: , K. Then we have

Original: [Equation]
Revised: [Equation]

Original: See Appendix D.3.1 for the proof and detailed discussions.
Revised: See Appendix D.3.1 for the proof and detailed discussions.

Original: For proper initial point ✓ (0) where W l (0) ’s are full-rank, we can always pick ⌧ l ,  l so that ✓ (0) 2 R .
Revised: For proper initial point ✓ (0) where W l (0) ’s are full-rank, we can always pick ⌧ l ,  l so that ✓ (0) 2 R .

Original: The trajectory { ✓ ( k ) } either stays in R forever (in which case K = 1 ), or leaves R at some ﬁnite iteration K .
Revised: The trajectory { ✓ ( k ) } either stays in R forever (in which case K = 1 ), or leaves R at some ﬁnite iteration K .

Original: In the former case, the loss converges to zero at a geometrical rate; in the latter case, the loss decreases to below (1  µ/  ) K k e (0) k 2 .
Revised: In the former case, the loss converges to zero at a geometrical rate; in the latter case, the loss decreases to below (1  µ/  ) K k e (0) k 2 .

Original: However, our theorem does not specify how large
Revised: However, our theorem does not specify how large

Original: K is for a given situation.
Revised: K is for a given situation.

Original: Previous works on convergence (e.g., Hu et al., 2020; Du et al., 2018;
Revised: Previous works on convergence (e.g., Hu et al., 2020; Du et al., 2018;

Original: Allen-Zhu et al., 2019; Zou et al., 2018) bound the movement of the weights with extra assumptions, so that the trajectory stays in a certain nice regime (related to R ).
Revised: Allen-Zhu et al., 2019; Zou et al., 2018) bound the movement of the weights with extra assumptions, so that the trajectory stays in a certain nice regime (related to R ).

Original: We do not attempt to prove the trajectory stays in R .
Revised: We do not attempt to prove the trajectory stays in R .

Original: Instead, we use this as a motivation for algorithm design: if we can improve the condition numbers of weights during training, then the trajectory may stay in R for a longer time, and thus lead to smaller loss values.
Revised: Instead, we use this as a motivation for algorithm design: if we can improve the condition numbers of weights during training, then the trajectory may stay in R for a longer time, and thus lead to smaller loss values.

Original: Next, we present the preconditioning layer as such a method.
Revised: Next, we present the preconditioning layer as such a method.

Original:  P RECONDITIONING L AYER
Revised:  P RECONDITIONING L AYER

Original: In the following, we ﬁrst introduce classical polynomial preconditioners in Sec. 3.1.
Revised: In the following, we ﬁrst introduce classical polynomial preconditioners in Sec. 3.1.

Original: We then present the preconditioning layer for deep nets in Sec. 3.2.
Revised: We then present the preconditioning layer for deep nets in Sec. 3.2.

Original: We explain how to compute a preconditioning polynomial afterwards in Sec. 3.3, and ﬁnally present an adaptive preconditioning in Sec. 3.4.
Revised: We explain how to compute a preconditioning polynomial afterwards in Sec. 3.3, and ﬁnally present an adaptive preconditioning in Sec. 3.4.

Original: 3. P RELIMINARY : P OLYNOMIAL P RECONDITIONER
Revised: 3. P RELIMINARY : P OLYNOMIAL P RECONDITIONER

Original: Preconditioning considers the following classical question: for a symmetric matrix Q , how to ﬁnd an operator g such that  ( g ( Q )) is small?
Revised: Preconditioning considers the following classical question: for a symmetric matrix Q , how to ﬁnd an operator g such that  ( g ( Q )) is small?

Original: Due to the importance of this question and the wide applicability there is a huge literature on preconditioning.
Revised: Due to the importance of this question and the wide applicability there is a huge literature on preconditioning.

Original: See, e.g., Chen (2005) for an overview, and Appendix B for a short introduction.
Revised: See, e.g., Chen (2005) for an overview, and Appendix B for a short introduction.

Original: In this work, we focus on polynomial preconditioners (Johnson et al., 1983).
Revised: In this work, we focus on polynomial preconditioners (Johnson et al., 1983).

Original: The goal is to ﬁnd a polynomial ˆ p such that ˆ p ( Q ) Q has a small condition number.
Revised: The goal is to ﬁnd a polynomial ˆ p such that ˆ p ( Q ) Q has a small condition number.

Original: The matrix ˆ p ( Q ) is often called preconditioner , and ˆ g ( Q ) , ˆ p ( Q ) Q is the precondtioned matrix.
Revised: The matrix ˆ p ( Q ) is often called preconditioner , and ˆ g ( Q ) , ˆ p ( Q ) Q is the precondtioned matrix.

Original: We call g the preconditioning polynomial .
Revised: We call g the preconditioning polynomial .

Original: Polynomial preconditioning has a special merit: the difﬁcult problem of manipulating eigenvalues can be transformed to manipulating a 1-d function, based on the following fact (proof in Appendix E.2.1).
Revised: Polynomial preconditioning has a special merit: the difﬁcult problem of manipulating eigenvalues can be transformed to manipulating a 1-d function, based on the following fact (proof in Appendix E.2.1).

Original: Claim 3.1 Suppose ˆ g is any polynomial, and Q 2 R m ⇥ m is a real symmetric matrix with eigenvalues 
Revised: Claim 3.1 Suppose ˆ g is any polynomial, and Q 2 R m ⇥ m is a real symmetric matrix with eigenvalues

Original: [Equation]
Revised: [Equation]

Original:  m .
Revised:  m .

Original: Then the eigenvalues of the matrix ˆ g ( Q ) are ˆ g
Revised: Then the eigenvalues of the matrix ˆ g ( Q ) are ˆ g

Original: (  i ) , i = 1 , . . .
Revised: (  i ) , i = 1 , . . .

Original: , m. As a corollary, if ˆ g ([  1 ,  m ]) ✓ [1  ✏ , 1] , then eig
Revised: , m. As a corollary, if ˆ g ([  1 ,  m ]) ✓ [1  ✏ , 1] , then eig

Original: (ˆ g ( Q ))
Revised: (ˆ g ( Q ))

Original: ✓ [1  ✏ , 1] .
Revised: ✓ [1  ✏ , 1] .

Original: To ﬁnd a matrix ˆ g ( Q ) =
Revised: To ﬁnd a matrix ˆ g ( Q ) =

Original: ˆ p ( Q ) Q that is well-conditioned, we need to ﬁnd a polynomial ˆ p
Revised: ˆ p ( Q ) Q that is well-conditioned, we need to ﬁnd a polynomial ˆ p

Original: such that
Revised: such that

Original: ˆ g ( x ) = ˆ p ( x ) x maps [  1 ,  m ] to [1  ✏ , 1] .
Revised: ˆ g ( x ) = ˆ p ( x ) x maps [  1 ,  m ] to [1  ✏ , 1] .

Original: This can be formulated as a function approximation problem: ﬁnd a polynomial ˆ g ( x ) of the form x ˆ p ( x ) that approximates a function ˆ f (  ) in [  1 ,  m ] .
Revised: This can be formulated as a function approximation problem: ﬁnd a polynomial ˆ g ( x ) of the form x ˆ p ( x ) that approximates a function ˆ f (  ) in [  1 ,  m ] .

Original: Under some criterion, the optimal polynomial is a variant of the Chebychev polynomial, and the solutions to more general criteria are also well understood.
Revised: Under some criterion, the optimal polynomial is a variant of the Chebychev polynomial, and the solutions to more general criteria are also well understood.

Original: See Appendix B.1 for more.
Revised: See Appendix B.1 for more.

Original: A scaling trick is commonly used in practice.
Revised: A scaling trick is commonly used in practice.

Original: It reduces the problem of preconditioning Q to the problem of preconditioning a scaled matrix Q sca =
Revised: It reduces the problem of preconditioning Q to the problem of preconditioning a scaled matrix Q sca =

Original: Q/  m .
Revised: Q/  m .

Original: Scaling employs two steps: ﬁrst, we ﬁnd a polynomial g that approximates f ( x )
Revised: Scaling employs two steps: ﬁrst, we ﬁnd a polynomial g that approximates f ( x )

Original: = 1 in x 2 [  1 /  m , 1] ; second, set ˆ g (  ) =
Revised: = 1 in x 2 [  1 /  m , 1] ; second, set ˆ g (  ) =

Original: g (  /  m ) and use ˆ g ( Q ) =
Revised: g (  /  m ) and use ˆ g ( Q ) =

Original: g ( Q/  m )
Revised: g ( Q/  m )

Original: =
Revised: =

Original: g ( Q sca ) as the ﬁnal preconditioned matrix.
Revised: g ( Q sca ) as the ﬁnal preconditioned matrix.

Original: It is easy to verify ˆ g approximates ˆ f (  )
Revised: It is easy to verify ˆ g approximates ˆ f (  )

Original: =
Revised: =

Original: 1 in [  1 ,  m ] .
Revised: 1 in [  1 ,  m ] .

Original: Thus this approach is essentially identical to solving the approximation problem in [  1 ,  m ] .
Revised: Thus this approach is essentially identical to solving the approximation problem in [  1 ,  m ] .

Original: Johnson et al.
Revised: Johnson et al.

Original: (1983) use this trick mainly to simplify notation since they can assume  m = 1 without loss of generality.
Revised: (1983) use this trick mainly to simplify notation since they can assume  m = 1 without loss of generality.

Original: We will use this scaling trick for a different purpose (see
Revised: We will use this scaling trick for a different purpose (see

Original: Section 3.3).
Revised: Section 3.3).

Original: 3. P RECONDITIONING L AYER IN D EEP N ETS
Revised: 3. P RECONDITIONING L AYER IN D EEP N ETS

Original: Suppose D ( W 1 , . . .
Revised: Suppose D ( W 1 , . . .

Original: , W L ) is a deep net parameterized by weights W 1 , . . .
Revised: , W L ) is a deep net parameterized by weights W 1 , . . .

Original: , W L for layers l{ 1 , . . .
Revised: , W L for layers l{ 1 , . . .

Original: , L } .
Revised: , L } .

Original: To control the spectrum of a weight W l , we want to embed a preconditioner ˆ g into the neural net.
Revised: To control the spectrum of a weight W l , we want to embed a preconditioner ˆ g into the neural net.

Original: Among various preconditioners, polynomial ones are appealing since their gradient is simple and permits natural integration with backpropagation.
Revised: Among various preconditioners, polynomial ones are appealing since their gradient is simple and permits natural integration with backpropagation.

Original: For this we present a preconditioning layer (PC-layer) as follows: a PC-layer ˆ g ( W ) =
Revised: For this we present a preconditioning layer (PC-layer) as follows: a PC-layer ˆ g ( W ) =

Original: g ( SN ( W )) is the concatenation of a preconditioning polynomial g and an SN operation of (Miyato et al., 2018)
Revised: g ( SN ( W )) is the concatenation of a preconditioning polynomial g and an SN operation of (Miyato et al., 2018)

Original: (see Appendix app sub: details of FPC and APC for details of SN ( W ) ).
Revised: (see Appendix app sub: details of FPC and APC for details of SN ( W ) ).

Original: The SN operator is used as a scaling operator (reason explained later).
Revised: The SN operator is used as a scaling operator (reason explained later).

Original: We describe an efﬁcient implementation of PC-layer in Appendix C.3.
Revised: We describe an efﬁcient implementation of PC-layer in Appendix C.3.

Original: In our case, we use A = SN ( W ) to indicate the scaled matrix.
Revised: In our case, we use A = SN ( W ) to indicate the scaled matrix.

Original: Prior work on polynomial preconditioners (Johnson et al., 1983; Chen, 2005) often study square matrices.
Revised: Prior work on polynomial preconditioners (Johnson et al., 1983; Chen, 2005) often study square matrices.

Original: To handle rectangular matrices, some modiﬁcations are needed.
Revised: To handle rectangular matrices, some modiﬁcations are needed.

Original: A na¨ıve solution is to apply a preconditioner to the symmetrized matrix A T A , leading to a matrix g
Revised: A na¨ıve solution is to apply a preconditioner to the symmetrized matrix A T A , leading to a matrix g

Original: ( A ) =
Revised: ( A ) =

Original: p ( A T A )
Revised: p ( A T A )

Original: A T A .
Revised: A T A .

Original: This solution works for linear models (see Appendix B.2 for details), but it is not appropriate for deep nets since the shape of p ( A T A )
Revised: This solution works for linear models (see Appendix B.2 for details), but it is not appropriate for deep nets since the shape of p ( A T A )

Original: A T A 2 R m ⇥ m differs from A .
Revised: A T A 2 R m ⇥ m differs from A .

Original: To maintain the shape of size n ⇥ m , we propose to transform A to g
Revised: To maintain the shape of size n ⇥ m , we propose to transform A to g

Original: ( A ) =
Revised: ( A ) =

Original: p ( AA T )
Revised: p ( AA T )

Original: A 2 R n ⇥ m .
Revised: A 2 R n ⇥ m .

Original: This transformation works for general parameterized models including linear models and neural nets.
Revised: This transformation works for general parameterized models including linear models and neural nets.

Original: For a detailed comparison of these two approaches, see Appendix B.2.
Revised: For a detailed comparison of these two approaches, see Appendix B.2.

Original: The following claim relates the spectrum of A and p ( AA T ) A ; see the proof in Appendix E.2.2.
Revised: The following claim relates the spectrum of A and p ( AA T ) A ; see the proof in Appendix E.2.2.

Original: Claim 3.2 Suppose A 2 R n ⇥ m has singular values  1  · · ·   m .
Revised: Claim 3.2 Suppose A 2 R n ⇥ m has singular values  1  · · ·   m .

Original: Suppose g ( x ) =
Revised: Suppose g ( x ) =

Original: p
Revised: p

Original: x 2 ) x where p is a polynomial.
Revised: x 2 ) x where p is a polynomial.

Original: Then the singular values of g ( A ) =
Revised: Then the singular values of g ( A ) =

Original: p ( AA T )
Revised: p ( AA T )

Original: A are | g (  1 ) | , . . .
Revised: A are | g (  1 ) | , . . .

Original: , | g (  m ) | .
Revised: , | g (  m ) | .

Original: To ﬁnd a polynomial p such that g ( A ) =
Revised: To ﬁnd a polynomial p such that g ( A ) =

Original: p ( AA T )
Revised: p ( AA T )

Original: A is well-conditioned, we need to ﬁnd a polynomial p such that g ( x )
Revised: A is well-conditioned, we need to ﬁnd a polynomial p such that g ( x )

Original: = p ( x 2 ) x maps [  1 ,  m ] to [1  ✏ , 1] for some ✏ .
Revised: = p ( x 2 ) x maps [  1 ,  m ] to [1  ✏ , 1] for some ✏ .

Original: This can be formulated as a function approximation problem: ﬁnd a polynomial g ( x ) in G k that approximates a function f (  ) = 1 in  2 [  1 ,  m ] where G k = { g ( x ) =
Revised: This can be formulated as a function approximation problem: ﬁnd a polynomial g ( x ) in G k that approximates a function f (  ) = 1 in  2 [  1 ,  m ] where G k = { g ( x ) =

Original: xp ( x 2 ) | p 2 P k } .
Revised: xp ( x 2 ) | p 2 P k } .

Original: We will describe the algorithm for ﬁnding the preconditioning polynomial g in Sec. 3.3.
Revised: We will describe the algorithm for ﬁnding the preconditioning polynomial g in Sec. 3.3.

Original: In principle, the PC-layer can be added to any deep net including supervised learning and GANs.
Revised: In principle, the PC-layer can be added to any deep net including supervised learning and GANs.

Original: Here, we focus on GANs for the following reason.
Revised: Here, we focus on GANs for the following reason.

Original: Current algorithms for supervised learning already work quite well, diminishing the effect of preconditioning.
Revised: Current algorithms for supervised learning already work quite well, diminishing the effect of preconditioning.

Original: In contrast, for GANs, there is a lot of room to improve training.
Revised: In contrast, for GANs, there is a lot of room to improve training.

Original: Following SN-GAN which applies SN to the discriminator of
Revised: Following SN-GAN which applies SN to the discriminator of

Original: GANs, in the experiments we apply PC to the discriminator.
Revised: GANs, in the experiments we apply PC to the discriminator.

Original: 3. F INDING P RECONDITIONING P OLYNOMIALS
Revised: 3. F INDING P RECONDITIONING P OLYNOMIALS

Original: In this subsection, we discuss how to generate preconditioning polynomials.
Revised: In this subsection, we discuss how to generate preconditioning polynomials.

Original: This generation is done off-line and independent of training.
Revised: This generation is done off-line and independent of training.

Original: We will present the optimization formulation and discuss the choice of a few hyperparameters such as the desirable range and the target function f .
Revised: We will present the optimization formulation and discuss the choice of a few hyperparameters such as the desirable range and the target function f .

Original: Optimization formulation .
Revised: Optimization formulation .

Original: Suppose we are given a range [  L ,  U ] , a target function f , and an integer k ; the speciﬁc choices are discussed later.
Revised: Suppose we are given a range [  L ,  U ] , a target function f , and an integer k ; the speciﬁc choices are discussed later.

Original: Suppose we want to solve the following approximation problem: ﬁnd the best polynomial of the form g ( x ) =
Revised: Suppose we want to solve the following approximation problem: ﬁnd the best polynomial of the form g ( x ) =

Original: x ( a 0 + a 1 x 2 + · · · + a k x 2 k ) that approximates f ( x ) in domain [  L ,  U ] , i.e., solve
Revised: x ( a 0 + a 1 x 2 + · · · + a k x 2 k ) that approximates f ( x ) in domain [  L ,  U ] , i.e., solve

Original: [Equation]
Revised: [Equation]

Original: (3) where G k = { g ( x ) =
Revised: (3) where G k = { g ( x ) =

Original: xp ( x 2 ) | p 2 P k } , d [  L ,  U ] is a distance metric on the function space
Revised: xp ( x 2 ) | p 2 P k } , d [  L ,  U ] is a distance metric on the function space

Original: C [  L ,  U ] , such as the ` 1 distance d [  L ,  U ] ( f, g ) =
Revised: C [  L ,  U ] , such as the ` 1 distance d [  L ,  U ] ( f, g ) =

Original: max t 2 [  L ,  U ] | f ( t )  g ( t ) | .
Revised: max t 2 [  L ,  U ] | f ( t )  g ( t ) | .

Original: We consider a weighted least-square problem suggested by Johnson et al.
Revised: We consider a weighted least-square problem suggested by Johnson et al.

Original: [Equation]
Revised: [Equation]

Original: where w ( x ) = x ↵ is a weight function used in (Johnson et al., 1983).
Revised: where w ( x ) = x ↵ is a weight function used in (Johnson et al., 1983).

Original: We discretize the objective and solve the ﬁnite-sample version of Eq.
Revised: We discretize the objective and solve the ﬁnite-sample version of Eq.

Original: (4) as follows:
Revised: (4) as follows:

Original: [Equation]
Revised: [Equation]

Original: where x i 

Revised: where x i

Original: [  L ,  U ] , 8 i (e.g., drawn from uniform distribution on [  L ,  U ] ).
Revised: [  L ,  U ] , 8 i (e.g., drawn from uniform distribution on [  L ,  U ] ).

Original: This is a weighted least squares problem (thus convex) that can be easily solved by a standard solver.
Revised: This is a weighted least squares problem (thus convex) that can be easily solved by a standard solver.

Original: Choice of desirable range [  L ,  U ] .
Revised: Choice of desirable range [  L ,  U ] .

Original: The range [  L ,  U ] within which we want to approximate the target function is often chosen to be the convex hull of the singular values of the matrix to be preconditioned.
Revised: The range [  L ,  U ] within which we want to approximate the target function is often chosen to be the convex hull of the singular values of the matrix to be preconditioned.

Original: For the original matrix W , the desirable range [  L ,  U ] =
Revised: For the original matrix W , the desirable range [  L ,  U ] =

Original: [  min ( W ) ,  max ( W )] .
Revised: [  min ( W ) ,  max ( W )] .

Original: However, this range varies across different layers and different iterations.
Revised: However, this range varies across different layers and different iterations.

Original: For this reason we scale each W by 1 / k W k 2 to obtain A so that its singular values lie in a ﬁxed range [0 , 1] .
Revised: For this reason we scale each W by 1 / k W k 2 to obtain A so that its singular values lie in a ﬁxed range [0 , 1] .

Original: Note that a more precise range is [  min ( A )  , 1] , but we can relax it to [0 , 1] .
Revised: Note that a more precise range is [  min ( A ) /  max ( A ) , 1] , but we can relax it to [0 , 1] .

Original: We follow Miyato et al.
Revised: We follow Miyato et al.

Original: (2018) to use one power iteration to estimate the spectral norm ˜ W ⇡ k W k 2 , and denote SN ( W ) = W/
Revised: (2018) to use one power iteration to estimate the spectral norm ˜ W ⇡ k W k 2 , and denote SN ( W ) = W/

Original: ˜ W .
Revised: ˜ W .

Original: Since ˜ W is not exactly k W k 2 , the range of singular values of A = SN ( W ) may not be exactly in [0 , 1] .
Revised: Since ˜ W is not exactly k W k 2 , the range of singular values of A = SN ( W ) may not be exactly in [0 , 1] .

Original: We have checked the empirical estimation and found that the estimated spectral norm during the training of SN-GAN is often less than 1 .
Revised: We have checked the empirical estimation and found that the estimated spectral norm during the training of SN-GAN is often less than 1 .

Original: 1 times the true spectral norm (see Fig.
Revised: 1 times the true spectral norm (see Fig.

Original: 5 in Appendix
Revised: 5 in Appendix

Original: E.1), thus we pick [  L ,  U ] =
Revised: E.1), thus we pick [  L ,  U ] =

Original: [0 , 1 . 1] in our implementation.
Revised: [0 , 1 . 1] in our implementation.

Original: Choice of target function f (  ) .
Revised: Choice of target function f (  ) .

Original: Previously, we discuss the ideal situation that [  L ,  U ] =
Revised: Previously, we discuss the ideal situation that [  L ,  U ] =

Original: [  1 ,  m ] , thus the target function is 1 .
Revised: [  1 ,  m ] , thus the target function is 1 .

Original: In the previous paragraph, we have relaxed the desirable range to [0 ,  U ] , then we cannot set f ( x ) = 1 in [0 ,  U ] , because any polynomial g (  ) 2 G k must satisfy g (0) = 0 , causing large approximation error at  = 0 .
Revised: In the previous paragraph, we have relaxed the desirable range to [0 ,  U ] , then we cannot set f ( x ) = 1 in [0 ,  U ] , because any polynomial g (  ) 2 G k must satisfy g (0) = 0 , causing large approximation error at  = 0 .

Original: We shall set f (0) = 0 .
Revised: We shall set f (0) = 0 .

Original: A candidate target function is PL b ( x ) , where PL b ( x ) = ⇢ x/b, x < b 1 , x  b is deﬁned as a piece-wise linear function with cutoff point b .
Revised: A candidate target function is PL b ( x ) , where PL b ( x ) = ⇢ x/b, x < b 1 , x  b is deﬁned as a piece-wise linear function with cutoff point b .

Original: If the cutoff point b <  min ( A ) , then PL b (  ) maps all singular values of A to 1 .
Revised: If the cutoff point b <  min ( A ) , then PL b (  ) maps all singular values of A to 1 .

Original: While setting all singular values to 1 is ideal for fast training, this may reduce the expressiveness for deep nets.
Revised: While setting all singular values to 1 is ideal for fast training, this may reduce the expressiveness for deep nets.

Original: More speciﬁcally, the set of functions { D ( W 1 , . .
Revised: More speciﬁcally, the set of functions { D ( W 1 , . .

Original: , W L ) | eig ( W Tl W l ) ✓ { 1 }} is smaller than { D ( W 1 , . .
Revised: , W L ) | eig ( W Tl W l ) ✓ { 1 }} is smaller than { D ( W 1 , . .

Original: , W L ) | eig ( W Tl W l )
Revised: , W L ) | eig ( W Tl W l )

Original: ✓ [  0 , 1] } , thus forcing all singular values to be may hurt the representation power.
Revised: ✓ [  0 , 1] } , thus forcing all singular values to be may hurt the representation power.

Original: Therefore, we do not want the target function to have value 1 in [  min ( A ) ,  U ] .
Revised: Therefore, we do not want the target function to have value 1 in [  min ( A ) ,  U ] .

Original: In practice, the value of  min ( A ) varies for different problems, therefore we permit a ﬂexible target function f , to be chosen by a user.
Revised: In practice, the value of  min ( A ) varies for different problems, therefore we permit a ﬂexible target function f , to be chosen by a user.

Original: In our implementation, we restrict target functions to a family of piece-wise linear functions.
Revised: In our implementation, we restrict target functions to a family of piece-wise linear functions.

Original: We use PL b ( x ) with a relatively large cutoff point b , such as 0 . 8 and 0 .
Revised: We use PL b ( x ) with a relatively large cutoff point b , such as 0 . 8 and 0 .

Original: We plot our candidate target functions PL 0 .
Revised: We plot our candidate target functions PL 0 .

Original: 3 , PL 0 .
Revised: 3 , PL 0 .

Original: 4 , PL 0 .
Revised: 4 , PL 0 .

Original: 6 and PL 0 .
Revised: 6 and PL 0 .

Original: 8 in Figure 3.
Revised: 8 in Figure 3.

Original: As the cutoff point b changes from 1 to 0 , the function PL b becomes more aggressive as it pushes more singular values to 1 .
Revised: As the cutoff point b changes from 1 to 0 , the function PL b becomes more aggressive as it pushes more singular values to 1 .

Original: As a result, the optimization will likely become easier, while the representation power becomes weaker.
Revised: As a result, the optimization will likely become easier, while the representation power becomes weaker.

Original: The exact choice of the target function is likely problem-dependent, and we discuss two strategies to select them in Section 3.4.
Revised: The exact choice of the target function is likely problem-dependent, and we discuss two strategies to select them in Section 3.4.

Original: Search space of preconditioning polynomial .
Revised: Search space of preconditioning polynomial .

Original: As mentioned earlier, the default search space is
Revised: As mentioned earlier, the default search space is

Original: G k = { g ( x ) =
Revised: G k = { g ( x ) =

Original: xp ( x 2 ) | p 2 P k } for a pre-ﬁxed k .
Revised: xp ( x 2 ) | p 2 P k } for a pre-ﬁxed k .

Original: The degree of g (  ) is an important hyperparameter.
Revised: The degree of g (  ) is an important hyperparameter.

Original: On the one hand, the higher the degree, the better the polynomial can ﬁt the target function f .
Revised: On the one hand, the higher the degree, the better the polynomial can ﬁt the target function f .

Original: On the other hand, higher degree leads to more computation.
Revised: On the other hand, higher degree leads to more computation.

Original: In our implementation, we consider k = 1 , 2 , 3 , 4 , i.e., polynomials of degree 3 , 5 , 7 and 9 .
Revised: In our implementation, we consider k = 1 , 2 , 3 , 4 , i.e., polynomials of degree 3 , 5 , 7 and 9 .

Original: The extra time is relatively small; see Section subsec: comutation time for details.
Revised: The extra time is relatively small; see Section C.4 for details.

Original: [Equation]
Revised: [Equation]

Original: IXED P RECONDITIONING AND A DAPTIVE PRECONDITIONING
Revised: AND A DAPTIVE PRECONDITIONING

Original: The preconditioning polynomial can be determined by the target function and the degree k .
Revised: The preconditioning polynomial can be determined by the target function and the degree k .

Original: Which polynomial shall we use during training?
Revised: Which polynomial shall we use during training?

Original: Candidate preconditioners .
Revised: Candidate preconditioners .

Original: At ﬁrst sight, there are two hyper-parameters b and k .
Revised: At ﬁrst sight, there are two hyper-parameters b and k .

Original: Nevertheless, if b is small (steep slope), then it is hard to approximate PL b by low-order polynomials.
Revised: Nevertheless, if b is small (steep slope), then it is hard to approximate PL b by low-order polynomials.

Original: For each degree k 2 { 3 , 5 , 7 , 9 } , there is a certain b k such that b < b k leads to large approximation error.
Revised: For each degree k 2 { 3 , 5 , 7 , 9 } , there is a certain b k such that b < b k leads to large approximation error.

Original: We ﬁnd that b 3 ⇡ 0 .
Revised: We ﬁnd that b 3 ⇡ 0 .

Original: 8 , b 5 ⇡ 0 .
Revised: 8 , b 5 ⇡ 0 .

Original: 6 , b 7 ⇡ 0 .
Revised: 6 , b 7 ⇡ 0 .

Original: 4 , b 9 ⇡ 0 .
Revised: 4 , b 9 ⇡ 0 .

Original: After ﬁtting PL 0 . 3 , PL 0 .
Revised: After ﬁtting PL 0 . 3 , PL 0 .

Original: 4 , PL 0 .
Revised: 4 , PL 0 .

Original: 6 and PL 0 .
Revised: 6 and PL 0 .

Original: 8 , the resulting polynomials are shown in Figure 3.
Revised: 8 , the resulting polynomials are shown in Figure 3.

Original: A natural approach is to add the PC-layer to all layers of the neural net, resulting in a preconditioned net D PC ( ✓ )
Revised: A natural approach is to add the PC-layer to all layers of the neural net, resulting in a preconditioned net D PC ( ✓ )

Original: =
Revised: =

Original: D ( g ( SN ( W 1 )) , . . .
Revised: D ( g ( SN ( W 1 )) , . . .

Original: , g ( SN ( W L )) .
Revised: , g ( SN ( W L )) .

Original: We call this method ﬁxed preconditioning (FPC).
Revised: We call this method ﬁxed preconditioning (FPC).

Original: Just like other hyperparameters, in practice, we can try various preconditioners and pick the one with the best performance.
Revised: Just like other hyperparameters, in practice, we can try various preconditioners and pick the one with the best performance.

Original: Not surprisingly, the best preconditioner varies for different datasets.
Revised: Not surprisingly, the best preconditioner varies for different datasets.

Original: Adaptive preconditioning (APC) .
Revised: Adaptive preconditioning (APC) .

Original: Motivated by adaptive learning rate schemes like Adam (Kingma & Ba, 2014) and LARS (You et al., 2017), we propose an adaptive preconditioning scheme.
Revised: Motivated by adaptive learning rate schemes like Adam (Kingma & Ba, 2014) and LARS (You et al., 2017), we propose an adaptive preconditioning scheme.

Original: In APC, we apply the preconditioner in a epoch-adaptive and layer-adaptive manner: at each epoch and for each layer the algorithm will automatically pick a proper preconditioner based on the current condition number.
Revised: In APC, we apply the preconditioner in a epoch-adaptive and layer-adaptive manner: at each epoch and for each layer the algorithm will automatically pick a proper preconditioner based on the current condition number.

Original: The standard condition number  ( A ) =
Revised: The standard condition number  ( A ) =

Original:  max ( A )  min ( A ) is not necessarily a good indicator for the optimization performance.
Revised:  max ( A )  min ( A ) is not necessarily a good indicator for the optimization performance.

Original: In APC, we use a modiﬁed condition number ˜  ( A ) = max ( A ) ( P m [Equation]
Revised: In APC, we use a modiﬁed condition number ˜  ( A ) =

Original: 1  i ( A )) /m

Original: [Equation]
Revised: [Equation]

Original: where A has m columns and m 0 =
Revised: where A has m columns and m 0 =

Original: d m 10 e .
Revised: d m 10 e .

Original: We prepare r preconditioning polynomials g 1 , . . .
Revised: We prepare r preconditioning polynomials g 1 , . . .

Original: , g r with different strength (e.g., the four polynomials g 1 , g 2 , g 3 , g 4 shown in Figure 3).
Revised: , g r with different strength (e.g., the four polynomials g 1 , g 2 , g 3 , g 4 shown in Figure 3).

Original: We set a number of ranges [0 , ⌧ 1 ] , [ ⌧ 1 , ⌧ 2 ] , . . .
Revised: We set a number of ranges [0 , ⌧ 1 ] , [ ⌧ 1 , ⌧ 2 ] , . . .

Original: , [ ⌧ r , 1 ] and let ⌧ 0 = 0 , ⌧ r +1 = 1 .
Revised: , [ ⌧ r , 1 ] and let ⌧ 0 = 0 , ⌧ r +1 = 1 .

Original: If the modiﬁed condition number of A falls into the range [ ⌧ i , ⌧ i +1 ] for i 2 { 0 , 1 , . . .
Revised: If the modiﬁed condition number of A falls into the range [ ⌧ i , ⌧ i +1 ] for i 2 { 0 , 1 , . . .

Original: , r } , we will use g i in the PC-layer.
Revised: , r } , we will use g i in the PC-layer.

Original: In our implementation, we set r = 4 .
Revised: In our im- plementation, we set r = 4 .

Original: To save computation, we only compute the modiﬁed condition number and update PC strength at a ﬁxed interval (e.g., every 1000 iterations).
Revised: To save computation, we only compute the modiﬁed condition number and update PC strength at a ﬁxed interval (e.g., every 1000 iterations).

Original: The summary of APC is presented in Table 3 in Appendix C.2.
Revised: The summary of APC is presented in Table 3 in Appendix C.2.

Original: Computation time.
Revised: Computation time.

Original: We use a few implementation tricks; see Appendix C.3.
Revised: We use a few implementation tricks; see Appendix C.3.

Original: In our implementation of FPC with a degree 3 , 5 , 7 or 9 polynomial, the actual added time is around 20  30% (Fig.
Revised: In our implementation of FPC with a degree 3 , 5 , 7 or 9 polynomial, the actual added time is around 20  30% (Fig.

Original: 4 (a)) of the original training time of SN-GAN.
Revised: 4 (a)) of the original training time of SN-GAN.

Original: Fig.
Revised: Fig.

Original: 4 (b) shows that the extra time of APC over SN is often less than 10% .
Revised: 4 (b) shows that the extra time of APC over SN is often less than 10% .

Original: See Appendix C.4 for more on the compputation time.
Revised: See Appendix C.4 for more on the computation time.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We will demonstrate the following two ﬁndings.
Revised: We will demonstrate the following two ﬁndings.

Original: First, SN-GAN still suffers from training instabilities, and the failure case is accompanied by large condition numbers.
Revised: First, SN-GAN still suffers from training instabilities, and the failure case is accompanied by large condition numbers.

Original: Second, PC layers can reduce the condition number, and improve the ﬁnal performance, especially for high resolution data (LSUN 256 ⇥ 256 ).
Revised: Second, PC layers can reduce the condition number, and improve the ﬁnal performance, especially for high resolution data (LSUN 256 ⇥ 256 ).

Original: We conduct a set of experiments for unconditional image generation on CIFAR-10 ( 32 ⇥ 32 ), STL10 ( 48 ⇥ 48 ), LSUN-bedroom ( 128 ⇥ 128 and 256 ⇥ 256 ), LSUN-tower ( 256 ⇥ 256 ) and LSUNliving-room ( 256 ⇥ 256 ).
Revised: We conduct a set of experiments for unconditional image generation on CIFAR-10 ( 32 ⇥ 32 ), STL10 ( 48 ⇥ 48 ), LSUN-bedroom ( 128 ⇥ 128 and 256 ⇥ 256 ), LSUN-tower ( 256 ⇥ 256 ) and LSUNliving-room ( 256 ⇥ 256 ).

Original: We also compare the condition numbers of the discriminator layers for different normalization methods to demonstrate the connection between the condition number and the performance.
Revised: We also compare the condition numbers of the discriminator layers for different normalization methods to demonstrate the connection between the condition number and the performance.

Original: The following methods are used in our experiments: standard SN; SVD with DOptimal Reg.
Revised: The following methods are used in our experiments: standard SN; SVD with DOptimal Reg.

Original: (Jiang et al., 2019);
Revised: (Jiang et al., 2019);

Original: FPC with degree 3 or 7 preconditioners; APC.
Revised: FPC with degree 3 or 7 preconditioners; APC.

Original: Following Miyato et al.
Revised: Following Miyato et al.

Original: (2018), we use the log loss GAN on the CNN structure and the hinge loss GAN on the ResNet structure.
Revised: (2018), we use the log loss GAN on the CNN structure and the hinge loss GAN on the ResNet structure.

Original: CIFAR and STL: Training failure of (1,1)-update .
Revised: CIFAR and STL: Training failure of (1,1)-update .

Original: Tuning a GAN is notoriously difﬁcult and sensitivity to hyper-parameters.
Revised: Tuning a GAN is notoriously difﬁcult and sensitivity to hyper-parameters.

Original: Even for low-resolution images, without prior knowledge of good hyper-parameters such as D it , G it , training a GAN is often not trivial.
Revised: Even for low-resolution images, without prior knowledge of good hyper-parameters such as D it , G it , training a GAN is often not trivial.

Original: On CIFAR10, SN-GAN uses
Revised: On CIFAR10, SN-GAN uses

Original: D it = 5 , G it = 1 for ResNet; for simplicity, we call it a (5 , 1) -update.
Revised: D it = 5 , G it = 1 for ResNet; for simplicity, we call it a (5 , 1) -update.

Original: However, using a (1 , 1) - update, i.e., changing D it = 5 to D it = 1 while keeping G it = 1 , will lead to an SN-GAN training failure: a dramatic decrease of ﬁnal performance and an FID score above 77 .
Revised: However, using a (1 , 1) - update, i.e., changing D it = 5 to D it = 1 while keeping G it = 1 , will lead to an SN-GAN training failure: a dramatic decrease of ﬁnal performance and an FID score above 77 .

Original: SN-GAN with (1 , 1) - update also fails on STL data, yielding an FID score above 147 .
Revised: SN-GAN with (1 , 1) - update also fails on STL data, yielding an FID score above 147 .

Original: We are interested in stabilizing the (1 , 1) -update for two reasons: ﬁrst, trainability for both (1 , 1) -update
Revised: We are interested in stabilizing the (1 , 1) -update for two reasons: ﬁrst, trainability for both (1 , 1) -update

Original: and (5 , 1) -update means improved training stability; second, the (1 , 1) -update requires only about 1 / 3 of the time compared to the (5 , 1) -update.
Revised: and (5 , 1) -update means improved training stability; second, the (1 , 1) -update requires only about 1 / 3 of the time compared to the (5 , 1) -update.

Original: Therefore, in the ﬁrst experiment, we explore GAN-training with (1 , 1) -update on CIFAR-10 and STL-10.
Revised: Therefore, in the ﬁrst experiment, we explore GAN-training with (1 , 1) -update on CIFAR-10 and STL-10.

Original: Failure mode: large condition numbers .
Revised: Failure mode: large condition numbers .

Original: Understanding the failure mode of training is often very useful for designing algorithms (e.g., Glorot & Bengio, 2010).
Revised: Understanding the failure mode of training is often very useful for designing algorithms (e.g., Glorot & Bengio, 2010).

Original: We suspect that a large condition number is a failure mode for GAN training.
Revised: We suspect that a large condition number is a failure mode for GAN training.

Original: As Table 1 shows, the high FID scores (bad case) of
Revised: As Table 1 shows, the high FID scores (bad case) of

Original: SN-GAN are accompanied by large condition numbers
Revised: SN-GAN are accompanied by large condition numbers.

Original: PC reduces condition numbers and rectiﬁes failures .
Revised: PC reduces condition numbers and rectiﬁes failures .

Original: Table 1 shows that FPC and APC can both greatly improve the training performance: they reduce FID from 77 to less than 20 for CIFAR-10, and reduce FID from 147 to less than 34 for STL in 200 k iterations.
Revised: Table 1 shows that FPC and APC can both greatly improve the training performance: they reduce FID from 77 to less than 20 for CIFAR-10, and reduce FID from 147 to less than 34 for STL in 200 k iterations.

Original: The evolution of the 5 smallest singular values of the adaptive preconditioned matrices and the condition numbers are showed in
Revised: The evolution of the 5 smallest singular values of the adaptive preconditioned matrices and the condition numbers are showed in

Original: Fig. 1(b) and Fig. 2(b) for STL-10 training on ResNet with D it = 1 .
Revised: Fig. 1(b) and Fig. 2(b) for STL-10 training on ResNet with D it = 1 .

Original: This shows that PC-GAN successfully improves the spectrum of weight matrices in this setting.
Revised: This shows that PC-GAN successfully improves the spectrum of weight matrices in this setting.

Original: Experiments on “good” case of SN-GAN 
Revised: Experiments on “good” case of SN-GAN .

Original: We report the results for the (5 , 1) -update on CIFARand STL-10 with ResNet in the Appendix.
Revised: We report the results for the (5 , 1) -update on CIFARand STL-10 with ResNet in the Appendix.

Original: For those FPC and APC achieve similar or slightly better
Revised: For those FPC and APC achieve similar or slightly better

Original: FID scores.
Revised: FID scores.

Original: We also report IS scores there.
Revised: We also report IS scores there.

Original: We also list the results of PC and multiple baselines on the CNN structure in the Appendix
Revised: We also list the results of PC and multiple baselines on the CNN structure in the Appendix.

Original: High resolution images LSUN.
Revised: High resolution images LSUN.

Original: Using high-resolution data is more challenging.
Revised: Using high-resolution data is more challenging.

Original: We present numerical results on LSUN bedroom (128 ⇥ 128, and 256 ⇥ 256) , LSUN tower (256 ⇥ 256) and LSUN living room (256 ⇥ 256) data in Table 2.
Revised: We present numerical results on LSUN bedroom (128 ⇥ 128, and 256 ⇥ 256) , LSUN tower (256 ⇥ 256) and LSUN living room (256 ⇥ 256) data in Table 2.

Original: The training time for one instance is 30 hours on a single
Revised: The training time for one instance is 30 hours on a single

Original: RTX 2080 Ti (200k iterations).
Revised: RTX 2080 Ti (200k iterations).

Original: Note, SN-GAN is unstable and results in FID > 80 for LSUN-bedroom 256 ⇥ 256.
Revised: Note, SN-GAN is unstable and results in FID > 80 for LSUN-bedroom 256 ⇥ 256.

Original: The SVD method, our FPC and APC generate reasonable FID scores on all three data sets.
Revised: The SVD method, our FPC and APC generate reasonable FID scores on all three data sets.

Original: Importantly, our
Revised: Importantly, our

Revised: FPC is comparable or better than SVD, and our APC consistently outperforms the SVD method by

Original: [Table] [Table]
Revised: [Table] [Table]

Original: FPC is comparable or better than SVD, and our APC consistently outperforms the SVD method by 4-6 FID scores in most cases.
Revised: 4-6 FID scores in most cases.

Original: Also note, the condition numbers of the failure case of SN-GAN are much higher than the two normal cases of SN-GAN.
Revised: Also note, the condition numbers of the failure case of SN-GAN are much higher than the two normal cases of SN-GAN.

Original: In all cases, FPC and APC achieve signiﬁcantly lower condition numbers than SN-GAN.
Revised: In all cases, FPC and APC achieve signiﬁcantly lower condition numbers than SN-GAN.

Original: APC achieves higher condition numbers than FPC
Revised: APC achieves higher condition numbers than FPC

Original: [Table]
Revised: [Table]

Original: and also better FID scores.
Revised: and also better FID scores.

Original: We suspect that FPC over-controls the condition numbers which leads to lower representation power.
Revised: We suspect that FPC over-controls the condition numbers which leads to lower representation power.

Original: In contrast, APC strikes a better balance between representation and optimization than FPC.
Revised: In contrast, APC strikes a better balance between representation and optimization than FPC.

Original: The generated image samples are presented in Appendix F.3.
Revised: The generated image samples are presented in Appendix F.5.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We prove that for a deep pyramid linear networks, if all weight matrices have bounded singular values throughout training, then the algorithm converges to a global minimal value at a geometric rate.
Revised: We prove that for a deep pyramid linear networks, if all weight matrices have bounded singular values throughout training, then the algorithm converges to a global minimal value at a geometric rate.

Original: This result indicates that small weight matrix condition numbers are helpful for training.
Revised: This result indicates that small weight matrix condition numbers are helpful for training.

Original: We propose a preconditioning (PC) layer to improve weight matrix condition numbers during training, by leveraging tools from polynomial preconditioning literature.
Revised: We propose a preconditioning (PC) layer to improve weight matrix condition numbers during training, by leveraging tools from polynomial preconditioning literature.

Original: It is differentiable
Revised: It is differentiable

Original: [Table]
Revised: [Table]

Original: and thus can be plugged into any neural net.
Revised: and thus can be plugged into any neural net.

Original: We propose two methods to utilize the PC-layer: in FPC (ﬁxed preconditioning), we add a ﬁxed PC-layer to all layers; in APC (adaptive preconditioning), we add
Revised: We propose two methods to utilize the PC-layer: in FPC (ﬁxed preconditioning), we add a ﬁxed PC-layer to all layers; in APC (adaptive preconditioning), we add

Original: PC-layers with different preconditioning power depending on the condition number.
Revised: PC-layers with different preconditioning power depending on the condition number.

Original: Empirically, we show that applying FPC and APC to GAN training, we can generate good images in a few cases that SN-GAN perform badly, such as LSUN-bedroom 256 ⇥ 256 image generation.
Revised: Empirically, we show that applying FPC and APC to GAN training, we can generate good images in a few cases that SN-GAN perform badly, such as LSUN-bedroom 256 ⇥ 256 image generation.