C ONTROLLING THE C OMPLEXITY AND L IPSCHITZ C ONSTANT IMPROVES P OLYNOMIAL N ETS
A BSTRACT
While the class of Polynomial Nets demonstrates comparable performance to neural networks (NN), it currently has neither theoretical generalization characterization nor robustness guarantees.
To this end, we derive new complexity bounds for the set of Coupled CP-Decomposition (CCP) and Nested Coupled CP-decomposition (NCP) models of Polynomial Nets in terms of the (cid:96) ∞ -operator-norm and the (cid:96) 2operator norm.
In addition, we derive bounds on the Lipschitz constant for both models to establish a theoretical certiﬁcate for their robustness.
The theoretical results enable us to propose a principled regularization scheme that we also evaluate experimentally in six datasets and show that it improves the accuracy as well as the robustness of the models to adversarial perturbations.
We showcase how this regularization can be combined with adversarial training, resulting in further improvements.
I NTRODUCTION
Recently, high-degree Polynomial Nets (PNs) have been demonstrating state-of-the-art performance in a range of challenging tasks like image generation (Karras et al., 2019; Chrysos and Panagakis, 2020), image classiﬁcation (Wang et al., 2018),
reinforcement learning (Jayakumar et al., 2020), non-euclidean representation learning (Chrysos et al., 2020) and sequence models (Su et al., 2020).
In particular, in public benchmarks like the Face veriﬁcation on MegaFace task 1 (KemelmacherShlizerman et al., 2016), Polynomial Nets are currently the top performing model.
A major advantage of Polynomial Nets over traditional Neural Networks 2 is that they are compatible with efﬁcient Leveled Fully Homomorphic Encryption (LFHE) protocols (Brakerski et al., 2014).
Such protocols allow efﬁcient computation on encrypted data, but they only support addition or multiplication operations i.e., polynomials.
This has prompted an effort to adapt neural networks by replacing typical activation functions with polynomial approximations (Gilad-Bachrach et al., 2016;
Hesamifard et al., 2018).
Polynomial Nets do not need any adaptation to work with LFHE.
Without doubt, these arguments motivate further investigation about the inner-workings of Polynomial
Nets.
Surprisingly, little is known about the theoretical properties of such high-degree polynomial expansions, despite their success.
Previous work on PNs (Chrysos et al., 2020; Chrysos and Panagakis, 2020) have focused on developing the foundational structure of the models as well as their training, but do not provide an analysis of their generalization ability or robustness to adversarial perturbations.
In contrast, such type of results are readily available for traditional feed-forward Deep Neural
Networks, in the form of high-probability generalization error bounds (Neyshabur et al., 2015;
Bartlett et al., 2017; Neyshabur et al., 2017; Golowich et al., 2018) or upper bounds on their Lipschitz constant (Scaman and Virmaux, 2018; Fazlyab et al., 2019; Latorre et al., 2020).
Despite their similarity in the compositional structure, theoretical results for Deep Neural Networks 2 do not apply to Polynomial Nets, as they are essentialy two non-overlapping classes of functions.
Why are such results important?
First, they provide key theoretical quantities like the sample complexity of a hypothesis class: how many samples are required to succeed at learning in the
PAC-framework.
Second, they provide certiﬁed performance guarantees to adversarial perturbations
(Szegedy et al., 2014; Goodfellow et al., 2015) via a worst-case analysis c.f. Scaman and Virmaux (2018).
Most importantly, the bounds themselves provide a principled way to regularize the hypothesis class and improve their accuracy or robustness.
For example, Generalization and Lipschitz constant bounds of Deep Neural Networks that depend on the operator-norm of their weight matrices (Bartlett et al., 2017; Neyshabur et al., 2017) have layed out the path for regularization schemes like spectral regularization (Yoshida and Miyato, 2017;
Miyato et al., 2018), Lipschitz-margin training (Tsuzuku et al., 2018) and Parseval Networks (Cisse et al., 2017), to name a few.
Indeed, such schemes have been observed in practice to improve the performance of Deep Neural
Networks.
Unfortunately, similar regularization schemes for Polynomial Nets do not exist due to the lack of analogous bounds.
Hence, it is possible that PNs are not yet being used to their fullest potential.
We believe that theoretical advances in their understanding might lead to more resilient and accurate models.
In this work, we aim to ﬁll the gap in the theoretical understanding of PNs.
We summarize our main contributions as follows:
Rademacher Complexity Bounds.
We derive bounds on the Rademacher Complexity of the Coupled CP-decomposition model (CCP) and Nested Coupled CP-decomposition model (NCP) of PNs, under the assumption of a unit (cid:96) ∞ -norm bound on the input (Theorems 1 and 3), a natural assumption in image-based applications.
Analogous bounds for the (cid:96) 2 -norm are also provided (Appendices E. and F.3).
Such bounds lead to the ﬁrst known generalization error bounds for this class of models.
Lipschitz constant Bounds.
To complement our understanding of the CCP and NCP models, we derive upper bounds on their (cid:96) ∞ -Lipschitz constants (Theorems 2 and 4), which are directly related to their robustness against (cid:96) ∞ -bounded adversarial perturbations, and provide formal guarantees.
results
Regularization schemes.
We identify key quantities that simultaneously control both Rademacher
Complexity and Lipschitz constant bounds that we previously derived, i.e., the operator norms of the weight matrices in the Polynomial Nets.
Hence, we propose to regularize the CCP and NCP models by constraining such operator norms.
In doing so, our theoretical results indicate that both the generalization and the robustness to adversarial perturbations should improve.
We propose a Projected
Stochastic Gradient Descent scheme (Algorithm 1), enjoying the same per-iteration complexity as vanilla back-propagation in the (cid:96) ∞ -norm case, and a variant that augments the base algorithm with adversarial traning (Algorithm 2).
Experiments.
We conduct experimentation in ﬁve widely-used datasets on image recognition and on dataset in audio recognition.
The experimentation illustrates how the aforementioned regularization schemes impact the accuracy (and the robust accuracy) of both CCP and NCP models, outperforming alternative schemes such as Jacobian regularization and the L 2 weight decay.
Indeed, for a grid of regularization parameters we observe that there exists a sweet-spot for the regularization parameter which not only increases the test-accuracy of the model, but also its resilience to adversarial perturbations.
Larger values of the regularization parameter also allow a trade-off between accuracy and robustness.
The observation is consistent across all datasets and all adversarial attacks demonstrating the efﬁcacy of the proposed regularization scheme.
R ADEMACHER C OMPLEXITY AND L IPSCHITZ CONSTANT BOUNDS FOR P OLYNOMIAL N ETS
Notation.
The symbol ◦ denotes the Hadamard (element-wise) product, the symbol • is the facesplitting product, while the symbol (cid:63) denotes a convolutional operator.
Matrices are denoted by uppercase letters e.g., V .
Due to the space constraints, a detailed notation is deferred to Appendix C.
Assumption on the input distribution.
Unless explicitly mentioned otherwise, we assume an (cid:96) ∞ -norm unit bound on the input data i.e., (cid:107) z (cid:107) ∞ ≤ 1 for any input z .
This is the most common assumption in image-domain applications in contemporary deep learning, i.e., each pixel takes values in [ − 1 , 1] interval.
Nevertheless, analogous results for (cid:96) 2 -norm unit bound assumptions are presented in Appendices E.3, E.4, F.3 and F.4.
We now introduce the basic concepts that will be developed throughout the paper i.e., the Rademacher
Complexity of a class of functions (Bartlett and Mendelson, 2002) and the Lipschitz constant.
Deﬁnition 1 (Empirical Rademacher Complexity) .
Let Z = { z 1 , . . .
, z n } ⊆ R d and let { σ j : j = 1 , . .
. , n } be independent Rademacher random variables i.e., taking values uniformly in {− 1 , +1 } .
Let F be a class of real-valued functions over R d .
The Empirical Rademacher complexity of F with respect to Z is deﬁned as follows:
[Equation]
Deﬁnition 2 (Lipschitz constant) .
Given two normed spaces ( X , (cid:107) · (cid:107) X ) and ( Y , (cid:107) · (cid:107) Y ), a function f : X → Y is called Lipschitz continuous with Lipschitz constant K ≥ 0 if for all x 1 , x 2 in X :
[Equation]
2. C OUPLED CP-D ECOMPOSITION OF P OLYNOMIAL N ETS (CCP MODEL )
The Coupled CP-Decomposition (CCP) model of PNs (Chrysos et al., 2020) leverages a coupled CP
Tensor decomposition (Kolda and Bader, 2009) to vastly reduce the parameters required to describe a high-degree polynomial, and allows its computation in a compositional fashion, much similar to a feed-forward pass through a traditional neural network.
The CCP model was used in Chrysos and
Panagakis (2020) to construct a generative model.
CCP can be succintly deﬁned as follows:
[Equation]
where z ∈ R d is the input data, f ( z ) ∈ R
o is the output of the model and U n ∈ R m × d , C ∈ R o × m are the learnable parameters, where m ∈ N is the hidden rank.
In Fig.
1 we provide a schematic of the architecture, while in Appendix D.1 we include further details on the original CCP formulation (and how to obtain our equivalent re-parametrization) for the interested reader.
In Theorem 1 we derive an upper bound on the complexity of CCP models with bounded (cid:96) ∞ -operatornorms of the face-splitting product of the weight matrices.
Its proof can be found in Appendix E.1.
For a given CCP model, we derive an upper bound on its (cid:96) ∞ -Lipschitz constant in Theorem 2 and its proof is given in Appendix E.4.1.
Theorem 1.
Let Z = { z 1 , . . .
, z n } ⊆ R d and suppose that (cid:107)
z j (cid:107) ∞ ≤ 1 for all j = 1 , . .
. , n .
Let
[Equation]
The Empirical Rademacher Complexity of CCP k
( k -degree CCP polynomials) with respect to Z is bounded as:
[Equation]
Proof sketch of Theorem 1 We now describe the core steps of the proof.
For the interested reader, the complete and detailed proof steps are presented in Appendix E.1.
First, Hölder’s inequality is used to bound the Rademacher complexity as:
[Equation]
[ σ j ◦ ki =1 ( U i z j )] (cid:43) ≤ E sup f ∈F
[Equation]
n (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
n
[Equation]
[ σ
[Equation]
◦
[Equation]
[Equation]
j )] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∞ .
[Equation]
This shows why the factor (cid:107) c (cid:107) 1 ≤ µ appears in the ﬁnal bound.
Then, using the mixed product property (Slyusar, 1999) and its extension to repeated Hadamard products (Lemma 7 in Appendix C.3), we can rewrite the summation in the right-hand-side of (1) as follows:
[Equation]
This step can be seen as a linearization of the polynomial by lifting the problem to a higher dimensional space.
We use this fact and the deﬁnition of the operator norm to further bound the term inside the (cid:96) ∞ -norm in the right-hand-side of (1).
Such term is bounded as the product of the (cid:96) ∞ -operator norm of • ki =1 ( U i ) , and the (cid:96) ∞ -norm of an expression involving the Rademacher variables σ j and the vectors ∗ ki =1 ( z j ) .
Finally, an application of Massart’s Lemma (Shalev-Shwartz and Ben-David (2014), Lemma 26.8) leads to the ﬁnal result.
Theorem 2.
The Lipschitz constant (with respect to the (cid:96) ∞ -norm) of the function deﬁned in Eq.
(CCP) , restricted to the set { z ∈ R d : (cid:107) z (cid:107) ∞ ≤ 1 } is bounded as:
[Equation]
2. N ESTED C OUPLED CP-D ECOMPOSITION (NCP MODEL )
The Nested Coupled CP-Decomposition (NCP) model leverages a joint hierarchical decomposition, which provided strong results in both generative and discriminative tasks in Chrysos et al.
A slight re-parametrization of the model (Appendix D.2) can be expressed with the following recursive relation:
[Equation]
where z ∈ R d is the input vector and C ∈ R o × m , A n ∈ R m × d , S n ∈ R m × m and s 1 ∈ R m are the learnable parameters.
In Fig.
1 we provide a schematic of the architecture.
In Theorem 3 we derive an upper bound on the complexity of NCP models with bounded (cid:96) ∞ -operatornorm of a matrix function of its parameters.
Its proof can be found in Appendix F.1.
For a given NCP model, we derive an upper bound on its (cid:96) ∞ -Lipschitz constant in Theorem 4 and its proof is given in
Appendix F.4.1.
Theorem 3.
Let Z = { z 1 , . . .
, z n } ⊆ R d and suppose that (cid:107)
z j (cid:107) ∞ ≤ 1 for all j = 1 , . .
. , n .
Deﬁne the matrix Φ( A 1 , S 1 , . . .
, A n , S n ) :
=
( A k • S k ) (cid:81) k − 1 i =
I ⊗ A
i • S i .
Consider the class of functions:
F k NCP :=
{ f ( z ) as in (NCP) : (cid:107) C (cid:107) ∞ ≤ µ, (cid:107) Φ( A 1 , S 1 , . .
A k , S k ) (cid:107) ∞ ≤ λ } , where C ∈ R 1 × m (single output), thus, we will write it as c , and the corresponding bound also becomes (cid:107) c (cid:107) 1 ≤ µ .
The Empirical Rademacher Complexity of NCP k (k-degree NCP polynomials) with respect to Z is bounded as:
[Equation]
Theorem 4.
The Lipschitz constant (with respect to the (cid:96) ∞ -norm) of the function deﬁned in Eq.
(NCP) , restricted to the set { z ∈ R d : (cid:107) z (cid:107) ∞ ≤ 1 } is bounded as:
[Equation]
A LGORITHMS
By constraining the quantities in the upper bounds on the Rademacher complexity (Theoremsand 3), we can regularize the empirical loss minimization objective (Mohri et al., 2018, Theorem 3.3).
Such method would prevent overﬁtting and can lead to an improved accuracy.
However, one issue with the quantities involved in Theorems 1 and 3, namely
[Equation]
is that projecting onto their level sets correspond to a difﬁcult non-convex problem.
Nevertheless, we can control an upper bound that depends on the (cid:96) ∞ -operator norm of each weight matrix:
Lemma 1.
It holds that (cid:13)(cid:13) [Equation]
=
[Equation] U
i [Equation]
Lemma 2.
It holds that (cid:13)(cid:13)(cid:13)
[Equation]
The proofs of Lemmas 1 and 2 can be found in Appendix E.2 and Appendix F.2.
These results mean that by constraining the operator norms of each weight matrix, we can control the overall complexity of the CCP and NCP models.
Projecting a matrix onto an (cid:96) ∞ -operator norm ball is a simple task that can be achieved by projecting each row of the matrix onto an (cid:96) 1 -norm ball, for example, using the well-known algorithm from
Duchi et al.
The ﬁnal optimization objective for training a regularized CCP is the following:
[Equation]
subject to
[Equation]
where ( x i , y i ) n i =1 is the training dataset, L is the loss function (e.g., cross-entropy) and µ, λ are the regularization parameters.
We notice that the constraints on the learnable parameters U i and C have the effect of simultaneously controlling the Rademacher Complexity and the Lipschitz constant of the CCP model.
For the NCP model, an analogous objective function is used.
To solve the optimization problem in Eq.
(2) we will use a Projected Stochastic Gradient Descent method Algorithm 1.
We also propose a variant that combines Adversarial Training with the projection step (Algorithm 2) with the goal of increasing robustness to adversarial examples.
Input: dataset Z , learning rate { γ t > 0 } T − 1 t =0 , iterations T , hyper-parameters R , f , Loss L .
Output: model with parameters θ .
Initialize θ .
for t = 0 to T − 1 do
Sample ( x , y ) from Z [Equation] ; x , y ) .
if t mod f = 0 then θ = [Equation]
Input: dataset Z , learning rate { γ t > 0 } T − 1 t =0 , iterations T and n , hyper-parameters R , f , (cid:15) and α , Loss L
Output: model with parameters θ .
Initialize θ .
for t = 0 to T − 1 do
Sample ( x , y ) from Z for i = 0 to n − 1 do
[Equation]
if t mod f = 0 then
[Equation]
In Algorithms 1 and 2 the parameter f is set in practice to a positive value, so that the projection (denoted by Π ) is made only every few iterations.
The variable θ represents the weight matrices of the model, and the projection in the last line should be understood as applied independently for every weight matrix.
The regularization parameter R corresponds to the variables µ, λ in Eq.
Convolutional layers Frequently, convolutions are employed in the literature, especially in the image-domain.
It is important to understand how our previous results extend to this case, and how the proposed algorithms work in that case.
Below, we show that the (cid:96) ∞ -operator norm of the convolutional layer (as a linear operator) is related to the (cid:96) ∞ -operator norm of the kernel after a reshaping operation.
For simplicity, we consider only convolutions with zero padding.
We study the cases of 1D, 2D and 3D convolutions.
For clarity, we mention below the result for the 3D convolution, since this is relevant to our experimental validation, and we defer the other two cases to Appendix G.
Theorem 5.
Let A ∈ R n × m × r be an input image and let K ∈ R
h × h ×
r
×
o be a convolutional kernel with o output channels.
For simplicity assume that k ≤ min( n, m ) is odd.
Denote by B = K (cid:63) A the output of the convolutional layer.
Let U ∈ R nmo × nmr be the matrix such that vec ( B )
=
U vec ( A ) i.e., U is the matrix representation of the convolution.
Let M ( K ) ∈
R
o × hhr be the matricization of
K , where each row contains the parameters of a single output channel of the convolution.
It holds that: (cid:107) U (cid:107) ∞ =
(cid:107) M ( K ) (cid:107) ∞ .
Thus, we can control the (cid:96) ∞ -operator-norm of a convolutional layer during training by controlling that of the reshaping of the kernel, which is done with the same code as for fully connected layers.
It can be seen that when the padding is non-zero, the result still holds.
N UMERICAL E VIDENCE
The generalization properties and the robustness of PNs are numerically veriﬁed in this section.
We evaluate the robustness to three widely-used adversarial attacks in sec.
We assess whether the compared regularization schemes can also help in the case of adversarial training in sec. 4.3.
Experiments with additional datasets, models (NCP models), adversarial attacks (APGDT, PGDT) and layer-wise bound (instead of a single bound for all matrices) are conducted in Appendix H due to the restricted space.
The results exhibit a consistent behavior across different adversarial attacks, different datasets and different models.
Whenever the results differ, we explicitly mention the differences in the main body below.
4. E XPERIMENTAL S ETUP
The accuracy is reported as as the evaluation metric for every experiment, where a higher accuracy translates to better performance.
Datasets and Benchmark Models: We conduct experiments on the popular datasets of FashionMNIST (Xiao et al., 2017), E-MNIST (Cohen et al., 2017) and CIFAR-10 (Krizhevsky et al., 2014).
The ﬁrst two datasets include grayscale images of resolution 28 × 28 , while CIFARincludes 60 , 000 RGB images of resolution 32 × 32 .
Each image is annotated with one out of the ten categories.
We use two popular regularization methods from the literature for comparison, i.e.,
Jacobian regularization (Hoffman et al., 2019) and L 2 regularization (weight decay).
Models: We report results using the following three models: 1) a 4 th -degree CCP model named "PN-4", 2) a 10 th -degree CCP model referenced as "PN-10" and 3) a 4 th -degree Convolutional CCP model called "PN-Conv".
In the PN-Conv, we have replaced all the U i matrices with convolutional kernels.
None of the variants contains any activation functions.
Hyper-parameters: Unless mentioned otherwise, all models are trained for 100 epochs with a batch size of 64 .
The initial value of the learning rate is 0 .
After the ﬁrst 25 epochs, the learning rate is multiplied by a factor of 0 .
2 every 50 epochs.
The SGD is used to optimize all the models, while the cross-entropy loss is used.
In the experiments that include projection or adversarial training, the ﬁrst 50 epochs are pre-training, i.e., training only with the cross-entropy loss.
The projection is performed every ten iterations.
Adversarial Attack Settings: We utilize two widely used attacks: a) Fast Gradient Sign Method (FGSM) and b) Projected Gradient Descent (PGD).
In FGSM the hyper-parameter (cid:15) represents the step size of the adversarial attack.
In PGD there is a triple of parameters ( (cid:15) total , n iters , (cid:15) iter ), which represent the maximum step size of the total adversarial attack, the number of steps to perform for a single attack, and the step size of each adversarial attack step respectively.
We consider the following hyper-parameters for the attacks: a) FGSM with (cid:15) = 0.1, b) PGD with parameters (0.1, 20, 0.01), c)
PGD with parameters (0.3, 20, 0.03).
4. E VALUATION OF THE ROBUSTNESS OF PN S
In the next experiment, we assess the robustness of PNs under adversarial noise.
That is, the method is trained on the train set of the respective dataset and the evaluation is performed on the test set perturbed by additive adversarial noise.
That is, each image is individually perturbed based on the respective adversarial attack.
The proposed method implements Algorithm 1.
The quantitative results in both Fashion-MNIST and E-MNIST using PN-4, PN-10 and PN-Conv under the three attacks are reported in Table 1.
The column ‘No-proj’ exhibits the plain SGD training (i.e., without regularization), while the remaining columns include the proposed regularization, Jacobian and the L 2 regularization respectively.
The results without regularization exhibit a substantial decrease in accuracy for stronger adversarial attacks.
The proposed regularization outperforms all methods consistently across different adversarial attacks. Interestingly, the stronger the adversarial
[Table]
In each dataset, the base networks are PN-4, i.e., a 4 th degree polynomial, on the top four rows, PN-10, i.e., a 10 th degree polynomial, on the middle four rows and PN-Conv, i.e., a 4 th degree polynomial with convolutions, on the bottom four rows.
Our projection method exhibits the best performance in all three attacks, with the difference on accuracy to stronger attacks being substantial.
attack, the bigger the difference of the proposed regularization scheme with the alternatives of Jacobian and L 2 regularizations.
Next, we learn the networks with varying projection bounds.
The results on Fashion-MNIST and
E-MNIST are visualized in Fig.
2, where the x-axis is plotted in log-scale.
As a reference point, we include the clean accuracy curves, i.e., when there is no adversarial noise.
Projection bounds larger than 2 (in the log-axis) leave the accuracy unchanged.
As the bounds decrease, the results gradually improve.
This can be attributed to the constraints the projection bounds impose into the U i matrices.
Similar observations can be made when evaluating the clean accuracy (i.e., no adversarial noise in the test set).
However, in the case of adversarial attacks a tighter bound performs better, i.e., the best accuracy is exhibited in the region of 0 in the log-axis.
The projection bounds can have a substantial improvement on the performance, especially in the case of stronger adversarial attacks, i.e., PGD.
Notice that all in the aforementioned cases, the intermediate values of the projection bounds yield an increased performance in terms of the test-accuracy and the adversarial perturbations.
Beyond the aforementioned datasets, we also validate the proposed method on CIFAR-10 dataset.
The results in Fig.
3 and Table 2 exhibit similar patterns as the aforementioned experiments.
Although the improvement is smaller than the case of Fashion-MNIST and E-MNIST, we can still obtain about 10% accuracy improvement under three different adversarial attacks.
4. A DVERSARIAL TRAINING (AT) ON PN S
Adversarial training has been used as a strong defence against adversarial attacks.
In this experiment we evaluate whether different regularization methods can work in conjunction with adversarial training that is widely used as a defence method.
Since multi-step adversarial attacks are computationally intensive, we utilize the FGSM attack during training, while we evaluate the trained model in all three
[Figure] [Figure]
[Table]
adversarial attacks.
For this experiment we select PN-10 as the base model.
The proposed model implements Algorithm 2.
The accuracy is reported in Table 3 with Fashion-MNIST on the top and E-MNIST on the bottom.
In the FGSM attack, the difference of the compared methods is smaller, which is expected since similar attack is used for the training.
However, for stronger attacks the difference becomes pronounced with the proposed regularization method outperforming both the Jacobian and the L 2 regularization methods.
[Table]
Our projection method exhibits the best performance in all three attacks
[Table]
The limitations of the proposed work are threefold.
Firstly, Theorem 1 relies on the (cid:96) ∞ -operator norm of the face-splitting product of the weight matrices, which in practice we relax in Lemma 1 for performing the projection.
In the future, we aim to study if it is feasible to compute the non-convex projection onto the set of PNs with bounded (cid:96) ∞ -norm of the face-splitting product of the weight matrices.
This would allow us to let go off the relaxation argument and directly optimize the original tighter Rademacher Complexity bound (Theorem 1).
Secondly, the regularization effect of the projection differs across datasets and adversarial attacks, a topic that is worth investigating in the future.
Thirdly, our bounds do not take into account the algorithm used, which corresponds to a variant of the Stochastic Projected Gradient Descent, and hence any improved generalization properties due to possible uniform stability (Bousquet and Elisseeff, 2002) of the algorithm or implicit regularization properties (Neyshabur, 2017), do not play a role in our analysis.
C ONCLUSION
In this work, we explore the generalization properties of the Coupled CP-decomposition (CCP) and nested coupled CP-decomposition (NCP) models that belong in the class of Polynomial Nets (PNs).
We derive bounds for the Rademacher complexity and the Lipschitz constant of the CCP and the
NCP models.
We utilize the computed bounds as a regularization during training.
The regularization terms have also a substantial effect on the robustness of the model, i.e., when adversarial noise is added to the test set.
A future direction of research is to obtain generalization bounds for this class of functions using stability notions.
Along with the recent empirical results on PNs, our derived bounds can further explain the beneﬁts and drawbacks of using PNs. A CKNOWLEDGEMENTS
We are thankful to Igor Krawczuk and Andreas Loukas for their comments on the paper.
We are also thankful to the reviewers for providing constructive feedback.
Research was sponsored by the
Army Research Ofﬁce and was accomplished under Grant Number W911NF-19-1-0404.
This work is funded (in part) through a PhD fellowship of the Swiss Data Science Center, a joint venture between EPFL and ETH Zurich. This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement number 725594 - time-data). A A PPENDIX INTRODUCTION
The Appendix is organized as follows:
• The related work is summarized in Appendix B. • In Appendix C the notation and the core Lemmas from the literature are described.
• Further details on the Polynomial Nets are provided in Appendix D. • The proofs on the CCP and the NCP models are added in Appendix E and Appendix F respectively.
• The extensions of the theorems for convolutional layers and their proofs are detailed in Appendix G. • Additional experiments are included in Appendix H.
B R ELATED W ORK
Rademacher Complexity: Known bounds for the class of polynomials are a consequence of more general result for kernel methods (Mohri et al., 2018, Theorem 6.12).
Support Vector Machines (SVMs) with a polynomial kernel of degree k effectively correspond to a general polynomial with the same degree.
In contrast, our bound is tailored to the parametric deﬁnition of the CCP and the NCP models, which are a subset of the class of all polynomials.
Hence, they are tighter than the general kernel complexity bounds.
Bounds for the class of neural networks were stablished in (Bartlett et al., 2017; Neyshabur et al., 2017), but they require a long and technical proof, and in particular it assumes an (cid:96) 2 -bound on the input, which is incompatible with image-based applications.
This bound also depend on the product of spectral norms of each layer.
In contrast, our bounds are more similar in spirit to the path-norm-based complexity bounds (Neyshabur et al., 2015), as they depend on interactions between neurons at different layers.
This interaction precisely corresponds to the face-splitting product between weight matrices that appears in Theorem 1.
Lipschitz constant: A variety of methods have been proposed for estimating the Lipschitz constant of neural networks.
For example, Scaman and Virmaux (2018) (SVD), Fazlyab et al.
(Semidefinite programming) and Latorre et al.
(Polynomial Optimization) are expensive optimization methods to compute tighter bounds on such constant.
These methods are unusable in our case as they would require a non-trivial adaptation to work with Polynomial Nets.
In contrast
we ﬁnd an upper bound that applies to such family of models, and it can be controlled with efﬁcient (cid:96) ∞ -operator-norm projections.
However, our bounds might not be the tightest.
Developing tighter methods to bound and control the Lipschitz constant for Polynomial Nets is a promising direction of future research.
C B ACKGROUND
Below, we develop a detailed notation in Appendix C.1, we include related deﬁnitions in Appendix C. and Lemmas required for our proofs in Appendix C.3.
The goal of this section is to cover many of the required information for following the proofs and the notation we follow in this work.
Readers familiar with the different matrix/vector products, e.g., Khatri-Rao or face-splitting product, and with basic inequalities, e.g., Hölder’s inequality, can skip to the next section.
C. N OTATION
Different matrix products and their associated symbols are referenced in Table 4, while matrix operations on a matrix A are deﬁned on Table 5.
Every matrix product, e.g., Hadamard product, can be used in two ways: a) A ◦ B , which translates to Hadamard product of matrices A and B , b) ◦ [-Ni =
-][Equation]
A i abbreviates the Hadamard products
[Equation]
The symbol x j i refers to j th element of vector x i .
[Table] [Table]
C. D EFINITIONS
For thoroughness
[Table]
we include the deﬁnitions of the core products that we use in this work.
Speciﬁcally, the deﬁnitions of the Hadamard product (Deﬁnition 3), Kronecker product (Deﬁnition 4), the KhatriRao product (Deﬁnition 5), column-wise Khatri-Rao product (Deﬁnition 6) and the face-splitting product (Deﬁnition 7) are included.
Deﬁnition 3 (Hadamard product) .
For two matrices A and B of the same dimension m × n, the
Hadamard product A ◦ B is a matrix of the same dimension as the operands, with elements given by ( a ◦ b ) i,j = a i,j b i,j .
Deﬁnition 4 (Kronecker product) .
If A is an m × n matrix and B is a p × q matrix, then the Kronecker product A ⊗ B is the pm × qn block matrix, given as follows:
[Equation]
Example: the Kronecker product of the matrices A ∈ R 2 × 2 and B ∈ R 2 × 2 is computed below:
[Equation]
Deﬁnition 5 (Khatri–Rao product) .
The Khatri–Rao product is deﬁned as:
[Equation]
in which the ( i, j ) -th block is the m i p i × n j q j sized Kronecker product of the corresponding blocks of A and B , assuming the number of row and column partitions of both matrices is equal.
The size of the product is then ( (cid:80) i m i p i ) ×
( (cid:80) i n j q j ) .
Example: if A and B both are 2 × 2 partitioned matrices e.g.:
[Equation]
[Equation]
then we obtain the following:
[Equation]
a 1 ,
[Equation]
1 ,a 1 , 3 b 1 , 3 a 2 ,
[Equation]
[Equation]
a 2 , 2 b 1 ,a 2 , 3 b 1 ,a 2 , 3 b 1 , 3 a 3 , 1 b 2 ,a 3 , 2 b 2 ,a 3 , 3 b 2 ,a 3 , 3 b 2 , 3 a 3 , 1 b 3 ,a 3 , 2 b 3 ,a 3 , 3 b 3 ,a 3 , 3 b 3 ,
[Equation]
 .
Deﬁnition 6 (Column-wise Khatri–Rao product) .
A column-wise Kronecker product of two matrices may also be called the Khatri–Rao product.
This product assumes the partitions of the matrices are their columns.
In this case m 1 = m , p 1 = p , n = q and for each j: n j = p
j = 1 .
The resulting product is a mp × n matrix of which each column is the Kronecker product of the corresponding columns of A and B .
Example: the Column-wise Khatri–Rao product of the matrices A ∈ R 2 × 3 and B ∈ R 3 × 3 is computed below:
[Equation]
(cid:123)(cid:122) (cid:125)
A ∗ (cid:34) b 1 ,b 1 ,b 1 , 3 b 2 ,b 2 ,b 2 , 3 b 3 ,b 3 ,b 3 ,(cid:35) (cid:124) (cid:123)(cid:122) (cid:125) B =   a 1 , 1 b 1 ,a 1 , 2 b 1 ,a 1 , 3 b 1 , 3 a 1 , 1 b 2 ,a 1 , 2 b 2 ,a 1 , 3 b 2 , 3 a 1 , 1 b 3 ,a 1 , 2 b 3 ,a 1 , 3 b 3 , 3 a 2 , 1 b 1 ,a 2 , 2 b 1 ,a 2 , 3 b 1 , 3 a 2 , 1 b 2 ,a 2 , 2 b 2 ,a 2 , 3 b 2 , 3 a 2 , 1 b 3 ,a 2 , 2 b 3 ,a 2 , 3 b 3 ,
 (cid:124) (cid:123)(cid:122) (cid:125)
A ∗ B .
From here on, all ∗ refer to Column-wise Khatri–Rao product.
Deﬁnition 7 (Face-splitting product) .
The alternative concept of the matrix product, which uses row-wise splitting of matrices with a given quantity of rows.
This matrix operation was named the face-splitting product of matrices or the transposed Khatri–Rao product .
This type of operation is based on row-by-row Kronecker products of two matrices.
Example: the Face-splitting product of the matrices A ∈ R 3 × 2 and B ∈ R 3 × 2 is computed below:
[Equation]
2 ,b 2 , 2 b 3 ,b 3 ,(cid:35) (cid:124) (cid:123)(cid:122) (cid:125) B = (cid:34) a 1 , 1 b 1 ,a 1 , 2 b 1 ,a 1 , 1 b 1 ,a 1 , 2 b 1 , 2 a 2 , 1 b 2 ,a 2 , 2 b 2 ,a 2 , 1 b 2 ,a 2 , 2 b 2 , 2 a 3 , 1 b 3 ,a 3 , 2 b 3 ,a 3 , 1 b 3 ,a 3 , 2 b 3 ,(cid:35) (cid:124) (cid:123)(cid:122) (cid:125)
A • B .
C. W ELL - KNOWN L EMMAS
In this section, we provide the details on the Lemmas required for our proofs along with their proofs or the corresponding citations where the Lemmas can be found as well.
Lemma 3.
(Federer, 2014) Let g , h be two composable Lipschitz functions.
Then g ◦ h is also
Lipschitz with Lip ( g ◦ h ) ≤ Lip ( g ) Lip ( h ) .
Here and only here ◦ represents function composition.
Lemma 4.
(Federer, 2014) Let f : X ⊆ R n → R m be differentiable and Lipschitz continuous.
Let J f ( x ) denote its total derivative (Jacobian) at x .
Then Lip p ( f ) = sup x ∈X (cid:107) J f ( x ) (cid:107) p where (cid:107)
J f ( x ) (cid:107) p is the induced operator norm on J f ( x ) .
Lemma 5 (Hölder’s inequality) .
(Cvetkovski, 2012) Let ( S, Σ , µ ) be a measure space and let p, q ∈ [1 , ∞ ] with 1 p + 1 q = 1 .
Then, for all measurable real-valued functions f and g on S, it holds that:
[Equation]
p (cid:107) g (cid:107) q .
Lemma 6 (Mixed Product Property 1) .
(Slyusar, 1999)
The following holds:
[Equation]
Lemma 7 (Mixed Product Property 2) .
The following holds:
[Equation]
Proof.
We prove this lemma by induction on N .
Base case ( N = 1 ): A 1 B 1 =
A 1 B 1 .
Inductive step: Assume that the induction hypothesis holds for a particular k , i.e., the case N = k holds.
That can be expressed as:
[Equation]
Then we will prove that it holds for N = k + 1 : ◦ k +1 i = [Equation] =
[Equation]
◦ ki =
[Equation]
A
[Equation]
B i )] ◦ [Equation] A [Equation]
B
[Equation]
= [ •
[Equation]
=
[Equation]
( A
[Equation]
∗
[Equation]
[Equation]
A
[Equation]
+1 B k
[Equation]
1 ) use inductive hypothesis (Eq.
{+= [ • ki =
 ( A i ) • A k +1 ][ ∗ ki =
 ( B i ) ∗ B k+}
[Equation]
] Lemma 6 [Mixed product property 1] = • k
+1 i
{+=
 ( A i ) ∗ k +1 i =
 ( B i ) .+}
That is, the case N = k
+ 1 also holds true, establishing the inductive step.
Lemma 8 (Massart Lemma.
Lemma 26.8 in Shalev-Shwartz and Ben-David (2014)) .
Let A =
{ a 1 , · · · , a N } be a ﬁnite set of vectors in R m .
Deﬁne ¯ a = 1 N (cid:80) Ni
=1 a i .
Then:
[Equation]
Deﬁnition 8 (Consistency of a matrix norm) .
A matrix norm is called consistent on C n,n , if (cid:107) AB (cid:107) ≤ (cid:107) A (cid:107) (cid:107) B (cid:107) . holds for A , B ∈ C n,n .
Lemma 9 (Consistency of the operator norm) .
(Lyche, 2020)
The operator norm is consistent if the vector norm (cid:107)·(cid:107) α is deﬁned for all m ∈ N and (cid:107)·(cid:107)
β
= (cid:107)·(cid:107)
α
Proof.
[Equation]
Lemma 10.
(Rao, 1970)
[Equation]
D D ETAILS ON POLYNOMIAL NETWORKS
In this section, we provide further details on the two most prominent parametrizations proposed in
Chrysos et al.
This re-parametrization creates equivalent models, but enables us to absorb the bias terms into the input terms.
Firstly, we provide the re-parametrization of the CCP model in
Appendix D.
and then we create the re-parametrization of the NCP model in Appendix D.2.
D. R E - PARAMETRIZATION OF CCP MODEL
The Coupled CP-Decomposition (CCP) model of PNs (Chrysos et al., 2020) leverages a coupled CP
Tensor decomposition.
A k -degree CCP model f
( ζ ) can be succinctly described by the following recursive relations:
[Equation]
where ζ ∈ R δ is the input data with δ ∈ N , f ( ζ ) ∈ R o is the output of the model and V n ∈ R µ × δ ,
Q ∈ R o × µ and β ∈ R o are the learnable parameters, where µ ∈ N is the hidden rank.
In order to simplify the bias terms in the model, we will introduce a minor re-parametrization in Lemma 11 that we will use to present our results in the subsequent sections.
Lemma 11.
Let z =
[ ζ (cid:62) , 1] (cid:62) ∈ R d , x n = [ y (cid:62) n , 1] (cid:62)
[Equation] C
[Equation] [ Q [Equation] β ] ∈
[Equation]
, d
[Equation]
µ + 1 .
Deﬁne:
[Equation]
[Equation]
where the boldface numbers 0 and 1 denote all-zeros and all-ones column vectors of appropriate size, respectively.
The CCP model in Eq.
(4) can be rewritten as f ( z ) =
C ◦ ki =
U
i z , which is the one used in Eq.
(CCP) .
As a reminder before providing the proof, the core symbols for this proof are summarized in Table 6.
[Table]
β
R o
[Table]
z
R d
Re-parametrization
[Table]
C
R o × m C = ( Q , β ) .
Proof.
By deﬁnition, we have: x 1 =
[ y (cid:62) 1 , 1] (cid:62) =
(cid:20) y 1 1 (cid:21) =
(cid:20) V 1 ζ 1 (cid:21) =
(cid:20) V 0 (cid:62) (cid:21) (cid:20)
ζ (cid:21) = (cid:20) V 0 (cid:62) (cid:21) [ ζ (cid:62) , 1] (cid:62) = U 1 z .
x n =
[ y (cid:62) n , 1] (cid:62) =
(cid:20) y n 1 (cid:21) =
(cid:20) ( V n ζ ) ◦ y n − 1 + y n − 1(cid:21)
= (cid:20) V n ζ + (cid:21) ◦
(cid:20)
y n − (cid:21) =
(cid:20) V n 0 (cid:62) (cid:21) (cid:20)
ζ (cid:21) ◦
(cid:20)
y n − (cid:21) =
(cid:20) V n 0 (cid:62) (cid:21)
[ ζ (cid:62) , 1] (cid:62) ◦
[ y (cid:62) n − 1 , 1] (cid:62) =
U
[Equation]
Hence [Equation] it holds that:
[Equation]
D. R EPARAMETRIZATION OF THE NCP MODEL
The nested coupled CP decomposition (NCP) model of PNs (Chrysos et al., 2020) leverages a joint hierarchical decomposition.
A k -degree NCP model f ( ζ ) is expressed with the following recursive relations:
[Equation]
where ζ ∈ R δ is the input data with δ ∈ N , f ( ζ ) ∈ R o is the output of the model and V n ∈ R µ × δ , b n ∈ R µ , U n ∈ R µ × µ , Q ∈ R o × µ and β ∈ R o are the learnable parameters, where µ ∈ N is the hidden rank.
In order to simplify the bias terms in the model, we will introduce a minor re-parametrization in Lemma 12 that we will use to present our results in the subsequent sections.
Lemma 12.
Let z =
[ ζ (cid:62) , 1] (cid:62) ∈ R d , x n = [ y (cid:62) n , 1]
[Equation]
[Equation]
, β ]
[Equation]
×
[Equation]
µ + 1 .
Let:
[Equation]
where the boldface numbers 0 and 1 denote all-zeros and all-ones column vectors of appropriate size, respectively.
The NCP model in Eq.
(5) can be rewritten as
[Equation]
In the aforementioned Eq.
[Table] we have written S n even for n = 1 , when s 1 is technically a vector, but this is done for convenience only and does not change the end result.
E R ESULT OF THE CCP MODEL E. P ROOF OF T HEOREM 1: R ADEMACHER C OMPLEXITY BOUND OF CCP UNDER (cid:96) ∞ NORM
To facilitate the proof below
[Table]
we include the related symbols in Table 7.
Below, to avoid cluttering the notation, we consider that the expectation is over σ and omit the brackets as well.
[Table]
Proof.
[Equation] n
[Equation]
n
[Equation]
n
[Equation]
, n
[Equation]
=
[-E sup f ∈F k CCP n n (cid:88) j = (cid:0) σ j (cid:10) c, ◦ ki =
 ( U i z j ) (cid:11)(cid:1) = E sup f ∈F k CCP n (cid:42) c, n (cid:88)-][Equation]
j )]
[Equation]
[-Lemma 
 [Hölder’s inequality] = E sup f ∈F k CCP n (cid:107) c (cid:107)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88)-][Table]
[Equation]
[Table] [Equation]
=
[Equation]
∞
[Equation]
[Equation]
∞
[Equation]
∞
[Equation]
µλ [Equation] ∞ .
Next, we compute the bound of E (cid:13)(cid:13)(cid:13)(cid:80) nj =1 [ σ j ∗ ki =1 ( z j )] (cid:13)(cid:13)(cid:13) ∞ .
Let Z j = ∗ ki =1 ( z j ) ∈ R d k .
For each l ∈ [ d k ] , let v l =
( Z l 1 , . . .
, Z ln ) ∈ R n .
Note that (cid:107) v l (cid:107) 2 ≤ √ n max j (cid:107) Z j (cid:107) ∞ .
Let V = { v 1 , . .
. , v d k } .
Then, it is true that:
E
[Equation]
j =
[Equation]
Using Lemma 8 [Massart Lemma] we have that:
[Equation]
Then, it holds that:
[Equation]
E. P ROOF OF L EMMA
[Table]
Proof.
(cid:13)(cid:13) • ki =
( U i
[Equation]
Deﬁnition of Face-splitting product
[Equation]
Multiplicativity of absolute value
[Equation]
E. R ADEMACHER C OMPLEXITY BOUND UNDER (cid:96) 2 NORM
Theorem 6.
Let Z = { z 1 , . . .
, z n } ⊆ R d and suppose that (cid:107)
z j (cid:107) ∞ ≤ 1 for all j = 1 , . .
. , n .
Let
F k CCP :
= (cid:8) f
( z ) =
(cid:10) c , ◦ k i
=1 U i z (cid:11) : (cid:107) c (cid:107) 2 ≤ µ, (cid:13)(cid:13) • k
i
=
1 U i (cid:13)(cid:13) 2 ≤ λ (cid:9) .
The Empirical Rademacher Complexity of CCP k
( k -degree CCP polynomials) with respect to Z is bounded as:
[Equation]
To facilitate the proof below, we include the related symbols in Table 9.
Below, to avoid cluttering the notation, we consider that the expectation is over σ and omit the brackets as well.
[Table]
[Equation]
[Table]
Proof.
R Z ( F k CCP )
=
E sup f ∈F k CCP n n (cid:88)
j = σ j f ( z j )
=
E sup f ∈F k CCP n n (cid:88)
j = (cid:0) σ j (cid:10) c , ◦ ki =
( U i z j ) (cid:11)(cid:1) =
E sup f ∈F k CCP n (cid:42) c , n (cid:88)
j
= [ σ j ◦ ki =1 ( U i z j )] (cid:43) ≤ E sup f ∈F k CCP n (cid:107) c (cid:107) 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Lemma
[Hölder’s inequality] =
[Equation]
[Equation]
Lemma 7 [Mixed product property] =
E sup
[Equation]
[Equation]
f
[Equation]
CCP
[Equation]
[Table]
[Equation]
So
[Equation]
E. L IPSCHITZ CONSTANT BOUND OF THE CCP MODEL
We will ﬁrst prove a more general result about the (cid:96) p -Lipschitz constant of the CCP model.
Theorem 7.
The Lipschitz constant (with respect to the (cid:96) p -norm) of the function deﬁned in Eq.
(CCP) , restricted to the set { z ∈ R d : (cid:107) z (cid:107) p ≤ 1 } is bounded as:
[Equation]
Proof.
Let g ( x ) =
Cx and h ( z )
= ◦
ki =
( U i z ) .
Then it holds that f ( z ) =
g ( h ( z )) .
By Lemma 3, we have: Lip p ( f ) ≤ Lip p ( g ) Lip p ( h ) .
We will compute an upper bound of each function individually.
Let us ﬁrst consider the function g ( x ) =
Cx .
By Lemma 4, because g is a linear map represented by a matrix C , its Jacobian is J g ( x ) =
C .
So:
[Equation]
where (cid:107) C (cid:107) p is the operator norm on matrices induced by the vector p -norm.
Now, let us consider the function h ( z )
= ◦
ki =
U i z .
Its Jacobian is given by:
[Equation]
Using Lemma 4 we have:
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
Lemma 9 [consistency]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
(cid:107) p ≤ sup z : (cid:107) z (cid:107) p ≤k (cid:88)
i =
k (cid:89)
j = ( (cid:107) U j (cid:107) p ) =
k k (cid:89)
j = (cid:107) U j
[Equation]
p .
So:
[Equation]
E.4. P ROOF OF T HEOREM
Proof.
This is a particular case of Theorem 7 when p = ∞ .
F R ESULT OF THE NCP MODEL
Proof.
R Z ( F k NCP )
=
E sup f ∈F k NCP n n (cid:88)
j = σ j f ( z j )
=
E sup f ∈F k NCP n n (cid:88)
j
= ( σ j (cid:104) c , x k ( z j ) (cid:105) )
=
E sup f ∈F k NCP n (cid:42) c , n (cid:88)
j
= [ σ j x k ( z j )] (cid:43) ≤ E sup f ∈F k NCP n (cid:107) c (cid:107)(cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) n (cid:88)
j
= [ σ j x k ( z j )] (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) ∞
Lemma
[Hölder’s inequality]
=
E sup f ∈F k NCP n (cid:107) c (cid:107) 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88)
j
= [ σ j (( A k z j ) ◦ ( S k x k − 1 ( z j )))] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∞
=
E sup f ∈F k NCP n (cid:107) c (cid:107) 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
∞
Lemma 7 [Mixed product property]
[Equation]
Now, because of the recursive deﬁnition of the Eq.
(NCP), we obtain: n
[Equation]
Lemma
[Equation]
Lemma
[Equation]
[Equation]
recursively applying this argument we have:
[Equation]
Combining the two previous equations (Eqs. (13) and (14)) inside Eq.
(12) we ﬁnally obtain
[Table]
[Equation] [Table] [Equation] [Table]
following the same arguments as in Eq.
(10) it follows that:
[Equation]
Proof.
[Equation]
Lemma 9 [consistent]
[Equation]
Lemma 1 .
F. R ADEMACHER C OMPLEXITY UNDER (cid:96) 2 NORM
Theorem 8.
Let Z = { z 1 , . . .
, z n } ⊆ R d and suppose that (cid:107)
z j (cid:107) ∞ ≤ 1 for all j = 1 , . .
. , n .
Deﬁne the matrix Φ( A 1 , S 1 , . . .
, A n , S n ) :
=
( A k • S k ) (cid:81) k − 1 i =
I ⊗ A
i • S i .
Consider the class of functions:
F k NCP := { f ( z ) as in (NCP) : (cid:107) C (cid:107) 2 ≤ µ, (cid:107) Φ( A 1 , S 1 , . . .
, A k , S k ) (cid:107) 2 ≤ λ } , where C ∈ R 1 × m (single output case).
The Empirical Rademacher Complexity of NCP k (k-degree
NCP polynomials) with respect to Z is bounded as:
[Equation]
Proof.
[Equation]
[Equation]
[Equation]
n
[Equation]
[Equation] )] (cid:43) ≤
[Equation]
sup f
[Equation]
k NCP n (cid:107) c (cid:107)
[Table]
n
[Table]
j =
[Table]
[ σ j x k
[Table]
j )]
[Table]
{+Lemma 
 [Hölder’s inequality]+}
=
E sup f ∈F
[Equation]
NCP n (cid:107) c (cid:107) 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) n
[Table]
j = [ σ j (( A k z j ) ◦ ( S k x k
[Equation]
1 ( z j )))] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)= E sup f ∈F k NCP n (cid:107) c (cid:107) 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88)
j
= [ σ j (( A k • S k )( z j ∗ x k − 1 ( z j )))]
[Table]
Lemma 7 [Mixed product property]
[Table]
[Equation]
n
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Eq.
Theorem 9.
Let F L be the class of functions deﬁned as
[Equation]
The Lipschitz Constant of F L (k-degree NCP polynomial) under (cid:96) p norm restrictions is bounded as:
[Equation]
Proof.
Let g ( x ) =
Cx , h ( z ) =
( A n z ) ◦ ( S n x n − 1 ( z )) .
Then it holds that f ( z ) =
g ( h ( z )) .
By Lemma 3, we have: Lip ( f )
≤ Lip ( g ) Lip ( h ) .
This enables us to compute an upper bound of each function (i.e., g, h ) individually.
Let us ﬁrst consider the function g ( x ) =
Cx .
By Lemma 4, because g is a linear map represented by a matrix C , its Jacobian is J g ( x ) =
C .
So:
[Equation]
where (cid:107) C (cid:107) p is the operator norm on matrices induced by the vector p-norm, and σ max ( C ) is the largest singular value of C .
Now, let us consider the function x n ( z )
= h ( z ) =
( A n z ) ◦ ( S n x n − 1 ( z )) .
Its Jacobian is given by:
[Equation]
Triangle inequality
[Equation]
Lemma 9 [consistent]
[Equation]
Then we proof the result by induction
[Equation]
Inductive hypothesis:
[Equation]
Case k
= 1 :
[Equation]
Case k = n :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
So:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
F.4. PROOF OF T HEOREM
Proof.
This is particular case of Theorem 9 with p = ∞ .
G R ELATIONSHIP BETWEEN A C ONVOLUTIONAL LAYER AND A F ULLY C ONNECTED LAYER
In this section we discuss various cases of input/output types depending on the dimensionality of the input tensor and the output tensor.
We also provide the proof of Theorem 5.
Theorem 10.
Let A ∈ R n .
Let K ∈ R
h be a 1-D convolutional kernel.
For simplicity, we assume h is odd and h ≤ n .
Let B ∈ R n , B = K (cid:63) A be the output of the convolution.
Let U be the convolutional operator i.e., the linear operator (matrix) U ∈ R n × n such that B = K (cid:63) A = UA .
It holds that (cid:107) U (cid:107) ∞ =
(cid:107) K (cid:107) 1 .
Theorem 11.
Let A ∈ R n × m , and let
K ∈ R
h
× h be a 2-D convolutional kernel.
For simplicity assume h is odd number and h ≤ min( n, m ) .
Let B ∈ R n × m , B = K (cid:63) A be the output of the convolution.
Let U be the convolutional operator i.e., the linear operator (matrix) U ∈ R nm × nm such that vec ( B ) =
U vec ( A ) .
It holds that (cid:107) U (cid:107) ∞ =
(cid:107) vec ( K ) (cid:107) 1 .
G. P ROOF OF T HEOREM
Proof.
From, B = K (cid:63) A =
UA we can obtain the following:
[Equation]
We observe that:
[Equation]
Then, it holds that:
[Equation]
G. P ROOF OF T HEOREM
Proof.
We partition U into n × n partition matrices of shape m × m .
Then the ( i, j ) th partition matrix U ( i,j ) describes the relationship between the B i and the A j .
So, U ( i,j ) is also similar to the
Toeplitz matrix in the previous result.
[Equation]
Meanwhile, the matrix M satisﬁes:
[Equation]
Then
[Equation]
we have the following:
[Equation]
In addition, by h ≤ n
[Equation]
h ≤ m we have:
[Equation]
Then, it holds that (cid:107) U
[Equation]
∞ =
[Equation] K i [Equation]
G. P ROOF OF T HEOREM
Proof.
We partition U into o × r partition matrices of shape nm × nm .
Then the ( i, j ) th partition of the matrix
U
( i,j ) describes the relationship between the i th channel of B and the j th channel of
A .
Then, the following holds: (cid:13)(cid:13) U ( i,j ) (cid:13)(cid:13) ∞ =
(cid:80) hi
=1 (cid:13)(cid:13)
K iij (cid:13)(cid:13) 1 , where K ij means the two-dimensional tensor obtained by the third dimension of K takes j and the fourth dimension of K takes i . (cid:107) U (cid:107) ∞ = max n × m × o i = n × m × r (cid:88) j = (cid:12)(cid:12) u ( i,j ) (cid:12)(cid:12) =
o − 1 max l = n × m max i =
r − 1 (cid:88) s = n × m (cid:88) j = (cid:12) (cid:12) u ( i + nml,j + nms ) (cid:12) (cid:12) ≤ o − 1 max l = r − 1 (cid:88) s =
( n × m max i = n × m (cid:88)
j = (cid:12)(cid:12)
u
( i + nml,j + nms ) (cid:12)(cid:12) )
=
o − 1 max l = r − 1 (cid:88) s =
(cid:13)(cid:13) U ( l +1 ,s +1)
(cid:13)(cid:13) ∞ =
o max l = r (cid:88) s =
(cid:13)(cid:13) U ( l,s )
(cid:13)(cid:13) ∞ =
o max l = r (cid:88) s
= h (cid:88) i =
(cid:13)(cid:13) K ils (cid:13)(cid:13)≤ o max l =1 (cid:13)(cid:13)(cid:13)
ˆ K l (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13) ˆ K (cid:13)(cid:13)(cid:13) ∞ .
Similar to Eq.
(15): for every nm rows, we choose k +12 th row.
Then its 1-norm is equal to this nm rows of the ˆ K ’s ∞ -norm.
So the equation holds.
H A UXILIARY NUMERICAL EVIDENCE
A number of additional experiments are conducted in this section.
Unless explicitly mentioned otherwise, the experimental setup remains similar to the one in the main paper.
The following experiments are conducted below:
The difference between the theoretical and the algorithmic bound and their evolution during training is studied in Appendix H.1.
An ablation study on the hidden size is conducted in Appendix H.2.
An ablation study is conducted on the effect of adversarial steps in Appendix H.3.
We evaluate the effect of the proposed projection into the testset performance in Appendix H.4. 5.
We conduct experiments on four new datasets,
i
e., MNIST, K-MNIST, E-MNIST-BY, NSYNTH in Appendix H.5.
These experiments are conducted in addition to the datasets already presented in the main paper.
In Appendix H.6 experiments on three additional adversarial attacks,
i
e., FGSM-0.01, APGDT and TPGD, are performed.
We conduct an experiment using the NCP model in Appendix H.7. 8.
The layer-wise bound (instead of a single bound for all matrices) is explored in Appendix H.8.
The comparison with adversarial defense methods is conducted in Appendix H.9.
H. T HEORETICAL AND ALGORITHMIC BOUND
As mentioned in Section 3, projecting the quantity θ = (cid:13)(cid:13) • ki =
U
i (cid:13)(cid:13) ∞ onto their level set corresponds to a difﬁcult non-convex problem.
Given that we have an upper bound
[Equation]
we want to understand in practice how tight is this bound.
In Fig.
4 we compute the ratio γθ for PN-4.
In Fig.
5 the ratio is illustrated for randomly initialized matrices (i.e., untrained networks).
H. A BLATION STUDY ON THE HIDDEN SIZE
Initially, we explore the effect of the hidden rank of PN-4 and PN-10 on Fashion-MNIST.
Fig.exhibits the accuracy on both the training and the test-set for both models.
We observe that PNhas a better accuracy on the training set, however the accuracy on the test set is the same in the two models.
We also note that increasing the hidden rank improves the accuracy on the training set, but not on the test set.
H. A BLATION STUDY ON THE EFFECT OF ADVERSARIAL STEPS
Our next experiment scrutinizes the effect of the number of adversarial steps on the robust accuracy.
We consider in all cases a projection bound of 1 , which provides the best empirical results.
We vary the number of adversarial steps and report the accuracy in Fig.
The results exhibit a similar performance both in terms of the dataset (i.e., Fashion-MNIST and K-MNIST) and in terms of the network (PN-4 and PN-Conv).
Notice that when the adversarial attack has more than 10 steps the performance does not vary signiﬁcantly from the performance at 10 steps, indicating that the projection bound is effective for stronger adversarial attacks.
[Figure]
[Figure]
[Figure]
[Figure]
H. E VALUATION OF THE ACCURACY OF PN S
In this experiment, we evaluate the accuracy of PNs.
We consider three networks, i.e., PN-4, PNand PN-Conv, and train them under varying projection bounds using Algorithm 1.
Each model is evaluated on the test set of (a) Fashion-MNIST and (b) E-MNIST.
The accuracy of each method is reported in Fig.
8, where the x-axis is plotted in log-scale (natural logarithm).
The accuracy is better for bounds larger than 2 (in the log-axis) when compared to tighter bounds (i.e., values less than 0 ).
Very tight bounds stiﬂe the ability of the network to learn, which explains the decreased accuracy.
Interestingly, PN-4 reaches similar accuracy to PN-10 and PN-Conv in Fashion-MNIST as the bound increases, while in E-MNIST it cannot reach the same performance as the bound increases.
The best bounds for all three models are observed in the intermediate values, i.e., in the region of 1 in the log-axis for PN-4 and PN-10.
[Figure]
We scrutinize further the projection bounds by training the same models only with cross-entropy loss (i.e., no bound regularization).
In Table 10, we include the accuracy of the three networks with and without projection.
Note that projection consistently improves the accuracy, particularly in the case of larger networks, i.e., PN-10. [Figure]
[Table]
H. E XPERIMENTAL RESULTS ON ADDITIONAL DATASETS
To validate even further we experiment with additional datasets.
We describe the datasets below and then present the robust accuracy in each case.
The experimental setup remains the same as in
Section 4.2 in the main paper.
As a reminder, we are evaluating the robustness of the different models under adversarial noise.
Dataset details: There are six datasets used in this work:
Fashion-MNIST (Xiao et al., 2017) includes grayscale images of clothing.
The training set consists of 60 , 000 examples, and the test set of 10 , 000 examples.
The resolution of each image is 28 × 28 , with each image belonging to one of the 10 classes.
E-MNIST (Cohen et al., 2017) includes handwritten character and digit images with a training set of 124 , 800 examples, and a test set of 20 , 800 examples.
The resolution of each image is 28 × 28 .
E-MNIST includes 26 classes.
We also use the variant EMNIST-BY that includes 62 classes with 697 , 932 examples for training and 116 , 323 examples for testing.
K-MNIST (Clanuwat et al., 2018) depicts grayscale images of Hiragana characters with a training set of 60 , 000 examples, and a test set of 10 , 000 examples.
The resolution of each image is 28 × 28 .
K-MNIST has 10 classes.
MNIST (Lecun et al., 1998) includes handwritten digits images.
MNIST has a training set of 60 , 000 examples, and a test set of 10 , 000 examples.
The resolution of each image is 28 × 28 .
CIFAR-10 (Krizhevsky et al., 2014) depicts images of natural scenes.
CIFAR-10 has a training set of 50 , 000 examples, and a test set of 10 , 000 examples.
The resolution of each RGB image is 32 × 32 . 6.
NSYNTH (Engel et al., 2017) is an audio dataset containing 305 , 979 musical notes, each with a unique pitch, timbre, and envelope.
We provide a visualization 3 of indicative samples from MNIST, Fashion-MNIST, K-MNIST and
E-MNIST in Fig.
We originally train PN-4, PN-10 and PN-Conv without projection bounds.
The results are reported in Table 11 (columns titled ‘No proj’) for MNIST and K-MNIST, Table 13 (columns titled ‘No proj’) for E-MNIST-BY and Table 14 (columns titled ‘No proj’) for NSYNTH.
Next, we consider the performance under varying projection bounds; the accuracy in each case is depicted in Fig.for K-MNIST, MNIST and E-MNIST-BY and Fig.
11 for NSYNTH.
The ﬁgures (and the tables) depict the same patterns that emerged in the two main experiments, i.e., the performance can be vastly improved for intermediate values of the projection bound.
Similarly, we validate the performance when using adversarial training.
The results in Table 12 demonstrate the beneﬁts of using projection bounds even in the case of adversarial training.
H. E XPERIMENTAL RESULTS OF MORE TYPES OF ATTACKS
To further verify the results of the main paper, we conduct experiments with three additional adversarial attacks: a) FGSM with (cid:15) = 0.01, b)
Projected Gradient Descent in Trades (TPGD) (Zhang et al., 2019), c) Targeted Auto-Projected Gradient Descent (APGDT)
(Croce and Hein, 2020).
In TPGD and APGDT, we use the default parameters for a one-step attack.
The quantitative results are reported in Table 15 for four datasets and the curves of Fashion-MNIST and E-MNIST are visualized in Fig.
12 and the curves of K-MNIST and MNIST are visualized in
Fig.
The results in both cases remain similar to the attacks in the main paper, i.e., the proposed projection improves the performance consistently across attacks, types of networks and adversarial attacks.
H. E XPERIMENTAL RESULTS IN NCP MODEL
To complement, the results of the CCP model, we conduct an experiment using the NCP model.
That is, we use a 4 th degree polynomial expansion, called NCP- for our experiment conduct an
[Figure] [Figure]
[Figure]
Log of Bound
[Figure]
u r a c y ( % )
PN-
[Figure] [Figure]
[Table] [Table]
[Table] [Table]
experiment in the K-MNIST dataset and present the result with varying bound in Fig.
Notice that the patterns remain similar to the CCP model, i.e., intermediate values of the projection bound can increase the performance signiﬁcantly.
H. L AYER - WISE BOUND
To assess the ﬂexibility of the proposed method, we assess the performance of the layer-wise bound.
In the previous sections, we have considered using a single bound for all the matrices, i.e., (cid:107) U
i (cid:107) ∞ ≤ λ , because the projection for a single matrix has efﬁcient projection algorithms.
However, Lemmaenables each matrix U i to have a different bound
λ
i .
We assess the performance of having different bounds for each matrix U i .
[Table]
being substantial.
[Table]
[Figure]
Log of Bound
[Figure]
A cc u r a c y ( % )
[Figure]
[Table]
We experiment on PN-4 that we set a different projection bound for each matrix U i .
Speciﬁcally, we use ﬁve different candidate values for each λ
i and then perform the grid search on the FashionMNIST FGSM-0.01 attack.
The results on Fashion-MNIST in Table 16 exhibit how the layer-wise bounds outperform the previously used single bound 4 .
The best performing values for PN-4 are λ 1 = 1 . 5 , λ 2 = 2 , λ 3 = 1 . 5 , λ 4 = 2 , µ = 0 .
The values of λ i in the ﬁrst few layers are larger, while the value in the output matrix C is tighter.
To scrutinize the results even further, we evaluate whether the best performing
λ
i can improve the performance in different datasets and the FGSM-0.1 attack.
In both cases, the best performing λ i can improve the performance of the single bound.
[Table] [Table]
H. A DVERSARIAL DEFENSE METHOD
One frequent method used against adversarial perturbations are the so called adversarial defense methods.
We assess the performance of adversarial defense methods on the PNs when compared with the proposed method.
We experiment on PN-4 in Fashion-MNIST.
We chose three different methods: gaussian denoising, median denoising and guided denoising (Liao et al., 2018).
Gaussian denoising and median denoising are the methods of using gaussian ﬁlter and median ﬁlter for feature denoising (Xie et al., 2019).
The results in Table 17 show that in both attacks our method performs favourably to the adversarial defense methods.