T OWARDS R OBUST N EURAL N ETWORKS VIA C LOSELOOP C ONTROL
A BSTRACT
Despite their success in massive engineering applications, deep neural networks are vulnerable to various perturbations due to their black-box nature.
Recent study has shown that a deep neural network can misclassify the data even if the input data is perturbed by an imperceptible amount.
In this paper, we address the robustness issue of neural networks by a novel close-loop control method from the perspective of dynamic systems.
Instead of modifying the parameters in a ﬁxed neural network architecture, a close-loop control process is added to generate control signals adaptively for the perturbed or corrupted data.
We connect the robustness of neural networks with optimal control using the geometrical information of underlying data to design the control objective.
The detailed analysis shows how the embedding manifolds of state trajectory affect error estimation of the proposed method.
Our approach can simultaneously maintain the performance on clean data and improve the robustness against many types of data perturbations.
It can also further improve the performance of robustly trained neural networks against different perturbations.
To the best of our knowledge, this is the ﬁrst work that improves the robustness of neural networks with close-loop control 1 .
I NTRODUCTION
Due to the increasing data and computing power, deep neural networks have achieved state-of-theart performance in many applications such as computer vision, natural language processing and recommendation systems.
However, many deep neural networks are vulnerable to various malicious perturbations due to their black-box nature: a small (even imperceptible) perturbation of input data may lead to completely wrong predictions (Szegedy et al., 2013; Nguyen et al., 2015).
This has been a major concern in some safety-critical applications such as autonomous driving (Grigorescu et al., 2020) and medical image analysis (Lundervold & Lundervold, 2019).
Various perturbations have been reported, including the (cid:96) p norm based attack (Madry et al., 2017; Moosavi-Dezfooli et al., 2016;
Carlini & Wagner, 2017), semantic perturbation (Engstrom et al., 2017) etc.
On the other side, some algorithms to improve the robustness against those perturbations have shown great success (Madry et al., 2017).
However, most robustly trained models are tailored for certain types of perturbations, and they do not work well for other types of perturbations.
Khoury & Hadﬁeld-Menell (2018) showed the non-existence of optimal decision boundary for any (cid:96) p -norm perturbation.
Recent works (E, 2017; Haber & Ruthotto, 2017) have shown the connection between dynamical systems and neural networks.
This dynamic system perspective provides some interesting theoretical insights about the robustness issue.
Given a set of data x 0 ∈ R d and its labels y ∈
R l with a joint distribution D , training a neural network can be considered as following
[Equation]
[Figure]
u
[Figure]
Layer
[Figure]
E
[Figure]
[Figure] Contu Layer· · · Layer T where θ are the unknown parameters to train, and f , Φ represent the forward propagation rule and loss function (e.g. cross-entropy) respectively.
The dynamical system perspective interprets the vulnerability of neural networks as a system instability issue, which addresses the state trajectory variation under small perturbations applied on initial conditions.
The optimal control theory focuses on developing a control model to adjust the system state trajectory in an optimal manner.
The ﬁrst work that links and extends the classical back-propagation algorithm using optimal control theory was presented in Li et al.
(2017), where the direct relationship between the Pontryagin’s Maximum
Principle (Kirk, 1970) and the gradient based network training was established.
Ye et al.
(2019) used control theory to adjust the hyperparameters in the adversarial training algorithm.
Han et al.
(2018) established the mathematical basis of the optimal control viewpoint of deep learning.
These existing works on algorithm development are open-loop control methods since they commonly treat the network weights θ as control parameters and keep them ﬁxed once the training is done.
The ﬁxed control parameters θ operate optimally for data sampled from the data distribution D .
However, various perturbation methods cause data distributions to deviate from the true distribution D (Song et al., 2017) and cause poor performance with the ﬁxed open-loop control parameters.
1. P APER C ONTRIBUTIONS
To address the limitation of using open-loop control methods , we propose the C lose-
L oop C ontrol
N eural N etwork ( CLC-NN ), the ﬁrst close-loop control method to improve the robustness of neural networks.
As shown in Fig.
1, our method adds additional blocks to a given T -layer neural network: embedding functions E t , which induce running losses in all layers that measure the discrepancies between true features and observed features under input perturbation, then control processes generate control variables u t to minimize the total running loss under various data perturbations.
The original neural network can be designed by either standard training or robust training.
In the latter case, our CLC-NN framework can achieve extra robustness against different perturbations.
The forward propagation rule is thus modiﬁed with an extra control parameter u t ∈ R d (cid:48)
[Equation] Fig.
1 should not be misunderstood as an open-loop control. From the perspective of dynamic systems, x 0 is an initial condition, and the excitation input signal is u t (which is 0 in a standard feed-forward network). Therefore, the forward signal path is from u t to the internal states x t and then to the output label y . The path from x t to the embedding function E t ( x t ) and then to the excitation signal u t forms a feedback and closes the whole loop.
The technical contributions of this paper are summarized below:
• The proposed method relies on the well accepted assumption that the data and hidden state manifolds are low dimensional compared to the ambient dimension (Fefferman et al., 2016).
We study the geometrical information of the data and hidden layers to deﬁne the objective function for control.
Given a trained T -layer neural network, a set of embedding functions E t are trained off-line by minimizing the reconstruction loss (cid:107)E ( x t ) − x t (cid:107) over some clean data from D only.
The embedding functions support deﬁning a running loss required in our control method.
• We deﬁne the control problem by dynamic programming and implement the online iterative solver based on the Pontryagin’s Maximum Principle to avoid the curse of dimensionality.
The proposed close-loop control formulation does not require prior information of the perturbation.
• We provide a theoretical error bound of the controlled system for the simpliﬁed case with linear activation functions and linear embedding.
This error bound reveals how the close-loop control improves neural network robustness in the simplest setting.
R ELATED W ORKS
Many techniques have been reported to improve the robustness of neural networks, such as data augmentation (Shorten & Khoshgoftaar, 2019), gradient masking (Liu et al., 2018), etc.
We review adversarial training and reactive defense which are most relevant to this work.
Adversarial Training.
Adversarial training is (possibly) the most popular robust training method, and it solves a min-max robust optimization problem to minimize the worse-case loss with perturbed data.
Adversarial training effectively regularizes the network’s local Lipschitz constants of the loss surface around the data manifold (Liu et al., 2018).
Zhang et al.
(2019) formulated the robustness training using the Pontryagon’s Maximum Principle, such open-loop control methods result in a set of ﬁxed parameters that operates optimally on the considered perturbation.
Liu et al.
(2020a;b) considered a close-loop formulation from the differential dynamic programming perspective, this algorithm is categorized as a open-loop control method because it utilizes the state feedback information to boost the training convergence and results in a set of ﬁxed controls for any unseen data.
On the contrary, the proposed CLC-NN formulation adaptively targets on the inputs with different control parameters and is capable of distinguishing clean data by generating no control.
Reactive Defense.
A reactive defense method tries to reject or pre-process the input data that may cause mis-classiﬁcations.
Metzen et al.
(2017) rejected perturbed data by using adversarial detectors that are trained with adversarial data to detect abnormal data during forward propagation.
Song et al.
(2017) estimated the input data distribution D with a generative model (Oord et al., 2016) to detect data that does not belong to D , it applies a greedy method to search the local neighbour of input data for a more statistically plausible counterpart.
This puriﬁcation process has shown improved accuracy with adversarial data contaminated by various types of perturbations.
Puriﬁcation can be considered as a one-step method to solve the optimal control problem that has the objective function deﬁned over the initial condition only.
On the contrary, the proposed CLC-NN solves the control problem by the dynamic programming principle and its objective function is deﬁned over the entire state trajectory, which guarantees the optimality for the resulted controls.
T HE C LOSE - LOOP C ONTROL F RAMEWORK FOR N EURAL N ETWORKS
Now we present a close-loop optimal control formulation to address the robustness issue of deep learning.
Consider a neural network consisting of model parameters θ equipped with external control policy π , where π ∈ Π is a collection of functions R d → R d (cid:48) acting on the state and outputting the control signal.
The feed-forward propagation in a T -layer neural network can be represented as
[Equation]
Given a trained network, we solve the following optimization problem
[Equation]
where π collects the control policies π 0 , · · · , π T − 1 for all layers.
Note that (2) differs from the open-loop control used in standard training.
An open-loop control that treats the network parameters as control variables seeks for a set of ﬁxed parameters θ to match the output with true label y by minimizing the terminal loss Φ , and the running loss L deﬁnes a regularization for θ .
However, the terminal and running losses play different roles when our goal is to improve the robustness of a neural network by generating some adaptive controls for different inputs.
Challenge of Close-loop Control for Neural Networks.
Optimal control has been well studied in the control community for trajectory optimization, where one deﬁnes the running loss as the error between the actual state x t and a reference state x t, ref over time interval [0 , T ] .
The resulting control policy adjusts x t and makes it approach x t, ref .
In this paper, we apply the idea of trajectory optimization to improve the robustness of a neural network via adjusting the undesired state of x t .
However, the formulation is more challenging in neural networks: we do not have a “reference” state during the inference process, therefore it is unclear how to deﬁne the running loss L .
In the following, we investigate manifold embedding of the state trajectory to precisely deﬁne the loss functions Φ and L of Eq.
(2) required for the control objective function of a neural network.
3. M ANIFOLD L EARNING FOR S TATE T RAJECTORIES
State Manifold.
Our controller design is based on the “manifold hypothesis”: real-world high dimensional data can often be embedded in a lower dimensional manifold M (Fefferman et al., 2016).
Indeed, neural networks extract the embedded features from M .
To fool a well-trained neural network, the perturbed data often stays away from the data manifold M (Khoury & Hadﬁeld-Menell, 2018).
We consider the data space Z ( x ∈ Z, ∀ x ∼ D ) as: Z = Z (cid:107) (cid:76) Z ⊥ , where Z (cid:107) contains the embedded manifold M and Z ⊥ is the orthogonal complement of Z (cid:107) .
During forward propagation, the state manifold embedded in Z (cid:107) varies at different layers due to both the nonlinear activation function f and state dimensionality variation.
Therefore, we denote Z t =
Z t (cid:107) (cid:76) Z t
⊥ as the state space decomposition at layer t and M t ∈ Z t (cid:107) .
Once an input data is perturbed, the main effects of causing misclassiﬁcations are in Z ⊥ .
Therefore, it is important to measure how far the possibly perturbed state x t deviates from the state manifold M t .
Embedding Function.
Given an embedding function E t that encodes x t onto the lowerdimensional manifold M t and decodes the result back to the full state space Z t , the reconstruction loss (cid:107)E t ( x t ) − x t (cid:107) measures the deviation of the possibly perturbed state x t from the manifold M t .
The reconstruction loss is nonzero as long as x t has components in Z t ⊥ .
The embedding functions are constructed ofﬂine by minimizing the total reconstruction losses over a clean training data set.
• Linear Case: E t ( · ) can be considered as V rt ( V rt ) T where V rt forms an orthonormal basis for Z t (cid:107) .
Speciﬁcally one can ﬁrst perform a principle component analysis over a collection of hidden states at layer t , then V rt can be obtained as the ﬁrst r columns of the resulting eigenvectors.
• Nonlinear Case: we choose a convolutional auto-encoder (detailed in Appendix B) to obtain a representative manifold embedding function E t due to its ease of implementation.
Based on the assumption that most perturbations are in the Z ⊥ subspace, the embeddings are effective to detect the perturbations as long as the target manifold is of a low dimension.
Alternative manifold learning methods such as Izenman (2012) may also be employed.
3. F ORMULATION FOR THE C LOSE -L OOP C ONTROL OF N EURAL N ETWORKS
Control Objectives.
The above embedding function allows us to deﬁne a running loss L :
[Equation]
Here the matrix R deﬁnes a regularization term promoting controls of small magnitudes.
In practical implementations, using a diagonal matrix R with small elements often helps to improve the performance.
Now we are ready to design the control objective function of CLC-NN.
Different from a standard open-loop control, this work sets the terminal loss Φ as zero because no true label is given during inference.
Consequently, the close-loop control formulation in Eq.
(2) becomes
[Equation]
Assume that the input data is perturbed by a bounded and small amount, i.e.,
[Equation]
where z can be either random or adversarial.
The proposed CLC-NN adjusts the perturbed state trajectory x (cid:15),t such that it stays at a minimum distance from the desired manifold M t while promoting small magnitudes of controls.
Intuition.
We use an intuitive example to show how CLC-NN controls the state trajectory of unseen data samples.
We create a synthetic binary classiﬁcation data set with 1500 samples.
We train a residual neural network with one hidden layer of dimension 2 , and adopt the fast gradient sign method (Goodfellow et al., 2014) to generate adversarial data.
Fig.
2 (a) and (b) show the states of clean data (red and blue) and of perturbed data (black and gray) at t = 0 and t = 1 , respectively.
The
CLC-NN adjusts the state trajectory to reduce the reconstruction loss as shown in Fig.
2 (c) and (d), where lighter background color represents lower reconstruction loss.
Comparing Fig.
2 (a) with (c), and Fig.
2 (b) with (d), we see that the perturbed states in Fig.
2 (a) and (b) deviate from the desired state manifold (light green region) and has a high reconstruction loss.
Running 1000 iterations of
Alg. 1 adjusts the perturbed states and improves the classiﬁcation accuracy from 86% to 100% . [Figure]
I MPLEMENTATION VIA THE P ONTRYAGIN ’ S M AXIMUM P RINCIPLE
Dynamic Programming for Close-Loop Control (4).
The control problem in Eq.
(4) can be solved by the dynamical programming principle (Bellman, 1952).
For simplicity we consider one input data sample, and deﬁne a value function V : T × R d
→ R (where T := { 0 , 1 , . .
, T − 1 } ).
Here V ( t, x ) represents the optimal cost-to-go function of Eq.
(4) incurred from time t at state x .
One can show that V ( t, x ) satisﬁes the dynamic programming principle
[Equation]
Eq.
(5) gives a necessary and sufﬁcient condition for optimality of Eq.
(4), and it is often solved backward in time by discretizing the entire state space.
The state dimension of a modern neural network is at the order of thousands or even higher, therefore, discretizing the state space and directly solving Eq.
(5) is intractable for real-world applications due to the curse of dimensionality.
Solving (5) via the Pontryagin’s Maximum Principle.
To overcome the computational challenge, the Pontryagin’s Maximum Principle (Kirk, 1970) converts the intractable dynamical programming into two ordinary differential equations and a maximization condition.
Instead of computing the control policy π of Eq.
(5), the Pontryagin’s Maximum Principle provides a necessary condition for the optimality with a set of control parameters [ u ∗ 0 , · · · , u ∗ T ].
The mean-ﬁeld Pontryagin’s Maximum Principle can be considered when the initial condition is a batch of i.i.d.
samples drawn from D .
Speciﬁcally, we trade the intractable computational complexity with processing time for solving the Hamilton equations and its maximization condition for every newly observed data.
To begin with, we deﬁne the Hamiltonian H :
T × R d × R d × R l × R m → R as
[Equation]
Let x ∗ denote the corresponding optimally controlled state trajectory.
There exists a co-state process p ∗ : [0 , T ] → R d such that the Hamilton’s equations
[Equation]
are satisﬁed.
The terminal co-state p T = 0 , since we do not consider the terminal loss Φ( x T , y ) .
Moreover, we have the Hamiltonian maximization condition
[Equation]
Instead of solving Eq.
(5) for the optimal control policy π ∗ ( x t ) , for a given initial condition, the
Pontryagin’s
Maximum Principle seeks for a open-loop optimal solution such that the global optimum of Eq.
(5) is satisﬁed.
The limitation of using the maximum principle is that the control parameters u ∗ t need to be solved for every unseen data to achieve the optimal solution.
Algorithm Flow.
The numerical implementation of CLC-NN is summarized in Alg. 1.
Given a trained network (either from standard or adversarial training) and a set of embedding functions, the controls are initialized as u t = 0 , ∀ t ∈ T , because adding random initialization weakens the
Input : Possibly perturbed data x (cid:15) , a trained neural network, embedding functions [ E 1 , · · · , E T − 1 ] , maxItr (maximum number of iterations).
Output: A set of optimal control parameters u ∗ 0 , · · · , u ∗ T − 1 .
1 for k = 0 to maxItr do
J k = 0 , for t = 0 to T − 1 do x t +1 ,k
= f ( x t,k , θ t , u t,k ) , where x 0 ,k
= x (cid:15) , (cid:46)
Forward propagation Eq.
J k = J k + L ( x t,k , u t,k , E t ( x t,k )) , (cid:46)
Objective function Eq.
(4), end for for t = T to 1 do p t,k = p
Tt +1 · ∇ x t f ( x t,k , θ t , u t,k ) − ∇ x t L ( x t,k , u t,k , E t ( x t,k )) , where p T,k = 0 , (cid:46) Backward propagation Eq.
(8) end for for t = 0 to T − 1 do u t,k [-+
-]+1 =
u t,k + (cid:0) p
Tt +1 ,k · ∇ u t f
( x t,k , θ t , u t,k ) − ∇ u t L ( x t,k , u t,k , E t ( x t,k )) (cid:1) , (cid:46) Maximization of Hamiltonian Eq.
(9) based on Eq.
(6) and gradient ascent.
end for 15 end for robustness performance in general, and clean trajectory often does not result in any running loss for the gradient update on the control parameters.
In every iteration, a given input x 0 is propagated forwardly with Eq.
(7) to obtain all the intermediate hidden states x t for all t and to accumulate cost
J .
Eq. (8) backward propagates the co-state p t and Eq. (9) maximizes the t th Hamiltonian with current x t and p t to compute the optimal control parameters u ∗ t .
E RROR A NALYSIS FOR S IMPLIFIED L INEAR C ASES
For the ease of analysis, we consider a simpliﬁed neural network with linear activation functions:
[Equation]
and reveal why our proposed method can improve robustness in the simplest setting.
Given a perturbed data sample x (cid:15), 0 , we denote its perturbation-free counterpart as x 0 so that z = x (cid:15), 0 − x 0 .
We consider a general perturbation where z is the direct sum of two orthogonal contributions: z (cid:107) , which is a perturbation within the data manifold (subspace), and z ⊥ , which is a perturbation in the orthogonal complement of the data manifold.
This case is general: if we consider adversarial attacks, then the perturbation along the orthogonal complement dominates.
In contrast, if we consider random perturbations, then the two perturbations are on the same scale.
Our formulation covers both such extreme scenarios, together with intermediate cases.
We use an orthogonal projection as the embedding function such that E t =
V rt
( V rt ) T , where V rt is the ﬁrst r columns of the eigenvectors computed by the Principle Component Analysis on a collection of states x t .
The proposed CLC-NN minimizes (cid:107) x (cid:15),t − x t (cid:107) 22 by reducing the components of x (cid:15),t that lie in the the orthogonal complement of Z t (cid:107) .
The following theorem provides an error estimation between x (cid:15),t and x t .
Theorem 1.
For t ≥ 1 , we have the error estimation
[Equation]
where γ t := max s ≤ t (cid:0) 1 + κ ( θ s ) 2 (cid:1) (cid:107) I − θ T s θ s (cid:107) 2 , and α = c 1+ c , c represents the control regularization.
In particular, the equality
[Equation]
holds when all θ t are orthogonal.
The detailed derivation is presented in Appendix A. Let us summarize the insights from Theorem 1.
[Table]
• The above error estimation is general for any input perturbation.
It shows the working principle behind the proposed CLC-NN on controlling the perturbation that lies in the orthogonal complement of input subspace ( z ⊥ ).
• The above error estimation improves as the control regularization c goes to 0 (so α → 0 ).
It is not the sharpest possible as it relies on a greedily optimal control at each layer.
The globally optimal control deﬁned by the Ricatti equation may achieve a lower loss when c (cid:54) = 0 .
• When the dimension of embedding subspace r decreases, our control becomes more effective in reducing (cid:107) x (cid:15),t − x t (cid:107) 22 .
This means that the control approach works the best when the data is constrained on a low dimensional manifold, which is consistent with the manifold hypothesis.
In particular, observe that as r → 0 , (cid:107) z (cid:107) (cid:107) {+
+}
→•
The obtained upper bound is tight: the estimated upper bound becomes the actual error if all the forward propagation layers are orthogonal matrices.
N UMERICAL E XPERIMENTS
We test our proposed CLC-NN framework under various input data perturbations.
Here we brieﬂy summarize our experimental settings, and we refer readers to Appendix B for the details.
• Original Networks without Close-Loop Control.
We choose residual neural networks (He et al., 2016) with ReLU activation functions as our target for close-loop control.
In order to show that CLC-NN can improve the robustness in various settings, we consider networks from both standard and adversarial trainings.
We consider multiple adversarial training methods: fast gradient sign method ( FGSM )
(Goodfellow et al., 2014), project gradient descent ( PGD ) (Madry et al., 2017), and the Label smoothing training ( Label Smooth )
(Hazan et al., 2017).
• Input Perturbations.
In order to test our CLC-NN framework, we perturb the input data within a radius of (cid:15) with (cid:15) = 2 , 4 and 8 respectively.
We consider various perturbations, including nonadversarial perturbations with the manifold-based attack (Jalal et al., 2017)
( Manifold ), as well as some adversarial attacks such as FGSM , PGD and the CW methods (Carlini & Wagner, 2017).
• CLC-NN Implementations.
We consider both linear and nonlinear embedding in our closeloop control.
Speciﬁcally, we employ a principal component analysis with a 1% truncation error for linear embedding, and convolutional auto-encoders for nonlinear embedding.
We use Adam (Kingma & Ba, 2014) to maximize the Hamiltonian function (9) and keep the same hyperparameters (learning rate, maximum iterations) for each model against all perturbations.
Result Summary: Table 1 and Table 2 show the results for both CIFAR- 10 and CIFARdatasets on some neural networks from both standard training and adversarial training respectively.
• CLC-NN signiﬁcantly improves the robustness of neural networks from standard training.
Table 1 shows that the baseline network trained on a clean data set becomes completely vulnerable (with almost 0% accuracy) under PGD and CW attacks.
Our CLC-NN improves its accuracy to nearly 40% and 80% under PGD and CW attacks respectively.
The accuracy under FGSM attacks has almost been doubled by our CLC-NN method.
The accuracy on clean data is slightly decreased because the lower-dimensional embedding functions cannot exactly capture Z (cid:107) or M . • CLC-NN further improves the robustness of adversarially trained networks.
Table 2 shows that while an adversarially trained network is inherently robust against certain types of perturbations, CLC-NN strengthens its robustness signiﬁcantly against various perturbations. For in- [Table]
[Table] stance, CLC-NN improves the accuracy of an FGSM trained network under PGD and CW attacks by a maximum of 59% and 81% , respectively.
• The robustness improvement of adversarially trained networks is less signiﬁcant.
This is expected because the trajectory of perturbed data lies on the embedding subspace Z (cid:107) if that data sample has been used in adversarial training.
However, our experiments show that applying CLCNN to adversarially trained networks can achieve the best performance under most attacks.
Comparison with PixelDefend (Song et al., 2017).
Our method achieves similar performance on CIFAR-10 with slightly different experimental setting.
Speciﬁcally, PixelDefend improved the robustness of a normally trained 62 -layer ResNet from 0% to 78% against CW attack.
Our proposed
CLC-NN improves the robustness of a 20 -layer ResNet from 0% to 81% against CW attacks.
Furthermore, we show that CLC-NN is robust against the manifold-based attack.
No result was reported for CIFAR-100 in Song et al.
Comparison with Reactive Defense Reactive defenses can be understood as only applying a control at the initial condition of a dynamical system.
Speciﬁcally, reactive defense equipped with linear embedding admits the following dynamics:
[Equation]
By contrast, CLC-NN controls all hidden states and results in a decreasing error as the number of layers T increases (c.f. Theorem 1).
To quantitatively compare CLC-NN with reactive defense, we implement them with the same linear embedding functions and against all perturbations.
In
Table 3, CLC-NN outperforms reactive defense in almost all cases except that their performances are case-dependent on clean data.
C ONCLUSION
We have proposed a close-loop control formulation to improve the robustness of neural networks.
We have studied the embedding of state trajectory during forward propagation to deﬁne the optimal control objective function.
The numerical experiments have shown that our method can improve the robustness of a trained neural network against various perturbations.
We have provided an error estimation for the proposed method in the linear case.
Our current implementation uses the Pontryagin’s Maximum Principle and an online iterative algorithm to overcome the intractability of solving a dynamical programming.
This online process adds extra inference time.
In the future, we plan to show the theoretical analysis for the nonlinear embedding case.
Acknowledgement Zhuotong Chen and Zheng Zhang are supported by NSF CAREER Award No.
1846476 and NSF CCF No. 1817037.
Qianxiao Li is supported by the start-up grant under the NUS
PYP programme.
A A PPENDIX A E RROR E STIMATION FOR THE P ROPOSED CLC-NN
Preliminaries We deﬁne the performance index at time t as
[Equation]
(13) where Q t =
I − V rt
( V rt ) T , V rt is the linear projection matrix at time t with only its ﬁrst r principle components corresponding to the largest r eigenvalues.
The optimal feedback control is deﬁned as
[Equation]
due to the linear system and quadratic performance index, the optimal feedback control admits an analytic solution by taking the gradient of performance index (Eq.
and setting it to 0 .
[Equation]
which leads to the analytic solution of u ∗ t ( x t ) as
[Equation]
The above analytic control solution u ∗ t optimizes the performance index instantly at time step t , the error measured by Eq.
(13) for the dynamical programming solution x (cid:15),t must be smaller or equal to the state trajectory equipped with u ∗ t deﬁne by Eq.
(14), which gives a guaranteed upper bound for the error estimation of the dynamic programming solution.
We deﬁne the feedback gain matrix K t =
( c · I + Q Tt Q t ) − 1 Q Tt Q t .
Thus, the one-step optimal feedback control can be represented as u ∗ t = − K t x t .
The difference between the controlled system applied with perturbation at initial condition and the uncontrolled system without perturbation is shown
[Equation]
The control objective is to minimize the state components that span the orthogonal complement of the data manifold ( I − V rt
( V rt ) T ), when the input data to feedback control only stays in the state manifold,
such that (cid:107) ( I − V rt
( V rt ) T ) x
t (cid:107) 22 = 0 , the feedback control K t x t = 0 .
The state difference of Eq.
(15) can be further shown by adding a 0 term of ( θ t K t x t )
[Equation]
In the following, we show a transformation on the control dynamic term ( I − K t ) based on its deﬁnition.
Lemma 1.
For t ≥ 0 , we have
[Equation]
where P t :=
V rt
( V rt ) T , which is the orthogonal projection onto Z t (cid:107) , and α :=
c 1+ c such that α ∈ [0 , 1] .
Proof.
Recall that K t =
( c · I + Q Tt Q t ) − 1 Q Tt Q t , and Q t
= I − V rt
( V rt ) T , Q t can be diagonalized as following
[Equation]
where the ﬁrst r diagonal elements have common value of 0 and the last ( d − r ) diagonal elements have common value of 1 .
Furthermore, the feedback gain matrix K t can be diagonalized as
[Equation]
where the last ( d − r ) diagonal elements have common value of 1 1+ c .
The control term ( I − K t ) thus can be represented as
[Equation]
where the ﬁrst r diagonal elements have common value of 1 and the last ( d − r ) diagonal elements have common value of c 1+ c .
By denoting the projection of ﬁrst r columns as V rt and last ( d − r ) columns as ˆ V r t , it can be further shown as
[Equation]
Oblique Projections.
Let P be a linear operator on R d ,
• We say that P is an projection if P 2 = P .
• P is an orthogonal projection if P = P T
=
P 2 .
• If P 2 = P but P (cid:54) =
P T , it is called an oblique projection.
Proposition 2.
For a projection P ,
If P is an orthogonal projection, then (cid:107) P (cid:107) 2 = 1 .
If P is an oblique projection, then (cid:107) P (cid:107) 2 > 1 . 3.
If P , Q are two projections such that range ( P )
=
range ( Q ) , then PQ = Q and QP
= P . 4.
If P is a projection, then rank ( P ) =
Tr ( P ) .
Furthermore, if P is an orthogonal projection, then rank ( P ) =
(cid:107) P (cid:107) 2 F =
Tr ( PP T ) .
Deﬁne for t ≥(cid:40) P 0 t := P t ,
P ( s +1) t := θ − 1 t − s − 1 P s t θ t − s − 1 , s = 0 , 1 , . . .
, t − 1 ,
Lemma 3.
Let P st be deﬁned as above for 0 ≤ s ≤ t .
Then
P st is a projection.
P st is a projection onto Z t − s (cid:107) ,
i
e. range [Equation]
P
[Equation] [Equation]
where κ ( A ) is the condition number of A , i.e. κ ( A )
= (cid:107) A (cid:107) 2 · (cid:107)
A − 1 (cid:107) 2 , and r = rank ( Z 0 (cid:107) )
=
rank ( Z 1 (cid:107) )
= . . .
=
rank ( Z t (cid:107) ) .
Proof.
We prove it by induction on s for each t .
For s = 0 , P 0 t =
P t , which is a projection by its deﬁnition.
Suppose it is true for s such that P st = P st P st , then for ( s + 1) ,
[Equation]
We prove it by induction on s for each t .
For s = 0 , P 0 t =
P t , which is the orthogonal projection onto Z t (cid:107) .
Suppose that it is true for s such that P st is a projection onto Z t − s (cid:107) , then for ( s + 1) , P s +1 t [Equation] θ
[Equation]
t
[Equation]
s − 1 P st θ t − s − 1 , which implies
[Equation]
We use the inequalities (cid:107) AB (cid:107) F ≤ (cid:107) A (cid:107) 2 (cid:107) B (cid:107) F , and (cid:107) AB (cid:107) F ≤ (cid:107) A (cid:107) F (cid:107) B (cid:107) 2 .
By the deﬁnition of P s t ,
[Equation]
we have the following
[Equation] [Equation]
θ t − 1 θ t − 2 · · · θ t − s (cid:1) − 1 (cid:107) 22 · (cid:107) (cid:0) θ t − 1 θ t − 2 · · · θ t − s (cid:1) (cid:107) 22 · (cid:107) P 0 t (cid:107) 2 F ,
[Equation] [Equation]
· r,
Lemma 2(4) .
The following Lemma uses the concept of oblique projection to show a recursive relationship to project any t th state space of Eq.
(16) back to the input data space.
Lemma 4.
Deﬁne for 0 ≤ s ≤ t ,
[Equation]
Then, Eq.
(16) can be written as
[Equation]
Proof.
We prove it by induction on t .
For t = 1 , by the deﬁnition of G st and transformation from
Lemma 1,
[Equation]
Suppose that it is true for ( x (cid:15),t − x t ) [Equation] by using Eq.
(16) and Lemma
[Equation]
we have
[Equation]
Lemma
[Equation]
Recall the deﬁnitions of P ( s +1)
t
[Equation]
, and G st := α · I + (1 − α ) P st , we have the following
[Equation]
which results in the equality for the oblique projections.
Furthermore,
[Equation]
Applying the above to Eq.
(17) results in
[Equation]
Lemma 5.
Let
[Equation]
Then,
[Equation]
Proof.
We prove it by induction on t .
Recall the deﬁnition of G st := α · I + (1 − α ) · P st .
When t = 1 ,
[Equation]
Suppose that it is true for t such that
[Equation]
for ( t + 1) ,
[Equation]
Recall Lemma 3, range ( P tt ) =
range ( P ss ) =
Z 0 (cid:107) .
According to Proposition 2 (3), P tt P ss =
P ss .
Hence,
[Equation]
Lemma 6.
Let V ∈ R
d
× r be a matrix whose columns are an orthogonal basis for a subspace D , and θ ∈ R
d × d be invertible.
Let P =
VV T be the orthogonal projection onto D .
Denote by ˆ P the orthogonal projection onto θ D :=
{ θ x : x ∈ D} .
Then
θ − 1 ˆ P θ is an oblique projection onto D . 2.
(cid:107) θ − 1 ˆ P θ − P (cid:107) 2 ≤ (cid:0) 1 + κ ( θ ) 2 (cid:1) · (cid:107) I − θ T θ (cid:107) 2 .
In general, the last inequality shows that θ − 1 ˆ P θ =
P , if θ is orthogonal.
Proof.
( θ − 1 ˆ P θ ) 2 = θ − 1 ˆ P 2 θ = θ − 1 ˆ P θ , therefore, θ − 1 ˆ P θ is an projection
Since ˆ P is orthogonal projection onto the row space of θ V , then
[Equation]
Furthermore,
[Equation]
We further bound (cid:107) [ V T θ T θ V ] − 1 (cid:107) 2 .
[Equation]
Hence, we have
[Equation]
Corollary 1.
Let t ≥ 1 .
Then for each s = 0 , 1 , · · · , t , we have
[Equation]
where
• θ := θ s − 1 · · · θ 0 , s ≥ 1 , • θ :=
I , s = 0 .
Observe that P ss =
( θ s ) − 1 P s θ s .
Using Lemma 6, we arrive at the main theorem.
Theorem 1.
For t ≥ 1 , we have the error estimation (cid:107) x
[Equation]
where γ t := max s ≤ t (cid:0) 1 + κ ( θ s ) 2 (cid:1) (cid:107) I − θ
Ts θ s (cid:107) 2 , and α = c 1+ c , c represents the control regularization.
In particular, the equality
[Equation]
holds when all θ t are orthogonal.
Proof.
The input perturbation z = x (cid:15), 0 − x 0 can be written as z = z (cid:107) + · z ⊥ , where z (cid:107)
∈ Z (cid:107) and z
⊥ ∈
Z ⊥ , where z (cid:107) and z ⊥ are vectors such that
• z (cid:107) ·
z ⊥ = 0 almost surely.
• z (cid:107) , z ⊥ have uncorrelated components.
• z (cid:107) ∈ D , and z ⊥ ∈ D ⊥ .
Since z (cid:107) and z ⊥ are orthogonal almost surely, recall Lemma 4,
[Equation]
( G t
[Equation]
[Equation]
z (cid:107) 22 , (18)
For the term (cid:107) ( G t − 1 t − 1 · · · G 00 )
z (cid:107) 22 , recall Lemma 5,
[Equation]
s
[Equation]
P
[Equation]
22 , = (cid:107) α t
z
+ (1 − α ) t − 1 (cid:88) s
= α s P
z
+ (1 − α ) t − 1 (cid:88) s
= α s ( P ss − P 0 )
z (cid:107) 22 , = (cid:107) α t
z
+ (1 − α t ) z (cid:107) +
(1 − α ) t − 1 (cid:88) s
=
α s ( P ss − P 0 )
z (cid:107) 22 , in the above, P 0 is an orthogonal projection on t =
(input data space), therefore, P
z =
z (cid:107) .
Furthermore, when s = 0 , P ss − P 0 = 0 .
Thus,
[Equation]
(cid:107)[-(cid:107) 
-]
[Equation]
1 − α )t − 1 (cid:88) s,q =
[Equation]
+ 2 α t (1 − α )
[Equation]
(cid:88) s = [Equation] ) t [Equation]
s =
α [Equation] z .
Using Corollary we have
• z T ( P ss − P 0 )
z ≤ (cid:107) z (cid:107) 22 · (cid:107) P ss − P 0 (cid:107) 2 , ≤ γ t (cid:107) z (cid:107) 22 .
• z T ( P ss − P 0 ) T
( P qq − P 0 )
z ≤ (cid:107) z (cid:107) 22 · (cid:107) P ss − P 0 (cid:107) 2 · (cid:107) P qq − P
(cid:107) 2 , ≤ γ 2 t (cid:107)
z (cid:107) 22 .
• ( z (cid:107) )
T ( P ss − P 0 )
z ≤ γ t (cid:107) z (cid:107) (cid:107) 2 · (cid:107) z (cid:107) 2 , ≤ γ t (cid:107) z (cid:107) 22 .
Thus, we have
(cid:107) ( G t − 1 t − 1 · · · G 00 )
z (cid:107) 22 ≤ α 2 t (cid:107) z ⊥ (cid:107)
+ (cid:107) z (cid:107) (cid:107)
+ α 2 (1 − α t − 1 ) 2 γ 2 t (cid:107) z (cid:107) 22 + 2 α t +1 (1 − α t − 1 )
γ t (cid:107) z (cid:107)+ 2 α (1 − α t )(1 − α t − 1 ) γ
[Equation]
Recall the error estimation in Eq.
[Equation]
[Table]
Conv2d (input channel = 3 , output channel = 16 , kernel size = 3 × 3 ),
BatchNorm2d(channel = 16 ),
Relu()
Residual Block{ Conv2d (input channel = 16 , output channel = 16 , kernel size = 3 × 3 ),
BatchNorm2d(channel = 16 ),
Relu(),
Shortcut() } ×
Residual Block
Conv2d (input channel = 16 , output channel = 32 , kernel size = 3 × 3 ),
BatchNorm2d(channel = 16 ),
Relu(),
Shortcut(), { Conv2d (input channel = 32 , output channel = 32 , kernel size = 3 × 3 ),
BatchNorm2d(channel = 32 ),
Relu(),
Shortcut() } ×
Residual Block
Conv2d (input channel = 32 , output channel = 64 , kernel size = 3 × 3 ),
BatchNorm2d(channel = 64 ),
Relu(),
Shortcut(), { Conv2d (input channel = 64 , output channel = 64 , kernel size = 3 × 3 ),
BatchNorm2d(channel = 64 ),
Relu(), Shortcut() } ×
Final
[Table]
Fully Connected (64 , 10)
In the speciﬁc case, when all θ t are orthogonal,
[Equation]
= 0 .
Thus,
[Equation]
B A PPENDIX B D ETAILS OF E XPERIMENTAL S ETTING B. N ETWORK C ONFIGURATIONS
Since the proposed CLC-NN optimizes the entire state trajectory, it is important to have a relatively smooth state trajectory, in which case, when the reconstruction loss (cid:107)E t ( x t ) − x t (cid:107) 22 at layer t is small, the reconstruction losses at its adjacent layers should be small.
For this reason, we use residual neural network (He et al., 2016)
as network candidate to retain smoother dynamics.
The conﬁguration of the residual neural network used for both CIFAR- 10 and CIFAR- 100 is shown in
Tab. 4.
Based on the conﬁguration of residual neural network shown in Tab. 4, we construct 4 embedding functions applied at input space, outputs of initial layer, residual block 1 and residual block 2 .
The output of residual block 3 is embedded with a linear orthogonal projection.
We randomly select 5000 clean training data to collect state trajectories at all 5 locations.
• For the linear orthogonal projections: we apply the Principle Component Analysis on each of the state collections.
We retain the ﬁrst r columns of the resulted basis, such that r = arg min { i : λ 1 + ...
+ λ i λ 1 ... + λ d ≥ 1 − δ } , where δ = 0 .
[Table]
Conv2d (input channel = c 1 , output channel =
c 2 , kernel size = 4 × 4 , stride = 2 × 2 , padding = 1 × 1 ),
ELU(alpha= 1 ),
BatchNorm2d(channel =
c 2 ),
Conv2d (input channel =
c 2 , output channel = c 3 , kernel size = 4 × 4 , stride = 2 × 2 , padding = 1 × 1 ), ELU(alpha= 1 ).
[Table]
ConvTranspose2d (input channel =
c 3 , output channel =
c 2 , kernel size = 4 × 4 , stride = 2 × 2 , padding = 1 × 1 ),
ELU(alpha= 1 ),
[Table]
(input channel =
c 2 , output channel = c 1 , kernel size = 4 × 4 , stride = 2 × 2 , padding = 1 × 1 ),
[Table]
• For the nonlinear embedding: we train 4 convolutional auto-encoders for the input space, outputs of the initial layer and residual blocks 1 , 2 .
All of the embedding functions are trained individually.
We adopt shallow convolutional auto-encoder structure to gain fast inference speed, in which case, CLC-NN equipped with linear embedding often outperform the nonlinear embedding as shown in Tab. 1.
The conﬁguration of all 4 convolutional auto-encoders are shown in Tab. 5.
B. P ERTURBATIONS AND D EFENSIVE T RAINING
In this section, we show details about the perturbations and robust networks that have been considered in this work.
For the adversarial training objective function,
[Equation]
where ∆( x 0 , (cid:15) ) generates a perturbed data from given input x 0 within the range of (cid:15) .
λ balances between standard accuracy and robustness.
We choose λ = 0 .
5 in all adversarial training.
For robust networks, we consider both perturbation agnostic and non-agnostic methods.
For the perturbation agnostic adversarial training algorithms equipped ∆( x 0 , (cid:15) ) , the resulted network that is the most robust against the ∆( x 0 , (cid:15) ) perturbation.
On the contrary, perturbation non-agnostic robust training methods are often robust against many types of perturbations.
• Adversarial training with the fast gradient sign method (FGSM) (Goodfellow et al., 2014) considers perturbed data as follows.
[Equation]
where sign ( · ) outputs the sign of the input.
In which case, FGSM considers the worse case within the range of (cid:15) along the gradient ∇ x 0 Φ( x T , y ) increasing direction.
Due to the worse case consideration, it does not scale well for deep networks, for this reason, we adversarially train the network with FGSM with (cid:15) = 4 , which is half of the maximum perturbation considered in this paper.
• The label smoothing training (Label Smooth) (Hazan et al., 2017) does not utilize any perturbation information ∆( x 0 , (cid:15) ) .
It converts one-hot labels into soft targets by setting the correct class as 1 − (cid:15) , while other classes have value of (cid:15)N − 1 , where (cid:15) is a small constant and N is number of classes.
Speciﬁcally, we choose
[Table]
• Adversarial training with the project gradient descent (PGD) (Madry et al., 2017) generates adversarial data by iteratively run FGSM with small step size, which results in stronger perturbations compared with FGSM within the same range (cid:15) .
We use 7 -step of (cid:15) = 2 to generate adversarial data for robust training.
For Perturbations, we consider the maximum range of (cid:15) = 2 , 4 , 8 to test the performance the network robustness against both strong and weak perturbations.
For this work, we test network robustness with the manifold-based attack (Jalal et al., 2017), FGSM (Goodfellow et al., 2014), 20 -step of PGD (Madry et al., 2017) and the CW attack (Carlini & Wagner, 2017).
B. O NLINE O PTIMIZATION
Optimization Methods.
we use Adam (Kingma & Ba, 2014) to maximize the Hamiltonian Eq.
(9) with default setting.
In which case, solving the PMP brings in extra computational cost for inference.
Each online iteration of solving the PMP requires a combination of forward propagation (Eq.
backward propagation (Eq.
(8)) and a maximization w.r.t.
the control parameters (Eq.
(9)), which has computational cost approximately the same as performing gradient descent on training a neural network for one iteration.
For the numerical results presented in the paper, we choose the maximum iteration that gives the best performance from one of [5 , 10 , 20 , 30 , 50] .
C M ORE N UMERICAL E XPERIMENTS
The proposed CLC-NN is designed to be compatible with existing open-loop trained.
We show extra experiments by employing the proposed CLC-NN on two baseline models, DenseNet- 40 (Table 6).
The layer-wise projection performs orthogonal projection on the hidden state.
We deﬁne the local cost function at the t th layer as follows
[Equation]
the layer-wise achieves the optimal solution at local time t ,
[Equation]
However, the layer-wise optimal control solution does not guarantee the optimum across all layers.
In Table 7, we launch comparisons between the proposed CLC-NN with layer-wise projection.
Speciﬁcally, under all perturbations the proposed CLC-NN outperforms layer-wise projection.
D R OBUSTNESS A GAINST M ANIFOLD - BASED A TTACK
The manifold-based attack (Jalal et al., 2017) (denoted as Manifold) has shown great success on breaking down the manifold-based defenses (Samangouei et al., 2018).
The proposed CLC-NN can successfully defend such speciﬁcally design adversarial attack for manifold-based defenses and improves the robustness accuracy from 1% to 81% for the standard trained model in Cifar- 10 , and 2% to 52% in Cifar- 100 .
[Table]
We provide detailed explanation for the successful defense of the proposed CLC-NN against such strong adversarial attack.
Exsiting manifold-based defense (Samangouei et al., 2018) focuses on detecting and de-noising the input components that do not lie within the underlying manifold.
The overpowered attack proposed in Jalal et al.
(2017) searches adversarial attack with in the embedded latent space, which is undetectable for the manifold-based defenses and caused complete failure defense.
In the real implementation, the manifold-based attack (Jalal et al., 2017) is detectable and controllable under the proposed framework due to the following reason.
The numerically generated manifold embedding functions are not ideal.
The error sources of non-ideal embedding functions are mainly due to the algorithm that used to compute the manifold, the architecture of embedding function, and the distribution shift between training and testing data (embedding functions of training data do not perfectly agree with testing data).
In which case, even the perturbation is undetectable and non-controllable at initial layer, as it is propagated into hidden layers, each layer ampliﬁes such perturbation, therefore, the perturbation becomes detectable and controllable in hidden layers.
We randomly select the batch of testing data to generate the manifold-based attack following the same procedure proposed in Jalal et al.
The proposed method improves the attacked accuracy from 1% to 78% .
More speciﬁcally, we compare the differences of all hidden states spanning the orthogonal complement between a perturbed testing data and its unperturbed counterpart, (cid:107) P ⊥ t x (cid:15),t −
P ⊥ t x (cid:15),t (cid:107) , where P ⊥ t is a projection onto the orthogonal complement.
The difference is growing such as 0 , 0 . 016 , 0 .
for hidden states at layer 0 , 1 , 2 , 3 , 4 respectively.
This validates the argument for how the proposed method is able to detect such perturbation and controls the perturbation in hidden layers.
Furthermore, we provide some insights about the reasons behind the success of such an adversarial attack.
This follows the same concept of the existence of adversarial attack in neural networks.
The highly nonlinear behaviour of neural networks preserves complex representative ability, meanwhile, its powerful representation results in its vulnerability.
For example, a constant function has 50% chance to make a correct prediction in binary classiﬁcation problem under any perturbation, but its performance is limited.
Therefore, we propose to use a linear embedding function that compensates between the embedding accuracy and robustness.
E D EFINITION OF THREAT MODEL
Generally, an attacker should not have access to the hidden states during inference, in which case, an attacker is not allowed to inject extra noise during inference.
To deﬁne the threat model of the proposed method, for the white-box setting, an attacker has access to both network and all embedding functions.
The condition that the perturbation (cid:15) · z makes our method vulnerable is deﬁned as follows,
[Equation]
In words, the perturbation (cid:15) · z applied on the input data must result in 0 reconstruction losses across all hidden layers, which means its corresponding state trajectory does not span any of all orthogonal complements of all hidden state spaces.
To obtain an effective attack satisfying the above equation, conventional gradient-based attackers cannot guarantee to ﬁnd an perfect attack.
A possible way is to perform grid-search backward in layers to ﬁnd such an adversarial attack satisfying the thread model condition, which is extremely costly.