O N
E XPLAINING
N
EURAL N ETWORK R OBUSTNESS WITH A CTIVATION P ATH
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite their verified performance, neural networks are prone to be misled by maliciously designed adversarial examples.
This work investigates the robustness of neural networks from the activation pattern perspective.
We find that despite the complex structure of the deep neural network, most of the neurons provide locally stable contributions to the output, while the minority, which we refer to as float neurons, can greatly affect the prediction.
We decompose the computational graph of the neural network into the fixed paths and float paths and investigate their role in generating adversarial examples.
Based on our analysis, we categorize the vulnerable examples into Lipschitz vulnerability and float neuron vulnerability.
We show that the boost of robust accuracy from randomized smoothing is the result of correcting the latter.
We then propose an SC-RFP (smoothed classifier with repressed float path) to further reduce the instability of the float neurons and show that our result can provide a higher certified radius as well as accuracy.
I NTRODUCTION
Despite their verified performance, neural networks are prone to be misled by maliciously designed adversarial examples.
In response to this issue, many studies focus on defensive algorithms that aim to increase the robustness of deep neural networks.
One of the emerging topics in this field is certifiable methods that aim to construct a guaranteed region, within which classifiers are able to provide stable results regardless of the perturbation.
The certifiable methods appear in two different forms: verifiable training and randomized smoothing.
This work introduces an SC-RFP (smoothed classifier with repressed float path) which builds on randomized smoothing algorithms and is able to further improve their robustness accuracy.
We decompose the local mapping function into fixed paths and float paths according to the stability of neurons on the path.
The fixed paths have a stable mapping relationship between input and output, while the float paths can result in a sudden change of the mapping function and alter the result.
We categorize the adversarial examples into Lipschitz vulnerable and float neuron vulnerable.
With respect to the ability of randomized classifiers in correcting misclassified data, we conclude that the essence of the smoothed classifier is to average the contribution of the float path and achieve a locally stable result.
Based on this, we further repress the float paths of the network and show that such a classifier can achieve better performance.
The theoretical basis of this work is developed from the analysis of the activation region that was initially proposed for explaining the performance of neural network with a piecewise linear activation function.
The input domain of such a neural network N is separated into many regions, within which the mapping of N is piecewise linear.
Previous investigation of this field includes the expressivity, sensitivity, and potential issues of the network.
However, due to the complexity of neural network, the theoretical investigation only provides insights into the neural network but has yet to be deployed downstream.
In this work, we use the theory to explain the model robustness and introduce a novel way to apply the complex theory to practical.
The contributions of this work are: (1) we introduce a complete framework to describe and decompose the neural network according to the activation status of each neuron; (2) we provide an explanation of adversarial examples and discuss the role of smoothed classifiers as well as their contribution in correcting misclassified example; (3) we introduce SC-RFP that achieves better performance in certifying the network.
R ELATED W ORKS
The adversarial examples are malicious inputs that are formed by applying an imperceptible perturbation to the original inputs but result in misclassification of a well-trained network (Biggio et al.
; Szegedy et al.
To explain the existence of adversarial example, previous works presented several hypotheses, such as linearity hypothesis (Szegedy et al.
; Luo et al. (2015)) and Evolutionary stalling (Rozsa et al.
Early works on increasing the robustness of neuron networks focused on adversarial training methods (Goodfellow et al.
; Wong et al.
Tram`er et al.
; Dong et al.
; Kurakin et al.
(2016)), while recent investigation shows adversarial training methods can be broken by more advanced attacks.
To address the issue, certifiable training and randomized smoothing methods aim to provide a certified region, within which the input data are free from attack.
By viewing the training as a convex optimization problem, dual relaxation approaches apply duality to provide a solid bound for training as well as verify the network (Wong & Kolter (2018); Wong et al.
An alternative is to estimate the Lipschitz boundary of the network and introduce constraints on either objective loss (Tsuzuku et al.
(2018)) or forward propagation (Lee et al.
; Weng et al.
; Zhang et al.
; Huang et al.
As verifiable training methods often come with a compromise of performance, recent works focus on bridging the gap between adversarial and verifiable training to address the scalability and accuracy issue (Xiao et al.
; Balunovi´c & Vechev (2020);
De Palma et al.
On the other hand, randomized smoothing introduces a smoothed classifier to the base classifier, therefore has a limited effect on the performance of standard models.
Cao & Gong (2017) first propose to ensemble the information around input data to smooth the prediction, but fail to provide a theoretical guarantee on the result.
Lecuyer et al.
(2019) certify the result of the smoothed classifier with differential privacy.
Cohen et al.
(2019) provides a theoretical analysis of the certifiable with
Monte Carlo, followed by Levine et al.
; Li et al.
Jeong & Shin (2020) introduces a regularized to improve the prediction constantly over noise, Jeong et al.
(2021) trains the model on a convex combination of samples and Salman et al.
(2019) employs PGD attack with randomized smoothing to further increases the robustness accuracy.
Another related topic is the explainability of neuron networks.
Lin et al.
(2017), Hornik et al.
(1989) and Park et al.
(2020) investigate how deep models approximate an objective function.
An inspiring observation is that the network is a piecewise function when the activation function is piecewise linear Pascanu et al.
The number of linear regions is then adopted as a proxy of network complexity (Montufar et al.
Hanin & Rolnick (2019a;b)).
Novak et al.
(2018) studies the network sensitivity by countering the transition density of trajectory in the input space.
Jiang et al.
(2022) compares the similarity of activation patterns globally to study the limitation of deep neural network.
Inspired by the theoretical investigation, Jordan et al.
(2019) introduces an algorithm named GeoCert that computes the l p bound of the network with a piecewise linear activation function.
Zhang et al.
(2022) proposes an algorithm that systematically searches the adversarial example based on the activation space of ReLU network.
P RELIMINARIES 3. N OTATIONS
Let N be a d block feedforward neural network for classification task with measure zero parameter set θ with respect to Lebesgue measure.
Each of the block h
i consists of a linear affine ϕ
i , an optional batch-normalization layer ψ i , and piecewise linear activation function σ i , while the last block h d omits the activation function.
Consider D as the distribution of a classification problem with c classes from R n 0 to Y = { 1 , 2 , . .
. , c } , network N computes a function f : R n 0 → R c , where f is a composition of d blocks f = h n ◦ h d − 1 · · · ◦ h 1 .
For every ( x, y ) ∼ D , the network computes a probability for each class f ( x ) ∈ R c and predicts the label of x as the class with the highest probability: ˆ y = arg max m ∈ Y f m ( x ) , where f m ( x ) is the m -th element of the network output vector.
We use x i ( x ; θ ) , y i ( x ; θ ) and z i ( x ; θ ) to denote the input, output, and pre-activation value of block i for data x .
We use a tuple pair ( i, j ) to denote the j -th neuron of layer i and a set I =
∪ d − 1 i =1 { ( i, j ) | j ∈ { 1 , . . . , n i }} to denote the index collection of neurons in intermediate layers, where n i is the output size of layer i .
3. R ANDOMIZED C ERTIFIABLE C LASSIFIER
The research on certifiable training aims to provide a guaranteed region for its input x , within which a classifier always provide stationary result.
To be specific, a classifier is regarded as robust for an input x for perturbation of size r if:
[Equation]
where B p ( x, r ) := { x ′ : ∥ x ′ − x ∥ p ≤ r } is the sphere with radius r measured by the metric induced by p -norm.
By taking the standard performance of the classifier into account, the robust accuracy of f with radius r is then defined as:
[Equation]
However, robust models often incur increased stableness and impaired expressivity.
As a concession for that, randomized algorithms are proposed to verify the network with a sound theoretical bound at the cost of slight additional computation other than model performance.
Let g be a randomized algorithm constructed based on classifier f .
Given a data ( x, y ) ∼ D , g employs a certain degree of randomness during the induction of f .
For instance, smoothed classifier g computes the probability of f ( x + ϵ ) belongs to class i given ϵ ∈ N (0 , σ 2 I )
(Cohen et al.
[Equation]
With certain confidence level α , the lower bound of p A of random variable g y ( x ) and the upper bound p B of the probability of second possible class max m ∈ Y g y
( x ) can be computed.
This induces a certified radius of classifier g ( x
[Equation] [Equation]
For every x ′ ∈ B 2 ( x, r ) , if arg max m ∈ Y g m ( x ) =
Y , then arg max m ∈ Y g m ( x ′ )
=
y .
In other words, with confidence level α , every x ′ within the radius can be correctly classifier by the smoothed classifier.
The smoothed classifier can also appear in other forms depends on the randomness and training algorithm (Levine et al.
Salman et al.
; Jeong & Shin (2020)).
3. A CTIVATION P ATTERN AND C OMPUTATIONAL P ATH
Intuitively, the input space is partitioned into small linear regions by a neural network with piecewise linear activation function.
Each of the regions is initially referred to as an activation region.
Previous works mostly focused on the general properties of linear regions in investigating the expressivity and limitation of neural network.
As our objective is to study the unit-wise reaction towards perturbation, introducing additional notations to describe the neurons is necessary for our analysis.
Definition 1 (Generalized Activation Pattern / Region) .
Let N be a network defined as Section 3.1.
Denote Γ = { γ 1 , γ 2 , . .
, γ q } as a set of breakpoints that separates the domain of activation function into q + 1 intervals U =
{ U 0 , U 1 , . . .
, U q } .
An activation pattern of N is defined as an indexed family that assigns each neuron a sign to represent the status of its pre-activation in U :
[Equation]
Given an activation pattern A , the activation region of A is defined as:
[Equation]
Conversely, given x ∈ R in
[Equation]
we denote the activation pattern of neuron ( i, j ) at x as:
[Equation]
The activation pattern of x is the collection of the all the pattern of neurons:
[Equation]
Activation Region Operator and its Inverse.
The generalized activation pattern describes the activation status of each unit at the intermediate layers.
Given an activation pattern, R ( · ) finds the region, such that, for every x ∈ R ( A ; θ, σ, Γ) , the pre-activation value of each neuron z ij ( x ; θ ) locates within the interval U a ij that determined by its pattern a ij ∈ { 0 , 1 , . .
. , q } .
On the basis of previous works (Hanin & Rolnick (2019b)Raghu et al.
, we further introduce an inverse operator ˆ A ( x ; θ, σ, Γ) that computes the activation pattern of an input x ∈ R n 0 in Definition 1.
This allows us to study the robustness of network by investigating the stability of activation pattern in the sphere B ( x, r ) .
With quantitatively analysis of the pattern, we decompose the computational graph of single input x in Lemma 2 and the prediction difference between inputs x, x ′ ∼ D in Lemma 1.
When dependencies are fixed, we abbreviate Equation 5 and 7 as R ( A ) and ˆ A ( x ) .
From Single Region to its Neighbor.
With the scale of network grows, the mean volume of single activation region decreases exponentially (Hanin & Rolnick (2019a)).
Therefore, it is necessary to generalize our investigation from single region to its neighbor.
Equation 5 suggests that the essence of operator R ( · ) is to find all the x that satisfies a certain constraint determined by the activation pattern.
It can be expressed as the intersection of a sequence of subspaces:
[Equation]
Given a subset of indexes I c ⊂ I , releasing the constraints of neuron set I\I c then equals to remove related intersection.
This can be viewed as merging several close activation regions into one.
From a geometric point of view, the input space is partitioned into small activation regions by a set of hyperplanes { H ijk ( θ ) } , where: H ijk ( θ ) :
= { x ∈ U | z ij ( x ; θ ) =
γ k } is determined by neuron ( i, j ) and breakpoint γ k .
This means that two adjacent activation regions R 1 , R 2 can be merged into one by removing the bent-hyperplanes that separate them.
Combining with Equation 8, the every data x in the merged region has same activation pattern expect for the neuron ( i, j ) based on which the removed hyperplane is defined on.
We use incomplete activation pattern A c to denote the stable neurons, and describe the merged region as:
Definition 2.
Let N be a network defined as Section 3.1.
Given an activation pattern A and a subset of the index set I c ∈
I , we denote A I c ⊂
A is an incomplete activation pattern of A :
A I c := { a ij | a ij ∈ A , ( i, j ) ∈ I C } .
The merged activation region of A I c is denoted as: R ( A c ) =
(cid:84) ( i,j
) ∈I c { x ∈ R in | z ij ( x ) ∈ U a ij } .
F ROM F LOAT P ATH TO M ODEL R OBUSTNESS 4. F LOAT N EURON AND P ATH
Definition 2 proposes a way to describe merged activation regions with incomplete activation patterns.
However, the merged regions are irregular spaces determined by bent-hyperplanes that defined by complex function.
Analyzing the properties of such space is not feasible.
Therefore, we use following definition to describe the neurons in a subset R ⊂ R n 0 .
Definition 3 (Float and Fixed Neuron) .
Let N be a network defined as Section 3.1.
For any space
R ⊂ R n 0 , if a neuron z has the same pattern for any x ∈ R , we refer it as a fixed neuron of R , otherwise it is a float neuron.
We denote the collection of fixed neurons and float neurons as I X ( R ) and I T ( R ) , respectively:
[Equation]
To build a connection between Definition 2 and Definition 3, we propose the following lemma to show: (1) the fixed neurons and float neurons are complementary sets in I and (2) any subset
R ∈ R n 0 can be covered by a merged region defined by the fixed neurons of R .
Lemma 1.
Let N be a neural network defined as Section 3.1.
Given R ⊂ R n 0 , denote I X
and I T are the set of float neuron and fixed neuron in R .
Then:
[Equation]
Now we are able to identify the stability of paths in the computational graph of network N with the following two definitions:
Definition 4.
Let N be a network defined as Section 3.1.
A path of N is a set of neuron indexes:
[Equation]
where [ n i ] = { 1 , 2 , . .
n i } is neuron index for layer i .
Definition 5 (Float Path and Fixed Path) .
Let N be a neural network defined as Section 3.1.
Given a subspace R ∈ R n 0 , a path ζ of N is a float path in R if there exist a neuron ( i, ζ i ) ∈ ζ is a float neuron, otherwise it is a fixed path.
The collection of float path and float path are denoted as:
[Equation]
To understand the motivation of above definition, consider the simple case that there is only 1 neuron ( i, j ) is float neuron in region R ⊂ R n 0 .
The only non-linearity in R is provided by neuron ( i, j ) .
The computational graph of N can be decomposed into float paths and fixed paths according to whether ( i, j ) it on the path.
This is formally described as the first statement in Theorem 1.
Moreover, given x, x ′ ∈ R , the difference between f ( x ) and f ( x ′ ) can be viewed the combination of a linear function and an unstable function with high non-linearity as suggested by the second statement in Theorem 1.
The remaining problem is how to decompose the computational graph of network N , which we discuss in the following section.
D ECOMPOSING
THE C OMPUTATIONAL G RAPH
Now that we introduced the framework for describing neuron status, we aim to build a connection between mapping function f and activation pattern.
Lemma 2 (Decomposition of Neural Network) .
Let N be a neural network defined as Section 3.1.
Given activation pattern A , for any x ∈ R ( A ; θ, σ, Γ) .
The k -th component of output of layer i can be represented as:
[Equation]
where x j,ζ j is the ζ j -th element of input in layer j , W ′ ( i ) and β ( i ) are the equivalent matrix and bias of linear transformation ψ i ◦
ϕ
i , A m,ζ m is the activation pattern of ζ m -th component of layer m and d A m,ζm is the slope of pattern A m,ζ m : d A m,ζm :=
σ ′ ( t ) , t ∈ U
A m,ζm .
Lemma 2 decomposes f in the manner of input and activation pattern.
Equation 11 shows that y ij ( x ) can be decomposed into summation of paths, while the value of each path depends on either input x (former part) or bias β at each layer (latter part), as well as the pattern of neurons on the path.
We use ζ ( i,j ) ( v, A ) to denote the value of a path ζ given v as the input of that path:
[Equation]
In a single activation region R , ∀ x, x ′ ∈ R have same activation pattern A .
The paths from bias ζ ( i,j ) ( β ( j ) ζ j , A ) are constant, while that from input ζ ( i,j ) ( x j,ζ j , A ) are linear to x j,ζ j .
On the other hand, for x, x ′ with different activation patterns A , A ′ , each of the path can be categorized as either fixed or float.
Since neurons on fixed paths has same pattern at x and x ′ , the above linearity remains unchanged, which means ζ ( i,j ) ( v, A )
=
ζ ( i,j ) ( v, A ′ )
if ζ is a fixed path.
In other words, given region
R ⊂ R n 0 , the float paths represent the linear part of f in R , while the fixed paths are the aggregation of all the non-linearity from the float neurons, as described bellow:
Theorem 1.
Let N be a neural network defined as Section 3.
Given R ⊂ X , for any x, x ′ ∈ R with activation pattern A and A ′ , we have :
[Equation]
where Z I ( x, A ; R ) = (cid:80) ζ ∈Z I ( R ) ζ ( x, A ) , Z T ( x, A ; R ) =
(cid:80) ζ ∈Z T ( R ) ζ ( x, A ) are the sum of fixed path and float path given the region R , J ( x ) is the Jacobian matrix of f at x .
The second statement of above theorem suggests that the difference between f ( x ) and f ( x ′ ) of network N can be decomposed into: (1) a linear part that depends on distance between x and x ′ as well as the slope at x , and (2) a non-linear part that resulted from instability of float neuron.
According to the prime contribution of prediction difference, we can categorize the unsecured data into Lipschitz vulnerable and float neuron vulnerable .
In the following section, we discuss the effect of smoothed classifier on the two types of data and introduce the SC-RFP based on our discussion.
4. F LOAT P ATH AND N ETWORK R OBUSTNESS
Consider ( x, y ) ∼ D has extreme large Lipschitz constant in B ( x, r ) , then there exists x ′ ∈ B ( x, r ) such that f ( x ′ )
− f ( x ) ≫ f y ( x ) − f m ̸
= y ( x ) .
We refer such an x as a Lipschitz vulnerable data.
For example, the Lipschitz constant of a large scale network without any defenses up to hundreds, which means small perturbation ϵ can alter the prediction of f ( x + ϵ ) .
By applying the randomized smoothing on Theorem 1, we show that the smoothed classifier fails to correct such data.
Theorem 2.
Let f be the base classifier.
Given ( x, y ) ∼ D with f ( x ) =
y , denote M ( f ( x ) , y ) :
=
min
y ′ ̸
= y | f ( x ) y − f ( x ) ′
y
| as the margin operator of prediction vector.
If ∥ J ( x ) ∥ > M ( f ( x ) , y ) + E [ M ( Z T ( x, A ; R ) , y )] r , then for any smoothed classifier g defined as above, there exist x ′ ∈ B ( x, r ) such that g ( x ′ ) ̸
= y .
On the other hand, consider x ′ ∈ B ( x, r ) as an adversarial example that misclassified by f but corrected by g , then given ϵ ∼ D noise , g y ( x ′ + ϵ ) > arg max m ∈ Y g m ( x ′ + ϵ ) .
This means that the adversarial example x ′ appears by change , while the majority of its neighbor are still voting for the correct label.
In other word, the instability of float path causes the misclassification, we refer such data as float neuron vulnerable .
[Figure]
The above discussion suggests that smoothed classifier fails to boost the performance of Lipschitz vulnerable example, but is able to correct float neuron vulnerable data by smoothing the sudden change of float path in a region.
Based on this, we introduce the SC-RFP in Algorithm 1.
Figure 1 illustrates the SC-RFP by showing the prediction, fixed path and float path.
We first present the prediction and the fixed path value in Figure 1(a).
The float path value is then computed as
Z T ( x, A ; R ) =
f ( x ) − Z I ( x, A ; R ) in Figure 1(b).
At last, by repressing the float value, we achieve a locally stable prediction from SC-RFP.
At each block, we first compute the pre-activation value of x and x + ϵ .
Prior to passes to the activation function, we compare the activation pattern between z i ( x ) and z i ( x + ϵ ) according to the separation Γ , and use a repression factor η to reduce the value of float path.
Algorithm
Inputs: Network f with parameter θ , randomized algorithm g , input x , repress ratio η .
Outputs: Predicted label ˆ y while g sample noise ϵ do for Block i in Block 1 , 2 , . . .
, d − 1 do
I Ti ← A ( z i ( x )) ̸
=
A ( z i ( x + ϵ ))
z i ( x + ϵ ) ← z
i ( x + ϵ ) − z i ( x + ϵ ) ×
I Ti × η x i +
( x + ϵ )
=
σ ( z i ( x + ϵ )) counts ← ϕ d ( x d ( x + ϵ ))
ˆ c A , ˆ c B ← top two indices in counts ˆ n A , ˆ n B ← counts [ˆ c
A ] , counts [ˆ c B ] if BinomPV alue ( n A , n A + n B , 0 . 5) ≤ α then return ˆ c
A else return Abstain
Intuitively, manipulating the computational graph can greatly change the prediction result, while we show that there are only a small proportion of path are affected by our method.
Figure 2 present the proportion of fixed neuron between x and x + ϵ of VGG16 network trained on CIFAR10.
It shows that models trained with noised sample have relatively more stable activation pattern, as the ratio of fixed neurons are lower.
In particular, the model trained with clean data has average fixed ratio around 72%, while after adding a minimum scale of noise σ = 0 .
05 , it increases to around 90%.
[Equation]
At the end of this section, we present Theorem 3 to link the certifiable boundary of proposed algorithm with previous works.
Theorem 3.
Let N be a network defined as Section 3.1.
Let g be a smoothed classifier that samples noise from distribution D noise and g ′ is the SC-RFP built on g .
Assume that the direction of ϵ is uniformly distributed.
[Equation]
If arg max m ∈ Y f y ( x ) =
y , then p ′
A > p A , p B [Equation] < p B , (14) where p ′ A , p A are the lower bound of g ′ y ( x ) and g y ( x ) , p B ′ , p B are the upper bound of g ′ m
̸ = y ( x ) , g ′ m
̸ = y ( x ) .
Moreover, arg max m ∈ Y g ′ ( x ) =
y for all ∥ ϵ ∥ ≤ R ,
[Equation]
Notice that in the above theorem, we do not specify the randomized classifier g but introduce a constraint of the randomness of g .
This means that the theorem holds for not only the naive smoothed classifier, but also its variants.
However, it does not hold for the classifiers that sample directed noise.
The proof of Theorem 3 can be divided into three steps.
First, we show that given ϵ ∼ D noise , the expectation of fixed paths for the any E ( Z X ( x + ϵ ))
=
Z X ( x ) .
This means that the fixed part of g as well as g ′ equals to that of f ( x ) .
Next, we discuss how the float paths affects the prediction f ( x + ϵ ) and prove that repressing the float path can increase the expectation of g ′ y ( x ) and its variance.
With
Chebyshev’s Inequality, we conclude the lower bound p ′
A is larger than that of p A , and similar to p ′ B .
Since both g and g ′ are random function that sample noise from same distribution, the certifiable radius holds for both g and g ′ .
E XPERIMENTS
[Figure]
As SC-RFP is built on a base smoothed classifier, we measure the method on different model structures and randomized smoothing algorithms.
The experiments are performed on
CIFAR10 and ImageNet datasets.
For the CIFAR10 dataset, we compare our method with the benchmark classifier proposed by Cohen et al.
(2019) with VGG16 network to show the effectiveness of our method.
For the ImageNet dataset, we choose ResNet50 as model architecture and add the adversarial smoothed classifier Salman et al.
as the base model.
We also compare our model with recent work
Jeong & Shin (2020); Jeong et al.
(2021) to obtain a general evaluation.
In line with previous works, we use certifiable accuracy at different radius computed by Cohen et al.
as the metric.
CIFAR10.
We train the VGG16 network with different scales of noisy samples.
Each of the models is trained for 200 epochs with SGD optimizer and an initial learning rate of 0.1, which decays after 60, 120, and 160 epochs with a rate of 0.2.
Table 1 compares the robust accuracy of
SC-RFP and benchmark.
Generally, our SC-RFP increases have an increasingly robust accuracy between 6% to 10% compared with the benchmark.
We notice that by repressing the float path,
SC-RFP is able to improve the accuracy of clean data, which supports our analysis that the float neuron vulnerability is one of the causes of misclassification.
Figure 3 present the robust accuracy at different noise scale.
We find that the performance boost from SC-RFP increases as the radius increases for all the models.
[Table]
Experiments on ImageNet.
We train ResNet50 using the benchmark model (Cohen et al.
and SmoothAdv (Salman et al.
on ImageNet-2012 for 80 epochs with same training sched-
[Figure]
uler and optimizers as the original work.
Each of the model are certified with same level of noise as they were trained.
Table 2 compares the certified robust accuracy of different models at different radius on ImageNet.
It shows that when deployed on the PGD+noise training (Salman et al.
, the SC-RFP has the highest robust accuracy at all the radius.
Moreover, an increasing boost in the robust accuracy is observed as the radius increase, which is consistent with our previous analysis of CIFAR10 result.
Figure 4 provides more details on our experiments.
For a noise level of σ = 0 .
25 , SmoothAdv and
SmoothAdv + SC-RFP has neglectable difference, while in 4(c), the SC-RFP ( σ = 0 .
25 ) increases the robust accuracy by around 5%.
By introducing SC-RFP ( σ = 0 . 1 ) to the benchmark classifier, it can achieve comparable results with other state-of-art models when l 2 radius is around 1.5.
l 2 radius 0.
Clean
Jeong & Shin (2020) 59.
Jeong et al.
Cohen et al.
+ SC-RFP (ours) 59.
Salman et al.
[Table]
+ SC-RFP (ours)
[Table]
We find two insightful observations from the above results.
First, introducing the repression factor to the float path has a minimum negative effect on the standard accuracy, while can greatly boost the robust accuracy.
This supports our discussion that instability vulnerable examples are caused by the float path, and can be cured by reducing their contribution.
Second, we find that as the radius grows, the boost from our method increases.
This is consistent with the theoretical basis of SC-RFP as well as the result from CIFAR10 dataset.
When size of the perturbation grows, the uncertainty of the randomized algorithm increases along with the number of float paths as Figure 2 suggests.
This results in a higher abstain rate and a higher possibility of misclassification.
By repressing the value of the float path, the deviation of g ( x + ϵ ) from f ( x ) is reduced, therefore a more stable result can be achieved.
C ONCLUSION
In this work, we introduce SC-RFP algorithm that can improve the performance of a randomized smoothing classifier.
We first introduce a framework for describing the local activation status of neurons and show that most of the neurons are locally stable, while the others can greatly affect the model prediction.
By decomposing the computational graph of the network, we find that the boost of robust accuracy provided by the smoothed classifier is averaging the deviation of float paths.
Based on this, we suggest further repressing the value of float paths with SC-RFP method.
The experiments show that our method can improve the performance of smoothing models.
A G EOMETRIC I LLUSTRATION
Figure 5 presents an illustration of the key concepts of this work.
Consider N as a neural network with 2D input, 4D output and 1 hidden layer with 4 neurons.
Assume that N has ReLU activation function.
The input space is partitioned into several regions by a set of hyperplanes.
Each of the region is referred as an activation region that marked in a unique color, within which the mapping is linear.
For example, the pink and green region are divided by the bent-hpyerplane H = x
| z 1 , 4 ( x ) = 0 , where z 1 , 4 is the pre-activation value of 4-th neuron in the 1st layer.
Now we consider a sphere
B ( x, r ) centered at x with radius r .
B ( x, r ) is covered by the union of pink and green region.
Since the two regions can be merged into one by removing the hyperplane, then the incomplete activation pattern of the merged region is I\{ (1 , 4) } .
Therefore, (1 , 4) is the only float neuron in B ( x, r ) .
This means that for every x ∈ B ( x, r ) , all the neuron has same activation pattern expect for the neuron (1 , 4) .
Now we consider the computational path of x ∈ B ( x, r ) .
Given a path ζ , if neuron (1 , 4) is not on this path, then ζ is a fixed path, which are the black lines in the figure.
This means that for every x ∈ B ( x, r ) , the value of those fixed path are linear function with respect to x .
On the other hand, all the non-linearity of function f ( x ) for x ∈ B ( x, r ) are contributed by the float path (Orange lines).
With the illustration above, we proof Lemma 1 as follows.
Lemma 1 (Restated) .
Let N be a neural network defined as Section 3.1.
Given R ⊂ R n 0 , denote
I X
and I T are the set of float neuron and fixed neuron in R .
Then:
[Equation] [Equation]
Proof.
Given a neuron ( i, j ) and a region R ∈ R n 0 .
For any x ∈ R , we use ˆ a ij ( x ) .
Since z ij ( x ) ∈ ( −∞ , ∞ ) , the pattern of x is either an index k of the region when z ij ( x ) ∈
U k
or -1 when z ij ( x ) locate on the bent-hyperplane H ijk := { x | z ij ( x ) = k } .
This means every neuron has a pattern for x ∈ R .
If for every x ∈ R n 0 , the pattern of neuron ( i, j ) remains the same, then ( i, j ) is a fixed neuron.
Otherwise, it is a float neuron.
In other words, a neuron is either fixed or float in R .
This implies that I X ( R ) (cid:83) I T
( R )
=
I .
Now we consider statement 2.
For any x 1 , x 2 ∈ R , and an index of neuron ( i, j ) .
If ˆ a ij ( x 1 ) =
ˆ a ij ( x 2 ) , ( i, j ) is a fixed neuron in R: ( i, j ) ∈
I X .
Denote the pattern a i j .
Then for all x 1 ∈ R , z ij ; x 1 ,θ = z ij ; x 2 ,θ .
Therefore, x 1 ∈ R ( A X ) .
B D ECOMPOSING THE N ETWORK
Lemma 2.
(Restate)
Let N be a neural network defined as Section 3.1.
Given activation pattern A , for any x ∈ R ( A ; θ, σ, Γ) .
The k -th component of output vector of layer i can be represented as:
[Equation]
where x j,ζ j is the ζ j -th element of input in layer j , W ′ ( i ) and β ( i ) are the equivalent matrix and bias of linear transformation ψ i ◦
ϕ
i , A m,ζ m is the activation pattern of ζ m -th component of layer m and d A m,ζm is the slope of pattern A m,ζ m : d A m,ζm :=
σ ′ ( t ) , t ∈ U
A m,ζm .
Proof (Lemma 2) .
We first show that the pre-activation transformation can be represented by a matrix.
As ϕ i is a linear affine, we denote ϕ
i ( x ) =
W i x i .
Combing with batch normalization layer, the mapping from input x i to pre-activation is:
[Equation]
(17) where D i ( · ) is the diagonal operator, ˆ µ and ˆ σ are the mean and variance parameter of the ψ i .
Denote W [Equation] ( i ) =
D i (
[Equation]
ˆ σ ) W i , β ′ i =
( β i − γ ˆ µ ˆ σ ) /n
i .
We proof this byh deduction.
For i = 1 :
[Equation]
where A 1 ,i is the activation pattern of z 1 i .
Since all the path in layer i ends at i are { (1 , i ) , (2 , i ) , . . .
, ( n 0 , i ) } .
Equation 11 holds.
Now we assume equation holds for i = p .
[Equation]
Lemma 2 decomposes the computational graph of network N , while Theorem 1 categorize the paths of f ( x ) into fixed part and float part.
Moreover, it describes the model robustness with by divide the variant of f into a linear part as well as a non-linear part.
Theorem 1 (Restate) .
Let N be a neural network defined as 3.
Given R ⊂ X , for any x, x ′ ∈ R with activation pattern A and A ′ , we have :
[Equation]
where Z I ( x, A ; R ) = (cid:80) ζ ∈Z I ( R ) ζ ( x, A ) , Z T ( x, A ; R ) =
(cid:80) ζ ∈Z T ( R ) ζ ( x, A ) are the sum of fixed path and float path given the region R , J ( x ) is the Jacobian matrix of f at x .
Before we proof above Theorem, we use the following Lemma to show that every fixed path in a region is linear regardless of the activation pattern.
Lemma 3.
Let N be a network defined as Section 3.3.
Let ζ be a fixed path in R ⊂ R n 0 .
Then for any x, x ′ ∈ R with activation pattern A , A ′ , ζ ( x ′ , A ) = ζ ( x ′ , A ′ ) .
Moreover, Z I ( x ′ , A ; R ) =
Z I ( x ′ , A ′ ; R ) .
Proof (Lemma 3) .
Given x ∈ R with activation pattern A ,
[Equation]
Since ζ is a fixed path in R , then every neuron alone ζ is fixed neuron, the activation pattern of ( m, ζ m ) is same regardless of input x .
Therefore, d A m,ζm is constant for any x ∈ R .
Then ζ ( x, A ) is a linear function of x : ζ ( x, A )
=
ζ ( x ) .
In other words, the change of activation pattern does not affect the neurons on ζ , therefore the slope of this path does not change.
ζ ( x ) is dependent on x in region R .
This means that ζ ( x ′ , A ) =
ζ ( x ′ , A ′ ) for any x, x ′ ∈ R with activation pattern A , A ′
Z I ( x ′ , A ; R ) is the aggregation of all the fixed path above.
Since the summation of linear function is still linear, we have:
[Equation]
Proof (Theorem 1) .
From Lemma 1, every neuron is either fixed neuron or float neuron.
For the neurons in path ζ , if there exists a float neuron, then the path is float path.
Otherwise, it is a fixed path.
The float path and fixed path are complementary set on the set of all paths.
Lemma 2 decomposes the f ( x ) into summation of paths.
As each path is either float or fixed, we have
[Equation]
For statement 2, we have:
[Equation]
Notice that, Z I ( x ′ , A ; R ) is the collection of all fixed path, therefore is unrelated with the change of A .
We have Z I ( x ′ , A ; R )
=
Z I ( x ′ , A ′ ; R ) .
The former part of above equation is equals to
J ( x )( x − x ′ ) , which reaches to statement 4.
C M ODEL R OBUSTNESS C. L IPSCHITZ VULNERABLE AND FLOAT NEURON VULNERABLE
In Section 4.3 we discuss two kinds of vulnerable data and discuss our motivation of SC-RFP.
In this section, we present formally statements of the proposed idea.
Before we move to Theorem 2, we need the following two lemmas.
Lemma 5 suggests that, given a fixed path ζ , the expectation for the randomized algorithm g ( ζ ( x )) is equal to the ζ ( x ) when g satisfies certain conditions.
In fact, the condition here just assume that the randomly sampled noise has equal probability at each direction, which is a natural assumption for most sampling procedure.
Before we start, we introduce the definition of a subpath of a path as below:
Definition 6.
A sub-path of a path consists of several consecutive elements of γ :
[Equation]
We use ζ ( i,j ) ( x ) to represent the computation graph from ( j, ζ j ) to ( i, ζ i ) :
[Equation]
Lemma 4.
Denote N as a neural network as Definition 3 with mapping function f , ζ is a fixed path in B 2 ( x, p ) .
Assume g is a randomized algorithm that applies noise ϵ from certain distribution
D noise on the computation path of ζ :
[Equation]
If the direction of ϵ is uniformly distributed, then the expectation of g ( ζ ( x )) is equals to ζ ( x ) , that is:
[Equation]
where ϵ is the noise generated for randomized algorithm g .
Proof.
Assume that g samples noise at layer k .
Then for any g , ζ ( k, 0) =
g ( ζ k, 0 ) , which we denote as x k .
Since the activation function is piecewise linear, the mapping of ζ d,k ( · ) is linear.
We have:
[Equation] [Equation]
ζ d,k is harmonic.
Therefore, given any B ( x, r ) with radius r > 0 :
[Equation]
[Equation]
where ω n is the volume of the unit ball in n dimensions, σ is the ( n − 1) dimensional surface measure.
Now we consider the expectation of g ( x ) :
[Equation]
where µ is the probability measure of ϵ D noise .
As the noise sampling is assumed to be direction irrelevant, the measure of p ( ϵ |∥ ϵ ∥ 2 = r ) is uniformly distributed given radius r .
Therefore:
[Equation] [Equation]
Equation 26 then equals:
[Equation]
Lemma 5.
Let f
and g be the mapping function and randomized classifier defined as above.
Given radius r such that, almost surely, Z T ( B ( x, r ))
= ∅ , ∀ ( x, y ) ∼ D , then the accuracy of base classifier and the naive smoothed classifier are same, that is:
[Equation]
Lemma 5 is the achieved by directly applying Lemma 4 on the all the computational graph of the network.
It shows that if all the neurons have locally stable activation pattern with respect to the distribution of dataset, the smoothed classifier provides identical accuracy with the base classifier.
Next, we present the proof of Theorem 2.
Theorem 2.
Let f be the base classifier.
Given ( x, y ) ∼ D with f ( x ) =
y , denote M ( f ( x ) , y ) :
=
min
y ′ ̸
= y | f ( x ) y − f ( x ) ′
y
| as the margin operator of prediction vector.
If
[Equation]
then for any smoothed classifier g defined as above, there exist x ′ ∈ B ( x, r ) such that g ( x ′ ) ̸
= y .
Proof.
Statement 4 of 1 suggests:
[Equation]
As ∥ J ( x ) ∥ > M ( f ( x ) ,y )+ M ( E [ Z T ( x, A ; R )] ,y ) r , there exist x ′ and i such that:
[Equation]
Therefore,
[Equation]
Next, we consider the case that x is an adversarial example that mislead f but correctly classified by g , which we referred to as a float neuron vulnerable example.
The following theorem suggests that, the float path at x are the cause of the altered prediction.
In other words, the network is locally correct around x , while there are sudden change from the float path that causes the misclassification.
Theorem 4.
Let f and g be the base and smoothed classifier defined above.
Given ( x, y ) ∼ D , denote x ′ ∈ B ( x, r ) as an adversarial example that mislead the base classifier but corrected by g without abstain:
[Equation]
then loss of float path is higher than that of g :
[Equation]
where CE ( · , · ) is the cross entropy loss, onehot ( y ) is the one hot embedding of label y .
Proof (Theorem 4) .
Since arg max m ∈ Y f m ( x ) ̸ = y , arg max m ∈ Y
g m ( x ) =
y , there exist i ∈ { 1 , . .
, c } :
[Equation]
Then we have:
[Equation]
Introducing Statement 4 of Lemma 1 and Lemma 5:
[Equation]
Moreover, since the prediction is not abstained.
[Equation]
This directly leads us to the result.
Theorem 2 and Theorem 4 show that a smoothed classifier is not able to boost the performance of fixed path, while it is applied to reduce the sudden change of float path in a region.
To be specific, if x ′ can be corrected by g , it is resulted from the sudden change provided by the float path.
A higher confidence score of g ( x ′ ) can be achieved by reducing the weight of float path during the computation path.
On the other hand, if x ′ is Lipschitz vulnerable, then regardless of the form of smoothed classifier, it cannot be fixed.
This lead us to the theoretical basis of SC-RFP algorithm.
C. V ERIFIABLE R ADIUS
At the end of section 4.3, we propose Theorem 3 to describe the upper and lower bound of SC-RFP as well as the certified radius.
We present the proof of Theorem 3 below.
Theorem 3.
Let N be a network defined as Section 3.1.
Let g be a smoothed classifier that samples noise from distribution D noise and g ′ is the SC-RFP built on g .
Assume that the direction of ϵ is uniformly distributed.
[Equation]
where p ′ A , p A are the lower bound of g ′ y ( x ) and g y ( x ) , p ′ B , p B are the upper bound of g ′ m
̸ = y ( x ) , g ′ m
̸ = y ( x ) .
Moreover, arg max m ∈ Y g ′ ( x ) =
y for all ∥ ϵ ∥ ≤ R ,
[Equation]
Proof.
The proof of Theorem 3 can be divided into three steps.
First, we show that given ϵ ∼ D noise , the expectation of fixed paths for the any E ( Z X ( x + ϵ ))
=
Z X ( x ) .
Consider ζ is a fixed path on a region R , then from Lemma
[Equation]
Notice that, since the SC-RFP does not affect the sample of noise, then above equation holds for both g and g ′ :
[Equation]
By applying randomized smoothing on statement 1 of Lemma 1, we have:
[Equation]
where we use g ′ ( Z T ( x, A ; R )) to denote applying smoothing algorithm g on deterministic function
Z I ( x, A ; R ) .
Since Z I ( x, A ; R ) is the aggregation of all the fixed path, with Equation 41 and 42, we have :
[Equation]
This means that the difference between E ( g ( x )) and E ( g ( x )) is same with that of g ( Z T ( x, A ; R )) and g ′ ( Z T ( x, A ; R )) .
Next, we discuss how the float path affect the prediction of g ( Z T ( x, A ; R )) and g ′ ( Z T ( x, A ; R )) .
Given ϵ ∼ D noise , we denote P 1 as the event
that f y ( x + ϵ ) >
f y ( x ) .
Since we have excluded the fixed path, P 1 holds means that the float path boosts the probability of f y ( x + ϵ ) .
However, as arg max m ∈ Y f y ( x ) =
y , repressing the float path does not alter the prediction of f y ( x ′ ) .
On the other hand, if f y ( x + ϵ ) < f y ( x ) , which we denote as P 2 .
This means that the float path
Z T ( x, A ; R ) < 0 and negatively contributes to the prediction.
Then repressing the float path means that can increase the probability of y .
In other words, for any x + ϵ , repressing the float path between x and x ′ can increase the predicted f y ( x + ϵ ) .
We assume that P 1 and P 2 happens with probability p 1 and p 2 , then given a certain number of sampling,
[Equation]
By applying Chebyshev Inequality, we have p ′
A > p A .
Similarly, we also have p ′ B < p B
Therefore, we conclude the lower bound p ′
A is larger than that of p A , and similar to p ′ B .
Computing the certified radius of x then is same with that of Cohen et al.
Since both g and g ′ are random function that sample noise from same distribution, the Neyman-Pearson theorem holds for both g and g ′ .
Therefore, the certified raidus remains unchanged.
D E XTRA T ABLES AND F IGURES
This section present the complete experiment results.
Table 3 shows the certified accuracy at different l 2 radius level.
Table 4 presents the experiment result of our model certified with different repression rate.
Among the different repression rate, we find that when η = 0 .
25 SC-RFP provides the best performance on increasing the model robust accuracy for perturbation with large size, while the standard accuracy and are slightly damaged.
This is also observed on previous works.
[Table] [Table]
Cohen
[Table] [Table]
+ SC
[Table]
Salman et al.
+ SC-RFP (ours η
[Table]
+ SC
[Table]
RFP (ours η
[Table]
+ SC
[Table]
RFP
[Table]
ours η
[Table] [Table]
Jeong
[Table]
Jeong
[Table]
al
[Table] [Table]
+ SC
[Table]
Salman et al.
[Table]
+ SC
[Table]
RFP (ours
[Table] [Table]
+ SC-RFP (ours
[Table]
+ SC
[Table] [Table]
Jeong &
[Table] [Table]
Jeong
[Table]
al
[Table] [Table]
Cohen
[Table]
al
[Table]
+ SC
[Table]
RFP (ours) 39.
Salman et al.
+ SC-RFP (ours η = 0 . 05 ) 40.
+ SC-RFP (ours η = 0 . 10 ) 39.
+ SC-RFP (ours η = 0 . 25 ) 39.
Figure 6 present the model trained with σ = 0 .
05 and σ = 0 .
25 , but under different level of noise.
We find the model trained with slightly noised sample can provide effective robustness.
[Figure]