C ERTIFYING N EURAL N ETWORK A UDIO C LASSIFIERS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present the ﬁrst end-to-end veriﬁer of audio classiﬁers.
Compared to existing methods, our approach enables analysis of both, the entire audio processing stage as well as recurrent neural network architectures (e.g., LSTM).
The audio processing is veriﬁed using novel convex relaxations tailored to feature extraction operations used in audio (e.g., Fast Fourier Transform) while recurrent architectures are certiﬁed via a novel binary relaxation for the recurrent unit update.
We show the veriﬁer scales to large networks while computing signiﬁcantly tighter bounds than existing methods for common audio classiﬁcation benchmarks: on the challenging
Google Speech Commands dataset we certify 95% more inputs than the interval approximation (only prior scalable method), for a perturbation of -90dB.
I NTRODUCTION
Recent advances in deep learning have enabled replacement of traditional voice recognition systems with a single neural network trained from data (Graves et al., 2013; Hannun et al., 2014; Amodei et al., 2016).
Wide adoption of these networks in consumer devices poses a threat to their safety when exposed to a malicious adversary.
Indeed, it was recently shown that an adversary can inject noise unrecognizable to a human and force the network to misclassify (Szegedy et al., 2013; Goodfellow et al., 2014; Zhang et al., 2017; Carlini & Wagner, 2018;
Carlini et al., 2016; Qin et al., 2019;
Neekhara et al., 2019; Yang et al., 2019; Esmaeilpour et al., 2019), exposing a serious security ﬂaw.
Ideally, when deploying an automated speech recognition system we would like to guarantee that the system is robust against noise injected by an adversary.
There has been substantial recent work on certifying robustness of computer vision models (Katz et al., 2017; Ehlers, 2017; Bunel et al., 2018;
Ruan et al., 2018; Tjeng et al., 2019; Anderson et al., 2018;
Wong et al., 2018; Dvijotham et al., 2018;
Raghunathan et al., 2018; Dvijotham et al., 2019; Weng et al., 2018; Zhang et al., 2018;
Salman et al., 2019; Gehr et al., 2018; Singh et al., 2018; 2019a; Wang et al., 2018; Singh et al., 2019b).
However, the audio domain poses unique challenges not addressed by prior certiﬁcation work for vision.
Differences between audio and vision models Concretely, while an input to a vision model is a raw image, audio models typically come with a complex preprocessing stage (that involves non-trivial non-linear operations such as logarithm) which extracts relevant features from the signal.
Additionally, audio systems typically use recurrent architectures (Chiu et al., 2017) which computer vision veriﬁers do not handle as they focus on fully-connected, convolutional and residual architectures.
s ( i )
DCT FB
FFT
ReLU
LSTM x
( i ) h
( i − 1)
h ( i ) s ( T )
ReLUDCT
FB FFT
LSTM x ( T ) h ( T − 1)
h
( T )
F C a
This work We address both of these challenges and propose an end-to-end veriﬁcation method for neural network based audio classiﬁers and an implementation of this method in a system called DAC (Deep Audio Certiﬁer).
Our threat model assumes an attacker can introduce a noise-based perturbation to the raw audio input signal.
The goal then is to certify that, for any signal that the attacker can produce, the neural network classiﬁes the signal to the correct label.
We perform veriﬁcation of this property using the framework of abstract interpretation (Gehr et al., 2018).
At a high level, the idea is to maintain an abstraction capturing all possible behaviors of both the audio processing stage and the neural network.
The ﬂow of DAC is shown in Fig.
1 where all abstractions are dark blue shapes.
Here, all possible signals an attacker can obtain are captured using an abstraction s ( i ) (a convex relaxation).
This abstraction is then propagated through the audio processing stage (shown in green boxes).
The key components of this step are abstract transformers .
For each audio processing operation (e.g.
FFT) we create an abstract transformer which receives an abstraction representing an approximation of all possible inputs to the operation and outputs a new abstraction which approximates all possible outputs of the operation.
The result of the audio processing stage is the abstraction x ( i ) .
The shape x ( i ) is then used as input to the recurrent LSTM unit (light blue) which maintains an abstraction of a hidden state h ( i − 1) .
LSTM consists of multiple operations and we create a custom abstract transformer for each of those.
The result of the transformers in LSTM is a new hidden state h ( i ) .
If this was the last frame in the signal (meaning i = T ), then hidden state h ( T ) is passed through the fully connected layer of the neural network and, again using the abstract transformer, the ﬁnal abstract shape a is obtained at the output (at the right of Fig. 1).
Finally, to certify the property we check if each concrete output in the abstraction a classiﬁes to the correct label (this is typically easy).
If this is true, the output of the network is correct for all inputs that the attacker can create.
Related work on RNN certiﬁcation The work of (Ko et al., 2019) proposes the POPQORN veriﬁer for recurrent neural networks (RNN).
We note that POPQORN does not handle the audio preprocessing pipeline.
Even though POPQORN cannot directly verify audio classiﬁers, their approximations for LSTM non-linearities can be integrated in DAC.
This results in ≈ 200 × slowdown with small decrease in the volume of the approximation.
The massive slowdown makes their approximations unsuitable for certifying audio classiﬁers.
In contrast, using our custom abstract transformers for LSTM non-linearities, DAC can precisely certify end-to-end robustness of challenging audio classiﬁers
in few minutes
Our main contributions are:
A novel and efﬁcient method to certify robustness of neural network audio classiﬁers to noise-based perturbations.
The method is based on new abstract transformers which handle non-linear operations used in both audio processing and recurrent architectures.
An implementation of both veriﬁcation and provably robust training in a system called DAC.
We evaluated DAC on common audio classiﬁcation benchmarks, showing it scales to realistic networks and is far more precise (97% to 2%) than the next best scalable method.
B ACKGROUND
We ﬁrst deﬁne a threat model that we work with and then present all operations that are part of the veriﬁcation procedure, including audio processing (MFCC) and LSTM updates.
We also discuss the type of veriﬁcation method we employ.
Threat model We follow the same attacker threat model as Carlini & Wagner (2018).
The assumption is that the attacker can add noise δ to the original signal s so to obtain a perturbed signal s (cid:48)
= s + δ .
The measure of signal distortion are decibels (dB) deﬁned as:
[Equation]
Note that the quieter the noise is, the smaller the decibel of perturbation dB s ( δ ) (it is usually a negative value as it is quieter than the signal).
We assume the attacker can generate noise δ such that dB s ( δ ) < (cid:15) where (cid:15) is a constant deﬁned by the threat model.
Our goal is to verify whether the neural network classiﬁes s (cid:48) correctly for each small perturbation δ (as constrained above).
Mel-Frequency Cepstral Coefﬁcients (MFCC) Though there have been number of works which operate directly on the raw signal (Pascual et al., 2017; Sainath et al., 2015), Mel-Frequency Cepstrum (MFC) is traditionally preferred for audio preprocessing in speech recognition systems e.g., DeepSpeech (Hannun et al., 2014).
The idea of MFC is to model non-linear human acoustic perception as power spectrum ﬁlters based on certain frequencies, called Mel-frequencies .
The ﬁnal result of the transformation is a vector of coefﬁcients whose elements contain log-scaled values of ﬁltered spectra, one for every Mel-frequency.
This resulting vector is a feature representation of the original signal and can now be used in a downstream task such as audio classiﬁcation.
Sahidullah & Saha (2012) presented an approach to represent MFCC computation using several matrix operations which we integrate with our veriﬁcation framework.
Given T frames of audio signals of the length N = 2 k , represented as matrix S =
[ s (1) · · · s ( T ) ] tr ∈ R T × N ( tr for transpose), audio preprocessing with MFCC is calculated using the following steps:
Pre-emphasizing and Windowing: Y = S ( I N − c
pe I +1 N ) (cid:12) H
Transform the signal with the pre-emphasizing and applying the Hamming window.
Here,
I +
N ∈ R N × N is the shifted diagonal identity matrix ( I +1 i,j = 1 if i + 1 =
j , otherwise 0 ),
H ∈ R T × N is the Hamming window, and c pe is the pre-emphasizing constant.
Power Spectrum of Fast Fourier Transform (FFT): Θ = ( Y W ) (cid:12) ( Y W )
Perform FFT on the windowed data and square it to get the real-value spectrum.
We can denote FFT on discrete domain (DFT) with the multiplication of Y and W ∈ C N × N/ 2 .
Filter Bank Log Energy: Ψ = log( ΘΛ )
Apply the Mel frequency ﬁlter bank to the power spectrum and get the log value of them.
Λ ∈ R N/ 2 × p is the ﬁlter bank given the number of ﬁlters p , and log is applied entry-wise.
DCT(Discrete Cosine Transformation): X =
Ψ D
Perform DCT on the previous result.
Again, this can be formulated using matrix multiplication.
We use the resulting X =
[ x (1) · · · x ( T ) ]
tr as the input for the neural network.
Long-Short Term Memory (LSTM)
LSTM architectures (Hochreiter & Schmidhuber, 1997) are a key part in modern state-of-the-art speech recognition systems (Hannun et al., 2014;
Amodei et al., 2016).
In our work, we consider the following deﬁnitions of updates in the LSTM unit.
[Equation]
, h ( t − 1) ] W i + b i c ( t ) =
σ ( f ( t ) 0 ) (cid:12) c ( t − 1) + σ ( i ( t ) 0 ) (cid:12) tanh(˜ c ( t ) 0 )
o
( t )
=
[ x ( t ) , h ( t − 1) ]
W o + b o ˜ c ( t )
=
[ x ( t ) , h ( t − 1) ]
W ˜ c
+ b ˜ c h ( t ) =
σ ( o ( t ) 0 ) (cid:12) tanh( c ( t ) ) where [ · , · ] is the horizontal concatenation two row vectors, W · and b · are kernel and bias of the cell, respectively.
At timestep t , vectors f ( t ) 0 , i ( t ) 0 , o ( t ) 0 ,
˜ c
( t ) 0 represent pre-activations of the forget, input, and output gate, respectively, and the pre-calculation of cell state.
Cell state c ( t ) and hidden state h ( t ) computed at timestep t are propagated to the LSTM unit at the next timestep, thus allowing it to maintain states.
This recurrent architecture allows inputs with arbitrary length, making it especially suited for audio inputs.
Robustness certiﬁcation In this work, our goal will be to certify the robustness of an audio classiﬁcation pipeline (including the LSTM) to noise perturbations of the input.
To build such a veriﬁer, we leverage the general method of abstract interpretation suggested by Cousot & Cousot (1977), successfully employed by some of the most recent state-of-the-art veriﬁers of neural networks (Gehr et al., 2018; Singh et al., 2019a).
The basic idea here will be to propagate the possible perturbations (captured by a convex region) through the operations of the entire audio pipeline and to then use the ﬁnal output region to certify the robustness property.
The most challenging step of this approach is deﬁning efﬁcient yet precise approximations of the non-linear operations (called abstract transformers) used in the audio pipeline.
In this work, we will introduce a number of such new abstract transformers that handle the non-linear operations used in audio processing.
Speciﬁcally, our over-approximations will be expressed in the recent D EEP P OLY abstraction (Singh et al., 2019a) (a restricted form of convex polyhedra) which aims to balance efﬁciency and precision.
That is, the abstract transformer of a non-linear function will take as input a convex polyhedra element expressible in D EEP P OLY and output another polyhedra which over-approximates the behavior of the non-linear operation when invoked with any concrete point inside the original polyhedra.
O VERVIEW OF V ERIFICATION P ROCESS
We now explain the workings of our veriﬁer on a (toy) example from the point where sound enters the processing stage to where the output of the classiﬁer is certiﬁed as robust under any allowed perturbation of the threat model.
Our goal is to provide an intuitive understanding, formal details are provided later.
Audio preprocessing Unlike in standard computer vision tasks where the input is fed directly to the neural network, audio models typically ﬁrst perform MFCC preprocessing to extract useful features
[Figure]
from the signal.
We represent all preprocessing operations as green boxes in Fig.
The calculation follows steps described formally in Section 2, using the same notation for resulting vectors.
Fast Fourier transform The ﬁrst operation is Fast Fourier Transform (FFT) of the pre-emphasized input signal.
It is a two-step process shown in dashed boxes in Fig.
We decompose it into afﬁne and square operations which transform signal s ( t ) into an intermediate representation θ ( t ) .
Using our novel and optimal abstract transformer for the square operation, formally deﬁned in Section 4 and visualized in Fig.
3b, we obtain linear bounds on θ ( t ) .
Filterbank transform The next operation is Filterbank transform (FB) which consists of afﬁne and logarithm operations, shown in solid boxes in Fig.
Note that if the approximations from the square transformer allows negative values, then the entire analysis will fail as the logarithm operation is undeﬁned for negative inputs.
Our transformers are carefully designed to avoid this scenario.
To obtain bounds on the output of the logarithm we apply our novel and optimal logarithm transformer, also formally described in Section 4 and visualized in Fig. 3a, and obtain linear upper and lower bounds on the log energy of the ﬁltered power spectrum, ψ ( t ) .
Logarithm operation is followed by
Discrete Cosine Transform (DCT) resulting in a vector ˜ x
( t ) which is then used as an input to the fully connected layer of the neural network followed by a ReLU.
Our analysis (detailed calculation in
Appendix C) produces ˜ x ( t ) 1 ∈ [0 . 87 , 5 . 56] , ˜ x ( t ) 2 ∈ [0 . 17 , 12 . 8] .
Since all the values are positive, the following ReLU has no effect on its input and thus we set x ( t ) =
˜ x ( t ) .
Using the back-substitution technique we describe later, we derive x ( t ) 1 ∈ [0 . 87 , 5 . 56] , x ( t ) 2 ∈ [0 .
In the next paragraph we describe bound propagation through LSTM in more detail.
LSTM bound propagation
Here we provide a technical overview of our LSTM transformer, formalized in Section 5 and visualized in Fig.
4, by calculating the result of the transformation on the ﬁrst neuron of the LSTM hidden layer.
We provide detailed mathematical basis of this process in
Appendix D. For our toy example, let the gates be updated as f ( t ) 1 = x ( t ) 1 / 4 , input gate i ( t ) 1 = 2 h ( t − 1), output gate o ( t ) 1 = h ( t − 1)and cell state ˜ c ( t )
=
h ( t − 1)− x ( t ) 2 / 4 .
Also, assume the previous states are bounded by: h ( t − 1),
h
[Equation]
[Equation] [Equation]
−
[Equation] [Equation]
∈ [0 . 90 , 1 . 00] .
We now apply our σ ( x ) · tanh( y ) and σ ( x ) · y transformers to get bounds of the cell state c ( t ) 1 = σ ( f ( t ) 1 ) ·
c ( t − 1)+
[Equation]
( i ( t ) 1 ) · tanh(˜ c ( t ) 1 ) .
[Equation]
[Equation] [Equation]
03˜
[Equation]
( t )
[Equation]
− 0 .
78 ≤
[Equation] [Equation]
( t )
[Equation]
tanh(
[Equation]
( t ) 1 ) ≤ 0 .
39˜ c ( t )
[Equation]
+ 0 .
Summing up the inequalities above results in c ( t )∈ [ − 0 .
For the hidden state h ( t )=
σ ( o ( t ) 1 ) · tanh( c ( t ) 1 ) , we again apply our abstract transformer and obtain
[Equation]
y l = log l [Equation] + [Equation]
− l [Equation] u x − l x log u x l x y u = 2 x u x + l x − 1 + log u x + l xl x u x (a) Log abstract transformer.
y u
= ( u x + l x ) x − u x l x y l
=
4 l x x − 4 l 2 x l x u x (
[Figure]
transformer.
i.e., h ( t ) 1 ∈ [ − 0 .
An analogous computation is performed for h ( t ) 2 .
For this example, we assume the computation results in h ( t ) 2 = 0 .
Robustness certiﬁcation using DeepPoly The hidden states in the LSTM at the ﬁnal timestep are passed through a fully connected layer without activation.
In our example we assume that logits after the layer are obtained with the following formula: l 1 = h ( t ) 1 + 0 .
24 and l 2 = h ( t ) 2 .
These are shown in the ﬁnal box of Fig.
To certify that the neural network classiﬁes the input to class 1, we need to prove that l 1 − l 2 > 0 .
We now apply the back-substitution technique as in Singh et al.
(2019a)
upto the input to the LSTM:
[Equation] [Equation]
) + 0 .
01( h ( t − 1)− x ( t ) 2 / 4) + 0 .
01 ≥ 0 .
As this value is greater than 0, robustness is established.
The process above, of replacing a variable with its constraints, is called back-substitution .
Here, we replaced l 1 with h ( t ) 1 + 0 .
24 , then h ( t ) 1 with 0 .
18 f ( t ) 1 + 0 .
03˜ c ( t ) 1 − 0 . 33 and so on. In Appendix C we show more detailed calculations to obtain tighter bound 0.0375 by also back-substituting in the preprocessing pipeline.
For our experiments in Section 6, we tune the number of back-substitution steps to achieve a good tradeoff between speed and precision.
Note that robustness cannot be proved if one concretizes the expression above to an interval, instead of performing back-substitution of linear bounds.
For instance, if we concretize h ( t )to [ − 0 . 25 , 0 .
65] , we obtain l 1 − l 2 ≥ h ( t ) 1 − h ( t ) 2 + 0 .
24 ≥ − 0 . 01 which is imprecise and fails to certify robustness.
Note that if t is not the last timestep, the hidden state and the cell state are passed to the next analysis timestep instead of computing the ﬁnal output.
A UDIO P ROCESSING T RANSFORMERS
As illustrated earlier, the ﬁrst part of the veriﬁcation process involves handling of the audio processing stage (performed using MFCC).
Here, most matrix multiplication parts can be captured exactly by our abstraction, but MFCC also includes the square operation in the FFT and the logarithm operation in the computation of energy from ﬁlterbanks.
Thus, to handle these non-linear operations, we need to create new abstract transformers, which we present next.
To ensure a minimal loss of precision, our abstract transformers minimize the area between the lower and upper bounds in the input-output plane.
This approach of minimizing the area has been used before for other transformers and has been shown to be practically effective in Singh et al.
(2018; 2019a).
We denote the set of all variables in the analysis as X .
For an element x ∈ X , we denote the functions corresponding to its linear lower and upper bound as x l and x u , respectively.
These are scalar functions deﬁned on R k where k is the number of other variables used to compute x .
For every element, we also maintain interval bounds, x ∈ [ l x , u x ] .
For ease of explanation, we introduce the log transformer followed by the square transformer.
Log abstract transformer
The logarithm operation is an assignment y :
= log( x ) where x, y ∈ X .
The output of this operation cannot be captured exactly and we need to carefully introduce an approximation.
Here, we ﬁrst compute the minimum l y = log( l x ) and the maximum u y = log( u x ) , which we use as interval approximation.
We deﬁne linear lower and upper bound functions y l , y u : R → R .
Using concavity of the logarithm on any subset of the domain, the lower bound function is a line connecting the points ( l x , log( l x )) and ( u x , log( u x )) .
The upper bound function is chosen as the tangent line to the function minimizing the area between the lower and the upper bound.
As a result, we obtain the following bounds as depicted in Fig.
3a:
[Equation]
Note that if l x ≤ 0 , y l ( x ) =
−∞ since the function is not deﬁned on that domain.
We make an exception when u x − l x < 10 − 4 to use interval bound for y l for avoiding the unstable ﬂoating point calculation caused by large denominator of x coefﬁcient.
Square abstract transformer
The square operation is an assignment y :
= x 2 where x, y ∈ X .
Similar to logarithm, this operation is non-linear and cannot be captured exactly in the abstraction.
We ﬁrst compute the interval bounds of the output l y and u y , and set the minimum value
l y to 0 when 0 ∈ [ l x , u x ] and min( l 2 x , u 2 x ) otherwise.
The maximum value u y is simply max( l 2 x , u 2 x ) .
Next, we deﬁne linear lower and upper bound functions y l , y u : R → R .
Using the convexity of the square function, we set the upper bound function y u to the linear function connecting ( l x , l 2 x ) and ( u x , u 2 x ) .
For the lower bound function y l , we have to be delicate: since x 2 : R → R ≥ 0 , our y l should also be greater or equal than 0 within any domain.
With the principle of minimizing the area between the bounds, we obtain the following bounds for cases as shown in Fig.
3b:
[Equation]
LSTM T RANSFORMERS
Most prior work on veriﬁcation of neural networks focuses on feed-forward or convolutional architectures whose operations consist of a sequence of afﬁne transforms and activation functions.
However, to perform veriﬁcation of recurrent architectures, one needs to handle the updates of the recurrent unit.
Following the equations updating the LSTM presented in Section 2, we can observe that pre-activations of gates are afﬁne transforms which can be captured exactly in our abstraction.
However, the operations updating the cell and hidden states are non-linear and require approximation.
Overall, we have three elementwise products – two are between a sigmoid and a tanh and one between a sigmoid and the identity.
A straightforward approach to handling such transforms would be to concretize the polyhedra DeepPoly element to an interval and perform the multiplication using intervals.
However, this approach would lose all relations between inputs and incur precision loss in future operations of the analysis.
Instead, we design custom binary approximation transformers speciﬁcally tailored to handle the elementwise product in the update of the recurrent unit.
Sigmoid (cid:12) Tanh abstract transformer We deﬁne elementwise multiplication between sigmoid and tanh as an assignment z :
= σ ( x ) tanh( y ) , where x, y, z ∈ X .
As before, our aim is to construct linear bounds z l , z u .
Unlike the previously deﬁned DeepPoly transformers which take as input a single abstract element, this transformer is the ﬁrst which receives two abstract elements as operands.
Hence, the bound functions have the form z l , z u : R 2 → R .
Let z
=
f
( x, y ) =
σ ( x ) · tanh( y ) .
Our goal is to bound this function between two planes such that the volume between the planes is as small as possible (following our previous heuristic).
We ﬁrst divide the computation into 3 cases based on the signs of l y and u y .
To simplify notation, we introduce the variable assignment in Table 1 so that we can reuse notation across all cases.
We ﬁrst deﬁne an anchor point a : a point in the box [ l x , u x ] × [ l y , u y ] where function f attains the max/min value deﬁned upon the case.
Also we need reference point r whereas the plane would meet z ux z
uy z lx z ly l y ≥Φ
min min min min a u x ,u y u x ,u y l x ,l y l x ,l y r l x ,u y u x ,l y u x ,l y l x ,u y l y < 0 ≤ u y Φ min min max min a u x ,u y u x ,u y u x ,l y u x ,l y r l x ,u y l x ,l y l x ,l y l x ,u y u y <
Φ max min max min a l x ,u y l x ,u y u x ,l y u x ,l y r u x ,u y l x ,l y l x ,l y u y ,u y
[Figure]
[Figure]
{+5 2 − 0 . 
+}
.y
x
z ﬁrst among the four corners if it is tilted in given direction, other than the anchor.
Our transformer computes two candidates planes for the upper bound ( z ux , z uy ) and two for the lower bound ( z lx , z ly ) as:
[Equation]
where D i f = ∂f∂x , D j f = ∂f∂y , and Φ , a , r are chosen from Table 1.
Finally, we choose bounds from two candidates which minimize the volume between the planes.
Fig.
4 is the visualization of this result.
We note that Sigmoid (cid:12) Identity transformer is handled in the same manner (we can directly apply the same transformer by replacing f ( x, y ) with σ ( x ) · y ).
Theorem 1.
Our Sigmoid (cid:12) Tanh transformer is optimal (it minimizes the volume between the lower and upper plane) under the assumptions: (1) bounding planes are tangent to the curve at the respective anchor points, (2) bounding planes are parallel to either x or y axis.
Planes computed using our transformer result in a volume strictly smaller than those computed using interval analysis.
We show the proof in Appendix B.
Our assumptions are based on the following reasoning.
The ﬁrst assumption is need so that our transformer produces bounds strictly better than interval analysis.
Unless z u passes through the point ( u x , u y ) , the concrete upper bound would be larger than σ ( u x ) · tanh( u y ) , making it worse than an interval bound(analogously for the lower bound).
The second assumption enables computation of optimal planes in O (1) while without this assumption one needs to solve a non-convex optimization problem which is not feasible at the scale of networks we consider.
E XPERIMENTS
We now evaluate the effectiveness of DAC on several datasets and neural architectures.
All our transformers are implemented in C (for performance) and exposed to the veriﬁer using a Python interface.
We will publicly release the datasets, trained networks and source code of DAC.
Veriﬁcation is performed on a Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz using 6 cores for each instance.
Experimental setup We evaluate on audio classiﬁcation benchmarks: Free Spoken Digit Dataset (FSDD) (Jackson, 2019) and Google Speech Commands (GSC) (Warden, 2018).
FSDD contains 2,000 samples of spoken digits while GSC is a more complex dataset containing 65,000 utterances of 30 short words spoken by thousands of different people.
State-of-the-art accuracy on GSC is 96.9% by de Andrade et al.
(2018) using attention RNN, while our veriﬁed model achieves 89%.
Robustness evaluation Following Singh et al.
(2019b)
, we deﬁne success rate of veriﬁcation as the ratio between certiﬁed samples and correctly predicted ones.
We randomly shufﬂed the test data and then, for every experiment, inferred labels one by one until the number of correctly classiﬁed samples reached 100.
We report the number of provably correct samples out of these 100 as our provability.
As a baseline, we consider a veriﬁcation method based on interval analysis, which only derives neuronwise lower and upper bounds using the previous layer’s bounds.
We used networks with [Figure] h = 40 hidden neurons and p = 10 ﬁlters for both FSDD and GSC.
Our results are shown in Fig.
They suggest that DAC substantially outperforms interval analysis due to better approximations of key functions in both audio preprocessing and neural network stages.
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
Effect of back-substitution depth In this experiment, we vary the depth of back-substitution used in DAC and study its effect on our performance.
We run this experiment on an FSDD network and report the results in Table 2.
We observe that increasing the back-substitution depth increases the provability.
This is due to the fact that we beneﬁt from cancellation of common terms in the propagated expressions, as demonstrated in our example in Section 3.
However, this comes at a cost of exponential increase in runtime.
Thus, we choose to use depth 3 in order to get high veriﬁcation rate while still having reasonable veriﬁcation speed.
Provable defense for audio classiﬁers We also for the ﬁrst time trained audio classiﬁers to be provably robust against noise-based perturbations.
Our training follows Mirman et al.
; Gowal et al.
(2018) – we perturb the input signal and propagate interval bounds through the audio processing and LSTM stages.
To train, we combine standard loss with the worst case loss obtained using interval propagation.
The resulting network shown in Fig.
5b achieves 80% of provability for -80 dB even with the imprecise intervals, outperforming the undefended network.
Even though this network was speciﬁcally trained to be veriﬁable using intervals, DAC still outperforms intervals and proves signiﬁcantly more robustness properties.
Also note that defended network has lower accuracy of 95%, compared to the 98% accuracy of the baseline.
Experimental comparison with prior work POPQORN (Ko et al., 2019) also proposes a method to certify recurrent neural networks by propagating linear bounds.
One of the key differences with our work is the approximation of σ ( x ) · tanh( y ) using linear bounds.
We found that, in practice, optimization approach used by POPQORN produces approximations of slightly smaller volume than our LSTM transformer (although non-comparable).
However, this smaller volume comes at a high cost in runtime.
We tried to integrate POPQORN bounds into our veriﬁcation framework, however we found it not feasible for our audio tasks as it increased our end-to-end runtime 200 × on one of the smaller networks.
In contrast, DAC takes only 1-2 minutes on average to verify a single example containing multiple frames.
Indeed, our observation is consistent with the evaluation of POPQORN in the paper where it takes hours on a GPU for verifying NLP and vision tasks only for a single frame.
C ONCLUSION
We presented the ﬁrst veriﬁer for certifying audio classiﬁers.
The key idea was to create abstract transformers for non-linear operations used in the audio processing stage and the recurrent network.
These transformers compute an optimal (area-wise) approximation under assumptions representable in the underlying convex relaxation and enable sound handling of the entire pipeline.
Our evaluation shows that DAC is practically effective and achieves high veriﬁcation rates on different datasets.
A S OUNDNESS P ROOF OF S IGMOID (cid:12) T ANH A BSTRACT T RANSFORMER
Suppose x and y are the abstract variable bounded by [ l x , u x ] , [ l y , u y ] respectively.
This suggested transformer ﬁnds the bounding planes of the function f
( x, y ) =
σ ( x ) tanh( y ) .
Let D i f = ∂f∂x and
D j f = ∂f∂y .
1. l y ≥
Choose the upper bound of the function between z
ux and z uy , where
[Equation]
by the smaller volume under the each plane.
Claim 1: z ux ( x, y ) ≥ f ( x, y ) in ( x, y ) ∈
[ l x , u x ] × [ l y , u y ] .
Let x 1 < x 2 ∈
[ l x , u x ] and y 1 < y 2 ∈ [ l y , u y ] .
Then for any x, y , f ( x, y 1 ) <
f
( x, y 2 ) and f ( x 1 , y ) <
f ( x 2 , y ) .
Thus, since z ux is independent to y , it is sufﬁcient to show z ux ( x, · ) ≥ f
( x, u y ) .
We can easily know that f ( x, u y ) is concave at x ≥ 0 and convex at x ≤ 0 by the second derivation of f .
a)
Consider
the case of u x >
Let x 0 be the x coordinate of the crossing
of f ( x, u y ) and g ( x )
=
D i f
( u x , u y )( x − u x ) − f
( u x , u y ) .
From the convexity of f ( x, u y ) in x < 0 , g (0) > f
(0 , u y ) , and g (cid:48) ( x ) > 0 , there exists a single x 0 < 0 .
If l x > x 0 , the slope of z ux ( x, · ) becomes D i f
( u x , u y ) , and z ux
and g become identical.
Hence, f ( x, u y ) ≤ g ( x ) =
z ux ( x, · ) .
If l x ≤ x 0 , the slope of z ux ( x, · ) becomes f ( u x ,u y ) − f
( l x ,u y )
u x − l x .
Then z ux and g shares a point of u x but z ux has smaller slope, so g ( x ) ≤ z ux ( x, · ) in [ l x , u x ] .
Also from f ( l x , u y ) =
z ux ( l x , · ) and f (0 , u y ) <
z ux (0 , · ) , f ( x, u y ) ≤ z ux ( x, · ) .
Thus, the claim holds when u x > 0 .
(b) Another case of u x ≤ 0 , x coefﬁcient of z ux will always be f ( u x ,u y )
− f ( l x ,u y ) u
x − l x .
Again, by convexity of f ( x, u y ) , z ux ( x, · ) ≥ f ( x, u y ) .
Hence z ux ( x, · ) ≥ f ( x, u y ) ≥ f
( x, y ) holds in ( x, y ) ∈
[ l x , u x ] × [ l y , u y ] .
Claim proved.
With analogous steps, z uy can be shown to cover above the curve.
Choosing the plane with smaller volume underneath it allows to minimize the expected difference between the true curve and the upper bound plane under the randomly chosen domain.
For the lower bound, choose the upper bound of the function between z lx and z ly , where z lx
[Equation]
x, y ) =
min
[Equation]
by the larger volume under the each plane.
Claim 2: z lx ( x, y ) ≤ f ( x, y ) in ( x, y ) ∈
[ l x , u x ] × [ l y , u y ] .
It is sufﬁcient to show z lx ( x, · ) ≥ f ( x, l y ) since f ( x, y 1 ) ≤ f ( x, y 2 ) for y 1 ≤ y 2 .
(a) Consider the case of l x < 0 .
Let x 0 be the x coordinate of the crossing of f ( x, l y ) and g ( x ) =
D i f ( l x , l y )( x − l x ) − f ( l x , l y ) .
From the concavity of f ( x, l y ) in x > 0 , g (0) < f
(0 , l y ) , and g (cid:48) ( x ) > 0 , there exists a single x 0 > 0 .
If l x < x 0 , the slope of z lx ( x, · ) becomes D i f ( l x , l y ) , and z lx and g become identical.
Hence, f ( x, l y ) ≤ g ( x ) =
z lx ( x, · ) .
If l x ≥ x 0 , the slope of z lx ( x, · ) becomes f ( l x ,l y ) − f
( u x ,l y ) l x − u x .
Then z lx and g shares a point of l x but z lx has smaller slope, so g ( x ) ≥ z lx ( x, · ) in [ l x , u x ] .
Also from f ( u x , l y )
=
z lx ( u x , · ) and f (0 , l y ) >
z lx (0 , · ) , f ( x, l y ) ≥
z lx ( x, · ) .
Thus, the claim holds when u x < 0 .
b)
Another
case of
l
x ≥
, x coefﬁcient of
z lx will always
be f
( l x ,l y ) − f ( u x ,l y ) l x − u x
Again, by convexity of
f ( x, l y ) , z lx ( x, · ) ≤ f ( x, l y ) .
Hence z lx ( x, · ) ≤ f ( x, l y ) ≥ f
( x, y ) holds in ( x, y ) ∈
[ l x , u x ] × [ l y , u y ] .
Claim proved.
With analogous steps, z ly can be shown to lie under the curve.
Choosing the plane with larger volume underneath it allows to minimize the expected difference between the true curve and the lower bound plane under the randomly chosen domain.
2. l y ≤ 0 ≤ u y
The proof of upper bounds will follow the same steps with the ﬁrst case.
z ux in this case is exactly same as before, but since f ( x, y ) goes below 0 when y < 0 , z uy has to anchor at ( l x , l y ) instead of ( u x , l y ) since f ( l x , l y ) ≥ f ( u x , l y ) and convexity of f in the region.
The proof steps do not differ much from the previous proofs.
Again, the proof for lower bound is similar as before, but note that z lx needs to choose maximum between the two slopes.
This is due to the sign of the values.
Since f ( u x , l y ) <is the minimum in the region and it grows along x gets smaller, both D i f
( u x , l y ) and f ( u x ,l y ) − f ( l x ,l y ) u x − l x are less than zero.
3. 0 ≥ u y
We will not provide the proof since this case is symmetric to the ﬁrst case.
B P ROOF OF T HEOREM
Theorem 1.
(copied) Our Sigmoid (cid:12) Tanh transformer is optimal (it minimizes the volume between the lower and upper plane) under the assumptions: (1) bounding planes are tangent to the curve at the respective anchor points, (2) bounding planes are parallel to either x or y axis.
Planes computed using our transformer result in a volume strictly smaller than those computed using interval analysis.
Proof.
First note that z ux , z uy , z lx , z ly are sound bounds by Appendix A. Assume
z ux is chosen as the upper bound.
W.l.o.g.
, suppose there exists another sound candidate upper bound ζ ux , satisfying (1) and (2), with smaller volume underneath it than z ux .
Then ζ ux would have larger absolute slope on x than z ux .
However, the slope of z ux is chosen as stiff as possible while remaining the soundness (Appendix A to see in detail).
Hence, ζ ux would not be sound anymore with larger absolute coefﬁcient.
Contradict to the existence of sound ζ ux .
Other cases can be proved with similar logic as well.
C V ERIFICATION BY E XAMPLE
We assume the last sound frame s ( t ) consists of two elements s ( t ) 1 and s ( t ) 2 such that: s ( t ) 1 ∈ [0 , 1] and s ( t ) 2 ∈ [ − 2 , − 1] .
These constraints capture a noise perturbation of the sound and are depicted in the white box in the left part of Fig.
We describe analysis only at the last timestep t (the same process is repeated for every timestep).
We note that the DeepPoly abstraction (which we build on in this work) maintains four constraints for every element: a lower and upper constraints which are linear in terms of the previous elements as well as two interval constraints.
This abstraction is exact for afﬁne transformations, however, to handle non-linear operations, one has to create new abstract transformers.
In our work we introduce such transformers and formally describe their operation in the next sections.
In what follows, we show their effect on our running example.
For the presentation below, our ﬁgure shows the linear constraints obtained by the veriﬁer, but to avoid visual clutter, we do not show the two interval constraints (however, we do list them in the text).
Fast Fourier Transform The ﬁrst operation in the audio processing stage is the Fast Fourier
Transform (FFT) of the pre-emphasized input signal.
It is a two-step process shown in dashed boxes in Fig.
The preemphasis is in fact an afﬁne transform so we perform it jointly with the afﬁne transform in the FFT.
As the composition of two afﬁne transforms is again afﬁne, this amounts to a single afﬁne transform on the input.
In our example, the composed afﬁne transform is y ( t )
= s ( t ) 1 − s ( t ) 2 and y ( t ) 2 = 2 s ( t ) 2 .
Afﬁne transforms can be captured exactly and here we obtain e.g., s ( t ) 1 − s ( t )
≤ y ( t ) 1 ≤ s ( t ) 1 − s ( t ) 2 .
We also obtain the interval bounds y ( t ) 1 ∈ [1 , 3] and y ( t ) 2 ∈ [ − 4 , − 2] .
The next step in the computation of FFT is elementwise square of y ( t ) .
We denote this operation as θ ( t )
=
( y ( t ) 1 ) 2 and θ ( t )
= ( y ( t ) 2 ) 2 .
The square operation is non-linear and cannot be captured exactly which means that we need to decide how to lose precision.
Here, we apply our new square abstract transformer (formally deﬁned in Section 4) which provides an optimal linear lower and upper bounds of the square function in terms of area.
After applying this transformer, we obtain the bounds:
[Equation]
The interval bounds are calculated by:
[Equation]
) − 8 =
Hence, θ ( t ) 1 ∈ [0 , 9] and θ ( t ) 2 ∈ [3 , 16] .
Note those bounds are not exact but sound; even y ( t ) 1 ∈ [1 , 3] and y ( t ) 2 ∈ [ − 4 , − 2] , DAC calculates the concrete lower and upper bound with the expression to be consistent with other transformers, so the bounds might not be exact as [1 , 9] and [4 , 16] .
In practice,
FFT is followed by another afﬁne transform which adds together the real and complex component of each element, but we omit this for clarity as it is captured as before, without loss of precision.
Filterbanks Transform Our analysis continues with the computation of ﬁlter banks of the input, shown in solid boxes in Fig.
The ﬁrst step is an afﬁne transform and in our example we use: ˜ ψ ( t ) 1 = 2 θ ( t ) 1 + θ ( t ) 2 and ˜ ψ ( t )
= θ ( t ) 1 + 3 θ ( t ) 2 .
Our abstraction is again exact here and additionally computes interval bounds ˜ ψ ( t ) 1 ∈ [3 , 34] and ˜ ψ ( t ) 2 ∈
The ﬁnal step in this transform is the elementwise logarithm of the input.
We denote the operation as ψ ( t ) 1 = log( ˜ ψ ( t ) 1 ) and ψ ( t ) 2 = log( ˜ ψ ( t ) 2 ) .
As this is again a non-linear operation, we apply our new log transformer so to obtain:
[Equation]
[-t ) 1 + 1 .0 . 
-]
[-˜ ψ ( t ) 2 + 1 . 
-]
[-≤ ψ ( t ) 2 ≤ 0 . 
-]
The interval bounds are calculated by:
[Equation]
t ) 1 − 4) + ( − 6 y ( t ) 2 − 9))
+ 0 .= 0 .
0783(2(4( s ( t ) 1 − s ( t ) 2 ) − 4)
+ ( − 6(2 s ( t ) 2 ) − 9) + 0 .= 0 .
6264 s ( t ) 1 − 1 .
566 s ( t ) 2 − 0 .
4674 ≥ 1 .ψ ( t ) 1 ≤ 0 . {+
+}
˜ ψ ( t ) 1 + 1 . 9178 = 0 .
0541(2 θ ( t ) 1 + θ ( t ) 2 ) + 1 .≤ 0 .
0541(2(4 y ( t ) 1 − 3)
+ ( − 6 y ( t ) 2 − 8)) + 1 .= 0 .
0541(2(4( s ( t ) 1 − s ( t ) 2 ) − 3)
+ ( − 6(2 s ( t ) 2 ) − 8)) + 1 .= 0 .
4328 s ( t ) 1 − 1 .
082 s ( t ) 2 + 1 . 1604 ≤ 3[-.ψ ( t ) 2 ≥ 0 . 
 ˜ ψ ( t ) 2 + 1 . 
 = 0-] .
[Equation]
In other words, ψ ( t ) 1 ∈ [1 . 0986 , 3 . 7572] and ψ ( t ) 2 ∈ [2 . 1967 , 4 . 2236] .
Again, these bounds are sound since [log 3 , log 34] ⊂ [1 . 0986 , 3 . 7572] , [log 9 , log 57] ⊂ [2 . 1967 , 4 . 2236] .
Discrete Cosine Transform and ReLU
After the Filterbanks Transform, the input is passed through the Discrete Cosine Transform (DCT) followed by a Lifting operation.
The analysis result of these steps is then provided as an input to a fully connected (FC) layer followed by a ReLU activation.
To ease presentation, in our example we combine DCT, Lifting and FC layer in a single afﬁne transform: ˜ x ( t ) 1 = ψ ( t ) 1 − ψ ( t ) 2 + 4 and ˜ x ( t ) 2 = 4 ψ ( t ) 1 − ψ ( t ) 2 .
We show this transform in a dotted box in Fig.
This is again captured exactly in our abstraction along with interval bounds ˜ x
( t )∈ [0 . 875 , 5 . 5605] ,
˜ x ( t )∈
The afﬁne transform is followed by a ReLU activation x ( t )= max(0 , ˜ x
( t ) 1 ) , x ( t )= max(0 , ˜ x ( t ) 2 ) .
In general, we use the ReLU transformer deﬁned in Singh et al. (2019a), which for this example produces: x ( t )
= ˜ x ( t ) 1 and x ( t ) 2 = ˜ x ( t ) 2 .
LSTM analysis After the veriﬁer completes the audio processing stage, its output is passed as input to the LSTM cell.
This cell also receives a hidden state and a cell state from the previous timestep (shown as a blue box in Fig. 2).
In our example, we assume these are given as: h ( t − 1), h ( t − 1), c ( t − 1), c ( t − 1)∈ [0 . 9 , 1] .
We note that these elements usually have different interval bounds, but for simplicity in our example we use the same intervals.
We focus on the ﬁrst neuron in the LSTM and compute pre-activations in our example for the forget gate f ( t ) 1 = x ( t ) 1 / 4 , input gate i ( t ) 1 = 2 h ( t − 1), output gate o ( t ) 1 = h ( t − 1)and cell state ˜ c ( t ) =
h ( t − 1)− x ( t ) 2 / 4 .
In order to update the cell state, we need to compute: c ( t ) 1 = σ ( f ( t ) 1 ) · c ( t − 1)+ σ ( i (
[Equation]
The left and right summands are computed using our new binary abstract transformers for σ ( x ) · y and σ ( x ) · tanh( y ) expressions, respectively.
Applying our abstract transformers on the summands produces the following bounds:
[Equation]
Summing up the above inequalities grants the upper and lower bounds of c ( t ) 1 in terms of x ( t ) , h ( t − 1) , and c ( t − 1) :
[Equation]
and get c ( t ) 1 ∈ [ − 0 .
We omit the back-substitution steps since we do not lose precision over the afﬁne transformations from x ( t ) , and note that this calculation is based on the assumption that the bounds of h ( t ) and c ( t − 1) are given constant.
In practice, we recursively apply the process for those vectors to express the bounds in terms of x ( t ) , x ( t − 1) , · · · .
The next hidden state is computed as h ( t ) 1 = σ ( o ( t ) 1 ) · tanh( c ( t ) 1 ) .
We apply our abstract transformer for the σ ( x ) · tanh( y ) expression, we obtain:
[Equation]
i.e., h ( t ) 1 ∈ [ − 0 .
An analogous computation is performed for h ( t ) 2 .
In this example we will assume h ( t ) 2 =
concretely.
Robustness certiﬁcation using DeepPoly The hidden states in the LSTM at the ﬁnal timestep are passed through a fully connected layer without activation.
In our example we assume that logits after the layer are obtained with the following formula: l 1 = h ( t ) 1 + 0 .
24 and l 2 = h ( t ) 2 .
These are shown in the ﬁnal box of Fig.
To certify that the neural network classiﬁes the input to class 1, we need to prove that l 1 − l 2 > 0 .
We now apply the same back-substitution technique as Singh et al.
(2019a):
[Equation] [Equation]
0167˜
[Equation] [Table] [Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] For the purpose of demonstration, we stop here and plug in previously obtained interval bounds for ψ ( t ) .
As this value is greater than 0, robustness is established.
D M ATHEMATICAL D ERIVATION OF LSTM T RANSFORMER
LSTM cell receives not only the resulting neurons from the pre-processing stage but a hidden state and a cell state from the previous timestep.
To incorporate with all those incoming values, we extend the scope of the LSTM equations deﬁned in Section 2.
Let d be the dimension of the input and h be the dimension of states of the cell.
To standardize the operations performed within the LSTM cell, we append 0 d × d = [0] d × d and 0 d = [0] d below to each kernel and bias respectively to express the formula as
[Equation]
[Figure]
Then the cell state and hidden state computations can be rewritten as
[Equation]
As is described in detail in Section 5, f ( x, y )
=
σ ( x ) · tanh( y ) and g ( x, y )
= σ ( x ) · y can be bounded with two planes regarding the operands’ lower and upper bounds.
In other words, for any given two real values, x, y ∈ R , x ∈ [ l x , u x ] , y ∈ [ l y , u y ] , our ﬁnding guarantees the planar bounds of f and g with the coefﬁcients built by linear combination of l x , u x , l y , u y .
For a single neuron c ( t ) j in c ( t ) , the upper bound can be expressed from Eq.
[Equation]
(4) where A u , · · · , F u are calculated real values by Section 5, and w · and b are the simpliﬁed linear coefﬁcients of each vector and bias term, respectively.
We also can get the upper bound of h ( t ) j by plugging Eq.
(4) in Eq.
(3), and similarly for the lower bounds of those states.
Note these linear bounds makes possible to recursively apply this step to get desired bounds, and can be branched in any depth for the trade-off between the precision and the complexity. E C OMPARISON BETWEEN DAC AND POPQORN
In this subsection we provide in-depth comparison between our approach and POPQORN (Ko et al., 2019) which provides another way to bound σ ( x ) · tanh( y ) and σ ( x ) · y non-linearities.
Before the experiments, we discuss comparability between three methods to bound the non-linearities: interval,
DAC and POPQORN.
Then, we perform three different experiments.
First, we compare volume of bounds produced by both POPQORN and DAC on a set of synthetic test cases.
As these test cases do not reﬂect the actual test cases which occur during the veriﬁcation of audio benchmarks, in the second set of experiments we compare the methods on a set of test cases from one of our benchmarks.
Finally, in the third experiment we plug in POPQORN bounds into our framework and show its certiﬁcation performance on the audio benchmarks.
[Figure]
[Figure]
Comparability of different bounding methods We say that methods A and B are pairwise noncomparable if there exists an input for which method A produces tighter bound than method B, and vice versa.
Given this deﬁnition, POPQORN is non-comparable with our method.
To demonstrate this, in Fig.
6 we show a case where this behavior is manifested.
Here, for y ≤ − 1 POPQORN (shown as orange plane) produces tighter bound than DAC (shown as blue plane).
However, for the other entire range of inputs where − 10 ≤ y ≤ − 2 , POPQORN bounds are substantially worse than our bounds.
Further, those bounds are even worse than interval bounds and the overapproximation error is not bounded.
Contrary to POPQORN, our bounds are always strictly better than interval bounds (this is proven in Theorem 1) and distance between the function and our planes is bounded.
Comparison on synthetic test cases In this experiment, we compare bounds produced by
POPQORN and DAC on a set of synthetic test cases, unrelated to the certiﬁcation of audio classiﬁers.
Both methods take l x , u x and l y , u y as bounds for the input to the sigmoid and tanh, respectively.
We sample those inputs uniformly from [ − 2 , 2] × [ − 2 , 2] and compared the volume between the curve σ ( x ) · tanh( y ) and bounding planes produced by both DAC and POPQORN.
The volume was computed using 1000 Monde Carlo samples in the set [ l x , u x ] × [ l y , u y ] .
Since there are two σ ( x ) · tanh( y ) and one σ ( x ) · y calculation appearing in a single LSTM cell, we run the experiment with such portion of data.
In other words, in 67% of experiments we bound σ ( x ) · tanh( y ) , and in 33% we bound σ ( x ) · y .
We sampled 100 such inputs and compared the volumes obtained by
POPQORN and DAC with the volume obtained using interval.
The distribution of volumes is shown in Fig.
Here, 1 stands for the same volume as interval bounds and values less than 1 indicate performance better than intervals.
We conclude that, for this experiment, POPQORN bounds produces smaller volume than our method - 0.2 compared to 0.37.
In terms of runtime, POPQORN takes 14.37 seconds on average while the bound calculation of DAC ﬁnishes in few milliseconds.
Direct comparison on test cases from audio benchmarks The previous experiment may not reﬂect the actual performance on audio benchmarks that we consider in this work.
We uniformly sampled the arguments l x , u x , l y , u y that our transformer for σ ( x ) · tanh( y ) and σ ( x ) · y was invoked with.
We distinguish between test cases corresponding to different perturbation ranges and then further split it into certiﬁed and uncertiﬁed samples.
For each of the cases, Fig. 8 shows the box plot of the distributions of volumes produced by both POPQORN and DAC.
[Figure]
We found that, while overall POPQORN bounds work well in practice, they frequently produce bounds less precise than interval bounds.
The reason of this malfunctioning comes from the limitation of gradient descent approach employed by POPQORN.
The gradient of the function, which POPQORN uses to search for the optimal bounds, is large when the inputs are distributed near 0.
However, if the inputs are far from the origin or are too close to each other, the function curve becomes almost ﬂat, gradients are close to zero and gradient descent has problems converging.
Also, in other cases, gradient descent is not guaranteed to ﬁnd the bounds with minimum volume.
Fig.
6 shows one of the examples where POPQORN fails to ﬁnd lower planar bound which produces minimum volume.
On the contrary, the resulting value of σ ( x ) · tanh( y ) is within [ − 1 , 1] regardless of the magnitude of arguments which means that the error produced by intervals is bounded.
As our planar bounds are strictly better than intervals, regardless of input conditions, our error is also bounded.
Plugging in POPQORN bounds into DAC
We also experimented with using POPQORN bounds instead of our bounds in the pipeline and compared the ﬁnal adversarial regions with those resulting from DAC.
As POPQORN is relatively slow (108 minutes per sample), we performed this experiment only on the ﬁrst 10 inputs with − 80 dB perturbation.
Using their bounds results in 0 veriﬁed samples, while DAC veriﬁes 4 samples.
We believe the reason here is the existence of many pathological cases as described in the previous point where gradient descent used by POPQORN converges to suboptimal solution which ends up being worse than interval bounds.
These errors are further propagated through each frame and resulting output can not be certiﬁed.
F S ENSITIVITY OF P ROVABILITY M ETRIC
In our experiments, we followed the convention of provability measurement from Singh et al.
(2019b).
Here, we also provide the result with error bars from 10 independent repetitions with randomly permuted test set.
For each repetition, we randomly permute the test set with the different seed and collect the ﬁrst 100 samples with the correct prediction under zero perturbation from the ordered set.
We then count the number of certiﬁed inputs from those samples to represent the provability under the given constant (cid:15) .
Fig.
9 shows that provabilities do not differ much from the reported results for multiple experiments.
G T RAINING WITH P ROVABLE D EFENSE WITH IBP
Here we give more details on our training procedure for provably defended network which follows
Mirman et al.
; Gowal et al.
Let z LB ( (cid:15) ) and z UB ( (cid:15) ) be the resulting lower and upper bounds for the ﬁnal logit z ∈ R d under the perturbation size (cid:15) , with the true label j ∈ { 0 , · · · , d − 1 } .
We deﬁne the worst-case logits ˆ z ( (cid:15) ) as
[Equation]
which corresponds to the worst concrete logits under the given input with the predeﬁned perturbation amount.
Recall that we say the input is certiﬁed when this worst-case logits satisfy j = arg max i
ˆ
z i .
[Figure]
[Figure]
[Table]
[Figure]
[Table]
Training loss L is a linear combination of the standard cross-entropy loss l ( z , e ( j ) ) and the worst-case cross-entropy loss l (ˆ z , e ( j ) ) where e ( j ) is target one-hot vector, i.e.,
[Equation]
Note that we set up the κ and (cid:15) as the function of t (cid:48) .
As in Gowal et al.
(2018), gradual increase/decrease of these parameters during the training was essential to get desired performance.
We set these functions with respect to the number of training epochs E as κ ( t ) =
(cid:26)
[Equation]
t/E t < E/1 /otherwise
[Equation]
Also, to track the training speed, we increase one by one t (cid:48) only if we are achieving 85% of standard accuracy and 80% provability at the current setting.
The models were built under the E = 60 .
We also attach the missing result of defended GSC with 82% concrete accuracy in Fig.
H M ODEL P ARAMETERS U SED IN THE E XPERIMENTS
We perform the experiments with the same architecture with different model parameters for FSDD and GSC.
Table 3 shows the parameters of the models we use.
Both defended and undefended networks share the same parameters for each dataset.