Targeted Adversarial Examples for Black Box Audio Systems
Abstract
The application of deep recurrent networks to audio transcription has led to impressive gains in automatic speech recognition (ASR) systems.
Many have demonstrated that small adversarial perturbations can fool deep neural networks intoincorrectly predicting a speciﬁed target with high conﬁdence.
Current work onfooling ASR systems have focused on white-box attacks, in which the model architecture and parameters are known.
In this paper, we adopt a black-box approachto adversarial generation, combining the approaches of both genetic algorithmsand gradient estimation to solve the task.
We achieve a 89.25% targeted attacksimilarity after 3000 generations while maintaining 94.6% audio ﬁle similarity.
Introduction
Although neural networks have incredible expressive capacity, which allow them to be well suitedfor a variety of machine learning tasks, they have been shown to be vulnerable to adversarial attacksover multiple network architectures and datasets [7].
These attacks can be done by adding smallperturbations to the original input so that the network misclassiﬁes the input but a human does notnotice the difference.
So far, there has been much more work done in generating adversarial examples for image inputsthan for other domains, such as audio.
Voice control systems are widely used in many productsfrom personal assistants, like Amazon Alexa and Apple Siri, to voice command technologies in cars.
One main challenge for such systems is determining exactly what the user is saying and correctlyinterpreting the statement.
As deep learning helps these systems better understand the user, onepotential issue is targeted adversarial attacks on the system, which perturb the waveform of whatthe user says to the system to cause the system to behave in a predetermined inappropriate way.
For example, a seemingly benign TV advertisement could be adversely perturbed to cause Alexa tointerpret the audio as “Alexa, buy 100 headphones.”
If the original user went back to listen to theaudio clip that prompted the order, the noise would be almost undetectable to the human ear.
There are multiple different methods of performing adversarial attacks depending on what informationthe attacker has about the network.
If given access to the parameters of a network, white box attacksare most successful, such as the Fast Gradient Sign Method [7] or DeepFool [11].
However, assumingan attacker has access to all the parameters of a network is unrealistic in practice.
In a black boxsetting, when an attacker only has access to the logits or outputs of a network, it is much harder toconsistently create successful adversarial attacks.
In certain special black box settings, white box attack methods can be reused if an attacker createsa model that approximates the original targeted model.
However, even though attacks can transfer across networks for some domains, this requires more knowledge of how to solve the task thatthe original model is solving than an attacker may have [10, 13].
Instead, we propose a novelcombination of genetic algorithms and gradient estimation to solve this task.
The ﬁrst phase of theattack is carried out by genetic algorithms, which are a gradient-free method of optimization thatiterate over populations of candidates until a suitable sample is produced.
In order to limit excessmutations and thus excess noise, we improve the standard genetic algorithm with a new momentummutation update.
The second phase of the attack utilizes gradient estimation, where the gradientsof individual audio points are estimated, thus allowing for more careful noise placement when theadversarial example is nearing its target.
The combination of these two approaches provides a 89.25%average targeted attack similarity with a 94.6% audio ﬁle similarity after 3000 generations.
1.1 Problem statement
Adversarial attacks can be created given a variety of information about the neural network, such asthe loss function or the output probabilities.
However in a natural setting, usually the neural networkbehind such a voice control system will not be publicly released so an adversary will only have accessto an API which provides the text the system interprets given a continuous waveform.
Given thisconstraint, we use the open sourced Mozilla DeepSpeech implementation as a black box system,without using any information on how the transcription is done.
We perform our black box targeted attack on a model M given a benign input x and a target t byperturbing x to form the adversarial input x (cid:48)
=
x + δ , such that M ( x (cid:48) )
=
t .
To minimize the audiblenoise added to the input, so a human cannot notice the target, we maximize the cross correlationbetween x and x (cid:48) .
A sufﬁcient value of δ is determined using our novel black box approach, so wedo not need access to the gradients of M to perform the attack.
1.2 Prior work
Compared to images, audio presents a much more signiﬁcant challenge for models to deal with.
Whileconvolutional networks can operate directly on the pixel values of images, ASR systems typicallyrequire heavy pre-processing of the input audio.
Most commonly, the Mel-Frequency Cepstrum(MFC) transform, essentially a fourier transform of the sampled audio ﬁle, is used to convert theinput audio into a spectogram which shows frequencies over time.
Models such as DeepSpeech (Fig.2) use this spectogram as the initial input.
In a foundational study for adversarial attacks, Cisse et al.
[5] developed a general attack frameworkto work across a wide variety of models including images and audio.
When applying their method toaudio samples, they ran into the roadblock of backpropagating through the MFC conversion layer.
Thus, they were able to produce adversarial spectograms but not adversarial .wav ﬁles.
Carlini and Wagner [3] overcame this challenge by developing a method of passing gradients throughthe MFC layer, a task which was previously proved to be difﬁcult [5].
They applied their methodto the Mozilla DeepSpeech model, which is a complex, recurrent, character-level network that candecode translations at up to 50 characters per second.
With a gradient connection all the way to theraw input, they were able to achieve impressive results, including generating samples over 99.9%similar with a targeted attack accuracy of 100%.
While the success of this attack opens new doorsfor white box attacks, adversaries in a real-life setting commonly do not have knowledge of modelarchitectures or parameters.
Alzantot et al.
[1] have demonstrated that black-box approaches for targeted attacks on ASR systemsare possible.
Using a genetic algorithm approach, they were able to iteratively apply noise to audio samples, pruning away poor performers at each generation, and ultimately end up with a perturbedversion of the input that successfully fooled a classiﬁcation system.
This attack was conductedon the Speech Commands classiﬁcation model [1], which is a lightweight convolutional model forclassifying up to 50 different single-word phrases.
Extending the research done by [1], we propose a genetic algorithm and gradient estimation approachto create targeted adversarial audio, but on the more complex DeepSpeech system.
The difﬁculty ofthis task comes in attempting to apply black-box optimization to a deeply-layered, highly nonlineardecoder model that has the ability to decode phrases of arbitrary length.
Nevertheless, the combinationof two differing approaches as well as the momentum mutation update bring new success to this task.
1.3 Background
Dataset For the attack, we follow [3] and take the ﬁrst 100 audio samples from the CommonVoicetest set.
For each, we randomly generate a 2-word target phrase and apply our black-box approachto construct an adversarial example.
More details on evaluation can be found in section 3.
Eachsample in the dataset is a .wav ﬁle, which can easily be deserialized into a numpy array.
Ouralgorithm operates directly on the numpy arrays, thus bypassing the difﬁculty of dealing with the
MFC conversion.
Victim model The model we attack is Baidu’s DeepSpeech model [8], implemented in Tensorﬂowand open-sourced by Mozilla.
2 Though we have access to the full model, we treat it as if in a blackbox setting and only access the output logits of the model.
In line with other speech to text systems[4, 5], DeepSpeech accepts a spectrogram of the audio ﬁle.
After performing the MFC conversion,the model consists 3 layers of convolutions, followed by a bi-directional LSTM, followed by afully connected layer.
This layer is then fed into the decoder RNN, which outputs logits over thedistribution of output characters, up to 50 characters per second of audio.
The model is illustrated inﬁgure 2.
Connectionist temporal classication While the DeepSpeech model is designed to allow arbitrarylength translations, there is no given labeled alignment of the output and input sequences duringtraining time.
Thus the connectionist temporal classication loss (CTC Loss) is introduced, as it allowscomputing a loss even when the position of a decoded word in the original audio is unknown [3].
DeepSpeech outputs a probability distribution over all characters at every frame, for 50 frames persecond of audio.
In addition to outputting the normal alphabet a-z and space, it can output specialcharacter (cid:15) .
Then CTC decoder C ( · ) decodes the logits as such: for every frame, take the characterwith the max logit.
Then ﬁrst, remove all adjacent duplicate characters, and then second, remove anyspecial (cid:15) characters.
Thus aab(cid:15)(cid:15)b will decode to abb [3].
As we can see, multiple outputs can decode to the same phrase.
Following the notation in [3], for anytarget phrase p , we call π an alignment of p if C ( π ) =
p .
Let us also call the output distribution ofour model y .
Now, in order to ﬁnd the likelihood of alignment π under y :
Pr ( p | y ) = (cid:88) π | C ( π )=
p
Pr ( π | y ) =
(cid:88)
π | C ( π )
= p (cid:89) i y iπ as noted by [3].
This is the objective we use when scoring samples from the populations in eachgeneration of our genetic algorithm as well as the score used in estimating gradients.
Greedy decoding As in traditional recurrent decoder systems, DeepSpeech typically uses a beamsearch of beam width 500.
At each frame of decoding, 500 of the most likely π will be evaluated,each producing another 500 candidates for a total of 2500, which are pruned back down to 500 forthe next timestep.
Evaluating multiple assignments this way increases the robustness of the modeldecoding.
However, following work in [3], we set the model to use greedy decoding.
At each timesteponly 1 π is evaluated, leading to a greedy assignment:
[Equation]
Thus, our genetic algorithm will focus on creating perturbations to the most likely sequence (ifgreedily approximated).
Black box attack algorithm
Algorithm 1 Black box algorithm for generating adversarial audio sample
Input: Original benign input x Target phrase t
Output: Adversarial Audio Sample x (cid:48) population ← [ x ] ∗ populationSize while iter < maxIters and Decode ( best )!
=
t do scores ← − CTCLoss ( population , t ) best ← population [ Argmax ( scores )] if EditDistance ( t, Decode ( best )) >
2 then // phase 1 - do genetic algorithm while populationSize children have not been made do
Select parent 1 from topk ( population ) according to softmax ( their score )
Select parent 2 from topk ( population ) according to softmax ( their score ) child ← Mutate ( Crossover ( parent1 , parent2 ) , p ) end while newScores ← − CTCLoss ( newPopulation , t )
p ← MomentumUpdate ( p , newScores , scores ) else // phase 2 - do gradient estimation top-element ← top ( population ) grad-pop ← n copies of top-element, each mutated slightly at one index grad ← ( − CTCLoss ( grad-pop ) − scores ) / mutation-delta pop ← top-element + grad end if end while return best
2.1 Genetic algorithm
As mentioned previously, Alzantot et al.
[1] demonstrated the success of a black-box adversarialattack on speech-to-text systems using a standard genetic algorithm.
The basic premise of ouralgorithm is that it takes in the benign audio sample and, through trial and error, adds noise to thesample such that the perturbed adversarial audio is similar to the benign input yet is decoded as thetarget, as shown in Figure 3.
A genetic algorithm works well for a problem of this nature because itis completely independent of the gradients of the model.
Alzantot et al.
[1] used a limited datasetconsisting of audio samples with just one word and a classiﬁcation with a predeﬁned number ofclasses.
In order to extend this algorithm to work with phrases and sentences, as well as with CTC
Loss, we make modiﬁcations to the genetic algorithm and introduce our novel momentum mutation.
[Figure]
The genetic algorithm works by improving on each iteration, or generation, through evolutionarymethods such as Crossover and Mutation [9].
For each iteration, we compute the score for eachsample in the population to determine which samples are the best.
Our scoring function was the
CTC-Loss, which as mentioned previously, is used to determine the similarity between an input audiosequence and a given phrase.
We then form our elite population by selecting the best scoring samplesfrom our population.
The elite population contains samples with desirable traits that we want to carryover into future generations.
We then select parents from the elite population and perform Crossover,which creates a child by taking around half of the elements from parent 1 and the other half fromparent 2 .
The probability that we select a sample as a parent is a function of the sample’s score.
With some probability, we then add a mutation to our new child.
Finally, we update our mutationprobabilities according to our momentum update, and move to the next iteration.
The population willcontinue to improve over time as only the best traits of the previous generations as well as the bestmutations will remain.
Eventually, either the algorithm will reach the max number of iterations, orone of the samples is exactly decoded as the target, and the best sample is returned.
2.2 Momentum mutation
Algorithm
2 Mutation
Input: Audio Sample x
Mutation Probability p
Output: Mutated Audio Sample x (cid:48) for all e in x do noise ← Sample ( N ( µ, σ 2 )) if Sample ( Unif (0 , 1)) < p then e (cid:48)
← e + filter highpass ( noise ) end if end for return x (cid:48)
The mutation step is arguably the most crucial component of the genetic algorithm and is our onlysource of noise in the algorithm.
In the mutation step, with some probability, we randomly add noiseto our sample.
Random mutations are critical because it may cause a trait to appear that is beneﬁcialfor the population, which can then be proliferated through crossover.
Without mutation, very similarsamples will start to appear across generations; thus, the way out of this local maximum is to nudge itin a different direction in order to reach higher scores.
Furthermore, since this noise is perceived as background noise, we apply a ﬁlter to the noise beforeadding it onto the audio sample.
The audio is sampled at a rate of f s = 16 kHz , which means thatthe maximum frequency response f max = 8 kHz .
As seen by Reichenbach and Hudspeth [14], giventhat the human ear is more sensitive to lower frequencies than higher ones, we apply a highpass ﬁlter at a cutoff frequency of f cutoff = 7 kHz .
This limits the noise to only being in the high-frequencyrange, which is less audible and thus less detectable by the human ear.
While mutation helps the algorithm overcome local maxima, the effect of mutation is limited by themutation probability .
Much like the step size in SGD, a low mutation probability may not provideenough randomness to get past a local maximum.
If mutations are rare, they are very unlikely tooccur in sequence and add on to each other.
Therefore, while a mutation might be beneﬁcial whenaccumulated with other mutations, due to the low mutation probability, it is deemed as not beneﬁcialby the algorithm in the short term, and will disappear within a few iterations.
This parallels the stepsize in SGD, because a small step size will eventually converge back at the local minimum/maximum.
However, too large of a mutation probability, or step size, will add an excess of variability and preventthe algorithm from ﬁnding the global maximum/minimum.
To combat these issues, we propose
Momentum Mutation , which is inspired by the Momentum Update for Gradient Descent.
With thisupdate, our mutation probability changes in each iteration according to the following exponentiallyweighted moving average update:
[Equation]
With this update equation, the probability of a mutation increases as our population fails to adaptmeaning the current score is close to the previous score.
The momentum update adds accelerationto the mutation probability, allowing mutations to accumulate and add onto each other by keepingthe mutation probability high when the algorithm is stuck at a local maximum.
By using a movingaverage, the mutation probability becomes a smooth function and is less susceptible to outliers in thepopulation.
While the momentum update may overshoot the target phrase by adding random noise,overall it converges faster than a constant mutation probability by allowing for more acceleration inthe right directions.
Algorithm
Input: Mutation Probability p
Scores for the new population newScores
Scores for the previous population scores
Output: Updated mutation probability p new currScore = max ( newScores ) prevScore = max ( scores ) p new = α × p old
+ return p
2.3 Gradient estimation
Genetic algorithms work well when the target space is large and a relatively large number of mutationdirections are potentially beneﬁcial; the strength of these algorithms lies in being able to searchlarge amounts of space efﬁciently [6].
When an adversarial sample nears its target perturbation,this strength of genetic algorithms turn into a weakness, however.
Close to the end, adversarialaudio samples only need a few perturbations in a few key areas to get the correct decoding.
In thiscase, gradient estimation techniques tend to be more effective.
Speciﬁcally, when edit distance ofthe current decoding and the target decoding drops below some threshold, we switch to phase 2.
When approximating the gradient of a black box system, we can use the technique proposed by Nitin
Bhagoji et al.
[Equation]

Here, x refers to the vector of inputs representing the audio ﬁle.
δ i refers to a vector of all zeros,except at the i th position in which the value is a small δ .
g ( · ) represents the evaluation function, which in our case is CTCLoss.
Essentially, we are performing a small perturbation at each index andindividually seeing what the difference in CTCLoss would be, allowing us to compute a gradientestimate with respect to the input x .
However, performing this calculation in full would be prohibitively expensive, as the audio is sampledat 16 kHz and so a simple 5-second clip would require 80,000 queries to the model for just onegradient evaluation!
Thus, we only randomly sample 100 indices to perturb each generation whenusing this method.
When the adversarial example is already near the goal, gradient estimation makesthe tradeoff for more informed perturbations in exchange for higher compute.
Evaluation
3.1 Metrics
We tested our algorithm by running it on a 100 sample subset of the Common Voice dataset.
For eachaudio sample, we generated a single random target phrase by selecting two words uniformly withoutreplacement from the set of 1000 most common words in the English language.
The algorithmwas then run for each audio sample and target phrase pair for 3000 generations to produce a singleadversarial audio sample.
We evaluated the performance of our algorithm in two primary ways.
The ﬁrst method is determiningthe accuracy with which the adversarial audio sample gets decoded to the desired target phrase.
Forthis, we use the Levenshtein distance, or the minimum character edit distance, between the desiredtarget phrase and the decoded phrase as the metric of choice.
We then calculated the percent similaritybetween the desired target and the decoded phrase by calculating the ratio of the Levenshtein distanceand the character length of the original input, ie. 1 − Levenshtein ( M ( x (cid:48) ) ,t ) len ( M ( x )) .
The second method isdetermining the similarity between the original audio sample and the adversarial audio sample.
Forthis, we use the accepted metric of the cross correlation coefﬁcient between the two audio samples.
3.2 Results
Of the audio samples for which we ran our algorithm on, we achieved a 89.25% similarity between theﬁnal decoded phrase and the target using Levenshtein distance, with an average of 94.6% correlationsimilarity between the ﬁnal adversarial sample and the original sample.
The average ﬁnal Levenshteindistance after 3000 iterations is 2.3, with 35% of the adversarial samples achieving an exact decodingin less than 3000 generations, and 22% of the adversarial samples achieving an exact decoding in lessthan 1000 generations.
One thing to note is that our algorithm was 35% successful in getting the decoded phrase to matchthe target exactly; however, noting from ﬁgure 5, the vast majority of failure cases are only a few editdistances away from the target.
This suggests that running the algorithm for a few more iterationscould produce a higher success rate, although at the cost of correlation similarity.
Indeed, it becomesapparent that there is a tradeoff between success rate and audio similarity such that this thresholdcould be altered for the attacker’s needs.
A comparison of white box targeted attacks, black box targeted attacks on single words (classiﬁcation),and our method: Metric White
[Table]
Attacks
Our Method
[Table]
Average
[Table]
Loss used for attack
CTC CTC Softmax Dataset tested on
[Table]
Speech Commands
Target
[Table]
generation
[Table]
phrases Single word One helpful visualization of the similarity between the original audio sample and the adversarialaudio sample through the overlapping of both waveforms, as shown in ﬁgure 4.
As the visualizationshows, the audio is largely unchanged, and the majority of the changes to the audio is in the relativelylow volume noise applied uniformly around the audio sample.
This results in an audio sample thatstill appears to transcribe to the original intended phrase when heard by humans, but is decoded asthe target adversarial phrase by the DeepSpeech model.
That 35% of random attacks were successful in this respect highlights the fact that black boxadversarial attacks are deﬁnitely possible and highly effective at the same time.
Conclusion
In combining genetic algorithms and gradient estimation we are able to achieve a black box adversarialexample for audio that produces better samples than each algorithm would produce individually.
Byinitially using a genetic algorithm as a means of exploring more space through encouragement ofrandom mutations and ending with a more guided search with gradient estimation, we are not onlyable to achieve perfect or near-perfect target transcriptions on most of the audio samples, we were ableto do so while retaining a high degree of similarity.
While this remains largely as a proof-of-conceptdemonstration, this paper shows that targeted adversarial attacks are achievable on black box modelsusing straightforward methods.
Furthermore, the inclusion of momentum mutation and adding noise exclusively to high frequenciesimproved the effectiveness of our approach.
Momentum mutation exaggerated the exploration at thebeginning of the algorithm and annealed it at the end, emphasizing the beneﬁts intended by combininggenetic algorithms and gradient estimation.
Restricting noise to the high frequency domain improvedupon our similarity both subjectively by keeping it from interfering with human voice as well asobjectively in our audio sample correlations.
By combining all of these methods, we are able toachieve our top results.
In conclusion we introduce a new domain for black box attacks, speciﬁcally on deep, nonlinear
ASR systems that can output arbitrary length translations.
Using a combination of existing and novelmethods, we are able to exhibit the feasibility of our approach and open new doors for future research.