VoiceBlock: Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models
Anonymous Author(s)
AffiliationAddress email
Abstract
As governments and corporations adopt deep learning systems to collect and analyze user-generated audio data, concerns about security and privacy naturallyemerge in areas such as automatic speaker recognition.
While audio adversarialexamples offer one route to mislead or evade these invasive systems, they are typically crafted through time-intensive offline optimization, limiting their usefulnessin streaming contexts.
Inspired by architectures for audio-to-audio tasks such asdenoising and speech enhancement, we propose a neural network model capable ofadversarially modifying a user’s audio stream in real-time.
Our model learns to apply a time-varying finite impulse response (FIR) filter to outgoing audio, allowingfor effective and inconspicuous perturbations on a small fixed delay suitable forstreaming tasks.
We demonstrate our model is highly effective at de-identifyinguser speech from speaker recognition and able to transfer to an unseen recognitionsystem.
We conduct a perceptual study and find that our method produces perturbations significantly less perceptible than baseline anonymization methods, whencontrolling for effectiveness.
Finally, we provide an implementation of our modelcapable of running in real-time on a single CPU thread.
Audio examples and codecan be found at https://master.d3hvhbnf7qxjtf.amplifyapp.com/ .
Introduction
Mass surveillance of voice communications is an ongoing and pervasive issue.
While section 702 ofthe United States Foreign Intelligence Surveillance Act allows the government to perform targetedmonitoring of foreign communications, bulk collection practices have resulted in the warrantlesssurveillance of large numbers of "incidental" foreign and domestic individuals [17].
Despite thefact that millions of these communications are obtained without warrants, they have been used inordinary criminal investigations [61], undermining a core purpose of the Fourth Amendment ofthe US constitution [60]: to protect the people against searches without probable cause.
Manycorporations also possess the capability for large-scale collection of voice data [23], which may beleveraged to profile users for advertising or accessed by government entities through upstream anddownstream surveillance [16].
As individuals are faced with these growing surveillance apparati, it isworth remembering that routine surveillance of private voice communications is also corrosive to freespeech and association and tends to disproportionately affect marginalized groups [29, 18].
In the absence of identifying metadata, automatic speaker recognition systems can facilitate masssurveillance by allowing an operator to search a database of recorded voice data for utterances from achosen speaker [7] or to diarise (assign utterances to individuals) transcripts of recordings [45].
Priorto the advent of automatic speaker recognition these tasks required human analysts, forming a naturalcheck on surveillance overreach.
We seek to restore this check by degrading the efficacy of speakerrecognition models while maintaining the original perceptual quality of the voice communication, astep that could grant users a measure of privacy from mass surveillance.
Modern speaker recognition systems rely on deep neural networks [6].
Deep networks have beenshown to be vulnerable to adversarial examples—natural instances (e.g. a recording of "Bob"speaking) modified to cause a model to make an incorrect prediction (the recording’s speaker islabeled as "Maria")
This presents opportunities for privacy-minded individuals to mislead orevade surveillance systems with adversarially-crafted inputs.
Researchers have proposed adversarialattacks against a variety of audio systems, including speaker recognition [63, 52].
To fool a given neural network-based system, many audio attacks modify a recording by addinga perturbation signal directly at the waveform representation [8].
This perturbation is typicallycrafted using gradient information obtained from the system’s neural network—or from a similarlyconstructed surrogate—and requires iterative optimization.
Once the optimization is complete, themodified recording is played to the system in an effort to induce arbitrary incorrect predictions (anuntargeted attack) or a specific incorrect prediction chosen by the attacker (a targeted attack).
In theory, such attacks might allow individuals to evade systems that surveil and analyze voicecommunications, thereby protecting privacy.
However, many existing attack algorithms require acostly optimization for each audio recording to which they are applied.
Continuous, real-time voicecommunication precludes the adoption of such approaches through online optimization (which istypically too slow) or through the use of a set of precomputed adversarial examples (which wouldnecessarily limit the user’s interaction).
This suggests a need for algorithms capable of modifyingspeech on-the-fly.
Recent years have seen the development of models for audio-to-audio tasks such as denoising, voiceconversion, and musical timbre transfer [13, 49, 14].
Such models have sufficient expressive power tomodify audio in coherent and task-specific ways, and are often designed to run in real-time.
Inspiredby these models, we propose VoiceBlock , a deep network that learns to apply a time-varying finiteimpulse response (FIR) filter to outgoing audio, producing highly effective adversarial perturbationson a small fixed delay suitable for streaming.
The main contributions of this work are:
• A highly effective method for de-identifying speech in real-time that is more perceptuallyinconspicuous than existing methods of equal effectiveness
• Objective and subjective experimental results that validate these claims• A system that embodies the claimed advances, and that runs in real-time on a single CPUthread
Through this work, we hope to encourage further exploration of audio-to-audio models for protectinguser privacy.
Audio examples and code, including our streaming implementation, can be found athttps://master.d3hvhbnf7qxjtf.amplifyapp.com/ .
Related work
Previous works in speech de-identification have proposed methods for obfuscating speech to evadesurveilling speaker-recognition systems.
One such method is voice conversion, which modifies thespeech of a source speaker to sound like that of another target speaker, both to humans and machines.
Jin et al.
[24] and Alegre et al.
[4] analyze the vulnerability of speaker recognition systems againstde-identification attacks performed with voice conversion models.
However, the conversion modelsconsidered are incapable of real-time operation.
While the recent availability of low-latency voiceconversion models [49, 50, 32] may make such approaches more practical, our aim is to modifyspeech in a way that is inconspicuous and minimally invasive to the user experience, and that has thepotential to generalize to tasks beyond speaker recognition.
This also rules out recent anonymizationapproaches catalogued through the VoicePrivacy Challenge, as all evaluated submissions to datenoticeably alter speaker characteristics [56].
Similarly, we do not consider obvious transformationslike pitch-shifting that significantly alter or degrade recorded audio.
Real-time audio attacks have recently been explored.
Chiquier et al.
[10] propose an attack onautomatic speech recognition systems that is capable of inducing significant transcription errors in anover-the-air environment (where attack audio is played to the system in a physical space), but whichintroduces conspicuous noise due to the use of additive perturbations at the audio waveform.
Theirapproach also necessitates both a large model (requiring approximately 16 GPU-days to train) andlengthy receptive field, resulting in an initial “idle" period of 2.5s during which the attack does not affect speech.
By contrast, we focus on an over-the-line setting, where attacks are passed to the victimmodel over a digital channel, and propose a lightweight network capable of producing inconspicuousperturbations through the use of filtering, with an initial delay of miliseconds rather than seconds.
Universal adversarial attacks optimize a short perturbation that can be played alongside arbitraryspeech, optionally in real-time, to evade a victim model [33, 63, 28].
However, these attacks tend tointroduce conspicuous noise due to the use of additive perturbations at the audio waveform.
The sameholds for the attack of Xie et al.
[62], which uses a Wave-U-Net [53] model to efficiently generateuniversal perturbations.
Rather than generate additive perturbations, we perform multiplicative attacksin the frequency domain through the use of filtering, and in doing so avoid the bias towards noise-likeartifacts typical of additive attacks.
Other works have proposed crafting adversarial attacks in the frequency domain .
The “Kenansville" attack of Abdullah et al.
[1] performs spectral gating, removing low-energy frequencycomponents in an effort to hinder classification.
The authors demonstrate the effectiveness of theirapproach against seen and unseen speaker- and speech-recognition systems, and the proposed methodis fast and gradient-free.
However, the spectral gating must be constrained to a small number ofkey audio frames to avoid conspicuous artifacts, which requires offline optimization using word- orphoneme-level alignment information.
Ahmed et al.
[3] optimize bandpass filters to perform targeted impersonation attacks on speakerrecognition models; these filters can then be physically realized as resonant tubes through which anattacker can speak to the victim system, allowing real-time operation.
However, once optimized andrealized, the bandpass filters are fixed in time and cannot adapt to the attacker’s speech.
The attack issignificantly less effective than traditional approaches, even in controlled acoustic environments, andrequires an active intervention on the part of the user to both realize and apply.
Finally, O’Reilly et al.
[42] adversarially optimize the parameters of a time-varying finite impulseresponse (FIR) filter in order to avoid noisy artifacts.
However, the attack operates offline and mustbe optimized separately for every instance to which it is applied.
In summary, we are aware of no current approach that is simultaneously capable of performing deidentification while remaining inconspicuous to human listeners and operating in real-time on-device.
VoiceBlock
We propose VoiceBlock, a system that applies adversarial time-varying filtering to user audio inreal-time.
VoiceBlock modifies speech by specifying the frequency magnitude response of a standardfinite impulse response (FIR) filter.
By carefully varying the filter’s response many times per second,the speech is de-identified to an automated system while speaker identity is preserved for humanlisteners.
This is accomplished without introducing noisy artifacts at the waveform, as filtering canonly amplify or attenuate frequency energy present in the original signal through multiplication in the
Fourier domain (see Section 3.3 for details of the filtering implementation used in our experiments).
VoiceBlock consists of three main modules, shown in Figure 1: (1) an encoder module extractsacoustic features from input audio frames, (2) a recurrent bottleneck module incorporates contextfrom past frames to predict a set of adversarial filter controls for each frame, and (3) a decodermodule regularizes the predicted filter controls and applies them to the corresponding frame of inputaudio.
We provide an overview of each component below, and further details in Appendix A.
3.1 Encoder
The encoder module extracts acoustic features from 16kHz audio (double the sample rate of telephonespeech) segmented into frames of 256 samples with 50% overlap.
Though our model can operatein fully causal fashion, we find we achieve stronger attacks using a lookahead of five frames (see Section 3.2), resulting in a theoretical latency of 56ms.
Verizon reports 25ms or less is typicallatency for voice-over-IP communication in North America and Europe [57]. Experimental data [27]show that 130ms of latency is acceptable for high-quality Voice communications. Therefore, adding VoiceBlocks to the signal chain leaves latency well within this limit.
Audio frames entering the encoder are passed to four sub-modules in parallel to obtain the followingfeatures.
Pitch features : Given the known sensitivity of speaker-recognition models to pitch [3] and theimportance of spectral structure in delineating linguistic content, we extract fundamental frequencyand aperiodicity estimates for each frame using the DIO algorithm
Pitch estimates are refinedby the StoneMask algorithm
For both stages, we use the pyworld [22] implementation of the
WORLD vocoder [38] (MIT license).
Loudness : We take an A-weighted average of each frame’s log-magnitude spectrum to obtain aloudness estimate [36].
Phonetic posteriorgrams : Following the method of Ronssin & Cernak [49], we use a trainedphoneme classifier with frozen weights to encode linguistic content.
The classifier consists of amulti-layer perceptron followed by two LSTM [21] layers and a linear classification layer, and takesas input 13 mel-frequency cepstral coefficients (MFCC) with first- and second-order deltas for eachframe.
We train our phoneme classifier on the "train-clean-100" subset of the LibriSpeech dataset[43] using frame-aligned phoneme labels [34, 35] (CC-BY 4.0 license).
Rather than directly use theclassifier’s predicted distributions over phoneme labels – known as phonetic posteriorgrams (PPGs)[19] – we discard the classification layer after training and pass along the output of the final LSTMlayer, which Ronssin & Cernak also refer to as PPGs in their architecture.
Spectrogram features : Finally, we use a simple network consisting of a gated linear unit andmulti-layer perceptron operating independently on each mel-spectrogram frame—a widely usedspeech representation [26]—to capture any residual information. We note that Engel et al.
[14]also make use of a residual spectrogram encoder, alongside pitch and loudness features, to controldifferentiable signal-processing components.
For each frame, the above features are concatenated and projected linearly to obtain a low-dimensionalencoding.
We then introduce additional speaker information by passing a fixed, pre-computedembedding of the source speaker through a FiLM layer [46] to modulate the encoder output.
Thisembedding helps to guide the de-identification task, allowing us to train a single model capable ofde-identifying arbitrary users.
This imposes a requirement that users must record a small amountof speech to obtain an embedding before first using VoiceBlock.
We find that less than a minute ofspeech is sufficient, and use a pre-trained ResNetSE34V2 model [20] to compute embeddings (seesections 4.1, 4.2).
3.2 Bottleneck
Encodings entering the bottleneck are passed through two LSTM layers, and the outputs are concatenated with a skip connection from the encoder.
To enable streaming, we use unidirectional
LSTM layers and pass concatenated outputs through a small lookahead convolutional network [58] toincorporate information from future frames at the expense of a small fixed delay.
We find a lookaheadof 5 frames (48 ms) is sufficient to craft strong de-identification attacks.
Finally, a linear projectionlayer maps the concatenated representations to a vector of filter controls, with each entry representingthe unnormalized frequency magnitude response of a filter band for the current frame.
3.3 Decoder
Our decoder applies time-varying filtering to the input audio based on frame-wise controls obtainedfrom the bottleneck.
For our task, we find that a filtering-based decoder affords a number ofadvantages over the synthesis-based (e.g. transposed-convolutional) decoder architectures present inmany audio-to-audio models [13]:
• The filtering module is not prone to the periodic upsampling artifacts that transposed-
convolutional architectures often introduce [47].
• Our decoder is a simple deterministic module with no trainable parameters, keeping VoiceBlock lightweight.•
The capacity of the decoder and conspicuousness of perturbations can easily be constrainedin terms of the number of filter bands or their allowed range of motion, providing interpretable control to the user.
To regularize and apply filter controls to each frame of audio, we use a method similar to that of
O’Reilly et al.
and Engel et al.
We apply sigmoid scaling to bound filter controls to therange [0 , 2] .
We clip deviations from unity beyond a fixed bound ϵ .
Each set of scaled filter controlsis transformed into a time-domain impulse response via the inverse Fourier transform.
We shift theimpulse response to zero-phase (symmetric) form, apply a Hann window, and finally convolve withthe corresponding input audio frame by taking the Fourier transform and performing element-wisemultiplication.
After compensating for shift, we overlap-add the resulting frames with a Hann windowto obtain the final filtered audio.
We discuss the details of our buffered streaming implementation in
Appendix A.3.
Our VoiceBlock model is implemented in PyTorch [2] and contains 6.3m trainableparameters, and 7.5m in total counting the frozen phoneme encoder.
3.4 Training objective
To demonstrate the ability of VoiceBlock to perform inconspicuous privacy-preserving audio transformations, we train our model to attack speaker recognition systems.
Given a large database ofspeech recordings and access to a user’s audio stream, a surveilling entity may seek to (a) identifythe user by matching their speech against recordings of known provenance in the database, or (b)retrieve other utterances of the user from the database.
Systems designed for these tasks – speakerrecognition and retrieval, respectively – often rely on neural network models to map speech utterancesto a low-dimensional embedding space in which distance corresponds to speaker similarity [15].
Inthis work, we consider models f for which the speaker distance D f between utterances u and v canbe measured via a cosine distance between embeddings f ( u ) and f ( v ) :
[Equation]
At inference time, query audio from the user is embedded and scored for similarity against enrolledutterances – pre-computed embeddings stored in the database.
Scores may be evaluated for allenrolled embeddings, or for only a representative of each unique speaker (e.g. speaker centroids).
The highest-ranking result or results (e.g. the closest speaker identity) are then returned.
We do notdistinguish between recognition and retrieval tasks, and refer to both under the umbrella of "speakerrecognition.
" This is because from a privacy standpoint, the objective in each task is identical: alterthe query audio to prevent valid matches, thereby de-identifying the user.
A variety of methods have been proposed to efficiently compute and search over low-dimensionalspeaker representations.
Generally, a hashing algorithm is applied to speaker embedding vectors to reduce storage and search costs [51, 31, 15].
Because the resulting hash representation merelyserves as an efficient point of access for embedding-space distances, we omit hashing algorithmsfrom consideration and define our attack objectives on the embedding space directly.
Given a speaker embedding model, we aim to modify query audio drawn from a user’s streamsuch that its embedding-space distance from any enrolled utterances of the user is large, and itsevaluated similarity is small.
We quantify this de-identification in terms of distance thresholds inthe embedding space, set according to percentiles of the estimated distribution of all inter-speakerembedding distances.
Let P r represent the distance corresponding to the r th percentile of thisdistribution; then for query audio u and enrolled utterance v , D f ( u, v ) > P r implies that roughly rpercent of database entries should be scored as more similar to u than v .
Thus, we can set distancethresholds that correspond directly to the strength of de-identification applied to user audio, andconstruct a loss function that penalizes our model when the embedding-space cosine distance betweenquery and enrolled utterances falls below the threshold.
To do so, we use a variant of the adversarialloss proposed by Zhang et al.
Let f represent the victim model, g our VoiceBlock network, uan utterance from our user’s audio stream, and P r our de-identification threshold; then
[Equation]
[Equation]
where ( · ) + = max( · , 0) and κ is a confidence parameter encouraging the attack to fully cross thethreshold.
To ensure that our VoiceBlock model learns to perturb user audio inconspicuously, weincorporate an additional loss function to penalize perceptible filtering artifacts.
We compute thecombined waveform L 1 and multi-resolution spectrogram losses proposed by Defosséz et al.
[13] onthe clean and adversarial audio:
[Equation]
where L stft is given by a sum of magnitude and spectral convergence losses computed over several spectrogram resolutions.
We provide further details in Appendix A.2.
Combining the aboveadversarial and auxiliary losses, we obtain our final attack objective:
[Equation]
Experimental design
We describe experiments used to validate the claimed advances of our work, namely that VoiceBlockcan de-identify speech in real-time while remaining significantly less conspicuous than existingmethods of similar effectiveness.
We first introduce the models, datasets, and attacks considered inour experiments.
Following this, we detail our experiment configurations and present the results ofboth objective and subjective evaluations.
4.1 Speaker recognition models
ResNetSE34v2: We train attacks against the ResNetSE34v2 model [20] provided in the
VoxCeleb Trainer repository [12] (MIT License).
The model takes mel-spectrogram inputs anduses 2D convolutions with residual connections, squeeze-and-excitation, and attentive statistics pooling to generate frame-level features and aggregate them into 512-dimensional speaker embeddings.
Y-Vector: To examine the transferability of our approach against unseen systems, we evaluatetrained attacks against the Y-Vector model proposed by Zhu et al.
The model uses a multiscale1D-convolutional waveform encoder to extract acoustic features, followed by squeeze-and-excitationblocks, feature aggregation, and a time-delayed neural network to map variable-length utterances to128-dimensional speaker embeddings.
Both the ResNetSE34v2 and Y-Vector models were trained on the development set of the VoxCeleb2dataset [11].
Note that while our attack is causal—modulo a short, five-frame lookahead—we do notimpose the same restriction on a hypothetical surveillance system; instead, we evaluate our attackagainst strong, non-causal systems capable of aggregating speaker characteristics from across fullutterances before rendering predictions.
4.2 Datasets
LibriSpeech: (CC-BY 4.0)
We use both the train-clean-100 and test-clean subsets of the
LibriSpeech dataset [43] for training VoiceBlock.
The former comprises 28,539 utterances from 251 speakers while the latter comprises 2,620 utterances from 40 speakers.
We trim or pad all utterancesto 4 seconds.
VoxCeleb1: (CC-BY 4.0) To simulate large-scale surveiling speaker recognition, we evaluate attackson the VoxCeleb1 dataset [40], comprising 153,516 utterances from 1,251 speakers.
This also ensuresthat no evaluation speakers are seen during training.
As with the LibriSpeech dataset, we trim or padall utterances to 4 seconds.
For all experiments, we carefully divide the data to imitate a realistic attack setting.
During training,we select fifteen utterances (one minute total) from each source speaker in the training set and computeembeddings using the ResNetSE34v2 (MIT License) model.
The centroid of these embeddings isthen used as an enrolled target for the computation of the adversarial loss with all utterances of thatspeaker.
We find that this produces stronger attacks than using individual utterance embeddings astargets, possibly by ensuring more consistent gradient information across the optimization.
For our
VoiceBlock attack (see Section 4.3), we select a further ten utterances (40s total) from each sourcespeaker in the training set and again compute embeddings using the ResNetSE34v2 model.
Thecentroid of these embeddings is then fed as a fixed conditioning vector to the VoiceBlock modelalongside all utterances from that speaker (see Section 3.1).
Similar to training, during evaluation weselect fifteen utterances per speaker as a query set.
We again select a further ten utterances to serve asconditioning for the VoiceBlock attack.
Finally, twenty utterances of each speaker are enrolled in thespeaker recognition system, serving as the database against which query utterances are matched.
4.3 Attack algorithms
We perform untargeted attacks using the following algorithms.
VoiceBlock: We implement theproposed VoiceBlock attack as described in Section 3 and Appendix A and train for 10 epochs on
NVidia RTX 2080 Ti GPUs; this takes approximately 40 minutes.
Universal: We optimize a short(2s) universal additive perturbation for 10 epochs using the established penalty method [33, 63, 28]and the same adversarial objective as VoiceBlock.
On 3 NVidia RTX 2080 Ti GPUs, this takesapproximately 16 minutes.
To limit the perceptibility of the attack, we scale the perturbation to have
L ∞ norm 0.08 times that of the unperturbed speech.
During training and evaluation, the perturbationis aligned arbitrarily with query utterances and looped to match durations, serving as a constantadversarial "background" signal.
White noise: We add Gaussian noise to utterances at the waveformrepresentation at a signal-to-noise ratio of − 10 dB. Spectral gating:
We modify the "Kenansville"attack of Abdullah et al.
[1] to allow for streaming use by performing spectral gating at all frames,using a threshold of 4dB relative to the maximum-energy spectral bin of each frame.
4.4 Objective evaluation
We evaluate attacks in a large-scale closed-set speaker recognition task.
First, we use the test-clean
LibriSpeech subset to obtain a rough estimate of the distribution of distances between an individualutterance and the centroids of each distinct speaker in the embedding space of the ResNetSE34V2model.
To encourage strong de-identification attacks, we take the distance corresponding to the 25 thpercentile of this distribution as the target threshold P 25 for our training loss (see Section 3.4).
Wetrain each attack as discussed above.
We evaluate the closed-set recognition of all attacks over the VoxCeleb1 dataset.
We computethe distance between each query embedding and the centroid of all embeddings of each speaker;recogntion is then performed by returning the speaker identity of the nearest speaker centroid.
Wefind this is slightly more accurate and robust to attack than using the identity of the nearest embeddedutterance.
We perform exact nearest-neighbors search over the embedding space using FAISS[25] (MIT license), and report the top-1 (T-1) and top-10 (T-10) accuracy of the relevant speakerrecognition model given both clean and adversarial queries.
Additionally, we compute the followingobjective speech quality metrics over the clean and adversarial query audio as a proxy measure ofthe imperceptibility of attacks: Perceptual Evaluation of Speech Quality (PESQ) [48], operatingin the wide-band configuration and using the python-pesq implementation [59] (MIT license);and Short-Time Objective Intelligibility (STOI) [55], using the pystoi implementation [44] (MITlicense).The results of our evaluation are presented in table 1.
We evaluate the real-time performance of the streaming implementation of the VoiceBlock attackdescribed in Appendix A.3 by measuring its average real-time factor (RTF).
We measure performance [Table] [Table] on a single thread on two different CPUs, an Intel i7-5600U @ 3.2 GHz and an Apple M1 Chip.
In the streaming configuration, VoiceBlock processes a chunk of 4 overlapping frames at a time,equivalent to 640 samples / 40 ms.
We compute the average RTF for processing a chunk over allchunks in a 4 second audio clip, and find that VoiceBlock has an average RTF of .255 on the Inteli7-5600U and .200 on the Apple M1.
4.5 Subjective evaluation
For our subjective evaluation, we sampled 100 clean speech recordings from the VoxCeleb1 dataset.
Each of the four attacks described in Section 4.3 was applied to all 100 recordings, resulting in 100comparison sets with five recordings per set: the original clean speech and the speech modified byeach of the four attacks.
These sets were then evaluated in a MUSHRA-style listening study [9]deployed on Amazon Mechanical Turk using the open-source, MIT-licensed Reproducible Subjective
Evaluation (ReSEval) [39] system.
IRB approval was obtained prior to conducting this study, andthere were no known risks to the participants in this study.
We recruited 20 participants.
Participants were screened with a listening test prior to beginning thestudy; those that passed rated 20 comparison sets.
For each comparison set, the participant was askedto listen to and rate the relative audio quality of each of the five audio files on a scale from 0 to 100.
We omitted responses by four participants who failed our prescreening listening test and nine whorated the white noise attack (our low anchor) as superior in quality to ground-truth clean audio (ourhigh anchor), giving us a total of 140 five-way comparisons.
For more details of our crowdsourcedsubjective evaluation, see Appendix C.
The results of our crowdsourced subjective evaluation can be found in Figure 2.
We find VoiceBlockis preferred to all other methods, and exhibits similar perceptual quality to ground-truth speechrecordings.
The difference between VoiceBlock and ground-truth audio is significant ( p < 0 .
05)using a Wilcoxon signed-rank test ( p = 0 . 024) but not significant using Welch’s T-test ( p = 0 . 071) , which assumes that human perceptual scores are normally distributed but with potentially differentvariances.
All other pairs of conditions are significantly different using either test.
4.6 Additional experiments
We conduct supplementary experiments demonstrating the robustness of VoiceBlock against a deepnetwork-based speech enhancement model, examining its performance under the assumption thatadversarial queries are enrolled by the surveilling speaker recognition system, and evaluating theintelligibility of processed audio using an automatic system.
These experiments are detailed inappendices B.1, B.2, and B.3 respectively.
Ethics
The main goal of our work is to show that large-scale, untargeted automated surveillance can behindered by introducing inconspicuous perturbations to audio in real-time.
Because VoiceBlock doesnot conceal speaker identity from human listeners, it still allows high-effort targeted surveillance (e.g.authorized human-attended wiretaps of criminal enterprises).
In this way, we hope to return to thestatus quo of the 20th and early 21st centuries – in which the need for human listeners provided animportant check on mass surveillance.
Those wishing to avoid all voice identification by human or machine may use existing voice conversiontechnologies.
Voice conversion is not, however, appropriate in many contexts where some measure ofprivacy from mass surveillance may be desired. Consider telehealth: a medical expert or therapistdiscussing sensitive topics with a patient may gain great insight from hearing the nuance of thepatient’s voice. In this setting, methods that alter one’s voice beyond human recognition (e.g. voiceconversion) would necessarily degrade the interaction. The approach of VoiceBlock could also, in concept, be used to fraudulenty access information orservices protected by speaker verification.
However, VoiceBlock performs perceptually inconspicuousfiltering and the speech produced would still sound like the original speaker.
As such, it could notpass human inspection.
Both voice-conversion and speech synthesis-based attacks are much moresuited to this purpose, as both produce speech in a voice perceptually similar to the target speaker.
Our approach can be thought of as leveraging the asymmetry between system and attacker attentioninherent to mass surveillance, using inconspicuous perturbations to evade large-scale systems thatmust render predictions in bulk.
This same asymmetry is not necessarily present when trying tobypass authentication mechanisms and access tightly-guarded services.
Our experimental results indicate VoiceBlock can de-identify speech from arbitrary unseen users inreal-time on a standard M1 CPU.
The de-identified speech is of significantly higher audio qualitythan competing methods, as reported by a subjective listener study and as measured through standardmetrics of speech intelligibility.
Our method also achieves nontrivial de-identification results againsta system it was not trained on, outperforming a far more perceptible universal attack.
This isnotable given that we take no explicit steps to improve the transferability of our method.
Whilepure signal-processing approaches such as spectral gating and white noise transfer between systemsmore successfully, they severely degrade audio quality, resulting in much lower listener qualityratings.
This limits their practicality in real-world voice communications.
By contrast, our methodproduces adversarial audio that is virtually indistinguishable from the clean source, as indicated byour subjective evaluation.
We view our method as an initial step towards protecting users from indiscriminate mass surveillance.
A number of obvious directions for future work stand out, such as improving the transferability ofadversarial examples [41], and evaluating attacks against real-world speaker recognition systems andover real-world communication channels.
We hope this work encourages further exploration of theapplications of audio-to-audio models for protecting user privacy.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We claim to have designed a system to make highly
effective, inconspicuous perturbations that can transfer to unforseen architectures in
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] It will be
anonymized in the supplemental materials
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We provide a distribution of subjective quality ratings
for each attack in in 2.
We, however, do not provide error bars over random seeds for
training.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Stated in 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite authors of
LibriSpeech, VoxCeleb, ResNetSE34v2, ReSEval and Y-Vector
(b) Did you mention the license of the assets?
[Yes] We mention the licenses for datasets
and models in 4.2 and for ReSEval in 4.5.
We note that the Y-Vector model does
nothave a License, to our knowledge.(c
) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
e) Did you discuss whether the data you are using /curating contains personally identifiableinformation or offensive content
[N/A
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Yes
In the Supplementary Materials(
b ) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes] We obtained IRB approval.
There are noknown risks to the participants in out subjective evaluation. 4.5(
c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation
Yes] In C
A VoiceBlock implementationA.1 Architecture
We provide additional details of the VoiceBlock architecture used in our experiments.
Phoneme encoder:
The phoneme encoder consists of a feedforward network with two hiddenlayers of size 256 and ReLU activations, followed by two LSTM layers with hidden size 256.
Usingthe LibriSpeech "train-clean-100" dataset and aligned phoneme labels provided by Lugosch et al.[34], we train the phoneme encoder with cross entropy loss for 25 epochs using the Adam optimizerwith learning rate 1e-3.
During training, a linear layer is appended to the phoneme encoder to renderpredictions over phoneme labels; we discard this layer after training.
Spectrogram encoder:
We pass 64-bin mel-spectrograms through a 1 × 1 convolutional layer andgated linear unit (GLU) operating along the frequency axis, followed by a two linear layers with leaky
ReLU activations and hidden size 512.
The convolution layer upsamples the channel (frequency)dimension to 1024, from which the GLU obtains a 512-dimensional representation to pass to thelinear layers.
Source speaker conditioning: For each frame of audio, we concatenate the 256-dimensionaloutput of the phoneme encoder, the 512-dimensional output of the spectrogram encoder, and the onedimensional pitch / periodicity / loudness features and linearly project to obtain a 512-dimensionalvector.
A fixed 512-dimensional source speaker embedding obtained from the ResNetSE34V2 modelis passed through a 2 linear layers with leaky ReLU activations and hidden size 512, followed by batchnormalization and a final linear layer to obtain a 1024-dimensional vector holding scales and biasesfor a feature-wise affine transformation (FiLM) of the encoder output.
The affine transformation isapplied and the resulting vector is passed to the bottleneck.
Bottleneck: Our bottleneck consists of two LSTM layers with hidden size 512.
The encoder outputis passed through the LSTM layers with a skip connection concatenated at the output.
Following theapplication of a lookahead convolutional network, the resulting vector is projected linearly to obtaina 128-dimensional vector of filter controls for the current frame.
Decoder:
We perform filtering with 128 bands per frame, and set L ∞ bound ϵ = 0 .
5 to furtherconstrain deviations from a "neutral" control configuration of ⃗ 1 .
We experimented with alternativemethods of regularizing predicted filter controls, such as L 2 normalization, but found that simple L ∞clipping yielded satisfactory results.
Training:
We train both VoiceBlock and the universal baseline attack using the Adam [30]optimizer with learning rate 1e − 4 for 10 epochs over the LibriSpeech "train-clean" subset
.A.2 Auxiliary loss
We briefly summarize the multi-resolution STFT loss of Defosséz et al.
[13] used in the training ofour attacks.
The loss is given by
[Equation]
where L stft is given by a sum of magnitude and spectral convergence losses computed over Mspectrogram resolutions:
[Equation]
Here || · || F and || · || 1 refer to the Frobenius and L 1 norms, respectively.
Following Defosséz et al.
,we use M = 3 spectrogram resolutions with hop sizes of 50, 120, and 240 samples, FFT lengths of512, 1024, and 2048, and window lengths of 240, 600, and 1200, respectively.
A.3 Streamer
We implement a version of VoiceBlock for processing audio streams in real time.
All components aremodified to support the computation of perturbations on overlapping windows of audio while theinput and output streams process non-overlapping chunks.
Where possible, components are complied [Table] to TorchScript to improve performance.
Three buffers are added to process a stream of audio into overlapping chunks
• Last Frame Input Buffer: Contains the last 128 samples of the last input audio to be appended [Table] to the oncoming chunk, before computing the overlapping windows.
• Lookahead Buffer: Contains overlapping frames to be used as lookahead.• Output Buffer:
Contains the last 128 samples output by VoiceBlock, to be overlap-addedwith the next processed chunk.
Code for our streaming implementation can be found at https://master.d3hvhbnf7qxjtf.amplifyapp.com/ .
B Additional experiments
B.1 Attack robustness to preprocessing
In real-world settings, user audio may travel through various preprocessing stages before reaching aspeaker recognition model.
To simulate the presence of such stages, we perform attacks using theexperiment configuration discussed in Section 4.3 but pass query utterances through a pretrained
Demucs [13] speech enhancement model en route to the ResNetSE34v2 recognition system; resultsare reported in table 2.
The speaker recognition system maintains its top-1 and top-10 accuracy onclean queries.
In contrast to Chiquier et al.
[10] we do not incorporate the enhancement model intoour adversarial optimization, meaning that Demucs essentially functions as an unseen adversarialdefense.
We find that VoiceBlock loses almost no effectiveness when queries are passed through Demucs; by comparison, the universal attack loses most of its effectiveness, as Demucs is able toseparate the low-magnitue but noisy perturbation from clean speech.
Both the white noise and spectralgating attacks retain their effectiveness, presumably because they degrade audio beyond Demucs’capability to provide coherent reconstructions of the original speech.
We leave as future work theexploration of the robustness of VoiceBlock against more sophisticated preprocessing pipelines andadversarial defenses.
B.2 Enrollment of adversarial queries
It is possible that adversarially-perturbed query utterances extracted from a VoiceBlock user’s audiostream may themselves be enrolled by a surveiling speaker recognition system.
In such cases, thereare two possibilities:
VoiceBlock successfully de-identifies user speech, and adversarial queries are enrolled as a separate speaker profile from any existing clean utterances of the user
VoiceBlock fails to de-identify user speech, and adversarial queries are incorporated into anexisting profile of the user containing clean utterances.
We examine both scenarios, again using the VoiceBlock attack discussed in Section 4.3.
Results arepresented in table 3.
We find that while VoiceBlock is highly effective at de-identifying users against [Table] [Table] a profile constructed from clean (unperturbed) utterances, its de-identification performance sufferssignificantly under both of the aforementioned conditions.
This suggests that additional work isrequired to ensure that a VoiceBlock user remains a "moving target" to surveilling speaker recognitionsystems.
One possible solution is to leverage targeted rather than untargeted attacks: by proactively"spoofing" specific (random) locations in the embedding space, VoiceBlock may hamper the creationof a single matching enrolled profile.
To measure the intelligibility of speech processed by the proposed and baseline methods, we evaluatethe word- and character-error rates of each attack described in Section 4.3 on the LibriSpeechtrain-clean-360 subset using a Wav2Vec 2.0 automatic speech recognition model [5]. Thetrain-clean-360 dataset contains utterances and accompanying transcriptions from 921 speakersnot seen during training. We do not use the VoxCeleb dataset because it does not have the necessarytranscriptions.
We partition each speaker’s data into query, enrollment, and conditioning partitions,as described in Section 4.2. Our results are reported in Table 4. Our results demonstrate that pure signal-processing attacks such as spectral gating and white noiseresult in word- and character-error rates 20+ times that of VoiceBlock. VoiceBlock also outperformsall methods except spectral gating on the de-ientification task, with which it performs competitively.
This indicates our method’s ability to preserve the intelligibility of speech while adversariallymodifying speaker characteristics.
C Listening Study
The listening study consists of 20 evaluation tasks, where the participants are shown the followingtext, along with 5 four second long audio examples and 5 corresponding sliders with values from 0 to100:
Listen to all recordings of a person speaking. Then, move the sliders to rate the quality of eachaudio file from 0 (worst) to 100 (best) .
The higher-quality audio files are the ones that aremore natural sounding, or have fewer audio artifacts (e.g., clicks, pops, noise, or otherwisesound ’unnatural’). Note - Each slider cannot be moved until its corresponding audio file hasbeen listened to in its entirety.
It takes approximately 15 minutes to complete all the evaluation tasks.
Participants are also asked tocomplete a listening test before proceeding to the audio evaluation.
The listening test involves theparticipant listening to two audio files, and reporting the number of tones they heard.
For each audiofile, the participants are given 3 tries to correctly report the number of tones played.
If a participantsucceeds in the listening test and completes the evaluation tasks, then they are paid 3.00 USD, orequivalently, 12.00 USD / hour.
If the participant fails the listening test, then they are paid 0.50 USD.