VoiceBox: Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models
Anonymous Author(s)
AffiliationAddress email
Abstract
As governments and corporations adopt deep learning systems to collect and analyze user-generated audio data, concerns about security and privacy naturallyemerge in areas such as automatic speaker recognition.
While audio adversarialexamples offer one route to mislead or evade these invasive systems, they are typically crafted through time-intensive offline optimization, limiting their usefulnessin streaming contexts.
Inspired by architectures for audio-to-audio tasks such asdenoising and speech enhancement, we propose a neural network model capable ofadversarially modifying a user’s audio stream in real-time.
Our model learns to apply a time-varying finite impulse response (FIR) filter to outgoing audio, allowingfor effective and inconspicuous perturbations on a small fixed delay suitable forstreaming tasks.
We demonstrate our model is highly effective at de-identifyinguser speech from speaker recognition and able to transfer to an unseen recognitionsystem.
We conduct a perceptual study and find that our method produces perturbations significantly less perceptible than baseline anonymization methods, whencontrolling for effectiveness.
Finally, we provide an implementation of our modelcapable of running in real-time on a single CPU thread.
Audio examples and codecan be found at https://master.d3hvhbnf7qxjtf.amplifyapp.com/ .
Introduction
Mass surveillance of voice communications is an ongoing and pervasive issue.
While section 702 ofthe United States Foreign Intelligence Surveillance Act allows the government to perform targetedmonitoring of foreign communications, bulk collection practices have resulted in the warrantlesssurveillance of large numbers of "incidental" foreign and domestic individuals [16].
Despite thefact that millions of these communications are obtained without warrants, they have been used inordinary criminal investigations [58], undermining a core purpose of the Fourth Amendment ofthe US constitution [57]: to protect the people against searches without probable cause.
Manycorporations also possess the capability for large-scale collection of voice data [22], which may beleveraged to profile users for advertising or accessed by government entities through upstream anddownstream surveillance [15].
As individuals are faced with these growing surveillance apparati, it isworth remembering that routine surveillance of private voice communications is also corrosive to freespeech and association and tends to disproportionately affect marginalized groups [27, 17].
In the absence of identifying metadata, automatic speaker recognition systems can facilitate masssurveillance by allowing an operator to search a database of recorded voice data for utterances from achosen speaker [6] or to diarise (assign utterances to individuals) transcripts of recordings [43].
Priorto the advent of automatic speaker recognition these tasks required human analysts, forming a naturalcheck on surveillance overreach.
We seek to restore this check by degrading the efficacy of speakerrecognition models while maintaining the original perceptual quality of the voice communication, astep that could grant users a measure of privacy from mass surveillance.
Modern speaker recognition systems rely on deep neural networks [5].
Deep networks have beenshown to be vulnerable to adversarial examples—natural instances (e.g. a recording of "Bob"speaking) modified to cause a model to make an incorrect prediction (the recording’s speaker islabeled as "Maria")
This presents opportunities for privacy-minded individuals to mislead orevade surveillance systems with adversarially-crafted inputs.
Researchers have proposed adversarialattacks against a variety of audio systems, including speaker recognition [60, 50].
To fool a given neural network-based system, many audio attacks modify a recording by addinga perturbation signal directly at the waveform representation [7].
This perturbation is typicallycrafted using gradient information obtained from the system’s neural network—or from a similarlyconstructed surrogate—and requires iterative optimization.
Once the optimization is complete, themodified recording is played to the system in an effort to induce arbitrary incorrect predictions (anuntargeted attack) or a specific incorrect prediction chosen by the attacker (a targeted attack).
In theory, such attacks might allow individuals to evade systems that surveil and analyze voicecommunications, thereby protecting privacy.
However, many existing attack algorithms require acostly optimization for each audio recording to which they are applied.
Continuous, real-time voicecommunication precludes the adoption of such approaches through online optimization (which istypically too slow) or through the use of a set of precomputed adversarial examples (which wouldnecessarily limit the user’s interaction).
This suggests a need for algorithms capable of modifyingspeech on-the-fly.
Recent years have seen the development of models for audio-to-audio tasks such as denoising, voiceconversion, and musical timbre transfer [12, 47, 13].
Such models have sufficient expressive power tomodify audio in coherent and task-specific ways, and are often designed to run in real-time.
Inspiredby these models, we propose VoiceBox , a deep network that learns to apply a time-varying finiteimpulse response (FIR) filter to outgoing audio, producing highly effective adversarial perturbationson a small fixed delay suitable for streaming.
The main contributions of this work are:
• A highly effective method for de-identifying speech in real-time that is more perceptuallyinconspicuous than existing methods of equal effectiveness
• Objective and subjective experimental results that validate these claims• A system that embodies the claimed advances, and that runs in real-time on a single CPUthread
Through this work, we hope to encourage further exploration of audio-to-audio models for protectinguser privacy.
Audio examples and code, including our streaming implementation, can be found athttps://master.d3hvhbnf7qxjtf.amplifyapp.com/ .
Related work
Previous works in speech de-identification have proposed methods for obfuscating speech to evadesurveilling speaker-recognition systems.
One such method is voice conversion, which modifies thespeech of a source speaker to sound like that of another target speaker, both to humans and machines.
Jin et al.
[23] and Alegre et al.
[4] analyze the vulnerability of speaker recognition systems againstde-identification attacks performed with voice conversion models.
However, the conversion modelsconsidered are incapable of real-time operation.
While the recent availability of low-latency voiceconversion models [47, 48, 30] may make such approaches more practical, our aim is to modifyspeech in a way that is inconspicuous and minimally invasive to the user experience, and that has thepotential to generalize to tasks beyond speaker recognition.
This also rules out recent anonymizationapproaches catalogued through the VoicePrivacy Challenge, as all evaluated submissions to datenoticeably alter speaker characteristics [54].
Similarly, we do not consider obvious transformationslike pitch-shifting that significantly alter or degrade recorded audio.
Real-time audio attacks have recently been explored.
Chiquier et al.
[9] propose an attack onautomatic speech recognition systems that is capable of inducing significant transcription errors in anover-the-air environment (where attack audio is played to the system in a physical space), but whichintroduces conspicuous noise due to the use of additive perturbations at the audio waveform.
Theirapproach also necessitates both a large model (requiring approximately 16 GPU-days to train) andlengthy receptive field, resulting in an initial “idle" period of 2.5s during which the attack does not affect speech.
By contrast, we focus on an over-the-line setting, where attacks are passed to the victimmodel over a digital channel, and propose a lightweight network capable of producing inconspicuousperturbations through the use of filtering, with an initial delay of miliseconds rather than seconds.
Universal adversarial attacks optimize a short perturbation that can be played alongside arbitraryspeech, optionally in real-time, to evade a victim model [31, 60, 26].
However, these attacks tend tointroduce conspicuous noise due to the use of additive perturbations at the audio waveform.
The sameholds for the attack of Xie et al.
[59], which uses a Wave-U-Net [51] model to efficiently generateuniversal perturbations.
Rather than generate additive perturbations, we perform multiplicative attacksin the frequency domain through the use of filtering, and in doing so avoid the bias towards noise-likeartifacts typical of additive attacks.
Other works have proposed crafting adversarial attacks in the frequency domain .
The “Kenansville" attack of Abdullah et al.
[1] performs spectral gating, removing low-energy frequencycomponents in an effort to hinder classification.
The authors demonstrate the effectiveness of theirapproach against seen and unseen speaker- and speech-recognition systems, and the proposed methodis fast and gradient-free.
However, the spectral gating must be constrained to a small number ofkey audio frames to avoid conspicuous artifacts, which requires offline optimization using word- orphoneme-level alignment information.
Ahmed et al.
[3] optimize bandpass filters in order to perform targeted impersonation attacks onspeaker-recognition models; these filters can then be physically realized as resonant tubes throughwhich an attacker can speak to the victim system, allowing real-time operation.
However, onceoptimized and realized, the bandpass filters are fixed in time and cannot adapt to the attacker’s speech.
The attack is significantly less effective than traditional approaches, even in controlled acousticenvironments, and requires an active intervention on the part of the user to both realize and apply.
Finally, O’Reilly et al.
[40] adversarially optimize the parameters of a time-varying finite impulseresponse (FIR) filter in order to avoid noisy artifacts.
However, the attack operates offline and mustbe optimized separately for every instance to which it is applied.
In summary, we are aware of no current approach that is simultaneously capable of performing deidentification while remaining inconspicuous to human listeners and operating in real-time on-device.
VoiceBox
We propose VoiceBox, a system that applies adversarial time-varying filtering to user audio inreal-time.
VoiceBox modifies speech by specifying the frequency magnitude response of a standardfinite impulse response (FIR) filter.
By carefully varying the filter’s response many times per second,the speech is de-identified to an automated system while speaker identity is preserved for humanlisteners.
This is accomplished without introducing noisy artifacts at the waveform, as filtering canonly amplify or attenuate frequency energy present in the original signal through multiplication in the
Fourier domain (see Section 3.3 for details of the filtering implementation used in our experiments).
VoiceBox consists of three main modules, shown in Figure 1: (1) an encoder module extractsacoustic features from input audio frames, (2) a recurrent bottleneck module incorporates contextfrom past frames to predict a set of adversarial filter controls for each frame, and (3) a decodermodule regularizes the predicted filter controls and applies them to the corresponding frame of inputaudio.
We provide an overview of each component below, and further details in Appendix A.
3.1 Encoder
The encoder module extracts acoustic features from input audio to guide adversarial filtering. Ournetwork accepts 16kHz audio (double the sample rate of telephone speech) segmented into frames of256 samples with 50% overlap.
Though our model is capable of operating in fully causal fashion, wefind that we achieve stronger attacks using a lookahead of five frames (see Section 3.2), resulting in aminimum theoretical latency of 56ms.
Taking cues from voice-conversion systems, we aim to disentangle speaker characteristics from thelinguistic content of input utterances so that our model can subtly manipulate the former.
Audioframes entering the encoder are passed to four sub-modules in parallel to obtain the following features.
Pitch features : Given the known sensitivity of speaker-recognition models to pitch [3] and theimportance of spectral structure in delineating linguistic content, we extract fundamental frequencyand aperiodicity estimates for each frame using the DIO algorithm
Pitch estimates are refinedby the StoneMask algorithm
For both stages, we use the pyworld [21] implementation of the
WORLD vocoder [36] (MIT license).
Loudness : We take an A-weighted average of each frame’s log-magnitude spectrum to obtain aloudness estimate [34].
Phonetic posteriorgrams : Following the method of Ronssin & Cernak [47], we use a trainedphoneme classifier with frozen weights to encode linguistic content.
The classifier consists of amulti-layer perceptron followed by two LSTM [20] layers and a linear classification layer, and takesas input 13 mel-frequency cepstral coefficients (MFCC) with first- and second-order deltas for eachframe.
We train our phoneme classifier on the "train-clean-100" subset of the LibriSpeech dataset[41] using frame-aligned phoneme labels [32, 33] (CC-BY 4.0 license).
Rather than directly use theclassifier’s predicted distributions over phoneme labels – known as phonetic posteriorgrams (PPGs)[18] – we discard the classification layer after training and pass along the output of the final LSTMlayer, which Ronssin & Cernak also refer to as PPGs in their architecture.
Spectrogram features : Finally, we use a simple network consisting of a gated linear unit andmulti-layer perceptron operating independently on each mel-spectrogram frame—a widely usedspeech representation [25]—to capture any residual information.
For each frame, the above features are concatenated and projected linearly to obtain a low-dimensionalencoding.
We then introduce additional speaker information by passing a fixed, pre-computedembedding of the source speaker through a FiLM layer [44] to modulate the encoder output.
Thisembedding helps to guide the de-identification task, allowing us to train a single model capable ofde-identifying arbitrary users.
This imposes a requirement that users must record a small amount ofspeech to obtain an embedding before first using VoiceBox.
We find that less than a minute of speechis sufficient, and use a pre-trained ResNetSE34V2 model [19] to compute embeddings (see sections4.1, 4.2).
3.2 Bottleneck
Encodings entering the bottleneck are passed through two LSTM layers, and the outputs are concatenated with a skip connection from the encoder.
To enable streaming, we use unidirectional
LSTM layers and pass concatenated outputs through a small lookahead convolutional network [55] toincorporate information from future frames at the expense of a small fixed delay.
We find a lookaheadof 5 frames (roughly 48 ms at our sample rate) is sufficient to craft strong de-identification attacks.
Finally, a linear projection layer maps the concatenated representations to a vector of filter controls,with each entry representing the unnormalized frequency magnitude response of a filter band for thecurrent frame.
3.3 Decoder
Our decoder applies time-varying filtering to the input audio based on frame-wise controls obtainedfrom the bottleneck.
For our task, we find that a filtering-based decoder affords a number ofadvantages over the synthesis-based (e.g. transposed-convolutional) decoder architectures present inmany audio-to-audio models [12]:
• The filtering module is not prone to the periodic upsampling artifacts that transposed-
convolutional architectures often introduce [45].
• Our decoder is a simple deterministic module with no trainable parameters, keeping VoiceBox lightweight.
• The capacity of the decoder and conspicuousness of perturbations can easily be constrained in terms of the number of filter bands or their allowed range of motion, providing interpretable control to the user.
To regularize and apply filter controls to each frame of audio, we use a method similar to that of
O’Reilly et al.
and Engel et al.
We apply sigmoid scaling to bound filter controls to therange [0 , 2] .
We clip deviations from unity beyond a fixed bound ϵ .
Each set of scaled filter controlsis transformed into a time-domain impulse response via the inverse Fourier transform.
We shift theimpulse response to zero-phase (symmetric) form, apply a Hann window, and finally convolve withthe corresponding input audio frame by taking the Fourier transform and performing element-wisemultiplication.
After compensating for shift, we overlap-add the resulting frames with a Hann windowto obtain the final filtered audio.
We discuss the details of our buffered streaming implementation in
A.3.
Our VoiceBox model is implemented in PyTorch [2] and contains 6.3m trainable parameters,and 7.5m in total counting the frozen phoneme encoder.
3.4 Training objective
To demonstrate the ability of VoiceBox to perform inconspicuous privacy-preserving audio transformations, we train our model to attack speaker recognition systems.
Given a large database ofspeech recordings and access to a user’s audio stream, a surveilling entity may seek to (a) identifythe user by matching their speech against recordings of known provenance in the database, or (b)retrieve other utterances of the user from the database.
Systems designed for these tasks – speakerrecognition and retrieval, respectively – often rely on neural network models to map speech utterancesto a low-dimensional embedding space in which distance corresponds to speaker similarity [14].
Inthis work, we consider models f for which the speaker distance D f between utterances u and v canbe measured via a cosine distance between embeddings f ( u ) and f ( v ) :
[Equation]
At inference time, query audio from the user is embedded and scored for similarity against enrolledutterances – pre-computed embeddings stored in the database.
Scores may be evaluated for allenrolled embeddings, or for only a representative of each unique speaker (e.g. speaker centroids).
The highest-ranking result or results (e.g. the closest speaker identity) are then returned.
We do notdistinguish between recognition and retrieval tasks, and refer to both under the umbrella of "speakerrecognition.
" This is because from a privacy standpoint, the objective in each task is identical: alterthe query audio to prevent valid matches, thereby de-identifying the user.
A variety of methods have been proposed to efficiently compute and search over low-dimensionalspeaker representations.
Generally, a hashing algorithm is applied to speaker embedding vectorsto reduce storage and search costs [49, 29, 14].
Because the resulting hash representation merelyserves as an efficient point of access for embedding-space distances, we omit hashing algorithmsfrom consideration and define our attack objectives on the embedding space directly.
Given a speaker embedding model, we aim to modify query audio drawn from a user’s streamsuch that its embedding-space distance from any enrolled utterances of the user is large, and itsevaluated similarity is small.
We quantify this de-identification in terms of distance thresholds inthe embedding space, set according to percentiles of the estimated distribution of all inter-speakerembedding distances.
Let P r represent the distance corresponding to the r th percentile of thisdistribution; then for query audio u and enrolled utterance v , D f ( u, v ) > P r implies that roughly rpercent of database entries should be scored as more similar to u than v .
Thus, we can set distancethresholds that correspond directly to the strength of de-identification applied to user audio, andconstruct a loss function that penalizes our model when the embedding-space cosine distance betweenquery and enrolled utterances falls below the threshold.
To do so, we use a variant of the adversarialloss proposed by Zhang et al.
Let f represent the victim model, g our VoiceBox network, u anutterance from our user’s audio stream, and P r our de-identification threshold; then L adv ( f, g, u [Equation] =
[Equation]
u
[Equation]
(2) where ( · ) + = max( · , 0) and κ is a confidence parameter encouraging the attack to fully cross thethreshold.
To ensure that our VoiceBox model learns to perturb user audio inconspicuously, weincorporate an additional loss function to penalize perceptible filtering artifacts.
We compute thecombined waveform L 1 and multi-resolution spectrogram losses proposed by Defosséz et al.
[12] onthe clean and adversarial audio:
[Equation]
where L stft is given by a sum of magnitude and spectral convergence losses computed over several spectrogram resolutions.
We provide further details in Appendix A.2.
Combining the aboveadversarial and auxiliary losses, we obtain our final attack objective:
[Equation]
Experimental design
We describe experiments used to validate the claimed advances of our work, namely that VoiceBoxcan de-identify speech in real-time while remaining significantly less conspicuous than existingmethods of similar effectiveness.
We first introduce the models, datasets, and attacks considered inour experiments.
Following this, we detail our experiment configurations and present the results ofboth objective and subjective evaluations.
4.1 Speaker recognition models
ResNetSE34v2: We train attacks against the ResNetSE34v2 model [19] provided in the
VoxCeleb Trainer repository [11] (MIT License).
The model takes mel-spectrogram inputs anduses 2D convolutions with residual connections, squeeze-and-excitation, and attentive statistics pooling to generate frame-level features and aggregate them into 512-dimensional speaker embeddings.
Y-Vector: To examine the transferability of our approach against unseen systems, we evaluatetrained attacks against the Y-Vector model proposed by Zhu et al.
The model uses a multiscale1D-convolutional waveform encoder to extract acoustic features, followed by squeeze-and-excitationblocks, feature aggregation, and a time-delayed neural network to map variable-length utterances to128-dimensional speaker embeddings.
Both the ResNetSE34v2 and Y-Vector models were trained on the development set of the VoxCeleb2dataset [10].
Note that while our attack is causal—modulo a short, five-frame lookahead—we do notimpose the same restriction on a hypothetical surveillance system; instead, we evaluate our attackagainst strong, non-causal systems capable of aggregating speaker characteristics from across fullutterances before rendering predictions.
4.2 Datasets
LibriSpeech: (CC-BY 4.0)
We use both the train-clean-100 and test-clean subsets of the
LibriSpeech dataset [41] for training VoiceBox.
The former comprises 28,539 utterances from 251speakers while the latter comprises 2,620 utterances from 40 speakers.
We trim or pad all utterancesto 4 seconds.
VoxCeleb1: (CC-BY 4.0) To simulate large-scale surveiling speaker recognition, we evaluate attackson the VoxCeleb1 dataset [38], comprising 153,516 utterances from 1,251 speakers.
This also ensuresthat no evaluation speakers are seen during training.
As with the LibriSpeech dataset, we trim or padall utterances to 4 seconds.
For all experiments, we carefully divide the data to imitate a realistic attack setting.
During training,we select fifteen utterances (one minute total) from each source speaker in the training set and computeembeddings using the ResNetSE34v2 (MIT License) model.
The centroid of these embeddings isthen used as an enrolled target for the computation of the adversarial loss with all utterances of thatspeaker.
We find that this produces stronger attacks than using individual utterance embeddings astargets, possibly by ensuring more consistent gradient information across the optimization.
For our
VoiceBox attack (see Section 4.3), we select a further ten utterances (40s total) from each sourcespeaker in the training set and again compute embeddings using the ResNetSE34v2 model.
Thecentroid of these embeddings is then fed as a fixed conditioning vector to the VoiceBox modelalongside all utterances from that speaker (see Section 3.1).
Similar to training, during evaluation weselect fifteen utterances per speaker as a query set.
We again select a further ten utterances to serve asconditioning for the VoiceBox attack.
Finally, twenty utterances of each speaker are enrolled in thespeaker recognition system, serving as the database against which query utterances are matched.
4.3 Attack algorithms
We perform untargeted attacks using the following algorithms.
VoiceBox: We implement theproposed VoiceBox attack as described in Section 3 and Appendix A and train for 10 epochs on
NVidia RTX 2080 Ti GPUs; this takes approximately 40 minutes.
Universal: We optimize a short(2s) universal additive perturbation for 10 epochs using the established penalty method [31, 60, 26]and the same adversarial objective as VoiceBox.
On 3 NVidia RTX 2080 Ti GPUs, this takesapproximately 16 minutes.
To limit the perceptibility of the attack, we scale the perturbation to have
L ∞ norm 0.08 times that of the unperturbed speech.
During training and evaluation, the perturbationis aligned arbitrarily with query utterances and looped to match durations, serving as a constantadversarial "background" signal.
White noise: We add Gaussian noise to utterances at the waveformrepresentation at a signal-to-noise ratio of − 10 dB. Spectral gating:
We modify the "Kenansville"attack of Abdullah et al.
[1] to allow for streaming use by performing spectral gating at all frames,using a threshold of 4dB relative to the maximum-energy spectral bin of each frame.
4.4 Objective evaluation
We evaluate attacks in a large-scale closed-set speaker recognition task.
First, we use the test-clean
LibriSpeech subset to obtain a rough estimate of the distribution of distances between an individualutterance and the centroids of each distinct speaker in the embedding space of the ResNetSE34V2model.
To encourage strong de-identification attacks, we take the distance corresponding to the 25 thpercentile of this distribution as the target threshold P 25 for our training loss (see Section 3.4).
Wetrain each attack as discussed above.
We evaluate the closed-set recognition of all attacks over the VoxCeleb1 dataset.
We computethe distance between each query embedding and the centroid of all embeddings of each speaker;recogntion is then performed by returning the speaker identity of the nearest speaker centroid.
Wefind this is slightly more accurate and robust to attack than using the identity of the nearest embeddedutterance.
We perform exact nearest-neighbors search over the embedding space using FAISS[24] (MIT license), and report the top-1 (T-1) and top-10 (T-10) accuracy of the relevant speakerrecognition model given both clean and adversarial queries.
Additionally, we compute the followingobjective speech quality metrics over the clean and adversarial query audio as a proxy measure ofthe imperceptibility of attacks: Perceptual Evaluation of Speech Quality (PESQ) [46], operatingin the wide-band configuration and using the python-pesq implementation [56] (MIT license); [Table] and Short-Time Objective Intelligibility (STOI) [53], using the pystoi implementation [42] (MITlicense).The results of our evaluation are presented in table 1.
We evaluate the real-time performance of the streaming implementation of the VoiceBox attackdescribed in Appendix A.3 by measuring its average real-time factor (RTF).
We measure performanceon a single thread on two different CPUs, an Intel i7-5600U @ 3.2 GHz and an Apple M1 Chip.
In thestreaming configuration, VoiceBox processes a chunk of 4 overlapping frames at a time, equivalent to640 samples / 40 ms.
We compute the average RTF for processing a chunk over all chunks in a 4second audio clip, and find that VoiceBox has an average RTF of .255 on the Intel i7-5600U and .200on the Apple M1.
4.5 Subjective evaluation
For our subjective evaluation, we sampled 100 clean speech recordings from the VoxCeleb1 dataset.
Each one of the four attacks described in Section 4.3 was applied to all 100 recordings, resulting in100 comparison sets with five recordings per set: the original clean speech and the speech modifiedby each of the four attacks.
These sets were then evaluated in a MUSHRA-style listening study [8]deployed on Amazon Mechanical Turk using the open-source, MIT-licensed Reproducible Subjective
Evaluation (ReSEval) [37] system.
IRB approval was obtained prior to conducting this study, andthere are no known risks to the participants in this study.
We recruited 20 participants.
Participants were screened with a listening test prior to beginning thestudy. Each participant that passed the screening rated 20 comparison sets.
For each comparison set,the participant was asked to listen to and rate the relative audio quality of each of the five audio fileson a scale from 0 to 100.
We omitted responses by four participants who failed our prescreeninglistening test and nine who rated the white noise attack (our low anchor) as superior in quality toground-truth clean audio (our high anchor), giving us a total of 140 five-way comparisons.
For moredetails of our crowdsourced subjective evaluation, see Appendix C.
The results of our crowdsourced subjective evaluation can be found in Figure 2.
We find VoiceBoxis preferred to all other methods, and exhibits similar perceptual quality to ground-truth speechrecordings.
The difference between VoiceBox and ground-truth audio is significant ( p < 0 . 05)
usinga Wilcoxon signed-rank test ( p = 0 . 024) but not significant using Welch’s T-test ( p = 0 . 071) , whichassumes that human perceptual scores are normally distributed but with potentially different variances.
All other pairs of conditions are significantly different using either test.
4.6 Additional experiments
We conduct supplementary experiments demonstrating the robustness of VoiceBox against a deepnetwork-based speech enhancement model and examining its performance under the assumption thatadversarial queries are enrolled by the surveiling speaker recognition system.
These experiments aredetailed in appendices B.1 and B.2, respectively.
Discussion
Our experimental results indicate VoiceBox can de-identify speech from arbitrary unseen users inreal-time on a standard M1 CPU.
The de-identified speech is of significantly higher audio qualitythan competing methods, as reported by a subjective listener study and as measured through standardmetrics of speech intelligibility.
Our method also achieves nontrivial de-identification results againsta system it was not trained on, outperforming a far more perceptible universal attack.
This isnotable given that we take no explicit steps to improve the transferability of our method.
Whilepure signal-processing approaches such as spectral gating and white noise transfer between systemsmore successfully, they severely degrade audio quality, resulting in word-error-rates 10 to 20 timeshigher than VoiceBox and much lower listener quality ratings.
This limits their practicality in realworld voice communications.
By contrast, our method produces adversarial audio that is virtuallyindistinguishable from the clean source, as indicated by our subjective evaluation.
We believe the benefits of real-time, imperceptible de-identification attacks for user privacy areself-evident. Our same approach, however, could be argued to hamper legitimate targeted surveillance(e.g. tracking of the communications of criminal organizations). This ability to evade trackingis, however, limited by the fact that VoiceBox’s output is perceptually quite close to the originalspeech.
Therefore, a human listener would still be able to readily identify the speaker.
As such, theeffect of VoiceBox would simply be to return us to the status quo that existed prior to automatedlarge-scale speaker recognition of requiring manual human evaluation.
VoiceBox could also, inconcept, be used to fraudulenty access information or services protected by speaker verification.
However, since VoiceBox performs perceptually inconspicuous filtering, the speech produced wouldstill sound like the original speaker and could not pass human inspection.
Both voice-conversionand speech synthesis-based attacks produce speech that is perceptually similar to the target andare, therefore, much more suited to this purpose.
Our approach can be thought of as leveraging theasymmetry between system and attacker attention inherent to mass surveillance, using inconspicuousperturbations to evade large-scale systems that must render predictions in bulk.
This same asymmetryis not necessarily present when trying to bypass authentication mechanisms and access tightly-guardedservices.
We view our method as an initial step towards protecting users from indiscriminate mass surveillance.
A number of obvious directions for future work stand out, such as improving the transferability ofadversarial examples [39], and evaluating attacks against real-world speaker recognition systems andover real-world communication channels.
We hope this work encourages further exploration of theapplications of audio-to-audio models for protecting user privacy.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We claim to have designed a system to make highly
effective, inconspicuous perturbations that can transfer to unforseen architectures in
real-time.
We show this in our objective 4.4and subjective evaluations 4.
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] It will be
anonymized in the supplemental materials
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We provide a distribution of subjective quality ratings
for each attack in in 2.
We, however, do not provide error bars over random seeds for
training.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Stated in 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite authors of
LibriSpeech, VoxCeleb, ResNetSE34v2, ReSEval and Y-Vector
(b) Did you mention the license of the assets?
[Yes] We mention the licenses for datasets
and models in 4.2 and for ReSEval in 4.5.
We note that the Y-Vector model does not
have a License, to our knowledge.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes]
In the Supplementary Materials
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes] We obtained IRB approval.
There are no known risks to the participants in out subjective evaluation.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[Yes] In C
A VoiceBox implementationA.1 Architecture
We provide additional details of the VoiceBox architecture used in our experiments.
Phoneme encoder:
The phoneme encoder consists of a feedforward network with two hiddenlayers of size 256 and ReLU activations, followed by two LSTM layers with hidden size 256.
Usingthe LibriSpeech "train-clean-100" dataset and aligned phoneme labels provided by Lugosch et al.[32], we train the phoneme encoder with cross entropy loss for 25 epochs using the Adam optimizerwith learning rate 1e-3.
During training, a linear layer is appended to the phoneme encoder to renderpredictions over phoneme labels; we discard this layer after training.
Spectrogram encoder:
We pass 64-bin mel-spectrograms through a 1 × 1 convolutional layer andgated linear unit (GLU) operating along the frequency axis, followed by a two linear layers with leaky
ReLU activations and hidden size 512.
The convolution layer upsamples the channel (frequency)dimension to 1024, from which the GLU obtains a 512-dimensional representation to pass to thelinear layers.
Source speaker conditioning: For each frame of audio, we concatenate the 256-dimensionaloutput of the phoneme encoder, the 512-dimensional output of the spectrogram encoder, and the onedimensional pitch / periodicity / loudness features and linearly project to obtain a 512-dimensionalvector.
A fixed 512-dimensional source speaker embedding obtained from the ResNetSE34V2 modelis passed through a 2 linear layers with leaky ReLU activations and hidden size 512, followed by batchnormalization and a final linear layer to obtain a 1024-dimensional vector holding scales and biasesfor a feature-wise affine transformation (FiLM) of the encoder output.
The affine transformation isapplied and the resulting vector is passed to the bottleneck.
Bottleneck: Our bottleneck consists of two LSTM layers with hidden size 512.
The encoder outputis passed through the LSTM layers with a skip connection concatenated at the output.
Following theapplication of a lookahead convolutional network, the resulting vector is projected linearly to obtaina 128-dimensional vector of filter controls for the current frame.
Decoder:
We perform filtering with 128 bands per frame, and set L ∞ bound ϵ = 0 .
5 to furtherconstrain deviations from a "neutral" control configuration of ⃗ 1 .
We experimented with alternativemethods of regularizing predicted filter controls, such as L 2 normalization, but found that simple L ∞clipping yielded satisfactory results.
Training:
We train both VoiceBox and the universal baseline attack using the Adam [28] optimizerwith learning rate 1e − 4 for 10 epochs over the LibriSpeech "train-clean" subset.
A.2 Auxiliary loss
We briefly summarize the multi-resolution STFT loss of Defosséz et al.
[12] used in the training ofour attacks.
The loss is given by
[Equation]
(5) where L stft is given by a sum of magnitude and spectral convergence losses computed over Mspectrogram resolutions:
[Equation]
[Equation]
Here || · || F and || · || 1 refer to the Frobenius and L 1 norms, respectively.
Following Defosséz et al.
,we use M = 3 spectrogram resolutions with hop sizes of 50, 120, and 240 samples, FFT lengths of512, 1024, and 2048, and window lengths of 240, 600, and 1200, respectively.
A.3 Streamer
We implement a version of VoiceBox for processing audio streams in real time.
All components aremodified to support the computation of perturbations on overlapping windows of audio while theinput and output streams process non-overlapping chunks.
Where possible, components are compliedto TorchScript to improve performance.
Three buffers are added to process a stream of audio intooverlapping chunks:
• Last Frame Input Buffer: Contains the last 128 samples of the last input audio to be appendedto the oncoming chunk, before computing the overlapping windows.
• Lookahead Buffer: Contains overlapping frames to be used as lookahead.
• Output Buffer: Contains the last 128 samples output by VoiceBox, to be overlap-added withthe next processed chunk.
Code for our streaming implementation can be found at https://master.d3hvhbnf7qxjtf.amplifyapp.com/ .
[Table]
B Additional experimentsB.1 Attack robustness to preprocessing
In real-world settings, user audio may travel through various preprocessing stages before reaching aspeaker recognition model.
To simulate the presence of such stages, we perform attacks using theexperiment configuration discussed in Section 4.3 but pass query utterances through a pretrained
Demucs [12] speech enhancement model en route to the ResNetSE34v2 recognition system; resultsare reported in table 2.
The speaker recognition system maintains its top-1 and top-10 accuracy onclean queries.
In contrast to Chiquier et al.
[9] we do not incorporate the enhancement model into ouradversarial optimization, meaning that Demucs essentially functions as an unseen adversarial defense.
We find that VoiceBox loses almost no effectiveness when queries are passed through Demucs; bycomparison, the universal attack loses most of its effectiveness, as Demucs is able to separate thelow-magnitue but noisy perturbation from clean speech.
Both the white noise and spectral gatingattacks retain their effectiveness, presumably because they degrade audio beyond Demucs’ capabilityto provide coherent reconstructions of the original speech.
We leave as future work the explorationof the robustness of VoiceBox against more sophisticated preprocessing pipelines and adversarialdefenses.
B.2 Enrollment of adversarial queries
It is possible that adversarially-perturbed query utterances extracted from a VoiceBox user’s audiostream may themselves be enrolled by a surveiling speaker recognition system.
In such cases, thereare two possibilities:
VoiceBox successfully de-identifies user speech, and adversarial queries are enrolled as aseparate speaker profile from any existing clean utterances of the user2.
VoiceBox fails to de-identify user speech, and adversarial queries are incorporated into anexisting profile of the user containing clean utterances.
We examine both scenarios, again using the VoiceBox attack discussed in Section 4.3.
Results arepresented in table 3.
We find that while VoiceBox is highly effective at de-identifying users againsta profile constructed from clean (unperturbed) utterances, its de-identification performance sufferssignificantly under both of the aforementioned conditions.
This suggests that additional work isrequired to ensure that a VoiceBox user remains a "moving target" to surveilling speaker recognitionsystems.
One possible solution is to leverage targeted rather than untargeted attacks: by proactively"spoofing" specific (random) locations in the embedding space, VoiceBox may hamper the
creation
of a single matching enrolled profile.
C Listening Study
The listening study consists of 20 evaluation tasks, where the participants are shown the followingtext, along with 5 four second long audio examples and 5 corresponding sliders with values from 0 to100:
[Table]
’unnatural’ has
It takes approximately 15 minutes to complete all the evaluation tasks.
Participants are also asked tocomplete a listening test before proceeding to the audio evaluation.
The listening test involves theparticipant listening to two audio files, and reporting the number of tones they heard.
For each audiofile, the participants are given 3 tries to correctly report the number of tones played.
If a participantsucceeds in the listening test and completes the evaluation tasks, then they are paid 3.00 USD, orequivalently, 12.00 USD / hour.
If the participant fails the listening test, then they are paid 0.50 USD.