VoiceBlock: Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models
Abstract
As governments and corporations adopt deep learning systems to collect and analyze user-generated audio data, concerns about security and privacy naturally emerge in areas such as automatic speaker recognition.
While audio adversarial examples offer one route to mislead or evade these invasive systems, they are typically crafted through time-intensive offline optimization, limiting their usefulness in streaming contexts.
Inspired by architectures for audio-toaudio tasks such as denoising and speech enhancement, we propose a neural network model capable of adversarially modifying a user’s audio stream in realtime.
Our model learns to apply a time-varying finite impulse response (FIR) filter to outgoing audio, allowing for effective and inconspicuous perturbations on a small fixed delay suitable for streaming tasks.
We demonstrate our model is highly effective at de-identifying user speech from speaker recognition and able to transfer to an unseen recognition system.
We conduct a perceptual study and find that our method produces perturbations significantly less perceptible than baseline anonymization methods, when controlling for effectiveness.
Finally, we provide an implementation of our model capable of running in real-time on a single CPU thread.
Audio examples and code can be found at https://interactiveaudiolab.github.io/project/voiceblock.html .
Introduction
Mass surveillance of voice communications is an ongoing and pervasive issue.
While section 702 of the United States Foreign Intelligence Surveillance Act allows the government to perform targeted monitoring of foreign communications, bulk collection practices have resulted in the warrantless surveillance of large numbers of "incidental" foreign and domestic individuals [17].
Despite the fact that millions of these communications are obtained without warrants, they have been used in ordinary criminal investigations [60], undermining a core purpose of the Fourth Amendment of the US constitution [59]: to protect the people against searches without probable cause.
Many corporations also possess the capability for large-scale collection of voice data [23], which may be leveraged to profile users for advertising or accessed by government entities through upstream and downstream surveillance [16].
As individuals are faced with these growing surveillance apparati, it is worth remembering that routine surveillance of private voice communications is also corrosive to free speech and association and tends to disproportionately affect marginalized groups [28, 18].
In the absence of identifying metadata, automatic speaker recognition systems can facilitate mass surveillance by allowing an operator to search a database of recorded voice data for utterances from a chosen speaker [7] or to diarise (assign utterances to individuals) transcripts of recordings [44].
Prior to the advent of automatic speaker recognition these tasks required human analysts, forming a natural check on surveillance overreach.
We seek to restore this check by degrading the efficacy of speaker recognition models while maintaining the original perceptual quality of the voice communication, a step that could grant users a measure of privacy from mass surveillance.
Modern speaker recognition systems rely on deep neural networks [6].
Deep networks have been shown to be vulnerable to adversarial examples—natural instances (e.g. a recording of "Bob" speaking) modified to cause a model to make an incorrect prediction (the recording’s speaker is labeled as "Maria")
This presents opportunities for privacy-minded individuals to mislead or evade surveillance systems with adversarially-crafted inputs.
Researchers have proposed adversarial attacks against a variety of audio systems, including speaker recognition [62, 51].
To fool a given neural network-based system, many audio attacks modify a recording by adding a perturbation signal directly at the waveform representation [8].
This perturbation is typically crafted using gradient information obtained from the system’s neural network—or from a similarlyconstructed surrogate—and requires iterative optimization.
Once the optimization is complete, the modified recording is played to the system in an effort to induce arbitrary incorrect predictions (an untargeted attack) or a specific incorrect prediction chosen by the attacker (a targeted attack).
In theory, such attacks might allow individuals to evade systems that surveil and analyze voice communications, thereby protecting privacy.
However, many existing attack algorithms require a costly optimization for each audio recording to which they are applied.
Continuous, real-time voice communication precludes the adoption of such approaches through online optimization (which is typically too slow) or through the use of a set of precomputed adversarial examples (which would necessarily limit the user’s interaction).
This suggests a need for algorithms capable of modifying speech on-the-fly.
Recent years have seen the development of models for audio-to-audio tasks such as denoising, voice conversion, and musical timbre transfer [13, 48, 14].
Such models have sufficient expressive power to modify audio in coherent and task-specific ways, and are often designed to run in real-time.
Inspired by these models, we propose VoiceBlock , a deep network that learns to apply a time-varying finite impulse response (FIR) filter to outgoing audio, producing highly effective adversarial perturbations on a small fixed delay suitable for streaming.
The main contributions of this work are:
• A highly effective method for de-identifying speech in real-time that is more perceptually inconspicuous than existing methods of equal effectiveness • Objective and subjective experimental results that validate these claims • A system that embodies the claimed advances, and that runs in real-time on a single CPU thread
Through this work, we hope to encourage further exploration of audio-to-audio models for protecting user privacy.
Audio examples and code, including our streaming implementation, can be found at https://interactiveaudiolab.github.io/project/voiceblock.html .
Related work
Previous works in speech de-identification have proposed methods for obfuscating speech to evade surveilling speaker-recognition systems.
One such method is voice conversion, which modifies the speech of a source speaker to sound like that of another target speaker, both to humans and machines.
Jin et al.
[24] and Alegre et al.
[4] analyze the vulnerability of speaker recognition systems against de-identification attacks performed with voice conversion models.
However, the conversion models considered are incapable of real-time operation.
While the recent availability of low-latency voice conversion models [48, 49, 31] may make such approaches more practical, our aim is to modify speech in a way that is inconspicuous and minimally invasive to the user experience, and that has the potential to generalize to tasks beyond speaker recognition.
This also rules out recent anonymization approaches catalogued through the VoicePrivacy Challenge, as all evaluated submissions to date noticeably alter speaker characteristics [55].
Similarly, we do not consider obvious transformations like pitch-shifting that significantly alter or degrade recorded audio.
Real-time audio attacks have recently been explored.
Chiquier et al.
[10] propose an attack on automatic speech recognition systems that is capable of inducing significant transcription errors in an over-the-air environment (where attack audio is played to the system in a physical space), but which introduces conspicuous noise due to the use of additive perturbations at the audio waveform.
Their approach also necessitates both a large model (requiring approximately 16 GPU-days to train) and lengthy receptive field, resulting in an initial “idle" period of 2.5s during which the attack does not affect speech.
By contrast, we focus on an over-the-line setting, where attacks are passed to the victim model over a digital channel, and propose a lightweight network capable of producing inconspicuous perturbations through the use of filtering, with an initial delay of miliseconds rather than seconds.
Universal adversarial attacks optimize a short perturbation that can be played alongside arbitrary speech, optionally in real-time, to evade a victim model [32, 62, 27].
However, these attacks tend to introduce conspicuous noise due to the use of additive perturbations at the audio waveform.
The same holds for the attack of Xie et al.
[61], which uses a Wave-U-Net [52] model to efficiently generate universal perturbations.
Rather than generate additive perturbations, we perform multiplicative attacks in the frequency domain through the use of filtering, and in doing so avoid the bias towards noise-like artifacts typical of additive attacks.
Other works have proposed crafting adversarial attacks in the frequency domain .
The “Kenansville" attack of Abdullah et al.
[1] performs spectral gating, removing low-energy frequency components in an effort to hinder classification.
The authors demonstrate the effectiveness of their approach against seen and unseen speaker- and speech-recognition systems, and the proposed method is fast and gradient-free.
However, the spectral gating must be constrained to a small number of key audio frames to avoid conspicuous artifacts, which requires offline optimization using word- or phoneme-level alignment information.
Ahmed et al.
[3] optimize bandpass filters to perform targeted impersonation attacks on speakerrecognition models; these filters can then be physically realized as resonant tubes through which an attacker can speak to the victim system, allowing real-time operation.
However, once optimized and realized, the bandpass filters are fixed in time and cannot adapt to the attacker’s speech.
The attack is significantly less effective than traditional approaches, even in controlled acoustic environments, and requires an active intervention on the part of the user to both realize and apply.
Finally, O’Reilly et al.
[41] adversarially optimize the parameters of a time-varying finite impulse response (FIR) filter in order to avoid noisy artifacts.
However, the attack operates offline and must be optimized separately for every instance to which it is applied.
In summary, we are aware of no current approach that is simultaneously capable of performing deidentification while remaining inconspicuous to human listeners and operating in real-time on-device.
VoiceBlock
We propose VoiceBlock, a system that applies adversarial time-varying filtering to user audio in real-time.
VoiceBlock modifies speech by specifying the frequency magnitude response of a standard finite impulse response (FIR) filter.
By carefully varying the filter’s response many times per second, the speech is de-identified to an automated system while speaker identity is preserved for human listeners.
This is accomplished without introducing noisy artifacts at the waveform, as filtering can only amplify or attenuate frequency energy present in the original signal through multiplication in the
Fourier domain (see Section 3.3 for details of the filtering implementation used in our experiments).
VoiceBlock consists of three main modules, shown in Figure 1: (1) an encoder module extracts acoustic features from input audio frames, (2) a recurrent bottleneck module incorporates context from past frames to predict a set of adversarial filter controls for each frame, and (3) a decoder module regularizes the predicted filter controls and applies them to the corresponding frame of input audio.
We provide an overview of each component below, and further details in Appendix A.
3.1 Encoder
The encoder module extracts acoustic features from 16kHz audio (double the sample rate of telephone speech) segmented into frames of 256 samples with 50% overlap.
Though our model can operate in fully causal fashion, we find we achieve stronger attacks using a lookahead of five frames (see Section 3.2), resulting in a theoretical latency of 56ms.
For reference, Verizon reports 25ms or less is typical latency for voice-over-IP communication in North America and Europe [56], while experimental data [26] show that 130ms of latency is acceptable for high-quality voice communications.
Therefore, adding VoiceBlock to the signal chain leaves latency well within this limit.
Audio frames entering the encoder are passed to four sub-modules in parallel to obtain the following features.
Pitch features : Given the known sensitivity of speaker-recognition models to pitch [3] and the importance of spectral structure in delineating linguistic content, we extract fundamental frequency and aperiodicity estimates for each frame using the DIO algorithm
Pitch estimates are refined by the StoneMask algorithm [37].
For both stages, we use the pyworld [22] implementation of the
WORLD vocoder [37] (MIT license).
Loudness : We take an A-weighted average of each frame’s log-magnitude spectrum to obtain a loudness estimate [35].
Phonetic posteriorgrams : Following the method of Ronssin & Cernak [48], we use a trained phoneme classifier with frozen weights to encode linguistic content.
The classifier consists of a multi-layer perceptron followed by two LSTM [21] layers and a linear classification layer, and takes as input 13 mel-frequency cepstral coefficients (MFCC) with first- and second-order deltas for each frame.
We train our phoneme classifier on the "train-clean-100" subset of the LibriSpeech dataset [42] using frame-aligned phoneme labels [33, 34] (CC-BY 4.0 license).
Rather than directly use the classifier’s predicted distributions over phoneme labels – known as phonetic posteriorgrams (PPGs) [19] – we discard the classification layer after training and pass along the output of the final LSTM layer, which Ronssin & Cernak also refer to as PPGs in their architecture.
Spectrogram features : Finally, we use a simple network consisting of a gated linear unit and multi-layer perceptron operating independently on each mel-spectrogram frame—a widely used speech representation [25]—to capture any residual information.
We note that Engel et al.
[14] also make use of a spectrogram encoder, alongside pitch and loudness features, to control differentiable signal-processing components.
For each frame, the above features are concatenated and projected linearly to obtain a low-dimensional encoding.
We then introduce additional speaker information by passing a fixed, pre-computed embedding of the source speaker through a FiLM layer [45] to modulate the encoder output.
This embedding helps to guide the de-identification task, allowing us to train a single model capable of de-identifying arbitrary users.
This imposes a requirement that users must record a small amount of speech to obtain an embedding before first using VoiceBlock.
We find that less than a minute of speech is sufficient, and use a pre-trained ResNetSE34V2 model [20] to compute embeddings (see sections 4.1, 4.2).
3.2 Bottleneck
Encodings entering the bottleneck are passed through two LSTM layers, and the outputs are concatenated with a skip connection from the encoder.
To enable streaming, we use unidirectional
LSTM layers and pass concatenated outputs through a small lookahead convolutional network [57] to incorporate information from future frames at the expense of a small fixed delay.
We find a lookahead of 5 frames (48 ms) is sufficient to craft strong de-identification attacks.
Finally, a linear projection layer maps the concatenated representations to a vector of filter controls, with each element representing the unnormalized frequency magnitude response of a filter band for the current frame.
3.3 Decoder
Our decoder applies time-varying filtering to the input audio based on frame-wise controls obtained from the bottleneck.
For our task, we find that a filtering-based decoder affords a number of advantages over the synthesis-based (e.g. transposed-convolutional) decoder architectures present in many audio-to-audio models [13]:
• The filtering module is not prone to the periodic upsampling artifacts that transposedconvolutional architectures often introduce [46].
• Our decoder is a simple deterministic module with no trainable parameters, keeping VoiceBlock lightweight.
• The capacity of the decoder and conspicuousness of perturbations can easily be constrained in terms of the number of filter bands or their allowed range of motion, providing interpretable control to the user.
To regularize and apply filter controls to each frame of audio, we use a method similar to that of
O’Reilly et al.
and Engel et al.
We apply sigmoid scaling to bound filter controls to the range [0 , 2] and clip deviations from unity beyond a fixed value ϵ .
Each set of scaled filter controls is transformed into a time-domain impulse response via the inverse Fourier transform.
We shift the impulse response to zero-phase (symmetric) form, apply a Hann window, and finally convolve with the corresponding input audio frame by taking the Fourier transform and performing element-wise multiplication.
After compensating for shift, we overlap-add the resulting frames with a Hann window to obtain the final filtered audio.
We discuss the details of our buffered streaming implementation in
Appendix A.3.
Our VoiceBlock model is implemented in PyTorch [2] and contains 6.3m trainable parameters, and 7.5m in total counting the frozen phoneme encoder.
3.4 Training objective
To demonstrate the ability of VoiceBlock to perform inconspicuous privacy-preserving audio transformations, we train our model to attack speaker recognition systems.
Given a large database of speech recordings and access to a user’s audio stream, a surveilling entity may seek to (a) identify the user by matching their speech against recordings of known provenance in the database, or (b) retrieve other utterances of the user from the database.
Systems designed for these tasks – speaker recognition and retrieval, respectively – often rely on neural network models to map speech utterances to a low-dimensional embedding space in which distance corresponds to speaker similarity [15].
In this work, we consider models f for which the speaker distance D f between utterances u and v can be measured via a cosine distance between embeddings f ( u ) and f ( v ) :
[Equation]
At inference time, query audio from the user is embedded and scored for similarity against enrolled utterances – pre-computed embeddings stored in the database.
Scores may be evaluated for all enrolled embeddings, or for only a representative of each unique speaker (e.g. speaker centroids).
The highest-ranking result or results (e.g. the closest speaker identity) are then returned.
We do not distinguish between recognition and retrieval tasks, and refer to both under the umbrella of "speaker recognition.
" This is because from a privacy standpoint, the objective in each task is identical: alter the query audio to prevent valid matches, thereby de-identifying the user.
A variety of methods have been proposed to efficiently compute and search over low-dimensional speaker representations.
Generally, a hashing algorithm is applied to speaker embedding vectors to reduce storage and search costs [50, 30, 15].
Because the resulting hash representation merely serves as an efficient point of access for embedding-space distances, we omit hashing algorithms from consideration and define our attack objectives on the embedding space directly.
Given a speaker embedding model, we aim to modify query audio drawn from a user’s stream such that its embedding-space distance from any enrolled utterances of the user is large, and its evaluated similarity is small.
We quantify this de-identification in terms of distance thresholds in the embedding space, set according to percentiles of the estimated distribution of all inter-speaker embedding distances.
Let P r represent the distance corresponding to the r th percentile of this distribution; then for query audio u and enrolled utterance v , D f ( u, v ) > P r implies that roughly r percent of database entries should be scored as more similar to u than v .
Thus, we can set distance thresholds that correspond directly to the strength of de-identification applied to user audio, and construct a loss function that penalizes our model when the embedding-space cosine distance between query and enrolled utterances falls below the threshold.
To do so, we use a variant of the adversarial loss proposed by Zhang et al.
Let f represent the victim model, g our VoiceBlock network, u an utterance from our user’s audio stream, and P r our de-identification threshold; then
[Equation]
where ( · ) + = max( · , 0) and κ is a confidence parameter encouraging the attack to fully cross the threshold.
To ensure that our VoiceBlock model learns to perturb user audio inconspicuously, we incorporate an additional loss function to penalize perceptible filtering artifacts.
We compute the combined waveform L 1 and multi-resolution spectrogram losses proposed by Defosséz et al.
[13] on the clean and adversarial audio:
[Equation]
where L stft is given by a sum of magnitude and spectral convergence losses computed over several spectrogram resolutions.
We provide further details in Appendix A.2.
Combining the above adversarial and auxiliary losses, we obtain our final attack objective:
[Equation]
Experimental design
We describe experiments used to validate the claimed advances of our work, namely that VoiceBlock can de-identify speech in real-time while remaining significantly less conspicuous than existing methods of similar effectiveness.
We first introduce the models, datasets, and attacks considered in our experiments.
Following this, we detail our experiment configurations and present the results of both objective and subjective evaluations.
4.1 Speaker recognition models
ResNetSE34v2: We train attacks against the ResNetSE34v2 model [20] provided in the
VoxCeleb Trainer repository [12] (MIT License).
The model takes mel-spectrogram inputs and uses 2D convolutions with residual connections, squeeze-and-excitation, and attentive statistics pooling to generate frame-level features and aggregate them into 512-dimensional speaker embeddings.
Y-Vector: To examine the transferability of our approach against unseen systems, we evaluate trained attacks against the Y-Vector model proposed by Zhu et al.
The model uses a multiscale 1D-convolutional waveform encoder to extract acoustic features, followed by squeeze-and-excitation blocks, feature aggregation, and a time-delayed neural network to map variable-length utterances to 128-dimensional speaker embeddings.
Both the ResNetSE34v2 and Y-Vector models were trained on the development set of the VoxCeleb dataset [11].
Note that while our attack is causal—modulo a short, five-frame lookahead—we do not impose the same restriction on a hypothetical surveillance system; instead, we evaluate our attack against strong, non
4.2 Datasets
LibriSpeech: (CC-BY 4.0)
We use both the train-clean-100 and test-clean subsets of the
LibriSpeech dataset [42] for training VoiceBlock.
The former comprises 28,539 utterances fromspeakers while the latter comprises 2,620 utterances from 40 speakers.
VoxCeleb1: (CC-BY 4.0) To simulate large-scale surveiling speaker recognition, we evaluate attacks on the VoxCeleb1 dataset [39], comprising 153,516 utterances from 1,251 speakers.
This also ensures that no evaluation speakers are seen during training.
As with the LibriSpeech dataset, we trim or pad all utterances to 4 seconds.
For all experiments, we carefully divide the data to imitate a realistic attack setting.
During training, we select fifteen utterances (one minute total) from each source speaker in the training set and compute embeddings using the ResNetSE34v2 (MIT License) model.
The centroid of these embeddings is then used as an enrolled target for the computation of the adversarial loss with all utterances of that speaker.
We find that this produces stronger attacks than using individual utterance embeddings as targets, possibly by ensuring more consistent gradient information across the optimization.
For our
VoiceBlock attack (see Section 4.3), we select a further ten utterances (40s total) from each source speaker in the training set and again compute embeddings using the ResNetSE34v2 model.
The centroid of these embeddings is then fed as a fixed conditioning vector to the VoiceBlock model alongside all utterances from that speaker (see Section 3.1).
Similar to training, during evaluation we select fifteen utterances per speaker as a query set.
We again select a further ten utterances to serve as conditioning for the VoiceBlock attack.
Finally, twenty utterances of each speaker are enrolled in the speaker recognition system, serving as the database against which query utterances are matched.
4.3 Attack algorithms
We perform untargeted attacks using the following algorithms.
VoiceBlock:
We implement the proposed VoiceBlock attack as described in Section 3 and Appendix A and train for 10 epochs on[-
-]
NVidia RTX 2080 Ti GPUs; this takes approximately 40 minutes.
Universal: We optimize a short (2s) universal additive perturbation for 10 epochs using the established penalty method [32, 62, 27] and the same adversarial objective as VoiceBlock.
On 3 NVidia RTX 2080 Ti GPUs, this takes approximately 16 minutes.
To limit the perceptibility of the attack, we scale the perturbation to have
L ∞ norm 0.08 times that of the unperturbed speech.
During training and evaluation, the perturbation is aligned arbitrarily with query utterances and looped to match durations, serving as a constant adversarial "background" signal.
White noise: We add Gaussian noise to utterances at the waveform representation at a signal-to-noise ratio of − 10 dB. Spectral gating:
We modify the "Kenansville" attack of Abdullah et al.
[1] to allow for streaming use by performing spectral gating at all frames, using a threshold of 4dB relative to the maximum-energy spectral bin of each frame.
4.4 Objective evaluation
We evaluate attacks in a large-scale closed-set speaker recognition task.
First, we use the test-clean
LibriSpeech subset to obtain a rough estimate of the distribution of distances between an individual utterance and the centroids of each distinct speaker in the embedding space of the ResNetSE34V model.
To encourage strong de-identification attacks, we take the distance corresponding to the 25 th percentile of this distribution as the target threshold P 25 for our training loss (see Section 3.4).
We train each attack as discussed above.
We evaluate the closed-set recognition of all attacks over the VoxCeleb1 dataset.
We compute the distance between each query embedding and the centroid of all embeddings of each speaker; recogntion is then performed by returning the speaker identity of the nearest speaker centroid.
We find this is slightly more accurate and robust to attack than using the identity of the nearest embedded utterance.
We perform exact nearest-neighbors search over the embedding space, and report the top- (T-1) and top-10 (T-10) accuracy of the relevant speaker recognition model given both clean and adversarial queries.
Additionally, we compute the following objective speech quality metrics over the clean and adversarial query audio as a proxy measure of the imperceptibility of attacks: Perceptual Evaluation of Speech Quality (PESQ) [47], operating in the wide-band configuration and using the python-pesq implementation [58] (MIT license); and Short-Time Objective Intelligibility (STOI) [54], using the pystoi implementation [43] (MIT license).The results of our evaluation are presented in table 1. [Table]
We evaluate the real-time performance of the streaming implementation of the VoiceBlock attack described in Appendix A.3 by measuring its average real-time factor (RTF).
We measure performance on a single thread on two different CPUs, an Intel i7-5600U @ 3.2 GHz and an Apple M1 Chip.
In the streaming configuration, VoiceBlock processes a chunk of 4 overlapping frames at a time, equivalent to 640 samples / 40 ms.
We compute the average RTF for processing a chunk over all chunks in a 4 second audio clip, and find that VoiceBlock has an average RTF of .255 on the Intel i7-5600U and .200 on the Apple M1.
4.5 Subjective evaluation
For our subjective evaluation, we sampled 100 clean speech recordings from the VoxCeleb1 dataset.
To standardize comparisons, we trimmed or padded all utterances to 4 seconds.
Each of the four attacks described in Section 4.3 was applied to all 100 recordings, resulting in 100 comparison sets with five recordings per set: the original clean speech and the speech modified by each of the four attacks.
These sets were then evaluated in a MUSHRA-style listening study [9] deployed on Amazon Mechanical Turk using the open-source, MIT-licensed Reproducible Subjective Evaluation (ReSEval) [38] system.
IRB approval was obtained prior to conducting this study, and there were no known risks to the participants in this study.
We recruited 20 participants.
Participants were screened with a listening test prior to beginning the study; those that passed rated 20 comparison sets.
For each comparison set, the participant was asked to listen to and rate the relative audio quality of each of the five audio files on a scale from 0 to 100.
We omitted responses by four participants who failed our prescreening listening test and nine who rated the white noise attack (our low anchor) as superior in quality to ground-truth clean audio (our high anchor), giving us a total of 140 five-way comparisons.
For more details of our crowdsourced subjective evaluation, see Appendix C.
The results of our crowdsourced subjective evaluation can be found in Figure 2.
We find VoiceBlock is preferred to all other methods, and exhibits similar perceptual quality to ground-truth speech recordings.
The difference between VoiceBlock and ground-truth audio is significant ( p < 0 . 05) using a Wilcoxon signed-rank test ( p = 0 . 024) but not significant using Welch’s T-test ( p = 0 . 071) , which assumes that human perceptual scores are normally distributed but with potentially different variances.
All other pairs of conditions are significantly different using either test.
4.6 Additional experiments
In Appendix B.1, we conduct ablation studies on the VoiceBlock encoder configuration, lookahead length, and choice of auxiliary loss.
We also conduct supplementary experiments demonstrating the robustness of VoiceBlock against a deep network-based speech enhancement model, examining its performance under the assumption that adversarial queries are enrolled by the surveilling speaker recognition system, and evaluating the intelligibility of processed audio using an automatic system.
These experiments are detailed in appendices B.2, B.3, and B.4 respectively.
Ethics
The main goal of our work is to show that large-scale, untargeted automated surveillance can be hindered by introducing inconspicuous perturbations to audio in real-time.
Because VoiceBlock does not conceal speaker identity from human listeners, it still allows high-effort targeted surveillance (e.g. authorized human-attended wiretaps of criminal enterprises).
In this way, we hope to return to the status quo of the 20th and early 21st centuries – in which the need for human listeners provided an important check on mass surveillance.
Those wishing to avoid all voice identification by human or machine may use existing voice conversion technologies.
Voice conversion is not, however, appropriate in many contexts where some measure of privacy from mass surveillance may be desired.
Consider telehealth: a medical expert or therapist discussing sensitive topics with a patient may gain great insight from hearing the nuance of the patient’s voice.
In this setting, methods that alter one’s voice beyond human recognition (e.g. voice conversion) would necessarily degrade the interaction.
The approach of VoiceBlock could also, in concept, be used to fraudulenty access information or services protected by speaker verification.
However, VoiceBlock performs perceptually inconspicuous filtering and the speech produced would still sound like the original speaker.
As such, it could not pass human inspection.
Both voice-conversion and speech synthesis-based attacks are much more suited to this purpose, as both produce speech in a voice perceptually similar to the target speaker.
Our approach can be thought of as leveraging the asymmetry between system and attacker attention inherent to mass surveillance, using inconspicuous perturbations to evade large-scale systems that must render predictions in bulk.
This same asymmetry is not necessarily present when trying to bypass authentication mechanisms and access tightly-guarded services.
Conclusions
Our experimental results indicate VoiceBlock can de-identify speech from arbitrary unseen users in real-time on a standard M1 CPU.
The de-identified speech is of significantly higher audio quality than competing methods, as reported by a subjective listener study and as measured through standard metrics of speech intelligibility.
Our method also achieves nontrivial de-identification results against a system it was not trained on, outperforming a far more perceptible universal attack.
This is notable given that we take no explicit steps to improve the transferability of our method, which leverages a lightweight model trained on a small corpus of clean speech without augmentation.
While pure signal-processing approaches such as spectral gating and white noise transfer between systems more successfully, they severely degrade audio quality, resulting in much lower listener quality ratings.
This limits their practicality in real-world voice communications.
By contrast, our method produces adversarial audio that is virtually indistinguishable from the clean source, as indicated by our subjective evaluation.
We view our method as an initial step towards protecting users from indiscriminate mass surveillance.
A number of obvious directions for future work stand out, such as improving the transferability of the adversarial examples crafted by our system [40], and evaluating attacks against real-world speaker recognition systems and over real-world communication channels.
We hope this work encourages further exploration of the applications of audio-to-audio models for protecting user privacy.
A VoiceBlock implementation A.1 Architecture
We provide additional details of the VoiceBlock architecture used in our experiments.
Phoneme encoder:
The phoneme encoder consists of a feedforward network with two hidden layers of size 256 and ReLU activations, followed by two LSTM layers with hidden size 256.
Using the LibriSpeech "train-clean-100" dataset and aligned phoneme labels provided by Lugosch et al. [33], we train the phoneme encoder with cross entropy loss for 25 epochs using the Adam optimizer with learning rate 1e-3.
During training, a linear layer is appended to the phoneme encoder to render predictions over phoneme labels; we discard this layer after training.
Spectrogram encoder:
We pass 64-bin mel-spectrograms through a 1 × 1 convolutional layer and gated linear unit (GLU) operating along the frequency axis, followed by a two linear layers with leaky
ReLU activations and hidden size 512.
The convolution layer upsamples the channel (frequency) dimension to 1024, from which the GLU obtains a 512-dimensional representation to pass to the linear layers.
Source speaker conditioning: For each frame of audio, we concatenate the 256-dimensional output of the phoneme encoder, the 512-dimensional output of the spectrogram encoder, and the onedimensional pitch / periodicity / loudness features and linearly project to obtain a 512-dimensional vector.
A fixed 512-dimensional source speaker embedding obtained from the ResNetSE34V2 model is passed through a 2 linear layers with leaky ReLU activations and hidden size 512, followed by batch normalization and a final linear layer to obtain a 1024-dimensional vector holding scales and biases for a feature-wise affine transformation (FiLM) of the encoder output.
The affine transformation is applied and the resulting vector is passed to the bottleneck.
Bottleneck: Our bottleneck consists of two LSTM layers with hidden size 512.
The encoder output is passed through the LSTM layers with a skip connection concatenated at the output.
Following the application of a lookahead convolutional network, the resulting vector is projected linearly to obtain a 128-dimensional vector of filter controls for the current frame.
Decoder:
We perform filtering with 128 bands per frame, and set L ∞ bound ϵ = 0 .
5 to further constrain deviations from a "neutral" control configuration of ⃗ 1 .
We experimented with alternative methods of regularizing predicted filter controls, such as L 2 normalization, but found that simple L ∞ clipping yielded satisfactory results.
Training:
We train both VoiceBlock and the universal baseline attack using the Adam [29] optimizer with learning rate 1e − 4 for 10 epochs over the LibriSpeech "train-clean" subset.
A.2 Auxiliary loss
We briefly summarize the multi-resolution STFT loss of Defosséz et al.
[13] used in the training of our attacks.
The loss is given by
[Equation]
where L stft is given by a sum of magnitude and spectral convergence losses computed over M spectrogram resolutions:
[Equation]
Here || · || F and || · || 1 refer to the Frobenius and L 1 norms, respectively.
Following Defosséz et al.
, we use M = 3 spectrogram resolutions with hop sizes of 50, 120, and 240 samples, FFT lengths of 512, 1024, and 2048, and window lengths of 240, 600, and 1200, respectively.
A.3 Streamer
We implement a version of VoiceBlock for processing audio streams in real time.
All components are modified to support the computation of perturbations on overlapping windows of audio while the input and output streams process non-overlapping chunks.
Where possible, components are complied to TorchScript to improve performance.
Three buffers are added to process a stream of audio into overlapping chunks:
• Last Frame Input Buffer: Contains the last 128 samples of the last input audio to be appended to the oncoming chunk, before computing the overlapping windows.
• Lookahead Buffer: Contains overlapping frames to be used as lookahead.
• Output Buffer: Contains the last 128 samples output by VoiceBlock, to be overlap-added with the next processed chunk.
Code for our streaming implementation can be found at https://interactiveaudiolab.github. io/project/voiceblock.html .
B Additional experiments B.1 Ablation study
To better understand the contribution of various aspects of the proposed model architecture and training procedure, we perform the following ablations.
Encoder: We examine the effectiveness of models using a subset of the encoder components discussed in Section 3.1.
Results are presented in Table 2.
Lookahead: we vary the number of lookahead frames passed to the model during training and inference.
At a lookahead of zero, the model operates in a causal manner. We report model effectiveness and theoretical latency as a function of lookahead in Table 3. Auxiliary loss : we replace the combined waveform and multi-resolution spectrogram loss of Defosséz et al. [13] with a waveform-only L 1 loss, the mel-frequency cepstral coefficient cosine-similarity loss used by
O’Reilly et al.
[41], and an ASR feature-matching loss.
Results are reported in table 4.
Note that the latter model variant outperforms the “main" VoiceBlock model presented in this work along all objective metrics, although we were unable to conduct a subjective evaluation to verify apparent gains in perceptual quality.
For all ablations, we perform attacks on the ResNetSE34 model using the experimental configuration detailed in Section 4.
B.2 Attack robustness to preprocessing
In real-world settings, user audio may travel through various preprocessing stages before reaching a speaker recognition model.
To simulate the presence of such stages, we perform attacks using the experiment configuration discussed in Section 4.3 but pass query utterances through a pretrained
[Table] [Table] Demucs [13] speech enhancement model en route to the ResNetSE34v2 recognition system; results are reported in table 5.
The speaker recognition system maintains its top-1 and top-10 accuracy on clean queries.
In contrast to Chiquier et al.
[10] we do not incorporate the enhancement model into our adversarial optimization, meaning that Demucs essentially functions as an unseen adversarial defense.
We find that VoiceBlock loses almost no effectiveness when queries are passed through
Demucs; by comparison, the universal attack loses most of its effectiveness, as Demucs is able to separate the low-magnitue but noisy perturbation from clean speech.
Both the white noise and spectral gating attacks retain their effectiveness, presumably because they degrade audio beyond Demucs’ capability to provide coherent reconstructions of the original speech.
We leave as future work the exploration of the robustness of VoiceBlock against more sophisticated preprocessing pipelines and adversarial defenses. [Table] [Table] [Table]
B.3 Enrollment of adversarial queries
It is possible that adversarially-perturbed query utterances extracted from a VoiceBlock user’s audio stream may themselves be enrolled by a surveiling speaker recognition system.
In such cases, there are two possibilities:
VoiceBlock successfully de-identifies user speech, and adversarial queries are enrolled as a separate speaker profile from any existing clean utterances of the user 2.
VoiceBlock fails to de-identify user speech, and adversarial queries are incorporated into an existing profile of the user containing clean utterances
We examine both scenarios, again using the VoiceBlock attack discussed in Section 4.3.
Results are presented in table 6.
We find that while VoiceBlock is highly effective at de-identifying users against a profile constructed from clean (unperturbed) utterances, its de-identification performance suffers significantly under both of the aforementioned conditions.
This suggests that additional work is required to ensure that a VoiceBlock user remains a "moving target" to surveilling speaker recognition systems.
One possible solution is to leverage targeted rather than untargeted attacks: by proactively "spoofing" specific (random) locations in the embedding space, VoiceBlock may hamper the creation of a single matching enrolled profile.
B.4 Word error rate evaluation
To measure the intelligibility of speech processed by the proposed and baseline methods, we evaluate the word- and character-error rates of each attack described in Section 4.3 on the LibriSpeech train-clean-360 subset using a Wav2Vec 2.0 automatic speech recognition model [5].
The train-clean-360 dataset contains utterances and accompanying transcriptions from 921 speakers not seen during training.
We do not use the VoxCeleb dataset because it does not have the necessary transcriptions.
We partition each speaker’s data into query, enrollment, and conditioning partitions, as described in Section 4.2.
Our results are reported in Table 7.
Our results demonstrate that pure signal-processing attacks such as spectral gating and white noise result in word- and character-error rates 20+ times that of VoiceBlock.
VoiceBlock also outperforms all methods except spectral gating on the de-ientification task, with which it performs competitively.
This indicates our method’s ability to preserve the intelligibility of speech while adversarially modifying speaker characteristics.
[Table]
C Listening Study
The listening study consists of 20 evaluation tasks, where the participants are shown the following text, along with 5 four second long audio examples and 5 corresponding sliders with values from 0 to 100:
Listen to all recordings of a person speaking.
Then, move the sliders to rate the quality of each audio file from 0 (worst) to 100 (best) .
The higher-quality audio files are the ones that are more natural sounding, or have fewer audio artifacts (e.g., clicks, pops, noise, or otherwise sound ’unnatural’).
Note - Each slider cannot be moved until its corresponding audio file has been listened to in its entirety.
It takes approximately 15 minutes to complete all the evaluation tasks.
Participants are also asked to complete a listening test before proceeding to the audio evaluation.
The listening test involves the participant listening to two audio files, and reporting the number of tones they heard.
For each audio file, the participants are given 3 tries to correctly report the number of tones played.
If a participant succeeds in the listening test and completes the evaluation tasks, then they are paid 3.00 USD, or equivalently, 12.00 USD / hour. If the participant fails the listening test