Throughout history, countries and their intelligence agencies have used advanced mass surveillance systems in their national security efforts [9] . Some mass surveillance systems collect a massive amount of data from landline communication, cellular, and endpoint devices. In recent decades, and especially during the COVID-19 pandemic, many people have moved to voice over IP (VoIP) communication via third party applications like Skype, Zoom, Discord, etc. Such calls can be monitored using surveillance systems for VoIP communication, which collect the audio data sent over the Internet, or directly from a user's endpoint. The increase in the amount of data collected has led to the development of automated surveillance systems that analyze the audio samples and route * Contributed equally conversations with sensitive content to a human agent [19, 11] .
At the heart of many audio-based mass surveillance systems is an automatic speech recognition (ASR) model that produces automatic transcriptions of the recorded conversations. The transcripts extracted by the ASR are sent to a natural language processing (NLP) algorithm to gain better understanding of the conversation. For example, an NLP model can be used to automatically identify sensitive topics of interest (e.g., drugs, crime, terror) in a conversation [18, 14] . Those models usually rely on artificial intelligence technologies, such as deep neural networks (DNNs).
Despite the benefits of mass surveillance systems for society, such tools raise questions related to privacy infringement of groups and individuals. In order to identify illegal behavior and identify criminals, the data of innocent people needs to be collected and analyzed, without their permission or knowledge. In addition, mass surveillance systems can also be exploited for hostile goals. For example, crime or terror organizations may spy on civilians for illegal purposes; mass surveillance can be used for industrial espionage, etc.
A potential means of addressing privacy concerns might derive from the DNNs' vulnerability to adversarial perturbation [35] , in which a carefully crafted perturbation to a legitimate input sample can cause a major change in the model's output. Since many ASR systems are based on deep learning models (e.g., RNN, LSTM, encoder-decoder), adversarial examples can serve as the basis of a privacy protection solution.
Although some studies have explored the use of adversarial examples in the audio domain [5, 31, 16] , they do not offer a solution for privacy protection in real-world settings. There are several challenges that need to be addressed in order to provide a real-world solution. First, the content of a conversation is usually unknown in advance, and as a result, this knowledge cannot be used to craft the perturbation. In addition, a perturbation with a constant length cannot be added to audio samples of different lengths, especially for live data streams commonly used in VoIP applications. Another challenge is balancing the perturbation's volume. The larger the perturbation is, the better it can fool the model. Yet, adding a large perturbation to a conversation can prevent the listener from understanding the speaker. Balancing this tradeoff is done by controlling the perturbation's amplitude. However, changing the amplitude means that a new perturbation needs to be created, but crafting one for every possible amplitude is time-consuming and therefore deceases the applicability of the solution. Therefore, although adversarial perturbation can offer a means of privacy protection, there are still many challenges that need to be addressed to create a viable solution.
In this study, we suggest a privacy preservation framework that enables participants in a VoIP conversation to avoid detection by a mass surveillance system based on their conversation topics. Our approach targets the ASR component of the surveillance system, and as a result, it is able to fool the entire system. To achieve that, our framework crafts a universal adversarial perturbation (UAP) that is agnostic to both the content of the conversation and the speaker. Moreover, the UAP we craft is short and can be played repeatedly, allowing the perturbation to be added to audio samples of any length, including audio streams. We also demonstrate how the perturbation amplitude can be controlled in real time by multiplying the UAP by a constant factor. Thus, the speaker can control the UAP's amplitude without the need to create multiple perturbations. We assume that if many users routinely utilize our proposed framework, an automatic mass surveillance system will be unable to distinguish between conversations about topics of interest and small talk.
We also present a real-world privacy preservation solution for a conversation held via VoIP applications. This is accomplished with a trusted external device (TED) that combines a microphone for audio recording and a Teensy microcontroller that adds the UAP to the audio in real time. Using this device, we show how two people can have a VoIP conversation that fools the ASR model (and thus the mass surveillance system itself) of a high privileged eavesdropper. We also show that our solution can be used for different popular VoIP applications (Skype and Zoom), speakers, and for data of any length. Moreover, we demonstrate that our UAP is transferable between two state-of-the-art ASR models (Deep Speech and Kaldi). By publishing our code and experimental setup [2] , we hope to support individuals and groups that aim to protect themselves in the digital age, where privacy has become a luxury.
The main contributions of our study are:
• A novel mechanism that allows groups and individuals to protect themselves from audio-based mass surveillance systems. By creating a UAP, our solution is robust to changes in the audio data, including unknown speakers and conversation context.
• An algorithm that creates a short repeatable perturbation that can be added to audio data of any length.
• A method that allows the user to control the UAP's volume in real time without crafting a new perturbation.
• A real-world implementation that maintains the privacy of VoIP calls. Our solution is designed to protect users from high privileged eavesdroppers that collect the audio data from the endpoints and communication channels.
• An evaluation of our framework in the real world for realtime conversations held via popular VoIP applications (Skype and Zoom) and a demonstration of the solution's performance against an unknown ASR model (Kaldi).
Adversarial examples were first introduced by Szegedy et al. [35] who showed that a small, carefully crafted perturbation added to a benign input sample can cause a major change in a neural network's output. Over the years, the optimization problem for crafting adversarial perturbations presented by the authors has been used to develop many attacks against learning models [10, 26, 6, 8] . Most of the methods focused on crafting a perturbation for single input sample and were mainly demonstrated in the image domain. The universal adversarial perturbation was introduced by Moosavi et al. [22] . Unlike its predecessors, this approach crafts one perturbation that fools the target model, regardless of the input sample it is added to. The perturbation fools the target model when it is added to both seen and unseen input samples, and it is also transferable between different models.
Carlini et al. [7] presented one of the first adversarial examples in the audio domain. In their work, the authors fooled an ASR system by adding unintelligible adversarial perturbations to prerecorded audio commands. A subsequent study by Carlini et al. [5] presented a white-box targeted attack against the state-of-the-art ASR model, Deep Speech [13] , which crafts a perturbation that should be undetectable to the human ear. Although the results show 100% success in fooling the model to output the attacker's target transcription, in both works the adversarial perturbations are not completely imperceptible.
To craft an imperceptible adversarial perturbation, Schonherr et al. [31] utilized the psychoacoustic hiding method. Psychoacoustic hiding is common in MP3 conversion, where a signal is compressed by eliminating tones, so that when they are played simultaneously, the human ear hears all of them together as one dominant tone. The attack limits the perturbation searching space to the tones that the human ear filters out. Although the attack created imperceptible adversarial examples, the perturbation is tailored to a specific audio sample. Due to the nature of psychoacoustic hiding, using it to craft a UAP is very challenging.
While initial research focused on white-box attacks, other works explored less permissive conditions. Taori et al. [37] presented a targeted attack against ASR models, based on an assumption that the attacker can access the model's output distribution (before the model decoding phase). Although the attack functions under grey-box settings, the target transcription is limited to two words. Khare et al. [16] utilized multiobjective evolutionary optimization to attack under black-box settings. In the real world, black-box attacks are considered a greater and more realistic threat than white-box or grey-box attacks. While some black-box attacks use complex optimizations and gradient approximations, a simpler approach is to use the transferability property of adversarial examples [35, 25] to transfer a perturbation from one model to another.
Additional work was done to craft adversarial perturbations that can be used in the physical world. Recent studies [28, 36, 45] presented attacks which can craft audio-based adversarial examples that can be played over-the-air. Both [28, 36] use the expectation over transformation framework [3] to improve perturbation robustness for real-world transformation. A simulator was used to simulate how each audio sample would be heard when played in rooms of different sizes. The additional data samples were used to craft the adversarial perturbation, which improves the success of the attack when evaluated with the same transformations in the real world. However, none of the attacks mentioned above are feasible in real-world scenarios, since they are implemented for prerecorded audio samples. Furthermore, they are also sensitive to background noise, limiting their use in the real world.
Neekhara et al. [23] crafted a UAP that fools an ASR model. The perturbation was shown to be effective when added to audio samples from both the training and test sets. However, their UAP has a fixed length and thus cannot be used for a conversation stream, where the audio records are sent continuously and are of unknown and variable lengths. In addition, the attack is created and tested using scores that are based on the character error rate (CER). Although this is a popular metric for evaluating ASR models, it is not suitable for evaluating the privacy preservation task investigated in our work. For example, a successful attack is defined for a perturbation that causes a CER that is greater than 0.5 between the original and distorted transcription. However, a topic detector can identify the subject of the sample based on a fraction of the conversation. As a result, crafting a UAP for the privacy preservation task should utilize a different metric.
Instead of exploiting the ASR model's vulnerability to adversarial examples, some studies utilized design flaws in the microphone hardware [46, 29, 30, 33] . Most microphones filter out frequencies that are inaudible to humans (frequencies > 20kHz). However, due to the nonlinear nature of microphone hardware, transmitting multiple inaudible tones to a microphone results in a shadow signal that falls within the range of tones that are audible to humans (20Hz to 20kHz). The studies exploit this vulnerability to transmit inaudible commands for voice controllable systems (e.g., Siri, Google Assistance, Alexa). This is achieved by using ultrasound speakers, whose output the microphone interprets as sound that is within the range of human auditory capabilities. Although this approach has shown promising results, it cannot be employed for a conversation between two people. When the person that sends the message uses an ultrasound speaker to mask his/her speech, the microphone will interpret both his/her voice and the inaudible waveform as above the threshold of hearing. As a result, the person that receives the message hears a mix of sounds, which is difficult to understand.
Several studies have utilized adversarial examples as a solution for privacy protection in the image domain. An early example is the work of Sharif et al. [32] , who created glasses with adversarial patterns that allow a person to avoid detection by a facial recognition system. Zhang et al. [47] proposed an end-to-end cloud solution to preserve the privacy of user portraits. The solution targets facial recognition-based crawlers that browse photo sharing clouds to build a user's portrait dataset. Mirjalili et al. [20] suggested a privacy model called "PrivacyNet." Their solution uses adversarial perturbations to modify the input in a way that allows the user to control which soft biometric attributes he/she wishes to obfuscate.
In the audio domain, Srivastava et al. [34] presented an encoder-decoder architecture for ASR that obfuscates the user's identity. Ahmed et al. [1] introduced an end-to-end cloud-based ASR service called Preech, a system that protects user privacy, including textual privacy. However, since the proposed solution requires the user to use a particular service, it is not a solution that can be implemented across multiple platforms. Moreover, Preech assumes offline decoding and does not support data streams and thus is less suitable for real-world VoIP applications.
In this study, we assume a scenario in which two people want to have a private VoIP conversation. For convenience, we simplify the problem so there is one Speaker and one Listener. Both the Speaker and Listener use their personal endpoints (i.e., computers) to communicate via a popular VoIP application. In addition, a malicious Eavesdropper uses a mass surveillance system to automatically collect audio data and identify the topic discussed.
To collect the conversation data, the Eavesdropper compromises the endpoints of the Speaker and/or Listener to intercept the VoIP call or by performing a man-in-the-middle attack on the communication channel. The digital waveform of the conversation is then sent to a deep learning-based system, with three main components: ASR, topic detector, and alert mechanism. The ASR component uses a deep learning model to create a transcription of the digital waveform. Then, the transcription is sent to a deep learning-based topic detector that analyzes the text and outputs a list of the topics of the conversation. If a specific topic is identified in the conversation, an alert will be raised to inform the Eavesdropper to assign a human agent who will listen to the call. If the topics is not of interest to the Eavesdropper, the call is ignored. Since the alert is usually implemented using a simple set of conditions with regard to the topic detector's output, we focus our evaluation on the two complex learning models (i.e., the ASR and topic detector).
We assume that the Eavesdropper's goal is to automatically identify specific topics in the conversation between the Speaker and Listener. To do so, the Eavesdropper can collect data from the victims' endpoints, as well as the communication channel.
On the other hand, the goal of the Speaker and Listener is to maintain a private communication channel. Hence, the victims want to talk via the VoIP application without the topic of their conversation being automatically detected by the Eavesdropper's system. Moreover, the victims aim to have a normal conversation. Therefore, the Speaker does not use a prerecorded message or recite a known transcript, and the Listener wants to be able to understand the Speaker but is willing to tolerate some noise on the call. In addition, we assume that the Speaker suspects that his/her endpoint might have been compromised by the Eavesdropper.
Our research provides a solution for the victims that allows them to maintain their privacy in the face of the Eavesdropper. We start by assuming that the victims have partial knowledge about the Eavesdropper's system and full knowledge on the ASR model, but later we will remove this assumption.
We suggest the framework presented in Figure 1 for the scenario of a private conversation between two parties described above. The Speaker uses the knowledge he/she has on the Eavesdropper's system to craft a UAP that can fool the ASR model. In addition, the Speaker uses a trusted external device as the input device (i.e., microphone) for his/her endpoint. During voice acquisition, the TED adds the UAP to the audio signal, and the perturbed audio is sent as digital audio input to the Speaker's endpoint. As a result, all of the audio signals that pass through the Speaker's computer contain the perturbation. The Speaker uses a third party VoIP application to communicate with the Listener. The Listener can understand the content of the perturbed audio, even though there is some audible noise in the conversation. We assume that some or all of the audio signals are automatically collected by the Eavesdropper's system. Those audio samples are fed to an ASR model, and the output transcript is fed to a topic detector that searches for topics of interest in the conversation. However, because the UAP has been added to the digital audio signal before it was sent to the Speaker's endpoint, the ASR model produces an incorrect transcript of the audio. As a result, the topic detector does not receive the correct transcript and thus fails to raise an alert on the conversation's topic.
Our solution aims to address the following objectives:
A real-time solution: The solutions proposed in many prior studies craft adversarial perturbations for a known digital audio signal, add the perturbation to the input signal, and feed the perturbed signal to the learning model. However, such a solution will not work for a normal, ongoing conversation in real life. Crafting an adversarial perturbation based on the Speaker's audio file is impossible, since in an actual conversation this data is unavailable in advance (i.e., the Speaker has not spoken, so the sample does not exist). To address this challenge, in this study we design a universal adversarial perturbation that can be added to any digital audio signal.
A live data stream: An adversarial perturbation for a prerecorded audio sample has a constant length. Hence, it cannot be used for digital audio signals of a different length or for a live audio stream. To address this challenge, we crafted a short perturbation that can be added repeatedly over a longer audio sample. As a result, the perturbation can be added to samples of different lengths, including a live data stream.
Effective against a high privileged eavesdropper: Most eavesdroppers collect data from the victims' endpoint or by performing a man-in-the-middle attack on the communication channel. If the perturbation is added after the Eavesdropper collects the audio signal, it will be useless. Therefore, the perturbation is added to the Speaker's digital audio signal in a TED. We assume that even if the Eavesdropper has compromised the Speaker's endpoint, he/she cannot compromise the TED connected to it. Thus, the digital audio signal that reaches the Speaker's endpoint already includes the perturbation. Such a setup also decreases the Eavesdropper's ability to automatically detect the presence of the perturbation.
Agnostic to third party applications: We assume that the Speaker and Listener are regular people and therefore communicate via popular, commonly-used applications. Thus, our solution aims to work even when the audio signal is sent via various third party applications that use common automatic compression processes (e.g., Skype, Zoom).
Blend into the crowd: We assume that if many users use the proposed framework, the Eavesdropper cannot build an automatic mitigation method. Even if the Eavesdropper can develop a method to detect perturbed calls, it will fail to automatically identify conversations about the topics of interest.
Our method aims to create one untargeted universal adversarial perturbation that fools an ASR model. The UAP can be added to any audio sample from a given dataset and cause the model to output an incorrect transcription. In addition, to Figure 1 : The proposed framework: The Speaker speaks into a microphone which passes the audio to a TED. The TED adds a UAP to the digital audio signal and sends it to the Speaker's endpoint. The perturbed data is sent, via a VoIP application, to the Listener side. The Eavesdropper may collect data from both endpoints or/and the communication channel. The collected data is sent to the Eavesdropper's surveillance system for analysis.
give the Speaker control of the volume of the perturbation, our method ensures that the amplitude of the UAP remains smaller than a given constraint. To support the ability to fool samples of different lengths, we define a short perturbation (several seconds) and concatenate it several times according to the sample's length.
Let M be a DNN-based ASR model that receives audio samples as input and outputs a transcription of the audio file. For a dataset of audio samples X, we aim to find a UAP δ that fools the model when added to most samples in X. Furthermore, to control the volume of the UAP, its amplitude (under ∞ norm) cannot exceed a given λ. Therefore, for dataset X and amplitude constraint λ, we search for the following UAP δ:
To create a UAP that can be used for streaming tasks, our method aims to find a UAP δ that can be added to audio samples of any length. We achieve this by creating a short perturbation that can be played repeatedly over longer audio samples. However, we must ensure that the perturbation is able to fool the ASR model when added to different parts of the sample. To do so, we integrate the replaying of the UAP as part of the perturbation's crafting process. Therefore, we define δ's length to be several seconds, denoted by l. For a chosen constant k, we trim the input samples in X such that the length of each x ∈ X equals l · k. To add δ to the samples, we define δ k , a k time concatenation of δ. As a result, Equation 1 is updated to find a UAP δ as follows:
Based on Equation 2, we defined an objective function L for finding δ. Function L is split into two parts: fooling the model and minimizing the UAP amplitude. To fool the model, we use M's loss function, L M , to maximize the distance between the true output and the result for a perturbed sample. To maintain a UAP with a small amplitude, we aim to minimize the Euclidean norm of δ. Inspired by [6] , we use a constant c to balance the tradeoff between the two parts. Therefore, for a given X, we aim to minimize:
To solve the objective problem in Equation 3, we employ the basic iterative method [17] . In each iteration we calculate the loss gradients of Equation 3 with respect to δ. Then, we multiply the sign of the gradients by a learning rate parameter α and add the result to the current δ. To improve the convergence time of the method, we start the process with a high α and gradually decrease it. Finally, we clip δ to ensure that −λ ≤ δ ≤ λ. Therefore, finding δ can be done using:
We use Equations 3 and 4 to define Algorithm 1 for crafting an audio-based UAP for an ASR model M. In line 1, we for all batch X B in X do 4:
if L stops improving then 8:
end if 10:
end for 11: δ ← Clip λ (δ) 12: end for 13: return δ start by initializing the UAP δ, of size l, with zeros. As part of the initialization, we also split X into batches to further improve the method's convergence time. We chose to use random batches with audio samples from different speakers, since it improves the UAP's robustness to speaker changes. Then, we perform E epochs to craft δ (line 2), such that in each epoch we iterate over all of the batches X B in X (line 3) and update δ according to Equation 4 . For each batch X B , we start by concatenating the current δ k times to create a l · k long perturbation δ k that can be added to any x ∈ X B (line 4). In line 5, we calculate the objective function presented in Equation 3 . Line 6 updates δ, as shown in Equation 4 , but without the clipping. After updating the UAP, we check to see whether the objective function L has stopped improving.
If it has, we decrease the learning rate α by one (lines 7-8). At the end of each epoch (line 11), we apply clipping to δ to ensure that −λ ≤ δ ≤ λ. We found that clipping after each epoch achieves better results than clipping at the end of each batch. The algorithm outputs an audio-based UAP δ.
The goal of our approach is to allow users to evade detection by an automatic surveillance system that searches for sensitive topics in VoIP calls. To achieve this, we use Algorithm 1 to create an audio-based UAP that targets the ASR component of the surveillance system. By adding the UAP to an audio sample, the ASR outputs an incorrect transcription of the digital waveform, and as a result, the system fails to detect the conversation's topic.
As described, the surveillance system has three components: an ASR, topic detector, and alert mechanism. While the alert mechanism can be implemented using a rule-based code, the first two components are more complex and usually use a DNN architecture. In our experiments, we focus on the ASR and topic detector models:
As done in past studies, we use open-source Deep Speech v0.4.1 [13] as the ASR model. The input for the model is a 16-bit 16 kHz mono signal, and it outputs the transcript of the audio sample. To do so, Deep Speech uses Mel-frequency cepstral coefficient (MFCC) preprocessing and a neural network architecture with a combination of dense and LSTM layers. The model was trained by optimizing the connectionist temporal classification (CTC) loss function presented in [12] , which we utilize as well.
For the topic detection model we use the TextRazor API [38]. The model receives speech transcription as input and employs state-of-the-art NLP methods to analyze and extract information from the text. The TextRazor API can be used for tasks, such as entity recognition, topic tagging, spelling correction, and more. In this study, we used the topic tagging component to perform topic detection on the output of Deep Speech.
LibriSpeech [24] is a corpus of approximately 1,000 hours of English speech and transcripts derived from the LibriVox project's audiobooks. The data contains audio samples from different speakers, including both clean and noisy records. Since LibriSpeech mainly contains short audio samples, it is less suitable for the topic detection task. As a result, we only use this dataset to craft the perturbations.
IBM's Project Debater [15] is a learning-based system that can participate in debates with people on complex subjects from various domains. The project's team released several free datasets, including the Debate Speech Analysis dataset [21] , which is used in this study. The dataset contains 60 debate records, from 10 different people, in 16 different domains. We can use this dataset to evaluate our framework, since each data sample contains a long audio file, a transcript, and the debate category.
The word error rate (WER) is a common evaluation metric for ASR systems. It measures the distance between two transcripts, according to the number of actions needed to turn one text into the other. The Levenshtein distance defines those actions as insertion, deletion, or substitution. Since the WER is normalized by the number of words in the first transcript, the error rate can be greater than one (i.e., when the number of actions is greater than the length of the reference transcript).
Since the topic detector analyzes the sentence words to detect the topic, we found that it is more important to understand the changes at the word level than the character level. It is important to note that evaluating the UAP's effectiveness in the privacy protection task should be done by evaluating the complete surveillance system (i.e., the topic detector). However, to allow future studies to reproduce our results, we use the WER to report the perturbation's effect on the ASR model.
The signal-to-noise ratio (SNR) is a measurement used to quantify the level of a signal relative to the level of the background noise. Usually, as done in this work, the ratio is measured in decibels (dB). To evaluate the perturbation's loudness, we measure the UAP's distortion P δ with respect to the original audio sample P signal as: SNR dB = 10 · log 10 P signal P δ
We use the SNR to evaluate the perturbation's distortion with respect to the Speaker's signal in the real world. The higher the SNR ratio is, the less distortion the perturbation causes.
To evaluate the UAP's effects on the topic detection model's predictions, we use Recall@10. To calculate it, we feed the clean input into the system and obtain a list of the top 10 topics. Then, we add the perturbation to the sample, feed it to the model, and calculate the percentage of topics that appear in both the clean and perturbed topics list. A low recall indicates that the UAP prevented the system from outputting the relevant list of topics.
The normalized discounted cumulative gain (NDCG) is a measure of ranking quality, which is often used in the information retrieval domain. The measurement assumes that highly relevant topics appear earlier in the list of results. In contrast to the recall metric, the NDCG considers the continuous confidence score for each topic. The NDCG is calculated as the discounted cumulative gain (DCG) score of the tested transcript, divided by the optimal query (i.e., IDCG). We use the NDCG to evaluate the output of the topic detector by comparing the list of topics for the clean and perturbed transcripts. For a given transcript, the topic detector outputs pairs that consist of a topic and its confidence score. For each topic, the confidence score is a number between zero and one, which reflects how sure the model is of its prediction. The list is sorted based on the confidence score, and thus the first topic is most relevant. We calculate NDCG@n: the NDCG based on the first n topics in the clean and perturbed output.
Let t 1 , . . . ,t n be the list of confidence scores that the topic detector output for the perturbed transcript. Thus, t i is the topic detector's confidence in the way the i'th topic represents the content of the input transcription. We define the NDCG@n score as:
where T ⊆ {1, . . . , n} is a list of indices that represent the topics in the perturbed topics list that also appear in the clean topics list. In this study we use NDCG@10.
We use Algorithm 1 to craft a UAP δ with a length of l = 3 seconds. For the training set we randomly sampled a subset of 2,500 audio files from the LibriSpeech dataset. Each file is cropped into a nine second sample, such that when the UAP is concatenated three times, the resulting δ 3 can be added to it. For the objective function in Equation 3, we use Deep Speech as the ASR model M and the CTC loss as the model's loss function L M , and set the tradeoff parameter c to 10 −4 .
To optimize the objective function, we split the dataset into batches of 16 samples and perform 100 epochs using the Adam optimizer. The learning rate α is initialized to 10 and is gradually deceased to one when the objective function stops improving. Since the samples in LibriSpeech are too short for topic detection, we use the Project Debater dataset for evaluation.
Based on this setup, we create six UAPs that differ by their amplitude constraint λ. We create the perturbations δ 70 , δ 100 , δ 150 , δ 200 , δ 300 , and δ 400 with the λ values of 70, 100, 150, 200, 300, and 400 respectively. The six perturbations are used to examine if our approach can improve the Speaker's ability to protect his/her privacy against the Eavesdropper, with respect to the UAP's amplitude constraint. In addition, we compare each of the UAPs to two types of random perturbations with the same amplitude constraint: random integer and random edge. The random integer is a perturbation that is uniformly sampled in the range of each amplitude constraint, while the random edge is uniformly sampled with only two values: the maximum and minimum constraint. For example, for δ 70 , the corresponding random integer and random edge are sampled from [−70, 70] and {−70, 70} respectively.
We start by evaluating our approach in the digital space: the perturbations are added to the digital audio samples and fed directly to the ASR model, which sends the output transcript to the topic detector model. Then we create an implementation for Figure 1 and perform a real-world evaluation.
We start by evaluating our approach in the digital space. In this setting, the perturbations are added to the raw digital audio sample which is then fed directly to Deep Speech and TextRazor. The evaluation allows us to report on the UAPs' effects in each component of the surveillance system, without the potential influence of real-world noise. We compare each of the UAPs δ 70 , δ 100 , δ 150 , δ 200 , δ 300 , and δ 400 to the random integer and random edge perturbations. To evaluate the perturbations' effect on the ASR model, we use the WER metric with respect to the perturbations' amplitude constraint. As shown in Figure 1 , the Speaker tries to fool the complete surveillance system (i.e., the topic detector's output). Therefore, we use Recall@10 and NDCG@10 to evaluate the perturbations' influence on the system's output. Figure 2 presents the effects of the UAPs and random perturbations with respect to the amplitude constraints of the perturbations. The results suggest that an increase in the amplitude results in an improved ability to fool the surveillance system, yet it is clear that the UAPs surpass the random perturbations' abilities for all amplitudes. Of the two random perturbations, it seems as though the random edge performs better than the random integer. This might stem from the perturbations' amplitude: random edge was designed to have values greater than (or equal to) the values of the random integer. As expected, a greater amplitude results in greater distortion of the input sample and thus has more effect on the surveillance system's output. Figure 2a shows that the WERs calculated for the UAPs are dramatically higher than the WERs for the the random perturbations. Therefore, we can conclude that the UAPs limit the ability of the ASR model to transcribe the audio samples. We note that the UAPs were trained using the LibriSpeech dataset yet evaluated on the Project Debater dataset (i.e., a completely different dataset). Therefore, the results obtained are satisfactory for unknown data, such as the data used in our experiments. Moreover, the results presented in Figures 2c  and 2b , which show the perturbations' effect on the topic detector, show that for δ 150 , the topic detector can correctly identify less than 20% of the topics; for δ 200 , it drops to less than 10%; and for δ 300 and δ 400 , the UAPs impair the ability of the system to output the correct topic list. Therefore, a UAP with an amplitude constraint greater than or equal to 150 is sufficient for fooling the surveillance system.
As shown in Figure 2 , an increase in the amplitude of the UAP results in a decrease in the surveillance system's ability to output the correct topic list. However, a higher amplitude causes a greater distortion, resulting in more background noise, and thus limits the Listener's ability to understand the content of the conversation. Therefore, this tradeoff should be considered when the Speaker is crafting the UAP. As a solution, the Speaker can control the conversation's privacy level by using multiple UAPs with different amplitude constraints.
However, crafting a UAP for each amplitude constraint is time-consuming and thus less applicable in the real world where real-time solutions are required. Moreover, the Speaker will be restricted to only using UAPs that were crafted in advance. We address this challenge by crafting one UAP and controlling its amplitude by multiplying the perturbation by a constant. For example, δ 100 could be increased to have an amplitude of 200 by multiplying the UAP by two. Similarly, to decrease the UAP to 70, the Speaker can multiply the δ 100 UAP by 0.7. Therefore, controlling the UAP's amplitude can be done in real-time solely by choosing the required constant.
We examine this method by multiplying each δ i by different constants to craft UAPs for different amplitudes. For instance, we multiply δ 100 by 0.7, 1.5, 2, 3, and 4 to create UAPs for amplitudes of 70, 150, 200, 300, and 400 respectively.
The results are presented in Figure 3 . For each λ value, only one UAP was crafted using this amplitude constraint, whereas the others are the result of constant multiplication. The results are similar to those presented in Figure 2 . Therefore, since the UAPs maintain their ability to fool the system, we can use one UAP to create perturbations for different amplitudes.
In this section, we present a real-world implementation for the framework presented in Figure 1 . The Speaker and Listener communicate via a third party VoIP application. The Eavesdropper runs a Trojan on each of their endpoints and collects all of the audio data. The collected data is analyzed to identify interesting conversation topics. The analysis includes creating a transcription of the audio and a topic detection phase. If the Eavesdropper's system identifies one of the targeted topics, it raises an alert. The Speaker and Listener aim to avoid being detected by the Eavesdropper's system.
In this experiment, we implement the endpoints of the Speaker and Listener using two kits of next unit of computing (NUC) that run Ubuntu. The parties communicate via the popular VoIP application Skype. To simulate the Eavesdropper's Trojan, we stream the Speaker and Listener output and input audio data (respectively) into the Deep Speech model, which transcribes the audio. Then, to identify the topic of the conversation, the transcript is sent to TextRazor. Based on the system's output, the Eavesdropper can be automatically notified of any relevant topic in the conversation.
To maintain a private communication channel, the Speaker uses a trusted external device as a microphone. As presented in Figure 4 , the TED has three main parts; a Teensy 3.2 USB board, an audio board, and a simple microphone. The microphone is connected to the audio board via an auxiliary port, and the board is connected to the Teensy via two sets of 14 pins. The TED is configured as an external microphone and can be connected to any computer via a USB, which is placed on the Teensy. We configured the TED as follows: The audio board samples the analog data from the microphone, converts the signal into a digital signal, and passes it to the Teensy at a 16-bit and 16kHz sample rate. Then, the Teensy adds a UAP (crafted in advance) to the audio and sends it to the computer as a digital audio stream. Finally, the perturbed sample is used for the communication between the Speaker and Listener, and thus is automatically collected from their endpoints by the Eavesdropper's system. Figure 5 presents the implementation of the real-world setup.
Since we assume an eavesdropper with high privileges, maintaining the conversation's privacy relies on the use of the TED to add the perturbation. The UAP cannot be added on the endpoint, since the Trojan might collect the clean data directly from the microphone's driver. Also, the Eavesdropper can automatically identify the existence of the UAP by comparing the clean audio on the Speaker side to the audio collected on the Listener side. Moreover, a privileged Trojan can also identify the process that adds the UAP and inform the Eavesdropper. As a result, the Eavesdropper will be able to automatically filter out the perturbation from the audio.
However, by using the TED to add the UAP to the clean audio, the parties can avoid detection. The TED is registered as a legitimate external microphone and feeds the NUC a perturbed speech signal. Hence, the UAP already exists in all of the audio samples on both sides. Therefore, the Eavesdropper cannot automatically detect or filter out the UAP from the clean audio sample. As discussed before, our experiment assumes one Speaker and one Listener. However, implementing a two-way conversation (or more) can be done easily by supplying an additional TED to the Listener side. 
We use the setup described above to evaluate the UAP in the real world. Each experiment follows similar steps: On the Speaker side, an external speaker device plays audio samples from the Project Debater dataset to the TED's microphone. Each sample is played twice: a clean sample without distortion and a perturbed sample in which the UAP was added to the clean sample by the TED. Then, the data is sent to the Speaker's NUC, which sends the audio to the NUC on the Listener side via Skype. We simulate the Eavesdropper in a controlled environment in which the Deep Speech streaming model runs on the NUC on each side. On the Speaker side, the audio is collected from the NUC's input source, while on the Listener side the data is collected from the application's output. On each side, the Deep Speech model receives the audio stream and transcribes it. Finally, the transcript is sent to the TextRazor API, which outputs a list of topics and their corresponding confidence scores. Using the output for both clean and perturbed audio samples, we measure the performance using the Recall@10 and NDCG@10 metrics.
The results presented in Figure 3 show that the distortion caused by one UAP can be controlled by using constant multiplication. Thus, to create a UAP that has similar results to δ 300 , we can multiply δ 150 by two in real time. To reflect realistic use of the UAP, we choose δ 150 as the UAP added by the TED and manually tune its volume. The distortion of the perturbation is measured using the SNR.
We start by evaluating our framework using the setup and configuration described above. Then, to better understand the framework's capabilities, we perform the following set of experiments that examine the different elements in a realworld scenario: 1) change the VoIP application from Skype to Zoom; 2) examine the UAP's effect on different speakers; 3) evaluate the UAP on audio samples of different lengths; and 4) assess whether the UAP is transferable to an unknown ASR model, Kaldi [27] . In all of the experiments we perform the same evaluation process and use the same metrics, but we change one of the components (application, speaker, sample length, target ASR model) in each experiment.
To evaluate whether the solution is feasible in the real world, we start by performing the baseline experiment. For the Speaker, we use all of the samples from the Project Debater dataset, the Deep Speech ASR model, and Skype. The perturbation's distortion is measured using the SNR. In addition, we evaluate the UAP's effects on the surveillance system using the mean of Recall@10 and NDCG@10 for all samples. The metrics are computed using the data sampled on the Speaker's endpoint (NUC) before the audio has been sent via Skype and on the Listener's endpoint on the VoIP application's output.
The UAP achieves mean values of 0.17 and 0.22 for Re-call@10 and NDCG@10 respectively, with a mean SNR of 16.71 on the Speaker's endpoint. Therefore, for a perturbation which is less than 3% of the Speaker's audio, the surveillance system was able to detect less than two topics on average.
For the Listener side, with an SNR of 17.45, we achieve 0.31 and 0.38 Recall@10 and NDCG@10 respectively. The difference might stem from Skype's data compression. While lossless compression restores the original sample, lossy compression loses some of the data in the decompression process. Since Skype [39] uses lossy compression, it might have deleted parts of the perturbation and thus reduced the UAP's impact on the surveillance system. As a result, we report a higher SNR and lower Recall@10 and NDCG@10.
The results reveal the importance of comparing the UAP's performance on both the Speaker and Listener endpoints and also show that our approach is feasible and applicable in the real world. This experiment serves as a baseline for the experiments that follow.
VoIP communication applications usually process the audio samples before sending them. For instance, the application can perform echo cancellation, audio enhancement, and data compression. The results for the baseline show that such preprocessing can affect the UAP's ability to fool the system. Therefore, we examine how the use of another VoIP application, with different preprocessing, affects our solution. Due to its popularity during the COVID-19 pandemic and multiple reports on security and privacy risks [40] , we chose Zoom [48] to replace Skype in this experiment.
Since we only change the application, we obtain the same results on the Speaker endpoint that we obtained for Skype. However, for the Listener endpoint, after the data is sent through the VoIP application, with 18.73 SNR we achieve 0.33 Recall@10 and 0.41 NDCG@10. The results indicate that Zoom filters the UAP more than Skype does (resulting in a higher SNR), which explains why the perturbation causes slightly less disruption when using Zoom. However, the results are very similar. Thus, this shows that our framework can be used with different VoIP applications.
Past studies that used adversarial learning for privacy protection in other domains presented a tailor-made perturbation that was crafted based on the data of a specific user. For example, since Sharif et al. [32] created the perturbation using images of a specific person, that person is the only one that can use the glasses; as a result, for the typical person, this solution is a less scalable than a user agnostic solution. In our domain, a speaker agnostic UAP (including gender, accent, etc.) means that the proposed solution will have increased accessibility.
We address this issue by creating the UAP using the Lib-riSpeech dataset, which contains multiple speakers of both genders. To ensure that the UAP is speaker agnostic, we evaluate its effects on each of the 10 speakers in the Project Debater dataset. The data contains two female speakers and eight male speakers, each of whom has a different speaking tone, pace, style, etc. We employ the same evaluation process and metrics used in the previous experiments, but each speaker is examined individually. Table 1 summarizes the results for each of the 10 speakers. As observed in the previous experiments, the audio on the Listener endpoint is harder to manipulate, resulting in higher Recall@10 and NDCG@10 values. It seems that on average, the UAP performs better with male speakers than female speakers (DZ and HE). However, it is important to note that the dataset contains fewer female speakers (two females out of the 10 speakers). In addition, while the UAP works well for speakers DJ, TL, and SF, this was not the case for SN, HE, and DZ. Moreover, while the UAP shows great results for speaker JL on the Speaker side, it is dramatically less effective on the Listener side. The difference between the speakers might stem from the length of the audio sample or the sample's topic, as well as the speaker's gender. Nevertheless, the results suggest that the perturbation can work for multiple speakers, while preserving the Speaker and Listener's privacy.
We also examine how the conversation length affects the ability of the Speaker and Listener to hold a private conversation. Does a longer conversation increase the Speaker's risk of detection? If it does, to what extent? Therefore, we sort the records in the Project Debater dataset by their length, where the shortest and longest samples are 168 and 458 seconds respectively. For each audio length we perform the same evaluation process as in the baseline.
The results presented in Figure 6 suggest that the longer the conversation is, the higher the Recall@10 and NDCG@10 are. Thus, the perturbation become less effective for longer conversations. However, we found that audio samples under five minutes maintain the Speaker and Listener's privacy.
Thus far, our approach has assumed that the Speaker has full knowledge of the ASR architecture and weights and limited knowledge regarding the rest of the surveillance system. However, in the real world, where an eavesdropper aims to remain as stealthy as possible, this is not a reasonable assumption. [27] .
In most experiments that test transferability, the perturbation is added to all the samples in a test set, and the perturbed samples are been used for attacking the surrogate model. Only the samples that fool the surrogate model are used for evaluating the perturbation's effect on the unknown model. However, in this study we aim to create an algorithm that will not be limited to specific audio samples or topics. Therefore, we examine the effects of the UAP on all audio samples, regardless of their performance on the surrogate model. Table 2 compares the effects on Deep Speech and Kaldi of one UAP with a SNR of 6.9 and 8.7 on the Speaker and Listener endpoints respectively. As expected, the UAP is less effective against Kaldi. We note, however, that the perturbation was digitally crafted using the LibriSpeech dataset and targeted the Deep Speech model but was evaluated in the real world by using the Project Debater dataset to fool Kaldi. Thus, the UAP was able to fool the unknown ASR model for unknown audio samples for almost half of the topics. The perturbation was not designed to to be imperceptible but rather to fool an automated surveillance system, and since the Listener can easily understand the Speaker, even with a low SNR, the UAP serves its goal.
While many studies in the adversarial learning domain aim to create an imperceptible perturbation, in automated surveillance systems it is less likely for a human to analyze the massive amount of audio data. Since there is no risk that a human will identify the noise, our approach puts less focus on the perturbation's amplitude. Therefore, as long as the Listener can understand the Speaker, the UAP is not required to be imperceptible. However, the Eavesdropper can utilize automatic anomaly detection methods to detect the parties trying to manipulate their conversation. For example, by analyzing the ASR output, the Eavesdropper can use language models to search for sentences with abnormal syntactic logic. Yet, if many people were to consistently use the proposed framework, the automatic mechanism for extracting the topics of interest would become futile. For instance, if a company integrates our solution for the endpoints of all its employees, it could protect itself from industrial espionage. Since the perturbation is constantly used by all of the workers, all of the transcriptions are abnormal.
However, using the same UAP for all endpoints increases the chance that the Eavesdropper will also get his/her hands on the perturbation. Therefore, we suggest using Algorithm 1 with different parameters to create a different UAP for each endpoint. Therefore, even if the Eavesdropper is able to identify one perturbation, it is not enough to compromise the entire company.
Only a few studies have presented defenses against input manipulation in the audio domain [43, 42, 4, 41, 44] . Some defenses aim to detect whether the audio sample sent over-the-air is a human speaking or being played through a speaker. Such defenses cannot detect our perturbation, since it is digitally added in real time to the speech of a person speaking into a microphone. In addition, while we target an ASR model, many defenses focus on speaker recognition systems. Furthermore, in our scenario the speaker aims to hide the conversation topic and not his/her identity. Yang et al. [44] suggested filtering the perturbation using a simple input transformation that includes quantization, local smoothing, sampling, and an autoencoder. However, this method might be less effective against our UAP. If the speaker is aware of this process, he/she can consider the mitigation in the crafting process to improve the UAP's robustness to those transformations. However, the perturbation can still fool the model even if the speaker doesn't do anything. As we showed, the UAP is robust to the lossy preprocessing done by Skype and Zoom, and thus it might also bypass this defense as well.
We propose a possible simple mitigation method that allows the Eavesdropper to reduce the perturbation's ability to fool the ASR model. The UAP is a pressure wave, and thus it can be cancelled by emitting a sound wave with the same amplitude but with an inverse phase (noise cancellation). However, since the UAP is unknown to the Eavesdropper, he/she cannot distinguish the perturbation from the background noise. Yet, we found that performing noise cancellation based on a small sample of the background audio without speech is enough to reduce most of the UAP's effects. Therefore, with a small reduction in the recording quality, the Eavesdropper can use noise cancellation based on the beginning or end of the audio sample to cancel most of the UAP. To overcome this mitigation, the speaker can use voice activity detection (VAD) to play the UAP only when speaking. Without any speechless background noise, the Eavesdropper's ability to cancel the perturbation is limited.
In real-world settings, the Eavesdropper's system is a wellprotected secret. Therefore, it is unlikely that the Speaker will have any knowledge of any part of the system, including the type of data preprocessing, ASR model, and topic detector. As a result, in future work we aim to improve our framework so it can work under more strict conditions. To do so, we plan to conduct an extensive study encompassing multiple settings and covering most of the methods such systems may employ. Based on the knowledge gained, we aim to improve the algorithm for crafting the UAP to make it more robust to changes in the surveillance system's implementation. In addition, given the benefits it provides to our framework, we wish to gain a deeper understanding of the transferability property of adversarial perturbations in the audio domain.
In this study, we suggested a framework that utilizes adversarial perturbation for privacy protection in the audio domain. To support an unlimited stream of data, we created a repeatable UAP that can be added to audio samples of any length. Moreover, we showed that controlling the UAP's amplitude can be done in real time using constant multiplication. We created a real-world implementation that uses a TED, which contains a Teensy microcontroller, to add the perturbation directly to the audio input coming from the microphone. Our setup was designed to overcome a high privileged eavesdropper with access to the endpoints of both the speaker and listener. Using this setup, we evaluate our solution for two popular VoIP applications, multiple speakers, different audio lengths, and an unseen state-of-the-art ASR model. The results support the feasibility of our solution for privacy protection.
Throughout our work, the UAPs that were trained on one dataset (LibriSpeech) were evaluated on completely different data (IBM's Project Debater). This indicates that solutions in the privacy protection domain do not need to rely on an individual user's data to safeguard his/her information; thus, future research should examine how the use of different datasets to craft adversarial perturbations affects their performance with unknown data. Moreover, we show that simple manipulations to the UAP maintain the perturbation's effects, while making the solution more accessible and simple. Therefore, we also suggest that future research consider the user experience when proposing novel solutions.
The COVID-19 pandemic has resulted in more people communicating over the Internet, increasing the need for privacy protection solutions. We believe that adversarial learning might be the key to an effective privacy preservation solution.