Adversarial machine learning
Adversarial machine learning is a machine learning technique that attempts to exploit models by taking advantage of obtainable model information and using it to create malicious attacks. The most common reason is to cause a malfunction in a machine learning model. Most machine learning techniques were designed to work on specific problem sets in which the training and test data are generated from the same statistical distribution (IID). When those models are applied to the real world, adversaries may supply data that violates that statistical assumption. This data may be arranged to exploit specific vulnerabilities and compromise  the results. The four most common adversarial machine learning strategies are evasion, poisoning, model stealing (extraction), and inference. History. In 2004, Nilesh Dalvi and others noted that linear classifiers used in spam filters could be defeated by simple "evasion attacks" as spammers inserted "good words" into their spam emails. (Around 2007, some spammers added random noise to fuzz words within "image spam" in order to defeat OCR-based filters.) In 2006, Marco Barreno and others published "Can Machine Learning Be Secure?", outlining a broad taxonomy of attacks. As late as 2013 many researchers continued to hope that non-linear classifiers  (such as support vector machines and neural networks) might be robust to adversaries, until Battista Biggio and others demonstrated the first gradient-based attacks on such machine-learning models (2012-2013). In 2012, deep neural networks began to dominate computer vision problems; starting in 2014, Christian Szegedy and others demonstrated that deep neural networks could be fooled by adversaries, again using a gradient-based attack to craft adversarial perturbations. Recently, it was observed that adversarial attacks are harder to produce in the practical world due to the different environmental constraints that cancel out the effect of noises. For example, any  small rotation or slight illumination on an adversarial image can destroy the adversariality. In addition, researchers such as Google Brain's Nicholas Frosst point out that it is much easier to make self-driving cars miss stop signs by physically removing the sign itself, rather than creating adversarial examples. Frosst also believe that the adversarial machine learning community incorrectly assumes models trained on a certain data distribution will also perform well on a completely different data distribution. He suggests that a new approach to machine learning should be explored, and is currently working on a unique neural network that has  characteristics more similar to human perception than state of the art approaches. While adversarial machine learning continues to be heavily rooted in academia, large tech companies such as Google, Microsoft, and IBM have begun curating documentation and open source code bases to allow others to concretely assess the robustness of machine learning models and minimize the risk of adversarial attacks. Examples. Examples include attacks in spam filtering, where spam messages are obfuscated through the misspelling of "bad" words or the insertion of "good" words; attacks in computer security, such as obfuscating malware code within network packets or modifying the characteristics  of a network flow to mislead intrusion detection; attacks in biometric recognition where fake biometric traits may be exploited to impersonate a legitimate user; or to compromise users' template galleries that adapt to updated traits over time. Researchers showed that by changing only one-pixel it was possible to fool deep learning algorithms. Others 3-D printed a toy turtle with a texture engineered to make Google's object detection AI classify it as a rifle regardless of the angle from which the turtle was viewed. Creating the turtle required only low-cost commercially available 3-D printing technology. A  machine-tweaked image of a dog was shown to look like a cat to both computers and humans. A 2019 study reported that humans can guess how machines will classify adversarial images. Researchers discovered methods for perturbing the appearance of a stop sign such that an autonomous vehicle classified it as a merge or speed limit sign. McAfee attacked Tesla's former Mobileye system, fooling it into driving 50 mph over the speed limit, simply by adding a two-inch strip of black tape to a speed limit sign. Adversarial patterns on glasses or clothing designed to deceive facial-recognition  systems or license-plate readers, have led to a niche industry of "stealth streetwear". An adversarial attack on a neural network can allow an attacker to inject algorithms into the target system. Researchers can also create adversarial audio inputs to disguise commands to intelligent assistants in benign-seeming audio; a parallel literature explores human perception of such stimuli. Clustering algorithms are used in security applications. Malware and computer virus analysis aims to identify malware families, and to generate specific detection signatures. Attack modalities. Taxonomy. Attacks against (supervised) machine learning algorithms have been categorized along three primary axes: influence on the  classifier, the security violation and their specificity. This taxonomy has been extended into a more comprehensive threat model that allows explicit assumptions about the adversary's goal, knowledge of the attacked system, capability of manipulating the input data/system components, and on attack strategy. This taxonomy has further been extended to include dimensions for defense strategies against adversarial attacks. Strategies. Below are some of the most commonly encountered attack scenarios: Evasion. Evasion attacks are the most prevalent type of attack. For instance, spammers and hackers often attempt to evade detection by obfuscating the content of spam emails and malware. Samples  are modified to evade detection; that is, to be classified as legitimate. This does not involve influence over the training data. A clear example of evasion is image-based spam in which the spam content is embedded within an attached image to evade textual analysis by anti-spam filters. Another example of evasion is given by spoofing attacks against biometric verification systems. Evasion attacks can be generally split into two different categories: black box attacks and white box attacks. Poisoning. Poisoning is adversarial contamination of training data. Machine learning systems can be re-trained using data collected during operations. For  instance, intrusion detection systems (IDSs) are often re-trained using such data. An attacker may poison this data by injecting malicious samples during operation that subsequently disrupt retraining. Model stealing. Model stealing (also called model extraction) involves an adversary probing a black box machine learning system in order to either reconstruct the model or extract the data it was trained on. This can cause issues when either the training data or the model itself is sensitive and confidential. For example, model stealing could be used to extract a proprietary stock trading model which the adversary could then use for their  own financial benefit. Inference. Inference attacks, leverages overgeneralization on training data, a common weakness of supervised machine learning models, in order to identify data used during model training. For attackers, this can be done even without knowledge or access to a target model's parameters, raising security concerns for models trained on sensitive data, including but not limited to medical records and/or personally identifiable information. With the emergence of transfer learning and public accessibility of many state of the art machine learning models, tech companies are increasingly drawn to create models based on public ones, giving attackers freely accessible information  to the structure and type of model being used. However, membership inference relies heavily on overfitting resulting from poor machine learning practices, meaning a model that generalizes well to the real distribution of data should theoretically be more secure to membership inference attacks. Specific attack types. There are a large variety of different adversarial attacks that can be used against machine learning systems. Many of these work on both deep learning systems as well as traditional machine learning models such as SVMs and linear regression. A high level sample of these attack types include: Adversarial examples. An adversarial example refers  to specially crafted input which is designed to look "normal" to humans but causes misclassification to a machine learning model. Often, a form of specially designed "noise" is used to elicit the misclassifications. Below are some current techniques for generating adversarial examples in the literature (by no means an exhaustive list). Black Box Attacks. Black box attacks in adversarial machine learning assumes that the adversary can only get outputs for provided inputs and has no knowledge of the model structure or parameters. In this case, the adversarial example is generated either using a model created from scratch, or without any  model at all (excluding the ability to query the original model). In either case, the objective of these attacks are to create adversarial examples that are able to transfer to the black box model in question. Square Attack. The Square Attack was introduced in 2020 as a black box evasion adversarial attack based on querying classification scores without the need of gradient information. As a score based black box attack, this adversarial approach is able to query probability distributions across model output classes, but has no other access to the model itself. According to the paper's authors, the proposed  Square Attack required less queries than when compared to state of the art score based black box attacks at the time. To describe the function objective, the attack defines the classifier as formula_1, with formula_2 representing the dimensions of the input and formula_3 as the total number of output classes. formula_4 returns the score (or a probability between 0 and 1) that the input formula_5 belongs to class formula_6, which allows the classifier's class output for any input formula_5 to be defined as formula_8. The goal of this attack is as followed: formula_9 In other words, finding some perturbed  adversarial example formula_10 such that the classifier incorrectly classifies it to some other class under the constraint that formula_10 and formula_5 are similar. The paper then defines loss formula_13 as formula_14 and proposes the solution to finding adversarial example formula_10 as solving the below constrained optimization problem: formula_16 The result in theory is an adversarial example that is highly confident in the incorrect class but is also very similar to the original image. To find such example, Square Attack utilizes the iterative random search technique to randomly perturb the image in hopes of improving the objective function. In each step , the algorithm perturbs only a small square section of pixels, hence the name Square Attack, which terminates as soon as an adversarial example is found in order to improve query efficiency. Finally, since the attack algorithm uses scores and not gradient information, the authors of the paper indicate that this approach is not affected by gradient masking, a common technique formerly used to prevent evasion attacks. HopSkipJump Attack. This black box attack was also proposed as a query efficient attack, but one that relies solely on access to any input's predicted output class. In other words, the HopSkipJump attack  does not require the ability to calculate gradients or access to score values like the Square Attack, and will require just the model's class prediction output (for any given input). The proposed attack is split into two different settings, targeted and untargeted, but both are built from the general idea of adding minimal perturbations that leads to a different model output. In the targeted setting, the goal is to cause the model to misclassify the perturbed image to a specific target label (that is not the original label). In the untargeted setting, the goal is to cause the model  to misclassify the perturbed image to any label that is not the original label. The attack objectives for both are as follows where formula_5 is the original image, formula_18 is the adversarial image, formula_2 is a distance function between images, formula_20 is the target label, and formula_21 is the model's classification class label function: formula_22 formula_23 To solve this problem, the attack proposes the following boundary function formula_24 for both the untargeted and targeted setting: formula_25 This can be further simplified to better visualize the boundary between different potential adversarial examples: formula_26 With this boundary function, the attack then  follows an iterative algorithm to find adversarial examples formula_18 for a given image formula_5 that satisfies the attack objectives. Boundary search uses a modified binary search to find the point in which the boundary (as defined by formula_24) intersects with the line between formula_5 and formula_18. The next step involves calculating the gradient for formula_5, and update the original formula_5 using this gradient and a pre-chosen step size. HopSkipJump authors prove that this iterative algorithm will converge, leading formula_5 to a point right along the boundary that is very close in distance to the original image. However, since HopSkipJump  is a proposed black box attack and the iterative algorithm above requires the calculation of a gradient in the second iterative step (which black box attacks do not have access to), the authors propose a solution to gradient calculation that requires only the model's output predictions alone. By generating many random vectors in all directions, denoted as formula_37, an approximation of the gradient can be calculated using the average of these random vectors weighted by the sign of the boundary function on the image formula_38, where formula_39 is the size of the random vector perturbation: formula_40 The result of  the equation above gives a close approximation of the gradient required in step 2 of the iterative algorithm, completing HopSkipJump as a black box attack. White Box Attacks. White box attacks assumes that the adversary has access to model parameters on top of being able to get labels for provided inputs. Fast Gradient Sign Method (FGSM). One of the very first proposed attacks for generating adversarial examples was proposed by Google researchers Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. The attack was called fast gradient sign method, and it consists of adding a linear amount of in-perceivable noise  to the image and causing a model to incorrectly classify it. This noise is calculated by multiplying the sign of the gradient with respect to the image we want to perturb by a small constant epsilon. As epsilon increases, the model is more likely to be fooled, but the perturbations become easier to identify as well. Shown below is the equation to generate an adversarial example where formula_5 is the original image, formula_42 is a very small number, formula_43 is the gradient function, formula_44 is the loss function, formula_45 is the model weights, and formula_46 is the true label. formula_47  One important property of this equation is that the gradient is calculated with respect to the input image since the goal is to generate an image that maximizes the loss for the original image of true label formula_46. In traditional gradient descent (for model training), the gradient is used to update the weights of the model since the goal is to minimize the loss for the model on a ground truth dateset. The Fast Gradient Sign Method was proposed as a fast way to generate adversarial examples to evade the model, based on the hypothesis that neural networks cannot  resist even linear amounts of perturbation to the input. Carlini & Wagner (C&W). In an effort to analyze existing adversarial attacks and defenses, researchers at the University of California, Berkeley, Nicholas Carlini and David Wagner in 2016 propose a faster and more robust method to generate adversarial examples. The attack proposed by Carlini and Wagner begins with trying to solve a difficult non-linear optimization equation: formula_49 Here the objective is to minimize the noise (formula_50), added to the original input formula_5, such that the machine learning algorithm (formula_21) predicts the original input with delta (or formula_53) as some other class  formula_54. However instead of directly the above equation, Carlini and Wagner propose using a new function formula_55 such that: formula_56 This condenses the first equation to the problem below: formula_57 and even more to the equation below: formula_58 Carlini and Wagner then propose the use of the below function in place of formula_55 using formula_60, a function that determines class probabilities for given input formula_5. When substituted in, this equation can be thought of as finding a target class that is more confident than the next likeliest class by some constant amount: formula_62 When solved using gradient descent, this equation  is able to produce stronger adversarial examples when compared to fast gradient sign method that is also able to bypass defensive distillation, a defense that was once proposed to be effective against adversarial examples. Defenses. Researchers have proposed a multi-step approach to protecting machine learning. Mechanisms. A number of defense mechanisms against evasion, poisoning, and privacy attacks have been proposed, including: