“Real Attackers Don’t Compute Gradients”: Bridging the Gap Between Adversarial ML Research and Practice
Author Abstract —Recent years have seen a proliferation of research on adversarial machine learning .
Numerous papers demonstrate powerful algorithmic attacks against a wide variety of machine learning (ML) models, and numerous other papers propose defenses that can withstand most attacks.
However, abundant real-world evidence suggests that actual attackers use simple tactics to subvert ML-driven systems, and as a result security practitioners have not prioritized adversarial ML defenses.
Motivated by the apparent gap between researchers and prac- titioners, this position paper aims to bridge these two domains.
We first present three real-world case studies from which we can glean practical insights unknown or neglected in research.
Next, we analyze all adversarial ML papers recently published in top security conferences and highlight positive trends and blind spots.
Finally, we state positions on precise and cost-driven threat modeling, collaboration between industry and academia, and reproducible research.
If adopted, our positions will increase the real-world impact of future endeavours in adversarial ML, bringing both researchers and practitioners closer to their shared goal of improving the security of ML systems.
I. I NTRODUCTION
Several recent surveys indicate that protecting ML models is not a leading security concern for practitioners [1]–[5]
According to the few recorded accounts of security failures “in the wild
” ML systems can be broken by na¨ıve attackers that are not systematically exploiting the vulnerabilities of
ML, but rather are developing attacks by guessing—either indiscriminately or by some coarse heuristic [6], [7].
Redteam exercises on ML systems often take advantage of security gaps that are agnostic to the existence of an ML model, and subsequent defensive recommendations are likewise more broad than, e.g., adversarial retraining [8], [9].
Additionally, the ML models deployed in production-grade ML systems are often not directly observable (and are sometimes even unreachable) by most attackers [10].
On the other hand, researchers assert that real ML implementations should not follow the principle of “security by obscurity” [11], and that security evaluations of ML models should assume worst-case scenarios [12]–[14].
Indeed, no one can exclude the possibility of future powerful adversaries turning their attention to the ML models embedded in real systems.
Fueled by such “what ifs,” thousands of papers [15] showcase successful security violations of ML models by means of sophisticated strategies such as gradient-based algorithms.
These few anecdotes suggest that the field of ML security suffers from a mismatch between the priorities of practitioners and the focus of researchers.
In this position paper we argue that the gap between adversarial ML research and practice is real and identify several underlying causes.
We aim to reduce this gap by proposing guidelines for future endeavours that reflect our observations of aspects overlooked by, or inconsistent within, existing literature.
To reach our goal, we first revisit the fundamentals of ML security (§II).
We argue that real deployed ML models are part of complex ML systems whose architectures are unknown to researchers, and that cybersecurity is rooted in binding economic considerations for both attackers and defenders.
To assist researchers in understanding crucial facets of operational ML security, we then present three original case studies from the real-world (§III) that elucidate: (a) the architecture of a complex ML system deployed in a large online social network; (b) the lack of evidence of adversarial examples in the context of a commercial-grade ML system for phishing webpage detection; (c) the role played by time and domain expertise in devising sophisticated attacks during an acclaimed ML evasion competition.
Next, we turn our attention to the research domain, and take a snapshot of the current landscape of adversarial ML as portrayed in scientific papers (§IV).
After surveying the proceedings of the “Top-4” security conferences fromto 2021, we systematically analyze all 88 papers that consider either attacks against ML or corresponding defenses.
Of these papers, 89% only consider algorithms based on neural networks, 63% focus on computer vision, and 80% perform their evaluation on “benchmark” data sets.
We also discover several inconsistencies in the definitions of the threat models presented in reputable prior work.
On the other hand, we identify several positive trends , such as an increasing amount of research that considers “weaker” attackers who “ignore” the ML model and reach their goal by targeting other elements of the ML system.
Finally, we coalesce all our observations by taking four positions that both researchers and practitioners in ML security can adopt to close the gap between these two domains (§V).
In particular, we encourage the community to (a) adapt threat models from ML models to ML systems, (b) integrate threat models with cost-driven assessments, (c) build collaborations between industry and academia, and (d) embrace a “just culture” for source-code disclosure.
In summary, we make three major contributions:
• We present three real-world case studies , showing practical insights unknown or neglected in research.
• We analyze all recent adversarial ML papers in top security venues, highlighting positive trends and blind spots.
• We state four positions that, if adopted, will help bridge the gap between research and practice in ML security.
Our contributions also include the detailed findings of our survey of recent security papers—found in the Appendix.
II. R EVISITING M ACHINE L EARNING S ECURITY
We begin by reviewing the foundations of the three pillars of our paper: machine learning systems (§II-A), security of machine learning (§II-B), and practical cybersecurity (§II-C).
We also compare our paper with related work (§II-D).
A. Overview of Machine Learning (Systems)
Machine learning in one paragraph.
The fundamental goal of machine learning (ML) is to create “machines that can make decisions by learning from data” [16].
At its core, the process entails applying a learning algorithm to training data , which yields a machine learning model .
The purpose of the ML model is to generalize to new data: given an input , the model produces an output that can be used to accomplish a given task, such as predicting future trends, detecting malware, recognizing speech patterns, or inferring objects in images [16], [17].
The effectiveness of the ML model is measured during a validation phase , during which the
ML model analyzes test data and its predictions are compared to some known ground truth (e.g., labels [18]).
Effectiveness can be measured through various metrics (e.g., accuracy ).
Informally, an ML model belongs to either of two “paradigms” that characterize the underlying learning algorithm.
Those based on neural networks [17] are denoted as deep learning (DL), whereas others are broadly denoted as shallow learning and include, e.g., SVMs and tree-based algorithms [19].
Using ML in practice.
If an ML model is determined to be effective, it can be integrated into a machine learning system , which itself can be a component of some product or service.
Indeed, an ML model by itself is usually insufficient for operational deployment.
First, it must receive an input that may itself be derived from a separate component (e.g., signal capture or preprocessing 1 ).
Second, the ML model’s output may be further analyzed for decision making.
We illustrate an
ML system with a linear data-processing pipeline in Fig.
[Figure]
Real ML systems may include pipelines having various components—each potentially having its own applications of
ML (e.g., image scaling [20] or word embeddings [21]).
Some
ML systems require constant updates, which can be done in an automated fashion [22].
Others may adopt companyspecific guidelines [23].
A given input may not even reach any ML model integrated in the ML system.
This is typical, e.g., in Network Intrusion Detection Systems (NIDS), wherein samples matching known “signatures” are put in a different pipeline that does not include any ML model [24].
O BSERVATION :
A machine learning model is merely a single component within a machine learning system .
Types of ML systems.
Many IT infrastructures already rely on ML systems at either small or large scales.
Without loss of generality, we identify two main categories of ML systems that distinguish the relationships (i.e., knowledge and degree of interaction) between the ML system and its users.
• O PEN .
The
source
code of
these ML
systems is
publicly available
This category may include custom-built ML systems where the code is subsequently released.
Obviously any ML system is OPEN for its developers
• C LOSED .
These ML systems
are developed in a proprietary setting, and distributed to their end users, who cannot inspect (i) the underlying source code, and
(ii
) the components enclosed in the ML system.
These ML systems are either: unrestricted , if they can be freely used; or restricted , if their use is subject to some restriction, such as a fee or a threshold based on queries.
We provide some examples of these ML systems in Table I.
[Table]
Both OPEN and CLOSED ML systems resemble the schematic shown in Fig.
1: they receive an input and provide an output .
Typically, the users of an ML system can control the input, and have visibility of the output (e.g., via an API [33]).
Such properties, however, may not hold for some special cases, which we denote as INVISIBLE ML systems.
Control and configuration of INVISIBLE ML systems are reserved for “highprivilege” users (e.g., developers or sys-admins), for whom the
ML system can be either OPEN or CLOSED .
The functionality of these systems depends on the interactions of “low-privilege” users (e.g., employees or customers), to whom the operation of the ML system is not readily apparent.
A low-privilege user may receive some feedback based on the ML system’s, but it may be a result of other systems or it may arrive after a very long and unpredictable delay.
Simply put, low-privilege users of INVISIBLE ML systems may not know (i) if something happened, (ii) when it happened, (iii) why it happened, and/or (iv) what changed.
We provide a schematic of an INVISIBLE
ML system for network management in Fig.
2, and we will present a case study in §III-A.
[Figure]
B. Security of Machine Learning
Adversarial Machine Learning.
Thousands of papers [15] addressed the security of machine learning, a research field typically denoted as “adversarial machine learning” [34].
Although work in this field has existed since the early 2000s [35], adversarial ML became popular when Szegedy et al.
[36] discovered that DL was vulnerable to adversarial examples .
In simple terms, adversarial examples are “similar” to “normal” inputs but induce a given ML model to behave incorrectly [37].
Adversarial examples can certainly be used in malicious ways: for instance, an attacker can craft an adversarial example that evades an ML model powering a security system, e.g., a malware classifier [38].
(We provide the definition of evasion attacks in Appendix A-A.)
Nevertheless, we will show that adversarial examples are only one of a myriad of threats that can violate the security of ML systems.
Threat Model.
Security assessments require the definition of a threat model , which establishes the relationship between an attacker and their target [10].
The seminal paper by Huang et al.
[34] was among the first to formalize the potential threats to ML.
Since then, more efforts followed—some more general (e.g., [37], [39]), others more specific (e.g., [40], [41]).
In the ML context, threat models characterize an attacker’s relationship to an ML model through four
G OAL .
Attackers can have diverse GOALS , such as violating the integrity or the availability of the ML model [42].
They can also, however, attempt to “steal” the ML model [43], or extract private data from it via membership inference [44] attacks that infer whether a person’s data is in the model’s training set.
K NOWLEDGE .
Srndic¸ and Laskov [41] pointed out that the attacker’s KNOWLEDGE spans over three elements of an ML model: the training data, the feature set, and the algorithm.
Depending on the information available to the attacker, most literature adopts a “box” terminology [39]: “white-box,” when the attacker knows everything ; “black-box,” when the attacker knows nothing ; and “gray-box,” for intermediate cases [40].
C APABILITIES .
Existing literature shows that attackers use various CAPABILITIES to tamper with ML models.
For instance, they can interfere with the training phase of the
ML model [45]; they can control inputs to a (trained)
ML model and observe its output [46]; and/or they can operate from diverse spaces of the data pipeline of a ML system (the so-called “problem/input” and “feature” spaces [47]).
In some circumstances [48], the attacker cannot interact with the ML model, but they can obtain a “surrogate” ML model— which is used, e.g., to transfer adversarial examples [42].
S TRATEGY .
To reach their GOAL , attackers exploit their
KNOWLEDGE and CAPABILITIES and enact a STRATEGY , e.g.: an attacker that fully knows a deep learning model can use its gradient to craft adversarial examples [49]; attackers who can manipulate training data may introduce a backdoor [50].
We observe, however, that the terminology above has a limitation: it focuses exclusively on the ML model (the inner and dotted box in Fig. 1), which is just a single component of the much more complex ML system (the outer box in Fig. 1).
We will elaborate on how to overcome this limitation in §V.
Adversarial ML in practice.
Several works investigated whether the explosive popularity of adversarial ML in research papers has been received with equal interest from industry practitioners.
Let us trace the major findings since 2020: (2020)
Kumar et al.
[3] conducted a survey of 28 companies.
Participants were asked about their perspectives on adversarial ML.
Most participants did not know how to respond and were primarily worried about poisoning attacks.
The following year, Boenisch et al.
[2] raised an important warning: many ML developers confessed that “I never thought about securing my ML models.”
(2021) Sun et al.
[1] studied ML models deployed on mobile apps and (perhaps unsurprisingly) found that: “41% of
ML apps do not protect their models at all.”
A year later, Bieringer et al.
[4] interviewed developers of
ML and concluded that “most lack adequate understanding to secure ML systems in production”, even though one-third “feel insecure about adversarial ML.”
Grosse et al.
[5] interviewed hundreds of practitioners asking their opinion on countermeasures to adversarial
ML attacks.
The general consensus was “Why do so?”
We find it surprising that the viewpoint of practitioners has barely changed, even in spite of warnings from sources beyond the research domain.
For example, in 2020 Gartner predicted that “by 2022, 30% of cyberattacks will leverage training-data poisoning, model theft, or adversarial examples.”
O BSERVATION :
Despite abundant evidence showing that ML models are vulnerable, practitioners persist in treating such threats as low priority.
C. (Economics of) Cybersecurity
An elementary principle.
It is well known that “there is no such thing as a foolproof system” [52] (the actual quote is from a renowned con artist [53]).
Given a sufficient amount of resources, any attacker can succeed in their goal [54].
Indeed, the purpose of security mechanisms is to raise the cost sustained by the attacker [55], either by requiring more resources (e.g., time to succeed) or by increasing the consequence of failure (e.g., getting caught).
However, countermeasures also have costs (e.g., include implementation efforts, preliminary evaluations, maintenance, and computational resources [56], [57]) that must be factored in to any decision about whether to build or deploy them.
Simply put, operational cybersecurity is rooted in economics [54], [58].
Cybersecurity in practice.
Let us substantiate our previous claims with public statements made by cybersecurity professionals.
On July 22nd, 2022, a tweet [59] by Sophos
AI’s employee revealed that “Adversarial examples are not the primary concern at Sophos”.
This tweet inspired an insightful discussion between renowned researchers and practitioners in this field.
Konstantin Berlin, Head of AI at Sophos, explains his thinking about adversarial examples [60] as follows:
Head of Sophos AI : “If you look at cybercrime in economical terms (as you should because it is a business) the optimization for an adversarial ex. is not the expensive part, it is the engineering part of building a tool that can create a diverse set of attacks with no obvious watermarks.”
( source : [61])
Konstantin Berlin also stated that “Given the existence of an already large number of more prevalent attacks that do bypass detections, why prioritize this one?”
Such a simple (and recent) use-case shows that operational decisions are dictated by economics, and practitioners prioritize threats that are deemed to be more important for their businesses.
O BSERVATION :
Economics is the main driver of practical cybersecurity, both for attackers and defenders.
For transparency, we report the rest of the discussion that transpired in such Twitter thread in Appendix A-B, where we will also provide our own viewpoint on this subject.
D. Related Work
Many papers share our goal of improving the security of real-world ML systems.
The authors of [14] and [62] provide practical guidelines on ML for security—whereas we focus on the security of ML.
Some related papers (e.g., [2]–[5]) focus exclusively on the industry perspective.
On the other hand, most literature surveys (e.g., [37], [39], [63]) focus just on the research perspective.
We consider both perspectives, because we aim to bridge the existing gap between these two sides.
Perhaps the closest effort to our paper is the (unpublished) work by Gilmer et al.
[64], which attempts to contextualize the implications of adversarial ML in the real-world.
Despite sharing our goal, [64] has two shortcomings: first, it mostly focuses on computer vision and deep learning—which, as we will show, represent only a subset of the conceivable applications of ML in reality; second, the remarks made in [64] relate to papers published before 2018—i.e., almost five years ago, when ML deployments were not as popular as today.
We briefly summarize two closely related research areas that are orthogonal to this paper.
(1) The performance of ML tends to degrade over time [65]-[66].
This phenomenon (called concept drift ) is due to changes in the data distribution that are physiological in nature and sometimes even beyond an attacker’s control.
(2) ML can also be used as an attack vector (e.g., “deepfakes” [67], or “attribute inference” [68]).
This research area (i.e., “offensive ML”) is complementary to ours, because we focus on the protection of ML systems.
To the best of our knowledge, this is the first position paper (on ML security) whose main positions stem from observations based on (i) a systematic analysis of recent research, and on (ii) case studies from real experience.
III. C ASE S TUDIES (R EAL E XPERIENCE )
We now describe three case studies fostering the contribution of industry practitioners.
Our intent is twofold: elucidate some practical aspects that may be obscure to researchers; and induce more practitioners to do so in the future.
The first (§III-A) presents the architecture of a real INVISIBLE ML system deployed in an online social network (OSN).
The second (§III-B) describes the triaging of phishing webpages that were poorly classified by the ML detector of a security company.
The third (§III-C) sheds light on the role of time and domain expertise to devise query-efficient attacks during an ML evasion challenge organized by renowned tech companies.
(In Appendix A-D, we provide a fourth case study showing that even ‘adversarially aware’ malware detectors can be easily fooled with expert knowledge.)
Because all of our case studies entail ML systems for detection tasks, this section will focus on “evasion attacks,” i.e., misclassifications at test time (of ‘malicious’ samples).
A. “The four A’s of abuse fighting”: ML Systems in an OSN
This case study reflects the experience of a real OSN that makes ample use of ML (in various forms) to manage its services and, in particular, to fight spam abuse.
(We further discuss this case study in Appendix A-G1.)
Scenario.
An attacker attempts to spread spam on a social network; for example, they want to post a pornographic image with some text, which may lure a user to click on an embedded
URL.
The OSN does not allow these activities on their platform, and hence employs an ML system to prevent this type of content from appearing.
The attacker—aware of the existence of the ML system—tries to evade the detector by perturbing the content and/or changing their behavior.
Problem.
On the surface, this scenario represents the “evading the ML spam detector” setting that is typically envisioned in research papers (e.g. [69], [70]).
However, most such papers assume that the ML system consists of (i) a single ML classifier that (ii) analyzes content and (iii) outputs the decision to either block or allow the content—a decision which (iv) is readily observable by the content’s author (e.g., [71]).
Such a “black-box” scenario (which has also been envisioned in its “white-box” variant [72]) assumes an adversary who can send an arbitrary number of requests to the ML model and use its response to craft adversarial examples—potentially with the added constraint of preserving the original input semantics.
Reality.
Production systems however, are far more complex (see §II-A).
In fact, content classification (block/allow) is the last line in a series of defenses erected by this OSN, which uses an entire funnel of countermeasures that work in concert to provide defense at different layers: to be successful, an attacker must evade all of them.
We denote this funnel as “the four
A’s of abuse fighting”: A UTOMATION , A CCESS , A CTIVITY , and
A PPLICATION .
We provide an overview in Fig. 3, and explain each term below.
Without loss of generality, we can assume that the input to the ML system is an action (e.g., posting an image) executed by a given entity (e.g., user account).
1) A UTOMATION layer.
At the top of the funnel is a simple question: “is the action due to automation?”
Anti-automation is the most general defense because it applies to both read and write surfaces and to logged-in and logged-out traffic.
Indeed, OSNs are constantly targeted by bots and automated scripts [73].
The AUTOMATION layer automatically prevents “black-box” evasion strategies based on repeatedly querying the system over short time frames.
A CCESS layer.
Automation detection is the only viable defense of the OSN against logged-out reads at scale.
In contrast, an account is needed for writes (e.g., posting content on the OSN) or logged-in scraping.
Most malicious activity is perpetrated by fake or compromised user accounts [74].
An attacker can, however, find other ways (e.g., fake advertiser accounts or stolen access tokens) to obtain the required permissions to launch an offense.
Simply put, the role of the ACCESS layer is to analyze any write request made to the OSN and determine whether the corresponding write permission was obtained legitimately by the requesting entity.
A CTIVITY layer.
If the OSN cannot stop malicious actions just for being automated (in the AUTOMATION layer), and cannot catch bad actors when they obtain access (in the ACCESS layer), then the next opportunity is when they begin carrying out their (malicious) activity on the OSN.
To reach their goal, spammers and other abusers must act in fundamentally different ways from normal users.
Such differences include how they interact with the social graph [75], the distribution of the actions undertaken on the OSN, and/or how fast they execute their actions.
A PPLICATION layer.
The first three layers of the funnel are generic and mostly boil down to well-known cybersecurity practices, some of which may not require ML.
However, if the funnel ended at the third layer, two concerns would arise: (i)
What happens if an attacker bypasses all three layers?
and (ii)
What about specific prohibited content such as nudity, hate speech, or gun sales—which are activities that, despite being against the OSN’s policies, can be unintentionally produced by “benign” users?
To address these concerns, the OSN has a fourth, APPLICATION layer.
Assuming that the above layers are mostly effective (i.e., low false positives/negatives), most of the violations in this layer will be small-scale and driven by real users.
This is the layer where ML can really shine, such as nudity detectors and hate-speech classifiers––i.e.
, the
DL models typically targeted in research papers [72].
Complexity.
A real spam-detection system has numerous additional complexities not shown in Fig.
For example, a blocked attacker typically has no way of knowing which layer “caught” them and therefore must try many different evasion strategies.
Each layer itself may contain both shallow learning and deep learning methods as well as heuristic rules, further complicating the attacker’s task.
The layers can inform each other even if an attacker is not blocked—for example, an account may be given a smaller rate limit threshold ( ACTIVITY layer) if a fake-account detector ( ACCESS layer) returns a “suspicious” score but is not confident enough to block outright.
Finally, even if some layers output a decision of “malicious,” such output is postprocessed by a response layer that decides which action to take: Ban the user?
Remove the content?
Show a CAPTCHA?
Wait and see?
The attacker may even observe different responses from multiple instances of the same input, depending on factors beyond the attacker’s control—a characteristic of an INVISIBLE ML system (§II-A).
O BSERVATION :
Real ML systems include many components, not necessarily all using ML.
Attackers must bypass all such components to be successful.
We remark, however, that the complexity of real ML systems does not necessarily mean that such systems are “omnipotent”.
The following case study will reveal that even full-fledged ML detectors can be thwarted via simple tactics.
B. Are there Adversarial Examples in the Wild (Web)?
To assess the prevalence and nature of adversarial examples “in the wild” (i.e., in the real world), we first consulted the AI Incident Database [76], containing over 1 600 reports of AI failures (as of Aug. 2022).
Unfortunately, querying the database with the keywords “evasion” and “adversarial machine learning” yielded only 2 and 6 results, respectively.
Apparently, only a limited number of (reported) incidents involve the types of attacks typically portrayed in adversarial ML papers.
To gain further insight, we asked a leading cybersecurity company whether they encounter adversarial examples when manually triaging security incidents.
This case study provides their answer.
Context.
We consider the problem of phishing webpage detection .
Similarly to the previous case study, the ML system is a commercial-grade detector, composed of diverse modules—all with the underlying goal of “catching phish.”
Specifically, the ML system (which is CLOSED and INVISIBLE to the attacker 3 ) consists of an ensemble of image-based classifiers, each responsible for a specific task (e.g., logo attribution, visual comparison, input form detection).
The ML system leverages the principle of active learning [77]: For each input, the output is a ‘phishing confidence’ score.
Besides using such score to make a decision (i.e., block/allow), security analysts use such score to improve the ML system, by triaging inputs for which the ML system is uncertain.
Such a setting fits our objective: most related research focuses on computer vision (§IV-A) and the ML system naturally suggests to the analysts which inputs can be linked to adversarial examples.
Method.
To identify adversarial examples on the web, we searched for them among the 40 domains that (according to the cybersecurity company) were most commonly targetedby phishing campaigns in July 2022.
During this timeframe, the ML system analyzed hundreds of thousands of inputs— among these, 9 174 were flagged as ‘uncertain’ by the ML system, and required manual triage.
We used these as a starting point, and we began to study all such samples (i.e., screenshots).
To make our in-depth analysis humanly feasible, we visually inspected each sample asking ourselves “can this be an adversarial example?”, and decided to stop once we obtained a positive answer for 100 samples.
We reached this number after going through 4 600 samples (half of our initial population): the discarded ones had nothing in common with adversarial examples, due to being “out of distribution” [78]— i.e., they were significantly different from any sample included in the training data of the ML system (e.g., a domain using a new logo, or shifting backgrounds—typical of, e.g., Netflix).
We then used the 100 ‘positive’ samples as basis for our indepth analysis.
Our aim was to determine the reasons that induced the ML system to be ‘uncertain’ and, in particular, if such reason could be traced back to gradient-based techniques typical of adversarial examples.
To reduce bias, this entire process (first inspection and in-depth analysis) entailed two authors who worked independently and had regular meetings to resolve issues and arrive at a consensus.
Results.
Our analysis suggests that attackers rely on an arsenal of relatively simple strategies—which are often effective.
We quantify the prevalence of these strategies in Table II and show their effects on real webpages (“from the wild”) in Fig. [Table]
The most prevalent attacks include cropping, masking, stretching, or blurring techniques—all of which induce optical character recognition (OCR) algorithms to incorrectly extract text from the page.
Such “anti-OCR” techniques have been employed for decades by phishers [79] and can (still) ably fool ML systems without relying on gradient computations.
These methods typically target the company name when it is part of a logo, mostly by manipulating the logo’s visual representation.
We also found samples with a different logo altogether, as well as and “company name style” attacks that alter the logo’s company name.
Even more rudimentary attacks simply remove the logo and/or the company name (or even both of them incases).
In rarer cases, we encountered “multiple form/image” attacks which either overlay duplicate forms on top of one another or provide a grid of forms or background images, and “background pattern” attacks that alter the background, especially behind company logos, to confuse the ML system.
Considerations.
Our analysis clearly indicates that real adversaries do attempt to evade anti-phishing ML systems that use image classification, and do so with some degree of success.
Although our results suggest that the strategies employed by real attackers have little in common with gradient-based adversarial examples, we cannot make such claims with certainty.
Indeed, proving whether a given “evasive sample” is a true adversarial example or just the result of educated guessing is extremely difficult today.
This problem could potentially be addressed with digital forensics techniques (which are covered by two recent works in the context of adversarial ML [80], [81]).
In a sense, the only way to be 100% certain that a sample has been generated with a gradient-based strategy would be to ask the attacker directly (i.e., a “probatio diabolica”).
Nonetheless, we are confident in the results of our analysis: the strategies we described above are simple yet effective.
Hence, it is sensible to conclude that an attacker would opt for these methods over more computationally expensive ones.
O BSERVATION : to evade phishing ML detectors, attackers employ tactics relying on cheap but effective methods— which can be hardly linked to gradient computations.
As an additional contribution, we publicly release theevasive webpages that we considered for our analysis [82].
We turn the attention to an Anti-Phishing Evasion competition organized by industry practitioners in ML security [83], who provided us with the data used for this case study.
Our intention is emphasize the role of time (and domain expertise ) in evading a ML detector. Although most contestants were security enthusiasts, the rules of this competition resembled a constrained and likely scenario—simulating a “real attack”.
Rules.
Contestants were given 10 phishing webpages (as
HTML) which had to be manipulated in such a way as to preserve the original rendering while evading seven ML-based detectors.
These detectors are CLOSED ML systems: contestants could input their manipulated webpages to the detectors, which provided a ‘phishing’ probability (within [0–1]) as output.
Winners were determined on two criteria: the number of successful evasions (probability below 0.1), and the number of queries issued to the detectors.
Contestants could send an unlimited amount of “submissions”, i.e., sets of manipulated webpages to be reviewed by the organizers at the end of the challenge.
A public leaderboard was regularly updated to reflect the current rankings, showing the number of successful evasions and queries by each contestant.
The challenge started on Aug. 6 and ended on Sept. 17 (42 days in total).
Results (high-level).
Four teams were able to craft variants of all 10 phishing webpages that evaded all 7 detectors, thereby achieving a perfect score of 70 points.
In order of ranking, these teams made 320 , 343 , 608 , and 9 982 queries [84].
The 1st-place (two scientists from Kaspersky) and 3rd-place (a single individual, Ryan Reeves) teams published their methodologies [85], [86].
We can reasonably assume that the strategy of the 4th-place team involved some automation that resulted in thousands of queries.
Using query count as a cost metric (as researchers often do, e.g., [32]), leads to the conclusion that the winning solution had the lowest cost: only 320 queries, i.e., an improvement of 47% over the 3rd-place ( 608 queries) and of 97% over the 4th-place ( 9 982 queries).
However, an in-depth analysis shows that this viewpoint is quite misleading.
Results (low-level).
The organizers of this competition provided us with details about the temporal distribution of the submissions made by the four top-ranked teams, whose cumulative submission history is shown in Fig.
We can assume that the last submission made by each team corresponds to the point in time in which they “finished” their attack.
The 3rd-place team (green line in Fig. 5) was the first to arrive at a perfect evasion, requiring 608 queries with the first submission on day#5 and the last on day#12.
In contrast, the perfect evasion achieved by the 1st-place team (blue line in Fig. 5) was the last to be submitted—they made their last submission after 42 days, i.e., on the last day of the challenge.
Indeed, the winning team adopted a “wait-andsee” approach, i.e., by taking into account the results of other participants.
Originally, they wanted to enact typical “blackbox” techniques (e.g., [87]) based on model replication, which
[Figure]
are query-intensive.
However, they changed their mind [85] when they noticed that the current leader (who ultimately finished 2nd) only had made a few hundred queries.
Hence, to win they needed to be very conservative with their API calls.
The winning strategy involved sophisticated manipulations of the HTML, rooted in extensive domain expertise.
In contrast, the 4th-place team (red line in Fig. 5) made a large number of queries and submissions, but these were all automated and (likely) required little human effort.
Moreover, the 3rd-place team required 12 days for a single person to achieve a perfect evasion, while the 1st-place team required 42 days by two people—an 8x increase in absolute time. O BSERVATION :
Measuring attack efficiency with query counts alone does not reveal the amount of time and domain expertise required to devise a successful offense. We release the code [88] and anonymised data [89] of our analysis (we received permission by the organizers).
Further Considerations. Insofar, we presented an objective analysis of the evidence available on this competition.
However, we believe that there is more to be learned from this case study: we provide our (hypotheses) in Appendix A-C, wherein we attempt to derive the human effort entailed in this competition. Moreover, we Finally, we point the reader to our fourth case study in Appendix A-D, further gleaning on the importance of domain expertise from both the offensive and defensive side of an anti-malware evasion competition.
IV. S NAPSHOT OF A DVERSARIAL ML R ESEARCH
We now turn our attention to the research domain.
We systematically analyze all papers within our scope that have been recently published in selected ‘top’ scientific venues.
Our objective is twofold: (i) to identify research trends and (ii) to pinpoint blind spots that may inspire novel studies.
We first present an overview of our analysis (§IV-A), and then focus on some positive trends of the threat models envisioned in prior work (§IV-B for attack papers, and §IV-C for defense papers); we then highlight some negative trends in inconsistent terminology adopted in research (§IV-D).
A detailed description of our study is provided in Appendix B.
Disclaimer :
Our study is not a finger-pointing exercise; it is a holistic review of prior peer-reviewed work aimed at reducing the gap between researchers and practitioners.
Methodology.
We are inspired by Arp et al.
[14] and
Apruzzese et al.
We consider papers accepted in four venues: ACM Conference on Computer and Communications
Security (CCS), USENIX Security Symposium (SEC), Network and Distributed Systems Security Symposium (NDSS), and IEEE Symposium on Security and Privacy (SP).
Because we want to focus on recent trends, we consider the papers published in the last three years: 2019, 2020, 2021.
We omit 2022 because, at the time of writing, it is still in progress, and we want to see yearly trends.
We identify 88 papers within our scope (shown in Table IV), which we then analyze.
Main Findings.
We report below the most relevant discoveries, which we will use to support some of the positions taken in this work (§V).
Out of 88 papers:
• 72% focus on attacks (28% on defenses); • 52% envision “evasion” attacks; • 89% consider only on deep learning algorithms; • 27% do not make any mention of economics; • 51% publicly release their source code; • 63% carry out their evaluations on image data (only 5% consider malware, phishing or intrusion detection); • 20% reproduce a pipeline by building an ML model only; • 20% experiment on real ML systems.
Positive Light : Taken individually , all of these papers are correct.
For example, it is legitimate to perform experiments on a self-developed ML system represented by a single ML model that analyzes image data by means of DL.
However, some general directions taken by the whole body of research generate blind spots—which we attempt to illuminate.
For additional descriptions, interpretations and figures showing ongoing trend, we refer the reader to Appendices B-B to B-H.
B. Positive findings in Threat Models of Attack Papers
Constrained Adversaries.
More and more attack papers are considering threat models that incorporate constrained adversaries.
For instance, the adversary may have limited knowledge of the targeted ML system, they may have restricted query budgets, or they may be able to observe (or interact with) only a specific segment of the data processing pipeline.
The underlying principle is that showing successful attacks in constrained (and thus more realistic) environments is more impactful.
In particular, we highlight:
• Nasr et al.
[90] consider attacks against an ML-NIDS.
Specifically, they envision a “blind” adversary whose perturbations can only be applied on live network traffic, i.e., without knowledge of the packets that will be generated after those manipulated—because such packets will be generated in the (unpredictable) future.
This work depicts an exemplary attack against an INVISIBLE ML system.
• Barradas et al.
[91] propose FlowLens, an ML system for website fingerprinting.
Here, the authors first consider attackers that are oblivious of the existence of FlowLens itself (which are unsuccesful), and then proceed to increase the knowledge/capability of the attacker.
Despite it not being surprising that attackers with limited power have little success, we believe it is important to also consider cases of unsuccessful attacks (which is also done in [52]).
These evaluations are useful in practice, because they portray attacks that are more likely to occur in the wild due to a lower entry barrier.
We do, however, advocate caution in assuming threat models that envision extremely constrained adversaries.
Successful attacks stemming from weak adversaries ( in terms of KNOWLEDGE and CAPABILITIES )
but
requiring sophisticated
STRATEGIES ,
may be as
unrealistic
as attacks stemming from powerful adversaries using simple STRATEGIES
(see §V-B)
Unusual Strategies.
In the context of
evasion
attacks,
some papers
propose STRATEGIES that have very little in common with established techniques for traditional adversarial examples ( e.g , those using gradients [46])
We highlight two such papers, both of which consider attacks against ML systems for automated speech recognition (ASR)
• Chen et al.
92] aim to generate samples that trick both the ML system and humans.
To do this, however, they rely on domain expertise, because prior attacks are useless against real ASR.
We quote [92]: “However, even with the estimated gradients, none of the existing gradient-based white-box methods can be directly used to attack ASR.”
• Zheng et al.
[93] consider (also) adversaries who (i) know nothing about, and (ii) cannot query the target ML system—which is a real ASR.
The attack is perpetrated through educated guesses stemming from expertise in the audio domain.
Indeed, we quote from [93]: “[because] the ultimate goal of ASR is [...] converting natural speech into text, we believe that the inclusion of the characteristics of natural command audios in the constructed adversarial examples may improve their transferability.” We
are pleased to see some research papers that propose evasion strategies that are outside the traditional realm of adversarial ML (which typically focuses on images).
Indeed, real attackers vastly rely on their domain expertise (§III -B
Broader Goals.
Some papers envision threat models where the attack succeeds despite the ML model’s correct response .
We share two that consider production-grade ML systems: • Xiao et al.
[20] “evade” an ML model for object recognition by eliciting an incorrect response of the preprocessing component of the broader ML system.
What is intriguing is that the ML model makes the correct prediction (i.e., it outputs the correct label) for the sample received as input—but only because the ground truth of the input sample was changed after preprocessing.
This example demonstrates how an attacker can succeed by fooling the ML system but not the ML model .
• Nassi et al.
[23] target the ML-based object detector embedded in autonomous cars.
Here, the authors (correctly) guess that such systems are often tuned to prioritize silhouettes of humans (crucial for decision-making in autonomous driving), meaning that these systems can be tricked via “phantom” figures with no spatial depth.
The authors prove their hypotheses by showing that real autonomous cars stop when a “phantom” appears.
In this case, the entire ML system makes the correct response, but the attacks are successful due to specific policies adopted by the system’s developers.
Simply put, attackers’ goals need not require fooling the ML model or even the entire ML system.
We are encouraged to see research that is exploring these scenarios.
O BSERVATION :
Assuming constrained adversaries is common.
Increasingly more attacks leverage domain expertise.
In some cases, attackers’ goals go beyond merely causing
ML models to misclassify samples.
C. Positive Findings in Threat Models of Defense Papers
Only 24 papers (out of 88) propose original defenses .
Compared to attack papers, defense-papers tend to put a stronger focus on the “cost” of the proposal—typically by measuring its overhead (i.e., baseline performance degradation).
Defending against knowledgeable attackers.
A common approach in defensive papers is to consider adversaries with perfect knowledge (e.g., [94], [95]).
Such “white-box” attacks can be difficult to stage in reality (as also hinted in §IV-B) because acquiring complete knowledge of the targeted ML system can be expensive.
In some cases, however, such knowledge can be easily acquired—especially if the targeted ML system is OPEN (see, e.g., the attacks by Sato et al.
Indeed, creating an OPEN ML system is increasingly dangerous, because doing so exposes the system to attacks at different layers.
For instance, Bagdasaryan and Shmatikov [96] show how to “poison” an ML system by manipulating its source code, i.e., without any change to the training data.
Defenses against limited-knowledge attackers.
Tang et al.
[97] propose a defense that considers (and is evaluated against) attackers who are powerful, but not omniscient; a similar “black-box” threat model is adopted also in SIGL [98].
Some (e.g., [13], [14]) may argue that similar defenses violate the “Kerchoff principle” [99], but we believe that the line of research taken by these works ([97], [98]) is worthy of being pursued, due to its high value for real-world deployments of
ML.
Indeed, attackers without perfect knowledge are a likely threat in the wild, and hence corresponding defenses also demand attention in research .
Notably, some works have shown that attackers with limited knowledge may be as dangerous as omniscient ones: Song and Mittal [100] show that “the gap between white-box attack accuracy and black-box attack accuracy is much smaller than previous estimates [101].”
O BSERVATION :
Most defenses assume attackers with perfect knowledge by default.
Some, however, are tailored for attackers with limited knowledge.
We further discuss and motivate our stance on defenses tailored for limited-knowledge attackers in Appendix A-G2.
D. Inconsistencies that May Harm Future Research
During our analysis, we focused our attention on the definitions of the threat models envisioned in each paper.
We report some of the ‘confusion’ we encountered, with a focus on the knowledge and capabilities of the envisioned adversaries.
What does the attacker know?
The (very common) terms “white-box” and “black-box” are used in different works to denote significantly different degrees of attacker knowledge.
We highlight such inconsistencies by reporting (verbatim, emphasis ours) the descriptions provided in some example papers [20], [102]–[104], all of which focus on “evasion” attacks (i.e., missclassifications at “test-time” [37]).
• According to Co et al.
[102]: “In white-box settings, the adversary has complete knowledge of the model architecture, parameters and training data .
In a blackbox setting, the adversary has no knowledge of the target model and no access to surrogate datasets .”
This description aligns with the one by Srndic and Laskov [41].
• Shan et al.
[103], however, consider a different “whitebox” setting: “We assume a basic white box threat model, where adversaries have direct access to the the ML model, its architecture, and its internal parameter values [...] but do not have access to the training data .”
• Xiao et al.
[20] do not specify anything about the training data: “In this paper, we focus on the white-box adversarial attack, which means we need to access the target model (including its structure and parameters).”
As a matter of fact, Xiao et al.
[20] also consider “black-box” attacks, which target an unknown ML model but which is trained on the exact same dataset as the “white-box” scenario.
• Suya et al.
[104] assume a “black-box” attacker that “does not have direct access to the target model or knowledge of its parameters”, but which “has access to pretrained local models for the same task as the target model” which could be “directly available or produced from access to similar training data. ”
Such a definition is in stark contrast to the “black-box” one by Co et al.
We also report the definition of the “gray-box” setting of Hui et al.
[105]: “The graybox setting gives full knowledge to the adversary in terms of the model details.
Specifically, except for the training data, the adversary knows almost everything about the model, such as the architecture and the hyper-parameters used for training.
Note that the adversary cannot know the training data (called a whitebox).”
This definition aligns with that of Shan et al.
[103]—for a “white-box” threat model!
What is the “box”?
Whenever “box”-based terminology is used, it is crucial to establish what is actually denoted by the “box”.
Recall that in the real world, attackers interact with
ML systems (i.e., the dotted rectangle in Fig. 1) and not with
ML models.
Hence, a “white-box” attacker (resp.
“black-box”) should have complete (resp.
zero) knowledge of the entire system .
This consideration is overlooked in research papers: for instance, Zhao et al.
[106] aim to attack “real world object detectors” and propose “white-box”/“black-box” attacks—but only consider the perspective of the single ML model.
A similar case can be made for the descriptions by Demontis et al.
[42]: here the attackers’ goal is first presented in terms of “ system operation”, but 10 lines later the knowledge is presented with a “white-box” terminology defined as follows: “the attacker has full knowledge of the target classifier ”.
Additional confusion stems from the so-called “no-box” attacks envisioned by Abdullah et al.
[107], whose definition conflicts with the original one by Li et al.
Specifically, the “nobox” attacker in Abdullah et al.
[107] knows nothing , which is the de facto assumption of “black-box” attackers.
In contrast,
Li et al.
[48] shift the focus from the KNOWLEDGE of the attacker to their CAPABILITIES : they cannot interact with the box (i.e., the ML model), but know that it behaves similarly to a surrogate ML model developed by the attacker, who also know a subset of its training data.
What can the attacker do ?
The attacker’s CAPABILITIES were also difficult to identify.
In some cases, this confusion resulted from imprecise usage of the term “access.”
For example, Shan et al.
[103] state that their attacker must have “direct access to the ML model” but do not specify whether such access includes read or write privileges; Suya et al.
[104] state that the (“black-box”) attacker has “no access to the target model,” but has “API access to the target model” (both statements are in the same line).
Another issue we encountered concerns the format of the output received by a given attacker in query-based strategies.
Output can come in many forms [108], but most papers merely state that the output is “a probability” without providing additional details.
O BSERVATION : Definitions of the KNOWLEDGE and CAPABILITIES of the envisioned attackers are inconsistent, especially when using “box”-based terminology.
V. R ECOMMENDATIONS
So far we have made a number of observations stemming from our preliminary discussion (§II), our real-world case studies (§III), and our detailed analysis of recent research (§IV).
We summarize these observations in Table III.
[Table]
We can now link all these contributions and use them to support four actionable positions: adapting threat models to
ML systems (§V-A), integrating threat models with cost-driven assessments (§V-B), encouraging industry and academia to collaborate (§V-C), and embracing a “just culture” for reproducible research (§V-D).
Let us discuss each of these positions.
Disclaimer : We phrase all our positions with ‘must’, because this is what we advocate to bridge the gap between research and practice.
However, not embracing any of our positions does not invalidate future contributions.
A. Adapting Threat Models to ML Systems
Our first position is motivated by the (unintended) inaccuracies shown in some papers (§IV-D), as well as by most papers tunnel-visioning on the single ML model (§IV-A), overlooking the much more complex ML systems (§III-A).
Note, however, that complexity does not imply robustness to evasion (as shown in our case study in Appendix A-D).
P OSITION :
Threat models must precisely define the viewpoint of the attacker on every component of the ML system.
Holistic vision.
The elements typically used when describing ML threat models (i.e., GOAL , KNOWLEDGE , CAPABILITY ,
STRATEGY ) must be extended to cover the entire ML system.
G OAL +.
Real attackers have broader GOALS than just attacking a single ML model.
For example, they may want to caus damage to all tenants of an ML system [10], or simultaneously fool humans and ML classifiers [109], [110].
K NOWLEDGE +.
To better portray real threats, KNOWLEDGE assumptions should cover all components of ML systems, e.g., any preprocessing steps [47].
For instance, an attacker who has perfect knowledge of an ML model, but zero knowledge on the preprocessing will have limited knowledge of the ML system.
(We provide our viewpoint on the interplay between
KNOWLEDGE and domain expertise in Appendix A-G3.)
C APABILITIES +.
As with KNOWLEDGE , attacker CAPABILITIES must consider all components of the ML system.
For example, attackers should only be assumed have the ability to give arbitrary inputs to the ML model (e.g., irrespective of preprocessing) if doing so is possible in reality (especially if operating only in the feature space [40]).
Moreover, the output of the ML system must be explicitly defined (e.g., label-only, label-and-score, top- k -scores, all-scores, logits [108]).
Finally, manipulations of the training dataset should also consider the components of the ML system that collect such data.
S TRATEGY +.
By extending their vision to the whole ML system, an attacker can adopt STRATEGIES that “ignore” the specific ML models.
E.g., Hong et al.
[111] apply manipulations at the hardware layer, Bagdasaryan et al.
[96] manipulate the source code used to develop the ML model, and Batina et al.
[112] derive additional information by spying on electromagnetic side-channels.
Notably, Rakin et al.
[113] describe their threat model (in terms of KNOWLEDGE and CAPABILITIES )
both from the system and the ML perspective—which is a promising first step towards more holistic threat models.
Precise terminology.
As the foundation of every security assessment, a threat model must be unambiguous—especially in research papers.
A poorly defined threat model is detrimental because it can lead to wrong estimations of attack power (e.g., the attacker may appear weaker), as well as unfair comparisons by future work.
In particular, we identify four terms that are subject to misinterpretation either from a researcherto-researcher, or from a researcher-to-practitioner viewpoint.
These terms are: BOX , ACCESS , EVASION , ADVERSARIAL .
Let us describe and take our stance on each of these terms.
B OX .
We have already highlighted the inconsistencies revolving around “box”-based terminology (§IV-D).
Our stance is that researchers should refrain from using this terminology.
Instead, we recommend (i) using perfect/limited/zero to define the attacker’s KNOWLEDGE and (ii) precisely specifying the components of the system to which the terms apply.
A CCESS .
We have already discussed some issues with this term (§IV-D).
Our stance is that researchers should be precise (e.g., read , or write ) when categorizing the access—avoiding ambiguous terms (e.g., “direct” or “internal”); and specify the type of access for every component of the ML system.
E VASION .
In related literature, this term denotes attacks which (i) cause “missclassifications” and which (ii) occur “at test time”.
In contrast, the generic definition of “evasion” in cybersecurity is to “bypass” a detection mechanism [114], i.e., to cause a malicious sample to be classified as benign 6 .
For instance backdoor attacks [50] are considered to be “poisoning” in literature [37] and not “evasion”.
However, from a security standpoint, a (real) attacker does not want to “poison” a dataset—rather, they may use “poisoning” (e.g., backdoors) to achieve their goal: evade the ML system (in this example).
Our stance is that “evasion” should be used exclusively to denote the attacker’s GOAL , which can be achieved via many strategies (e.g., poisoning [50], exploiting adversarial examples [115], or also by ignoring the ML model completely [111]).
A DVERSARIAL .
In research, it is typical to link the term “adversarial” with the ML context—most likely due to the popularity of “adversarial examples.”
In many conversations with practitioners, however, we found that this term generates a lot of confusion: in a security context, “everything is adversarial.”
Our stance on this term is to use it only when referring to established, well-defined notions, 7 such as adversarial example or adversarial ML .
Nonetheless, let us make an important observation, inspired by [62]: adversarial examples are not malicious per-se; what is malicious is crafting and using an adversarial example.
Studying adversarial examples can have applications that are orthogonal to security, such as improving robustness of ML methods [116].
B. Cost-driven Security Assessments
Our second statement is motivated by the poor consideration given by recent papers (§IV-A, but also see Appendix B-E) to the economics of cybersecurity: 27% do not mention cost at all; and although 50% perform some measurements, the human factor—which can be significant (§III-C)—is often neglected.
P OSITION : Threat models must include cost-driven assessments for both the attacker and the defender.
Context.
Biggio and Roli [37] point out that the best way to approach cybersecurity is through proactive defense: the system designer should first model the adversary, simulate the attack, and then develop a countermeasure “if the attack has a relevant impact.”
We stress, however, that all of these steps should also account for the cost to both the attacker and the defender.
For instance, developing a countermeasure just because an attack has “high impact” may not be wise if the corresponding attack is unlikely to happen in the first place.
System designers, when conducting their security assessments, prioritize threats that are more likely to occur in the wild (see §III-B).
Real attackers, as pointed out by Wilson et al.
[58], operate with a cost/benefit mindset: they will only attempt to evade a detector if they perceive the benefits to outweigh the costs.
At the same time, no defense is foolproof [52].
Disclaimer :
From a research perspective, it is appropriate to assume (and evaluate) attackers with perfect knowledge who break any ML system.
From a practical perspective, however, security evaluations are costly, and it is reasonable that real companies prioritize more common threats [117].
Proposal.
To promote future research that has a greater impact in practice , we extend the recommendations of Biggio and Roli [37] with respect to cost-driven threat modeling.
We provide an overview in Fig.
When establishing a given threat model, the system designer (i.e., the researcher) weighs the potential benefit to the attacker and the corresponding cost to reap such benefit (Fig. 6a).
This preliminary analysis yields the likelihood of the attack [117].
The designer can then simulate the attack and gauge the corresponding damage, yielding the risk [118], [119] of the attack (Fig. 6b, left).
Finally, the designer can conceive any given countermeasure, estimate its costs (e.g., implementation, overhead, upkeep), and—depending on the risk of the corresponding attack— assess whether such a countermeasure should be deployed or not (Fig.
6b, right).
Simply put, quoting from [62], the driver of cybersecurity is “Paying x (for the defense) to avoid paying y (if the attack succeeds), with y ≫ x ”.
[Figure]
We make three remarks on our cost-driven threat modeling.
• Although some papers (attempt to) measure the “cost” of an attack/defense, they overlook the human cost factor (§III-C).
Hence, we endorse future studies to account for human effort .
A possible (albeit imperfect) way of doing so is by reporting the time needed to write the source-code [120] for a given attack/defense—which can be facilitated by, e.g., using GIT repository’s commit history [121], [122]; or by process-mining techniques [123].
• It is wrong to consider the attacker/defender battle as a zero-sum game in economic terms.
Indeed, in many cases, the gain of a (successful) attacker may not correspond to an equal loss by a (broken) defender.
We discuss a (toy) example in Appendix A-E that makes use of the fact that cybersecurity is typically outsourced [124].
• Researchers proposing “novel” attacks should assume the viewpoint of a real adversary.
An attack will not be launched if—despite initial success—such an attack can be defused via simple heuristics (refer to §III-A).
Finally, a precise and holistic threat model (§V-A) is greatly beneficial for cost-driven assessments in research .
Future work can use such descriptions to estimate the (cumulative) cost of the attack at a later point in time, and then use such an estimate as reference for comparison (even if the threat model envisions a nation-state adversary—see Appendix A-G5).
C. Collaborations between Industry and Academia
Our third position stems from the facts that (i) merely 20% of our analyzed papers (§IV-A) consider real ML systems; and (ii) 90% of papers only consider DL algorithms—which are not necessarily those used in practice (e.g., owing to a long history of tabular data stored in production databases).
P OSITION :
Practitioners and academics must actively collaborate to pursue their common goal of improving the security of ML systems.
The gap between research and practice would diminish if researchers had easier access to production-grade ML systems.
Indeed, most real ML systems are INVISIBLE or CLOSED , preventing researchers from truly understanding how they work (as also mentioned in our twitter thread—see Appendix A-B).
For example, a researcher can reasonably assume that an
OSN uses an ( INVISIBLE ) ML system for spam detection (e.g., §III-A); however, without knowing its internal architecture, they could hardly develop a ‘realistic’ ML system with equivalent functionality.
Similar issues exist also for CLOSED ML systems.
Consider the attack by Pajola et al.
[30], in which specific character modifications induce Google Translate to output a wrong translation.
From the researcher’s viewpoint, it is impossible to determine precisely why the attack was successful.
(E.g., was it due to a problem of the preprocessing phase?
Or maybe of the ML model itself?)
We acknowledge that asking technology companies to publicly release their own ML systems is unrealistic (although some examples do exist [125]).
However, companies can still make it easier for academics to do security assessments.
For instance, a significant barrier faced by researchers is getting in contact with practitioners.
A potential compromise could be to offer “bug-bounty” programs that clearly define guidelines for interaction between researchers and platforms (e.g., [126]), and/or dedicating resources to processing researcher requests (e.g., [127]); in both cases companies benefit by gaining thorough, independent security assessments of their ML systems.
Another possibility would be providing high-level schematics of ML systems (as showed in §III-A), inspiring researchers to run experiments on more realistic ML pipelines.
D. Just Culture and Reproducible Research
Our last position is because only half of our analyzed papers release their source code (§IV-A).
We acknowledge that there may be legitimate reasons that prevent complete disclosure (discussed in Appendix A-F).
We conjecture, however, that this low number is in large part due to fear of being “blamed.”
P OSITION :
In ML security, source-code disclosure must be promoted with a just culture [128].
A “just culture” [128] denotes an approach in which mistakes are assumed to occur, and derive from organizational issues.
Mistakes are avoided by understanding their root causes and using them as constructive learning experiences.
This definition is in contrast to a blame culture [129], which seeks to avoid mistakes by actively blaming the corresponding authors—with the intention of discouraging individuals from making mistakes out of fear of shaming.
We believe that our community should avoid the latter and embrace the former.
A just culture encourages source-code disclosure, enabling the gradual improvement that is the foundation of research.
In the ML security context, some attacks (or defenses) may be incorrectly implemented or evaluated, as was the case with
DeepSec [130], whose evaluation was found to be flawed after its publication [131].
Some of such flaws, however, were spotted because the implementation of [130] was publicly accessible.
Fear of blaming could have induced its authors [130] to not disclose the details of their platform—but they rightly released their code , and our community learned from such mistakes, turning a negative result into a positive outcome .
(We are glad that [130] is still included in SP19’s proceedings.)
Although we endorse future researchers to apply as much rigor as possible when performing their evaluations (i.e., before submitting their papers), we believe that authors should not be afraid of publicly releasing their source code due to the risk of others discovering flaws.
Indeed, there is much to learn from flawed implementations : the ML domain constantly evolves, and errors can be systematized into practical guidelines (e.g., [132]) to help future research to avoiding mistakes.
VI. C ONCLUSIONS
Through our positions, we hope to see (i) more reproducible research, which (ii) adopts threat models on entire ML systems described via precise terminology, making (iii) cost-driven assessments factoring the human effort, and that (iv) fosters active collaboration with industry.
As a final remark, we also endorse developments of novel techniques for the forensics of
ML incidents: perhaps real attackers do compute gradients— but we cannot prove it (yet).
A DDITIONAL C ASE S TUDIES AND C ONSIDERATIONS A. Evasion via Adversarial Examples (and Perturbations)
We provide the formal definition of an evasion attack via adversarial examples, and provide a consideration on the perturbations typically used to craft an adversarial example.
Definition.
Let M be a (trained) ML model; and let x be any input sample, corresponding to a given ground truth y ; let us assume that such x is correctly analyzed by M , i.e., M ( x )=
y .
An adversarial example is defined by means of an “adversarial perturbation” that, when applied to x , will yield the adversarial example x ′ =
x + ε .
Such ε corresponds to the perturbation, which must be chosen so that: (1) the ground truth of both the original and adversarial example is the same (i.e., both x and x ′ correspond to the same y ); and (2) the targeted ML model
M will output an incorrect label—i.e., M ( x ′ )=
y ′ ̸
= y .
Finding a suitable perturbation can be expressed as an optimization problem (subject to some constraint).
Such notation describes an evasion attack against ML (a missclassification at test-time).
Consideration.
In related literature, the “adversarial perturbation” (i.e., ε ) used to craft an adversarial example is typically very small, i.e., imperceptible by humans [39].
However, as pointed out by several works [13], [62], this constraint does not hold universally.
Some attackers may want adversarial examples to be noticed by humans [110]), while in other domains (e.g., malware [47]) humans do not inspect inputs to the ML model, meaning that perturbations can be of higher magnitude—as long as they are physically realizable [115].
B. Researchers vs Practitioners (Twitter thread in §II-C)
We report the tweets of the Twitter thread [59] mentioned in §II-C, and then provide our viewpoint on this discussion.
Discussion (verbatim).
We consider the comments between
Battista Biggio (B), a well-known researcher in adversarial
ML [37], [45]; Joshua Saxe (J), an employee of Sophos AI; and Konstantin Berlin (K), head of Sophos AI.
J1) Why robustness to adversarial examples isn’t a first-priority concern on the Sophos AI team.
[image showing the ‘simplistic’ vision of a ML malware detector by researchers, compared with the ‘complex’ pipeline of a real ML malware detector.] B1)
I’d tend to agree.
But still optimizing over a set of transformations in a black-box manner may help find blind spots even in more complex systems – surely lp norms are not that useful here – I am talking about combinations of more interesting transformations that abuse the PE.
K1)
Given the existence of an already large number of more prevalent attacks that do bypass detections why prioritize this one? B2)
The goal of all these defenses in the end is about raising the bar for the attacker, and this work in my opinion is useful in that direction.
It is important that people know that ML is not learning what we expect it should learn... K2)
If you look at cybercrime in economical terms (as you should because it is a business) the optimization for an adversarial ex. is not the expensive part, it is the engineering part of building
a tool that can create a diverse set of attacks with no obvious watermarks.
follows ] Think of Sophos as human adversarial attackers against your adversarial attack.
My bet is that if you deployed most published ML adversarial attacks at scale Sophos would block you within a few hours in such a way that you would have to rewrite your PE generator , no
ML will help
B3)
It depends.
The transformations used are perfectly legit (e
g. adding sections).
Btw we never thought of bypassing a real AV pipeline , we know that is more complicated.
We only wanted
to show that some ideas (using DL from raw bytes are much more insecure and need improvement.
The last message (B3) was sent on July 24, 2022 at 10:06AM, while the first message (J1) was sent on July 22, 2022 at 8:17PM (both times are GMT+1).
Our viewpoint.
There are several messages that can be gleaned from this discussion.
However, as researchers, we anticipate that our viewpoint mostly aligns with that of Biggio.
First, we believe that comment K2 was an overstatement— or rather, it is a “convenient statement”.
Indeed, a specific term makes K2 unfair: “if you deployed most published ML adversarial attacks at scale ”.
Clearly, as shown in our first case study (§III-A) threats that operate “at scale” are prioritized by companies, and it is not surprising that Sophos would likely do all they can to block it as soon as possible.
However, the attackers envisioned in some adversarial ML papers (and, more generally, in most security papers [133]–[135]) tend to be subtle.
Hence, it is unfair for Konstantin Berlin to use such term, against which it is simply not possible to respond.
Second, even assuming that the ML detectors employed by Sophos are different than the ML systems envisioned in research papers, this does not mean that such detectors are “omnipotent”.
All our case-studies (§III) clearly demonstrate this fact: maybe the corresponding samples are not generated by exploiting gradient computations, but this does not make such papers (and the corresponding findings) any less valuable.
Hopefully, our paper will inspire Sophos AI to change their viewpoint on the relevance of adversarial ML literature for real-world deployments of ML.
Third, we agree with B3.
As researchers, we are “doing our best”.
Clearly, a better cooperation between industry and academia would kickstart novel research efforts that assess the robustness of full-fledged ML systems against adversarial examples.
However, there is still much to learn from the results of recent literature—even though the envisioned ML systems are an inaccurate representation of real ones.
C. Further Considerations on our third case study (§III-C)
To allow a complete understanding of our anti-phishing evasion case study (§III-C), we provide a further description of the rules of this competition.
Then, we provide our interpretation of the facts that transpired during the natural discourse of this challenge in light of the possible human effort (and decision making) entailed by each top-ranked team.
1) Queries
and Submissions: Our intent is clarifying Fig.
by explaining the difference between a “submission” y axis and a “query” (legend).
During the challenge, participants could do the following: query the detectors—either a single one, or all of them (there were 7 detectors in total); or submit their (adversarial) webpages.
Doing the latter counts as a “submission” (y-axis) and automatically generates 8x the queries per webpage included in such submission.
For example, a participant that submits 2 webpages automatically triggers 14 queries, because each webpage is analyzed by all 7 detectors (to determine how much points to award).
The organizers of MLSEC logged each submission, but not the individual queries (i.e., the single API calls).
This is why the plot in Fig.
5 only shows the submissions in the y-axis.
We are aware that a more fine-grained analysis would entail studying the history of the single queries.
However, as is evident by Fig.
5, there is a strong correlation between the submissions and the queries (although, technically, one could issue 10k queries and making only one submission), so we reported such metric in the y-axis.
2) Interpretation (human effort): Let us summarize the facts (for brevity, we denote each team as 1st, 2nd, 3rd, 4th).
• On day#0 all teams had a common objective “use as little queries as possible”.
• On day#5, 3rd made his first submission.
• On day#12, 3rd made his last submission, thereby completing his attack and leading the ranking withqueries.
Such information was publicly available: after this moment, all teams teams knew that, to win, they had to use less than 608 queries • (From day#17 to #42, 4th made various submissions and reached a perfect evasion by making 9 982 queries) • On day#31, both 1st and 2nd made their first submission (this does not mean that they did nothing before this day)
• On day 41, 2nd team made their last submission, warranting them a perfect evasion with 343 queries. Such information was publicly available. • On day 42, 7 hours after 2nd last’s submission , 1st made their second submission, which included 9 webpages (out of 10), all of which evaded all the ML detectors. • Also on day 42, 1st team made their last submission, warranting them a perfect evasion with 320 queries • At the end of the challenge, 1st stated [85]: “At first, we thought of attempting a classic model replication attack [...] but as we started working on the competition, we noted that the leader had already achieved the highest possible score using just 343 API calls”
Now, let us interpret these facts.
Firstly, the statement by 1st is technically incorrect, because the first submission by 1st was on day#31, and during such time-frame (i.e., between day#31 and day#42 – but also before day31) 1st were able to do anything (albeit there is no record of this).
Furthermore, 1st knew that 3rd had been leading with 608 queries since day#12: with such knowledge, why would 1st even consider attempting a “model replication attack” on day#41?
Finally, the second submission of 1st (on day#42) occurred only 7 hours after the last submission by 2nd (on day#41), and such submission included 9 webpages all of which achieved a perfect evasion.
Hence, we find it hard to believe that 1st “started working on the competition” only after noticing that the leader hadcalls (but we cannot claim this with 100% certainty).
Secondly, we observe that 2nd arrived at their perfect evasion after knowing that they needed less than 608 queries (because such was the leading score, by 3rd).
Hence, we conjecture that without the efforts of 3rd, it is unlikely that 2nd reached their perfect evasion in 343 queries: 2nd knew they were hard constrained, and hence had to be more careful.
Nonetheless, 2nd made their first submission on day#31, and the last on day#41: which is a larger window (from a submission perspective) than the one needed by 3rd to reach the same result (although 2nd required roughly half the queries); and it is also a larger amount in terms of absolute time since the start of the competition.
We can hence conclude that “3rd required less time than 2nd to accomplish the attack”.
Finally, let us go back to 1st; specifically, we focus (again) on their public statement [85].
Here, they stated that they wanted to do a model replication (which we can conjecture required “less effort”), but they changed their mind after seeing the score of 2nd—which achieved their score after 3rd place.
Conclusions.
By connecting all these observations, we derive: without the “effort” of 3rd – which (likely) influenced the actions of 2nd – the perfect evasion of 320 queries by 1st would not have been eached.
Alternatively, had the leaderboards not been publicly visible, 1st would have opted for a model replication attack (this is what 1st stated!).
We do not know what 2nd would have done, but 3rd would still have used their domain expertise to reach their solution— in only 12 days, and requiring 608 queries (3rd was the first contestant to reach a perfect evasion).
From a different perspective: model replication attacks require more queries (as clearly shown by 4th place), but require less human effort (as can be extrapolated from 1st actions and statement).
D. Case Study: Malware Competition (2020)
We extend our case studies with an in-depth analysis of the Malware Competition organized in the 2020 edition of
MLSEC.
Here, we elucidate how ‘competition-grade’ ML malware detectors can be (i) hardened, but also (ii) evaded.
Rules.
This competition had two phases, each with a specific challenge: the first focused on defense, the second on attack.
Let us describe what each challenge entailed.
• (Defense)
In the first phase, participants were asked to submit ML systems for malware detection.
To be accepted, such solutions had to meet some performance metrics—at most 1% false positive rate (fpr), 90% true positive rate (tpr), and 5s response time—measured on an unknown test set.
Accepted solutions will be used for the attack phase.
Winners were determined depending on how well their solutions performed against the attacks launched in the second phase.
• (Attack)
In the second phase, participants (which could be different from those of the first phase) were given 50 PE malware samples, which had to be manipulated in any way that preserved the original malicious functionality.
The manipulated PE had to evade as many ML-based detectors as possible—including those accepted in the previous defense phase.
These ‘attackers’ were given API access to the detectors, which resembled CLOSED ML systems: attackers could only submit an input and observe the output (i.e., the probability that the sample was malicious).
Winners were determined on the basis of how each of the 50 manipulated PE performed against all the considered detectors.
What did the winners do?
Let us focus our attention on the methodologies adopted by the winners of each challenge.
• (Defense) Quiring et al.
[136] won the defensive challenge.
Their solution integrated typical approaches to counter malware, such as signature-matching, and MLbased detection based on static analysis; surprisingly, however, they also integrated a defensive layer for anticipating adaptive attackers who would query the detector— i.e., what would be done by the participants of the second challenge.
The solution of [136] was evaded only by 23% of the PE submitted during the attacker’s challenge.
• (Attack) Ceschin et al.
[137] won the attackers challenge by having all their PE to evade all the considered detectors—including the one by Quiring et al.
Notably, the strategy adopted by [137] was simple .
Specifically they show that “embedding the malware payload into another binary eliminates most detection capabilities presented by the models”
Indeed, to quote from [137]: “[although] there are approaches for adversarial attacks generation based on complex techniques [...] we show that it is possible to generate attacks using known, simple techniques.”
O BSERVATION : bypassing some ML systems may not require sophisticated techniques (e.g., those based on gradients) commonly adopted in research papers.
E. Example: Gains and Losses is not a zero-sum game
The gain of an attacker may not correspond to the loss of a defender.
Consider the following scenario: a company, D , whose main business is the development of ML-based malware detectors.
Such a company sells their products to a customer, C .
Now, consider an attacker, A , that successfully evades one of the detectors developed by D and manages to inject some ransomware to the systems owned by customer C , asking for a ransom amounting to R .
Two scenarios can occur.
• Pay.
If C pays the ransom, then C loses R while A gains R .
However, D (which is the developer of the ‘evaded’ detector) loses very little.
In most cases, the service level agreement (SLA) between D and C protect D in cases of ‘faults’ in their systems.
In contrast, if the SLA determine that D is liable of the losses incurred by C , then D will pay R to C ; however, in this case, D is typically insured by other companies, who will reimburse R to D .
Hence, if the ransom is paid to A , then A gains R , and D loses nothing (assuming that C regains control of their systems).
• Recover.
If C does not pay the ransom, then A gains nothing (actually, A may lose something, e.g., the time spent to carry out the attack).
However, C will incur a loss, because their systems are compromised.
Two cases can follow depending on the SLA: if D is liable, then they must help C to recover their data; hence, D may either incur a loss (i.e., human labour), or not (i.e., if they are insured); otherwise, if D is not liable, then D may gain something: this may happen if C asks D to assist in the recovery of their systems (under a payment).
Simply put, when looking at cybersecurity from the economical (and operational) viewpoint, there are many nuances that can determine the deployment of a countermeasure in an ML system.
Such considerations further complicate the decision of whether to develop a countermeasure or not—whose main goal is to be beneficial for the system designer (or their employer).
F. Unreproducible Papers
Unavoidable truth.
The ideal of a publication that releases all its source code and uses public datasets containing attacks from the real world is difficult to meet in practice—and especially in cybersecurity.
The most common and understandable constraint is that of user privacy: real-world data is tied to consumers or employees and custodians of this data must respect its confidentiality.
The employers of security and
ML practitioners may also impose constraints with regard to releasing source code to avoid giving attackers complete vision of an ML systems or a portion thereof.
When constraints of this nature do not exist, researchers and practitioners must provide both the descriptive details and the technical artifacts to maximize reproducibility.
What can be done.
Publications that face privacy constraints and/or cannot release their source code can, however, still be valuable so long as they work within those constraints to make reproducibility a priority.
To this purpose, we propose the following actionable principles, which can be adopted by research papers to improve their scientific value:
1) ML systems and models should be described with sufficient clarity that a reader can re-create the model with a high degree of accuracy after reading the paper.
Features used by the model should be described in detail.
2) Practitioners should evaluate their models on public datasets whenever data that is sufficiently similar to the private data is available.
This recommendation need not prevent additional evaluation on private datasets.
3) Papers should describe as many of the nuances of private datasets as possible (e
g., examples of feature values and their distributions, correlation between features, and other properties of the data) while not providing individual data points when doing so would violate user privacy. 4)
Attacks that appear in private datasets should be released when possible .
If they are not releasable, they should be described with sufficient detail to enable researchers to reproduce similar attacks in their own studies and to understand attacker threats that appear in real systems.
We observe, however, that providing all such details can be tough in a research paper: as highlighted in [62], some venues (especially conferences) have a strict page-limit.
Hence, we endorse conference organizers to remove such limits when they are used to provide additional technical details—which are not part of the contribution of a paper, but which are crucial to provide at least some form of reproducibility.
G. Discussion (with the Reviewers)
We discuss some questions that may be inspired by our paper.
1) Spammers and Regular users:
According to the description in §III-A, the malicious user wishing to upload problematic image seems to perform a very similar action to millions of users uploading any sort of images every day on any OSN.
Hence, although some malicious user’s action pattern may appear different from those of normal users (especially when automated tooling is used to perform such actions) this is far from being an universal truth.
Our Response: This statement is correct, and we will happily clarify. According to the OSN employee,“professional spammers” are ultimately trying to make money.
They may do this by, e.g., posting links to counterfeit goods; ‘phishing’ legitimate users and selling their accounts’ credentials; engaging in 419 scams [138]; or in other ways.
Since the expected value of a single spam post is very low (most people don’t buy fake Ray-Bans!), spammers need to post a lot to reach their goal.
So they necessarily need to turn to techniques—usually, but not always, automation—that allow them to distribute content quickly across a wide audience.
This type of behavior does not match that of a typical OSN user.
Clearly, “stealthy spammers” who invest a lot of effort in mimicing the behaviour of legitimate users are harder to detect (with or without ML).
Such a risk is deemed acceptable by the OSN: the attack requires a high resource investment by the spammer, who will only be successful against a relatively small amount of users.
2) Defenses vs limited-knowledge attackers (§IV-C):
It is true that perfect-knowledge attacks are rare, but we should be very careful to draw a distinction between “defenses that are robust in a query-only setting” (a reasonable threat model) and “defenses that only work because the attacker doesn’t know what the defender is doing” (an UNreasonable threat model).
Our Response: This is a legitimate remark. We do not advocate that “all (defensive) papers must consider attackers without perfect knowledge”. Clearly, it would be beneficial if all papers proposed defenses that work against perfect knowledge attackers.
Our intention, however, was emphasizing that “real” attackers may not have perfect knowledge, and hence we find it positive that some papers propose defenses specifically addressed to such attackers.
Put differently, we think it is unfair that papers may be rejected because “the threat model considers an attacker without perfect knowledge”: if the threat is (justified to be) likely to occur in reality, then an effective defense can be beneficial to the real world. We believe there is much to learn from these evaluations as well! 3)
Domain Expertise:
In §III-C, the 1st–3rd place teams all reached a perfect evasion by exploiting their domain knowledge.
Does assuming attackers with different degrees of domain knowledge describe different threat models?
Our Response: Tough question.
In this case study, all participants had the same KNOWLEDGE , i.e., they knew nothing of the ML system (aside from that it analyzed webpages in the form of HTML).
However, clearly, all participants had different “knowledge” of the respective domain.
Note, however, that such knowledge is impossible to quantify: maybe they all perfectly knew the in-and-outs of HTML, but a participant was simply “lucky enough” to make a correct initial guess, which they then leveraged to build the remaining parts of the attack.
In a sense, this case study (§III-C) shows that guessing is a crucial part of a real attack (and we find promising that increasingly more papers are incorporating ‘guessing’ in their attacks, e.g., [10], [96]).
Yet, we believe that such different degrees of “domain knowledge” should not be classified as different threat models.
From a security standpoint, it is crucial to envision an attacker that is expert on the general domain.
Note, however, that this is different from assuming that the attacker has “perfect KNOWLEDGE of the target ML system”!
Hence, we conclude that the threat model should simply state the KNOWLEDGE of the attacker with respect to the ML system, and then assume that the attacker is an expert in any of the components of such ML system that they are aware of.
4) “Adversarial” in practice: Can you clarify what you meant by “everything is adversarial” in §V-A?
Our Response: Put simply, from the perspective of a security practitioner, an event is either ‘benign’ or ‘malicious’, i.e., either the system is under attack, or not.
Hence, the term “adversarial attack” is redundant: if there is an attack, it is ‘obvious’ that such an attack is “adversarial”.
At the same time, a “threat model” implicitly assumes that there is “a threat”, i.e., an attacker—which is ‘obviously’ adversarial in nature.
During our conversations with security practitioners, they are often confused when we state “adversarial attacks”: from a research perspective, such term refers to “attacks that exploit the vulnerability of ML models to adversarial examples”; however, from a practitioner perspective, the focus is “the attack”.
In other words: the term “adversarial attacks” implicitly means that “some attacks are not adversarial” – which is just illogical.
To provide examples, sometimes in our exchanges with practitioners (P), the following occurs:
• P: “Our systems are constantly under attack” • We: “Are they adversarial?”
• P: “Well of course they are!”
Alternatively: • We: “Do you test your systems vs adversarial attacks?”
• P: “Are there attacks that are not adversarial?”
(An even more awkward situation would be asking practitioners: “Are you more scared of adversarial attacks, or of non-adversarial attacks?”)
In short, we believe that a more precise (and less redundant) usage of the term “adversarial” will benefit bridging the gap between research and practice in the context of... adversarial ML!
5) Nation-state adversaries: If we consider nation-state adversaries, is human cost measurable here (or relevant)
Out Response: It depends.
Generally speaking, if by “nation-state adversaries” it is implied “adversaries with unlimited resources”, then we agree that it makes little sense to report such a cost measure.
However, such (extreme) cases should be reported in the paper when presenting the threat model.
For instance, a paper should state “we assume an attacker that is backed up by an entire country, and which is hence willing to invest a lot of resources to carry out the strategy presented in our paper”.
Doing this would automatically put such a paper in a different league w.r.t. costs. However, if the attack presented in such a paper is “devastating” (but not just because of a single ML model making a misclassification, but of the entire ML system) then real companies would be more interested in studying such paper and devise appropriate countermeasures. (In a sense, what we have just described is a clear use-case of “Adapting Threat Models to ML Systems” presented in §V-A).
Nevertheless, in reality, even nation-state adversaries do not have unlimited resources: hence, making such an assumption by default is not very realistic.
A PPENDIX B S TATE - OF - THE -A RT : LITERATURE REVIEW
We describe our review of the SotA (summarized in §IV).
Given the detailed methodology, the original interpretations, and the systematic analysis of recent literature, we consider this appendix as a complementary contribution of our paper.
A. Methodology
Inspired by by [14] and by [77], we carry out our review by adopting a structured approach organized two phases: selection and inspection .
Both phases entailed exchanges of opinions among the authors of this paper, aimed at ensuring fairness in the review process and removing potential sources of bias.
First phase: Selection.
We conduct our survey between
July and August 2022.
During such timeframe, we performed two following steps: we get the proceedings and filter them.
i)
Get.
We downloaded the proceedings of the ‘top4’ cybersecurity conferences (SEC, NDSS, CCS, SP).
We considered the editions of these conferences held in the last three years, i.e., in 2019, 2020 and 2021.
We only consider the papers published in the main event (i.e., we do not consider Workshop publications).
Overall, these proceedings amounted to 435 papers for 2019, 470 for 2020, and 644 for 2021.
ii) Reduce.
To identify papers within our scope, we started by considering papers included in those sections of the proceedings that specifically focused on ‘machine learning’.
We then extended our search by also performing a summary scan of the titles and abstracts of the remainder of the proceedings, identifying additional candidate papers 8 .
At the end of this step, we obtain 30 papers for 2019, 35 papers for 2020, and 66 papers for 2021.
Second phase: Inspection.
We further analyze ourcandidate papers, with a twofold objective: filtering those papers that, despite being related to ML security, were beyond our scope; and distilling the knowledge that we want to communicate in our work.
i)
Filter.
Our focus is on papers on the security of ML and, in particular, on attacks (and defenses) against ML systems.
We have already discussed (see §II-D) some orthogonal research areas.
Upon inspecting our candidate papers, however, we found 3 additional areas that we excluded from our main analysis due to assumptions that significantly deviate from our main focus.
Specifically: federated learning [140], because the corresponding issues mostly pertain to the distributed systems domain (such as byzantine fault tolerance [141]); papers considering robust, efficient, or secure computation , because they mostly focus on technical implementations and do not propose or consider any adversarial ML attack; as well as those papers that provide high-level analyses , because they do not propose any original attack or defense in the
ML security context.
Eventually, we obtain 23 papers for 2019, 24 papers for 2020, and 41 papers for 2021.
ii) Distill.
Finally, we thoroughly analyze the final set of 88 papers, We do this by going through each paper and asking ourselves a set of 12 questions—each focused on deriving knowledge significant for our paper, as well as for promoting future work in this research field.
Although most of our questions can be easily answered, some are not straightforward (as we will discuss).
To minimize the likelihood of errors and reduce subjective bias, we carry out our analysis in pairs: two authors independently reviewed each paper and then discussed the findings in a series of meetings.
We repeat such process five times.
Before we delve deeper into our analysis, we provide some interesting trivia that we noticed while scanning the proceedings.
In 2019, most papers were put in sessions entirely devoted to “ML security”.
In 2021, however, we had sections entirely dedicated to specific attacks (e.g., the “attacks on speech recognition”, or “inference” at SP2021).
Some papers have very similar contributions across the same conference.
For instance, [92] and [142] (both at SP21) propose attacks on speech recognition (and both, surprisingly, evaluate their proposals also on real humans); whereas [143] (SEC20) and [22] (CCS20) propose membership inference attacks that leverage the updates of ML models.
B. Research Questions
We then inspected 88 papers by asking ourselves 12 research questions, divided into two groups.
Generic Questions (G).
The first eight ( G1 to G8 ) are generic, and are meant to provide a broad overview of the latest trends in research—some of which are reported in our main paper (§IV).
For each of the 88 works, we ask ourselves:
G1) Does the paper focus on an attack or on a defense? G2)
What is the main attack family (
i
e., poisoning, stealing, evasion, membership inference)?
G3)
What paradigm of the ML model that is subject to the attacks (i.e., does it rely on shallow or deep learning)?
G4)
Are the costs taken into account (in any way)?
G5)
What are the data-types (
i
e., images, audio, text, or other) considered in the evaluation ?
G6) Has the source-code been publicly released? G7)
Has a complex pipeline been reproduced in the evaluation (i.e., does the ML system consist in just an ML model)?
G8) Does the paper consider an ML system deployed in the real world (and, if yes, what is its type )?
The answers to G1 to G8 are shown in Figs.
7 to 13, all sharing the same structure.
Each figure refers to a specific question, and presents four stacked bars, one per year [2019–2021] while the rightmost bar is an aggregate.
Each stacked bar reports the answers to the corresponding question, both in terms of relative (y-axis) and absolute (written on the specific bar) frequency over a given stack.
Descriptions and interpretations on these results are provided in appendices B-C to B-G.
Threat-Model Questions (T).
The last four ( T1 to T4 ) are more detailed and focus on the threat model envisioned in the paper.
Specifically, we focus our attention at the weakestattacker considered in any paper, and then ask ourselves:
T1)
Does the attacker know the semantics of the input data fed to the ML model?
T2) Does the attacker know the ML model’s parameters ?
T3)
Does the attacker have any power on the training set ? T4)
Can the attacker observe the output of the ML model (and, if yes, how )?
We explain how we answered T1 to T4 in Appendix B-H.
The answers to all our questions for all our considered papers are provided in Table IV, described in Appendix B-I.
C. Answers to G1 and G2: Focus, and Attack Family
The answers to the first two questions are shown in Figs.
F r equen cy 0.
[Figure]
G1 : Focus.
This question is straightforward to answer: does the paper focus on attacks against ML, or on defenses to attacks against ML?
In some rare cases, some papers simply proposed a ML method to solve a given task (e.g., intrusion detection [144]) but also evaluated such method against some evasion attacks: we considered these as attack papers, because the proposed ML method is not typically designed to withstand adversarial ML attacks.
From Fig.
7a, we can see that 75% of papers put a stronger emphasis on the attack—although most of these papers (e.g., [52]) also discuss some countermeasures to the corresponding attack.
Such a result is not surprising, as ‘attack papers’ are typically considered to have a higher novelty value—because the attack stems from a different threat model, or considers a new domain.
G2 : Attack Family.
Inspired by [4], we consider four main attack families: poisoning , model stealing , evasion , as well as membership inference .
Fig.
7b shows that most papers envision evasion attacks ( ∼ 50%), whereas poisoning, model stealing, and membership inference attacks are less prominent (20%, 10% and 20%, respectfully).
However, we can see a decreasing trend of evasion, and a rising trend for poisoning attacks , especially in 2021.
We conjecture that such a phenomenon is due to the 2020 paper by Kumar et al.
[3], highlighting that industry is particularly worried about poisoning.
D. Answer to G3: ML paradigm
This question is also straightfoward: is the underlying algorithm used by the ‘targeted’ ML model based on shallow or deep learning?
The results are in Fig.
[Figure]
It is apparent that most papers consider ML systems exclusively based on DL (i.e., those using neural networks).
Only 10 papers (out of 88) evaluate ML systems based on shallow learning (SL) algorithms (interestingly, two papers consider only SL: [47] and [91]).
We believe that future efforts should put more focus on SL, for a twofold reason:
• in some domains, SL is better than DL [19], [145].
E.g., we quote from [139]: “Even though in our experiments we used SVM, deep neural networks, and different variants of decision-tree learners, like random forest, we only discuss the results of the random forest approach as (1) we observed similar findings for these approaches, with random forest being the best classifier in most experiments, and (2) random forest allows for better interpretation of the results compared to neural networks.”
• SL exhibit different properties than DL.
We find insightful to quote a statement by Xu et al.
[146] (emphasis ours): “we mainly detect AI Trojans on neural networks.
We do not include other ML models in our discussion mainly because there is no current research showing that they suffer from backdoor attacks .”
Moreover, some SL methods (e g., Decision Trees) do not employ gradients, meaning that some gradient-based attacks (as well as gradient-based defenses) may not work on them.
We also mention that our case study (§III-A) elucidated that SL methods are used also in commercial-grade ML systems.
E. Answer to G4: Cost
As we discussed (§II-C), the cost (i.e., the economical factor) plays a crucial role in operational cybersecurity.
When analyzing prior work, we considered three possible answers to G4 : whether a paper measured the cost in some way (e.g., performance tradeoff, required queries); whether such cost was at least mentioned or discussed; and whether no consideration on the cost was made.
These answers can cover both attack and defense papers.
The results are shown in Fig.
[Figure]
These results show a positive trend.
In 2019, less than 30% of the papers did some measurements of the cost, and 30% made no consideration whatsoever.
Such occurrence changes in 2020 and 2021, as less than 25% papers did not take into account the economical factor, whereas more than 60% performed actual measurements.
Cost: Attack papers.
We highlight three papers which attempted to quantify the cost of an attack in real currency.
• (Equipment) Lovisotto et al.
[147] evade object detectors via artificial light manipulation through commercial projectors .
Some of these projectors have an MSRP of few hundreds dollars, whereas others are more expensive.
• (Queries)
The authors of [32] and [145] alongside measuring the cost of the proposed attack in terms of API calls (i.e., queries), but also quantified the actual cost required to make such API calls.
Specifically, they considered the prices of popular MLaaS , and derived that their attacks only necessitate few dollars.
Cost: Defensive papers .
The most typical way to measure the cost in defensive papers is via the ‘tradeoff’, i.e., the change in performance after applying a given countermeasure (e.g., [148]).
We mention two works that make insightful considerations on the ‘defensive’ cost—or rather, on how the principle of a defense is to increase the ‘cost’ of the attacker.
• Chen et al.
[149] propose to measure the cost of feature manipulation against shallow ML models.
The intuition is to carry out such measurements during the evaluation of ML models, with the underlying principle of “if an attacker wants to evade such ML model by doing a , they have to spend c a ; and if they want to do b , they have to spend c b .”
We remark, however, that [149] does not evaluate any attack: the main contribution is measuring the (computational) effort in manipulating some features.
(which is why we do not include [149] in our Table IV.)
• Heisenhofer et al.
[150] acknowledge that attackers can use adversarial examples to evade a ML system for automated speech recognition.
Hence, they aim to increase the cost of the attacker to craft such adversarial examples.
An alternative formulation of the “raise the attacker’s cost” is intrinsic of papers that focus on the detection of security violations.
We observe, however, that such detection can occur at different granularities .
For instance, some papers aim to detect individual adversarial examples (e.g., [94]) or poisoned
ML models (e.g., [97]); whereas others have a broader scope and aim to detect attackers (e.g., [103]).
F. Answers to G5 and G6: Data-type and Source-code
We now turn the attention to G5 and G6 , which are strongly related to the reproducibility [151] of research.
We make a statement on this subject at the end of our main paper (§V-D).
G5: Data-type.
Past research investigated a plethora of application domains of ML, which we divide in four categories.
Three correspond to the broad areas of images (e.g., computer vision), text (e.g., natural language processing), audio (e.g., automated speech recognition); the fourth includes all other domains (e.g., malware).
These results are in Fig.
10; some papers (e.g., [100]) consider multiple domains, which is why the y-axis goes above 1.
More details are shown in Table IV.
[Figure]
We can see an interesting trend from Fig.
10: in 2019, over 75% of the papers focused on images (typically using well-known datasets such as MNIST and CIFAR), whereas such percentage dropped to 50% in 2021.
The audio domain has also been neglected in 2019, but became prominent in 2021.
In particular, papers of 2021 covered much more ‘novel’ domains, such as graphs [152] or games [153].
G6: Source-code.
We looked for links pointing to the source-code of the experiments carried out in all our considered papers.
The results are in Fig.
We can see an encouraging trend, as 75% of papers did not release their source-code in 2019, whereas only 35% did not F r equen cy 0. TOTAL
Not Available
Public do so in 2021.
To facilitate future research, we provide also the actual links to every open code repository of each paper in
Table IV.
Nonetheless, we stress that even those papers that do not make their evaluation publicly accessible had valid reasons for doing so.
For instance, [52] describes plenty of details, uses well-known public datasets, and mostly rely on existing and publicly accessible code.
On the other hand, the experiments in [144] are based on sensitive data that cannot be disclosed.
G. Answers to G7 and G8: pipeline and type of ML system
The last generic questions pinpoint the degree of ‘realism’ considered in the evaluations of prior work.
Answering these two questions was not simple: the authors who took part in this activity had several debates before reaching a consensus.
G7: processing pipeline.
This question was among the hardest to answer during our survey.
Our goal was determining whether a given paper (i) envisioned a ML system, and (ii) developed more components of such ML system than just an ML model.
For example, if a paper considered a ML system (by, e.g., providing a schematic), but the evaluation spanned over a single ML model, the answer to G7 was negative.
The answer was also negative if the paper made some preliminary checks to the inputs of the ML model (e.g., to ensure that such samples do not violate domain constraints [154]).
Simply put, to answer G7 we embraced Abdullah et al.
[142] statement, which we quote: “Processing pipelines for modern systems are comprised of signal preprocessing and feature extraction steps, whose output is fed to a machine-learned model.
Prior work has focused on the models, using white-box knowledge to tailor model-specific attacks.”
Answers to G7 are in Fig.
F r equen cy 0. TOTAL Complex Pipeline
Just an ML model As expected, nearly 80% of papers perform their attacks by considering a single ML model.
Even some papers that attack
MLaaS (e.g., [155]) ultimately simply have the MLaaS train a given ML model, and then attempt to violate such model without reproducing a real pipeline.
Nonetheless, there are some notable efforts, such as [47] and [20].
These works reproduce a custom ML system by developing also a preprocessing component: such component represents an additional layer that an attacker must bypass to reach the actual ML model that will be ultimately targeted.
However, we find it concerning that the overall trend shown in Fig.
12 is stable .
We strongly endorse future work to start developing more components
G8: type of ML system.
This question is split in two parts.
First, we asked ourselves whether, in any part of the evaluation of a given paper, there was an ML model that can be considered as ‘deployed in practice’.
For example, papers that attack commercial ML systems (e.g., [23]) automatically fall into this category; however, we also considered papers that used ‘extremely popular’ ML models (e.g., Yolo [156]) that have been shown to have realistic applications besides benchmarking.
The answers to the ‘first’ part of G8 are shown in Fig.
Conversely, if a paper used some ‘popular’ techniques as a form of preprocessing but the attacked ML model is trained on benchmarks, then the answer to our question was negative—this is the case, e.g., of [47].
Then, if the answer to the first part of G8 was positive (i.e., the paper attacked a ‘deployed’ ML system), we proceeded to identify the type of (real) ML system.
Recall (§II-A) that there exist two main types of ML systems: OPEN and CLOSED (and also the INVISIBLE ones).
The answer to the second part of G8 can be seen in the detailed Table IV.
[Figure]
[Figure]
By observing Fig.
13, we are not surprised to find out that over 80% of the papers attack ML models (or ML systems) that are entirely self-developed—either by using private data, or by using well-known benchmark datasets.
We conjecture that this is due to the difficulty of researchers to acquire permission to use ‘commercial’ ML systems for research.
For example, the notable work by Nassi et al.
[23] clearly explains what the authors had to do in order to attack the ML system for object recognition integrated in real autonomous cars.
We commend the work by Nassi et al.
[23], but similar experiments may be “outside the reach” of most researchers.
Nevertheless, a detailed look at Table IV reveals that, among those papers that attack real ML systems, there is not a single one which is INVISIBLE .
This makes sense: such systems are typically protected by NDA and researchers can hardly use them for experimental purposes.
We hope that our statement (§V-C) will promote practitioners to collaborate more actively with researchers, so that future works can assess the robustness of real ML systems against against security violations. H. Answers to the Threat-Model Questions
These questions were also difficult to answer.
There is a remarkable lack of clarity in the way that research papers present their threat models (as also hinted in §IV-D).
T1: Parameters.
We focused on determining whether the attacker knew anything about the ML model itself, such as the underlying algorithm (e.g., a Decision Tree or a Neural
Network), its initial configuration (e.g., the learning rate or the architecture of the neural network) or its learned weights.
However – for those papers in which the attacker was assumed to have some knowledge on the ML model – it was difficult to specifically determine what was known.
Hence, we considered three possible answers to this question: full , zero and intermediate knowledge.
An exemplary case of the latter is the work by Tang et al.
[97], stating that their (“black-box”) attacker: “does not have information about the inner parameters of the target model [but] he knows the target model’s architecture, used optimization algorithm and hyper-parameters.”.
T2: Semantics.
For semantics , we consider whether the attacker was aware of what was fed as input to the ML model.
For instance: does the attacker know that the ML model analyzes images/malware/text?
The answer was binary:
yes or no .
In most cases, the answer was positive.
However, in some rare occurrences the attacker was also oblivious of such information.
An example is [111], in which the attacker simply does not need such knowledge because the manipulation affects the ML system at the hardware layer.
T3:
Output.
The main difficulty we encountered when answering this question was the format in which the attacker received the output of the ML system.
As highlighted by
Jagielski et al.
[108], an attacker can receive many forms of feedback by a given ML system.
For simplicity, we considered four answers: decision (e.g., [12]) if the attacker can only observe the decision of the whole ML system; label (e.g., [157]), if the attacker received the class with the highest probability; probability , if the attacker received any additional information beyond the label (e.g., [106]); as well as none , when the attacker did not receive any feedback (or was not needed).
T4: Training.
As stated in our main paper (see §IV-D), identifying if the attacker had some knowledge on the training data was very confusing.
To be as specific as possible, we considered the following answers: full , if the attacker had full access (read and write) to the training data; read , if the attacker can observe the entire training set; subset (e.g., [50], if the attacker had a subset of the actual training dataset; surrogate (e.g., [143]), if the attacker had samples not included in the training data, but having the same distribution; distribution (e.g., [158]), if the attacker only knows the class distribution of the training data; and none if the attacker knows nothing.
Quantitative analyses that take into account the cumulative answers to T1 to T4 are not possible: every paper ultimately considers a different scenario.
Therefore, we refrain from deriving any ‘trend’ from T1 to T4 .
Considerations should be made on a paper-by-paper basis, and we discuss some of them in the main paper (§IV-B and §IV-C).
Nevertheless, the individual answers to these questions are provided in Table IV.
I. Complete Table
We summarize our complete literature review in Table B-I.
This table lists each of the 88 papers considered in our analysis, presented in temporal order: from 2019 until 2021, and from NDSS (typically held in January) until CCS (typically held in November).
For each paper, we report the first author name, as well as concise answers to our 12 questions.
Generic Questions ( G1 to G8 ).
Most of these are straightforward.
For readability, every ‘negative’ answer is denoted with a blank cell.
For G1 (focus), an asterisk denotes a paper whose main contribution is a ML-based solution to a given problem, which is also evaluated in adversarial scenarios (e.g., [139]).
For G4 (cost), we use: ✗ when no consideration is made; when it is just mentioned; and ✓ when some measurement is performed.
For G5 , we also provide the specific data-type (e.g., malware) in the ‘other’ column (if there are more than one ‘other’ data-type, we report ‘+’).
For G6 (source code), the ✓ is a link that leads to the actual repository.
For G7 , we use ✓ if the paper develops an actual pipeline, i.e., at least another component besides the ML model (an asterisk denotes attacks at a different layer of the pipeline—e.g., hardware [111]).
Threat-Model Questions ( T1 to T4 ).
For T1 (parameters), we use ✗ if nothing is known, ✓ if the ML model is fully known, and if the attacker knows a portion of its architecture, or has a surrogate model.
For T11 (output), we use l if it is the actual label, p if it is a probability, s if it is the decision of the whole ML system, and ✓ if it is not needed.
For T11 (training data), we use: ✓ if the attacker has complete read and write access; R if they have complete knowledge of the training data; ⊂ if the attacker has a subset of the training data; and S if they have a surrogate dataset of the same distribution; and D if they only know the distribution of the training set; ✗ if they know (and can do) nothing about the training data.
However, if the answer to T2 is ‘poisoning’, it is implicitly assumed that the attacker always has some form of write access to the training data (the only exception is [96]).
We note that some (actually, most) papers envisioned attackers conforming to diverse threat models.
Due to some of statements made in our main paper (V-B), we show in Table IV
the assumptions that correspond to weakest attackers (e.g., if a paper considers both a black- and white-box attacker, we will report the description of the black-box attacker in
Table IV).
The motivation is that attacks stemming from weaker adversaries tend to be more representative of the real world.
Moreover, some papers consider dual attacks (e.g., [42] and [159]).
In these cases, we only report the one that is given a greater emphasis in the paper.
[Table]
p
[Table] [Table]
[Table] [Table]
[Table]
✗ ✗ ✗
SP (5/115)
Xu [146] def
Poison.
DL ✓ ✓ ✓ ✓ ✓ ✓ p ✓ Abdelnabi [179] atk
Evasion
DL ✓ ✓ ✓ ✗ ✓ p ✗ Chen [92] atk
Evasion
DL ✓ ✓ ✓ ✓
BOTH ✗ ✓ s
S Abdullah [142] atk
Evasion
DL ✓ ✓ ✓
C LOSED ✗ ✓ ✗ ✗ Nasr [33] def
Evasion
DL ✓ ✓
Finance ✗ ✓ p ✗
SEC (24/246)
Sato [26] atk
Evasion
DL ✓ ✓ ✓ ✓
O PEN ✓ ✓ p ✗ Nasr [90] atk
Evasion
DL ✓
Network ✓ ✓ ✗ ✓ p
D He [180] atk
Member.
DL
Graph ✗ ✓ p ✗ Severi [181] atk
Poison.
DL+SL ✓
Malware ✓ ✗ ✓ p ✓ Bagdasaryan [96] atk
Poison.
DL ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ Xi [152] atk
Poison.
DL ✓
Graph ✓ ✓ ✗
S Tang [97] def
Poison.
DL ✗ ✓ ✓ ✓ p ⊂ Schuster [182] atk
Poison.
DL ✓ ✓
O PEN ✗ ✓ ✗ ⊂ Carlini [52] atk
Poison.
DL ✓ ✓ ✗ ✓ ✗ ⊂ Vicarte [183] atk
Poison.
DL ✓ ✓ ✗ ✓ p ✗ Lovisotto [147] atk
Evasion
DL ✓ ✓ ✓
O PEN ✗ ✓ ✗ ⊂ Carlini
[Table]
28] atk
Member.
DL ✓ ✓ ✓
O PEN ✗ ✓ p ✗ Han
[Table]
98] atk
Evasion*
DL ✗
Graph ✗ ✓ p ✗ Eisenhofer
[Table]
] def
Evasion
DL ✓ ✓ ✓ ✓
O PEN ✓ ✓ p
R Wu
[Table]
] atk
Poison.
DL ✓
Games ✓ ✗ ✓ s ✗
He
[Table]
184] atk
Stealing
DL ✓ ✓ ✗ ✓ l
S Rakin
[Table]
113] atk
Evasion
DL ✓ ✓ ✓ * ✗ ✓ p ✗ Jia [185] def
Stealing
DL ✓ ✓ ✓ ✓ ✓ l ⊂
[Table]
] def
Stealing
DL ✗ ✓ * ✗ ✓ p ✗
[Table]
] def
Evasion
DL ✓ ✓ ✓ ✓ ✓ p ✗ Lin
[Table]
] atk
[Table]
DL ✓ ✓
Phishing ✓ ✓ ✗ ✓ p ✗
[Table]
def
Poison. [Table]
✓ ✓ ✓ ✓ p ✓ Hussain
[Table]
] def
[Table]
DL ✗ ✓ ✓ ✓ ✓ p ✗ Song
[Table]
100] def
Member.
DL ✓ ✓ + ✓ ✗ ✓ l ✗
CCS (9/196)
Zheng
[Table]
93] atk
[Table]
DL ✓ ✓
C LOSED ✗ ✓ ✗ ✗ Mu [190] atk
Evasion
DL ✓
Graphs ✗ ✓ l ✗ Bahramali [191] atk
Evasion
DL ✓
Network ✗ ✓ ✗
S Sheatsley
[Table]
] atk
[Table]
DL ✗
Network ✓ ✓ p
R Du
[Table]
] def
[Table]
DL ✓ ✓ ✓ ✓ ✓ p
R
[Table]
] def
[Table]
DL ✓ ✓ ✓ ✗ ✓ ✗ ✗
He
[Table]
def
Member.
DL ✗ ✓ ✓ ✓ l
S
[Table]
157] atk
Member.
DL ✓ ✓ ✓ ✗ ✓ l ✗ Chen [194] def
Member.
DL+SL ✗ ✓ + ✓ ✗ ✓ p ✗
Year (subs)
[Table]
[Figure]
[Table]
Type
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
G G
G
G5 (Evaluation
[Figure]
G
G G
T T
T T