O
NLINE A DVERSARIAL A TTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial attacks expose important vulnerabilities of deep learning models, yet little attention has been paid to settings where data arrives as a stream.
In this paper, we formalize the online adversarial attack problem, emphasizing two key elements found in real-world use-cases: attackers must operate under partial knowledge of the target model, and the decisions made by the attacker are irrevocable since they operate on a transient data stream.
We ﬁrst rigorously analyze a deterministic variant of the online threat model by drawing parallels to the well-studied ksecretary problem in theoretical computer science and propose V IRTUAL +, a simple yet practical online algorithm.
Our main theoretical result show V IRTUAL + yields provably the best competitive ratio over all single-threshold algorithms for k < 5 —extending previous analysis of the k -secretary problem.
We also introduce the stochastic k -secretary —effectively reducing online blackbox transfer attacks to a k -secretary problem under noise—and prove theoretical bounds on the performance of V IRTUAL +adapted to this setting.
Finally, we complement our theoretical results by conducting experiments on both MNIST and CIFARwith both vanilla and robust classiﬁers, revealing not only the necessity of online algorithms in achieving near-optimal performance but also the rich interplay of a given attack strategy towards online attack selection, enabling simple strategies like FGSM to outperform classically strong whitebox adversaries.
I NTRODUCTION
In adversarial attacks, an attacker seeks to maliciously disrupt the performance of deep learning systems by adding small but often imperceptible noise to otherwise clean data (Szegedy et al., 2014; Goodfellow et al., 2015).
Critical to the study of adversarial attacks is specifying the threat model Akhtar & Mian (2018), which outlines the adversarial capabilities of an attacker and the level of information available in crafting attacks.
Canonical examples include the whitebox threat model Madry et al.
(2017), where the attacker has complete access, and the less permissive blackbox threat model where an attacker only has partial information, like the ability to query the target model (Chen et al., 2017; Ilyas et al., 2019;
Papernot et al., 2016).
Previously studied threat models (e.g., whitebox and blackbox) implicitly assume a static setting that permits full access to instances in a target dataset at all times (Tramèr et al., 2018).
However, such an assumption is unrealistic in many real-world systems.
Countless real-world applications involve streaming data that arrive in an online fashion (e.g., ﬁnancial markets or real-time sensor networks).
Understanding the feasibility of adversarial attacks in this online setting is an essential question.
As a motivating example, consider the case where the adversary launches a man-in-the-middle attack depicted in Fig.
Here, data is streamed between two endpoints—i.e., from sensors on an autonomous car to the actual control system.
An adversary, in this example, would intercept the sensor data, potentially perturb it, and then send it to the controller.
Unlike classical adversarial attacks, such a scenario presents two key challenges that are representative of all online settings.
Transiency:
At every time step, the attacker makes an irrevocable decision on whether to attack and if she fails, or opts not to attack, then that datapoint is no longer available for further attacks.
Online Attack Budget: The adversary—to remain anonymous from stateful defences—is restricted to a small selection budget and must optimally balance a passive exploration phase before selecting high value items in the data stream (e
g. easiest to attack) to submit an attack on.
[Figure]
To the best of our knowledge, the only existing approaches that craft adversarial examples on streaming data (Gong et al., 2019a; Lin et al., 2017; Sun et al., 2020) require multiple passes through a data stream and thus cannot be applied in a realistic online setting where an adversary is forced into irrevocable decisions.
Moreover, these approaches do not come with theoretical guarantees.
Consequently, assessing the practicality of adversarial attacks— to better expose risks—in a truly online setting is still an open problem, and the focus of this paper.
Main Contributions .
We formalize the online threat model to study adversarial attacks on streaming data.
In our online threat model, the adversary must execute k successful attacks within n streamed data points, where k (cid:28) n .
As a starting point for our analysis, we study the deterministic online threat model in which the actual value of an input—i.e., the likelihood of a successful attack—is revealed along with the input.
Our ﬁrst insight elucidates that such a threat model, modulo the attack strategy, equates to the k -secretary problem known in the ﬁeld of optimal stopping theory Dynkin (1963); Kleinberg (2005), allowing for the application of established online algorithms for picking optimal data points to attack.
We then propose a novel online algorithm V IRTUAL
+ that is both practical, simple to implement for any pair ( k, n ) , and requires no additional hyperparameters.
Besides, motivated by attacking blackbox target models, we also introduce a modiﬁed secretary problem dubbed the stochastic k -secretary problem , which assumes the values an attacker observes are stochastic estimates of the actual value.
We prove theoretical bounds on the competitive ratio— under mild feasibility assumptions—for V IRTUAL +in this setting.
Guided by our theoretical results, we conduct a suite of experiments on both toy and standard datasets and classiﬁers (i.e., MNIST and
CIFAR-10).
Our empirical investigations reveal two counter-intuitive phenomena that are unique to the online blackbox transfer attack setting: 1.)
In certain cases attacking robust models may in fact be easier than non-robust models based on the distribution of values observed by an online algorithm.
Simple attack strategies like FGSM can seemingly achieve higher online attack transfer rates than stronger PGD-attackers when paired with a online selection algorithm, demonstrating the importance of carefully selecting which data points to attack.
Our key contributions are summarized as follows:
• We formalize the online adversarial attack threat model as an online decision problem and rigorously connect it to a generalization of the k-secretary problem.
• We introduce and analyze V IRTUAL +, an extension of V IRTUAL for the k -secretary problem yielding a signiﬁcant practical improvement ( 60% ).
We then provide, via novel techniques, a tractable formula for its competitive ratio, partially answering one of Albers & Ladewig (2020)’s open questions (see footnote 1 ) and achieving a new state-of-the-art competitive ratio for k < 5 .
• We propose Alg. 2 that leverages (secretary) online algorithms to perform efﬁcient online adversarial attacks.
We compare different online algorithms including V IRTUAL + via experiments on MNIST and CIFAR-10 in the challenging Non-Interactive BlackBox transfer (NoBox) setting.
B ACKGROUND AND P RELIMINARIES
Classical Adversarial Attack Setup .
We are interested in constructing adversarial examples against some ﬁxed target classiﬁer f t : X → Y which consumes input data points x ∈ X and labels them with a class label y ∈ Y .
The goal of an adversarial attack is then to produce an adversarial example x (cid:48) ∈ X , such that f t ( x (cid:48) )
(cid:54) =
y , and where the distance d ( x, x (cid:48) ) ≤ γ .
Then, equipped with a loss (cid:96) used to evaluate f t , an attack is said to be optimal if (Carlini & Wagner, 2017; Madry et al., 2017), x (cid:48) ∈ argmax x (cid:48) ∈X (cid:96)
[Equation]
Note that the formulation above makes no assumptions about access and resource restrictions imposed upon the adversary.
Indeed, if the parameters of f t are readily available, we arrive at the familiar whitebox setting, and problem in Eq.
1 is solved by following the gradient ∇ x f t that maximizes (cid:96) .
k-Secretary Problem .
The secretary problem is a well-known problem in theoretical computer science Dynkin (1963); Ferguson et al.
Suppose that we are tasked with hiring a secretary from a randomly ordered set of n potential candidates to select the secretary with maximum value.
The secretaries are interviewed sequentially and reveal their actual value on arrival.
Thus, the decision to accept or reject a secretary must be made immediately, irrevocably, and without knowledge of future candidates.
While there exist many generalizations of this problem, in this work, we consider one of the most canonical generalizations known as the k -secretary problem Kleinberg (2005).
Here, instead of choosing the best secretary, we are tasked with choosing k candidates to maximize the expected sum of values.
Typically, online algorithms that attempt to solve secretary problems are evaluated using the competitive ratio, which is the value of the objective achieved by an online algorithm compared to an optimal value of the objective that is achieved by an ideal “ofﬂine algorithm,” i.e., an algorithm with access to the entire candidate set.
Formally, an online algorithm A that selects a subset of items S A is said to be C -competitive to the optimal algorithm OPT which greedily selects a subset of items S ∗ while having full knowledge of all n items, if asymptotically in n
[Equation]
where V is a set-value function that determines the sum utility of each algorithm’s selection, and the expectations are over permutations sampled from the symmetric group of n elements, S n , acting on the data.
In §4, we shall further generalize the k -secretary problem to its stochastic variant where the online algorithm is no longer privy to the actual values but must instead choose under uncertainty.
O NLINE A DVERSARIAL A TTACKS
Motivated by our more realistic threat model, we now consider a novel adversarial attack setting where the data is no longer static but arrives in an online fashion.
3. A DVERSARIAL A TTACKS AS S ECRETARY P ROBLEMS
The deﬁning feature of the online threat model—in addition to streaming data and the fact that we may not have access to the target model f t —is the online attack budget constraint.
Choosing when to attack under a ﬁxed budget in the online setting can be related to a secretary problem.
We formalize this online adversarial attack problem in the boxed online threat model below.
In the online threat model we are given a data stream D
=
{ ( x 1 , y 1 ) , . . .
, ( x n , y n ) } of n samples ordered by their time of arrival.
In order to craft an attack against the target model f t , the adversary selects, using its online algorithm A , a subset S A ⊂ D of items to maximize:
[Equation]
where A TT ( x ) denotes an attack on x crafted by a ﬁxed attack method A TT that might or might not depend on f t .
From now on we deﬁne x (cid:48) i =
A TT ( x i ) .
Intuitively, the adversary chooses k instances that are the “easiest" to attack, i.e. samples with the highest value.
Note that selecting an instance to attack does not guarantee a successful attack.
Indeed, a successful attack vector may not exist if the perturbation budget γ is too small.
However, stating the adversarial goal as maximizing the value of
S A leads to the measurable objective of calculating the ratio of successful attacks in S A versus S ∗ .
If the adversary knows the true value of a datapoint then the online attack problem reduces to the original k -secretary.
On the other hand, the adversary might not have access to f t and instead, the adversary’s value function may be an estimate of the true value—e.g. the loss of a surrogate classiﬁer, and the adversary must make selection decisions in the face of uncertainty.
The theory developed in this paper will tackle both the case where values v i := (cid:96) ( f t ( x (cid:48) i ) , y i ) for i ∈ { 1 , . . . , n } :=
[ n ] are known (§3.2), as well as the richer stochastic setting with only estimates of v i , i ∈ [ n ] (§4).
Practicality of the Online Threat Model .
It is tempting to consider whether in practice the adversary should forego the online attack budget and instead attack every instance.
However, such a strategy poses several critical problems when operating in real-world online attack scenarios.
Chieﬂy, attacking any instance in D incurs a non-trivial risk that the adversary is detected by a defense mechanism.
Indeed, when faced with stateful defense strategies (e.g. Chen et al.
every additional attacked instance further increases the risk of being detected and rendering future attacks impotent.
Moreover, attacking every instance may be infeasible computationally for large n or impractical based on other real-world constraints.
Generally speaking, as conventional adversarial attacks operate by restricting the perturbation to a fraction of the maximum possible change (e.g., (cid:96) ∞ -attacks) online attacks analogously restrict the time window to a fraction of possible instances to attack.
Similarly, knowledge of n is also a factor that the adversary can easily control in practice.
For instance, in the autonomous control system example, the adversary can choose to be active for a short interval—e.g., when the autonomous car is a certain geospatial location—and thus set the value for n .
Online Threat Model.
The online threat model relies on the following key deﬁnitions:
• The target model f t .
The adversarial goal is to attack some target model f t : X → Y , through adversarial examples that respect a chosen distance function, d , with tolerance γ .
• The data stream D .
The data stream D contains the n examples ( x i , y i ) ordered by their time of arrival.
At any timestep i , the adversary receives the corresponding item in D and must decide whether to execute an attack or forever forego the chance to attack this item.
• Online attack budget k .
The adversary is limited to a maximum of k attempts to craft attacks within the online setting thus imposing that each attack is on a unique item in D .
• A value function V .
Each item in the dataset is assigned a value on arrival by the value function V : X × Y → R + which represents the utility of selecting the item to craft an attack.
This can be the likelihood of a successful attack under f t (true value) or a stochastic estimate of the incurred loss given by a surrogate model f s ≈ f t .
The online threat model corresponds to the setting where the adversary seeks to craft adversarial attacks (i) against a target model f t ∈ F , (ii) by observing items in D that arrive online, (iii) and choosing k optimal items to attack by relying on (iv) an available value function V .
The adversary’s objective is then to use its value function towards selecting items in D that maximize the sum total value of selections V (Eq. 3).
3. V IRTUAL + FOR A DVERSARIAL S ECRETARY P ROBLEMS
Let us ﬁrst consider the deterministic variant of the online threat model, where the true value is known on arrival.
For example consider the value function V ( x i , y i )
=
(cid:96) ( f t ( x (cid:48) i ) , y i ) =
v i i.e. the loss resulting from the adversary corrupting incoming data x i into x (cid:48)
i .
Under a ﬁxed attack strategy, the selection of high-value items from D is exactly the original k -secretary problem and thus the adversary may employ any A that solves the original k -secretary problem.
Well known single threshold-based algorithms that solve the k -secretary problem include the V IRTUAL , O PTIMISTIC Babaioff et al.
and the recent S INGLE -R EF algorithm Albers & Ladewig (2020).
In a nutshell, these online algorithm consists of two phases—a sampling phase followed by a selection phase —and an optimal stopping point t (threshold) that is used by the algorithm to transition between the phases.
In the sampling phase, the algorithms passively observe all data points up to a pre-speciﬁed threshold t .
Note that t itself is algorithm-speciﬁc and can be chosen by solving a separate optimization problem.
Additionally, each algorithm also maintains a sorted reference list R containing the top- k elements.
Each algorithm then executes the selection phase through comparisons of incoming items to those in R and possibly updating R itself in the process (see §D).
[Table]
and
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
{// Select element i } … t
[Figure]
and maintains R during the sampling phase.
Items are then picked into S A , after threshold t ,
Indeed, the simple structure of both the V IRTUAL and O PTIMISTIC algorithms—e.g., having few hyperparameters and not requiring the algorithm to involve Linear Program’s for varying values of n and k —in addition to being (1 /e ) -competitive (optimal for k = 1 ) make them suitable candidates for solving Eq.
However, the competitive ratio of both algorithms in the small k regime—but not k = 1 —has shown to be sub-optimal with S INGLE -R EF provably yielding larger competitive ratios at the cost of an additional hyperparameter selected via combinatorial optimization when n → ∞ .
We now present a novel online algorithm, V IRTUAL +, that retains the simple structure of V IRTUAL and O PTIMISTIC , with no extra hyperparameters, but leads to a new state-of-the-art competitive ratio for k < 5 .
Our key insight is derived from re-examining the selection condition in the V IRTUAL algorithm and noticing that it is overly conservative and can be simpliﬁed.
The V IRTUAL + algorithm is presented in Algorithm 1, where the removed condition in V IRTUAL (L2-3) is in pink strikethrough.
Concretely, the condition that is used by V IRTUAL but not by V IRTUAL + updates R during the selection phase without actually picking the item as part of S A .
Essentially, this condition is theoretically convenient and leads to a simpler analysis by ensuring that the V IRTUAL algorithm never exceeds k selections in S A .
V IRTUAL
+ removes this conservative R update criteria in favor of a simple to implement condition, | S A | ≤ k line 4 (in pink).
Furthermore, the new selection rule also retains the simplicity of V IRTUAL leading to a painless application to online attack problems.
Competitive ratio of V IRTUAL + .
What appears to be a minor modiﬁcation in V IRTUAL + compared to V IRTUAL leads to a signiﬁcantly more involved analysis but a larger competitive ratio.
In Theorem 1, we derive the analytic expression that is a tight lower bound for the competitive ratio of V IRTUAL + for general - k .
We see that V IRTUAL + provably improves in competitive ratio for k < 5 over both
V IRTUAL , O PTIMISTIC , and in particular the previous best single threshold algorithm, S INGLE -R EF .
Theorem 1.
The competitive ratio of V IRTUAL + for k ≥ 2 with threshold t k
= αn can asymptotically be lower bounded by the following concave optimization problem,
[Equation]
0 , 1] f
[Equation] [Equation]
where a m := (cid:0) k k ( k −
[Equation]
k
[Equation]
m − k m (cid:1) (
[Equation]
1) m +1 m ! .
Particularly, we get C 2 ≥ 0 .
427 , C 3 ≥ .
457 , C 4 ≥ .
4769 outperforming Albers & Ladewig (2020).
Connection to Prior Work .
The full proof for Theorem 1 can be found in §B along with a simple but illustrative proof for k = 2 in §A. Theorem 1 gives a tractable way to compute the competitive ratio of V IRTUAL + for any k , that improve the previous state-of-the-art (Albers & Ladewig, 2020) in terms of single threshold k -secretary algorithms for k < 5 and k > 100 .
However, it is also important to contextualize V IRTUAL + against recent theoretical advances in this space.
Most prominently,
Buchbinder et al.
(2014) proved that the k -secretary problem can be solved optimally (in terms of competitive ratio) using linear programs (LPs), assuming a ﬁxed length of n .
But these optimal algorithms are typically not feasible in practice.
Critically, they require individually tuning multiple thresholds by solving a separate LP with Ω( nk 2 ) parameters for each length of the data stream n , and the number of constraints grows to inﬁnity as n → ∞ .
Chan et al.
(2014) showed that optimal algorithms with k 2 thresholds could be obtained using inﬁnite LPs and derived an optimal algorithm for k =
Nevertheless they require a large number of parameters and the scalability of inﬁnite LPs for k > 2 remains uncertain.
In this work, we focus on practical methods with a single threshold (i.e., with O (1) parameters, e.g. Algorithm 1) that do not require involved computations that grow with n .
Open Questions for Single Threshold Secretary Algorithms .
By proposing new non-asymptotic results on the k -secretary problem that outperform asymptotically optimal algorithms, Albers &
Ladewig (2020) opened a new range of open questions for the k -secretary problem.
While this problem is considered solved when working with probabilistic algorithms 2 with Θ( nK 2 ) parameters (Buchbinder et al., 2014), ﬁnding optimal non-asymptotic single-threshold ( O (1) parameters) algorithms is still an open question.
As a step towards answering this question our work propose a practical algorithm that improves upon Albers & Ladewig (2020) for k = 2 , . .
, 4 with an optimal threshold that can be computed easily as it has a closed form.
S TOCHASTIC S ECRETARY P ROBLEM
In practice, online adversaries are unlikely to have access to the target model f t .
Instead, it is reasonable to assume that they have partial knowledge.
Following Papernot et al.
; Bose et al.
(2020) we focus on modeling that partial knowledge by equipping the adversary with a surrogate model or representative classiﬁer f s .
Using f s as opposed to f t means that we can compute the value V i :=
(cid:96) ( f s ( x (cid:48) i ) , y i ) of an incoming data point.
This value V i acts as an estimate of the value of interest v i := (cid:96) ( f t ( x (cid:48) i ) , y i ) .
The stochastic k -secretary problem is then to pick, under the noise model induced by using f s , the optimal subset S A of size k from D .
Thus, with no further assumptions on f s it is unclear whether online algorithms, as deﬁned in §3.2, are still serviceable under uncertainty.
Sources of randomness .
Our method relies on the idea that we can use the surrogate model f s to estimate the value of some adversarial examples on the target model f t .
We justify here how partial knowledge on f t could provide us an estimate of v i .
For example, we may know the general architecture and training procedure of f t , but there will be inherent randomness in the optimization (e.g., due to initialization or data sampling), making it impossible to perfectly replicate f t .
Moreover, it have been observed that, in practice, adversarial examples transfer across models (Papernot et al., 2016; Tramèr et al., 2017).
In that context, it is reasonable to assume that the random variable
V i :=
(cid:96) ( f s ( x (cid:48) i ) , y i ) is likely to be close to v i :
= (cid:96) ( f t ( x (cid:48) i ) , y i ) .
We formalize this idea in Assumption
4. S TOCHASTIC S ECRETARY A LGORITHMS
In the stochastic k -secretary problem, we assume access to random variables
V i and that v i are ﬁxed for i = 1 , . . .
, n and the goal is to maximize a notion of stochastic competitive ratio.
This notion is similar to the standard competitive ratio deﬁned in Eq.
2 with a minor difference that in the stochastic case, the algorithm does not have access to the values v i but to V i that is an estimate of v i .
An algorithm is said to be C s -competitive in the stochastic setting if asymptotically in n ,
[Equation]
Here the expectation is taken over S n (uniformly random permutations of the datastream D of size n ) and over the randomness of V
i , i = 1 , . . . , n .
S A and S ∗ are the set of items chosen by the stochastic online and ofﬂine algorithms respectively (note that while the online algorithm has access to V i , the ofﬂine algorithm picks the best v i ) and V is a set-value function as deﬁned previously.
Analysis of the algorithms .
In the stochastic setting all online algorithms observe V i that is an estimate of the actual value v i .
Since the goal of the algorithm is to select the k -largest values by only observing random variables ( V i ) it is requisite to make a feasibility assumption on the relationship between values v i and V i .
Let us denote top k { v i } as the set of top- k values among ( v i ) .
Assumption 1 (Feasibility) .
∃ γ > 0 such that P [ V i ∈ top
k
{V i } | v i ∈ top k { v i } ] ≥ γ, ∀ n ≥ 0 .
Assumption 1 is a feasibility assumption as if the ordering of ( V i ) does not correspond at all with the ordering of ( v i ) then there is no hope any algorithm—online or an ofﬂine oracle—would perform better than random when picking k largest v i by only observing ( V i ) .
In the context adversarial attacks, such an assumption is quite reasonable as in practice there is strong empirical evidence between the transfer of adversarial examples between surrogate and target models (see §C.1, for the empirical caliber of assumption 1).
We can bound the competitive ratio in the stochastic setting.
Theorem 2.
Let us assume that V IRTUAL + observes independent random variables
V i following
Assumption 1.
Its stochastic competitive ratio C s can be bounded as follows,
[Equation]
The proof of Thm.
2 can be found in §C. Such a theoretical result is quite interesting as the stochastic setting initially appears signiﬁcantly more challenging due to the non-zero probability that the observed ordering of historical values, V i , not being faithful to the true ranking based on v i .
4. R ESULTS ON S YNTHETIC D ATA
We assess the performance of classical single threshold online algorithms and V IRTUAL + in solving the stochastic k -secretary problem on a synthetic dataset of size n = 100 with k ∈ [1 , 10] .
The value of a data point is its index in D prior to applying any permutation π ∼ S n plus noise N (0 , σ 2 ) .
We compute and plot the competitive ratio over 10 k unique permutations of each algorithm in Figure 3.
[Figure]
Inputs: Permuted Datastream: D π , Online
Algo- rithm:
A , Surrogate classiﬁer: f s , Target classiﬁer: f t ,
Attack method: A TT , Loss: (cid:96) , Budget: k , Online Fool rate: F A π = 0 .
1: for ( x i , y i ) in D π do 2: x (cid:48)
i ← A TT ( x i ) {// Compute the attack} 3:
V i ← (cid:96) ( f s ( x (cid:48) i ) , y i ) {// Estimate v i } 4: if A ( V 1 , . . . , V i , k ) = =
T RUE then 5:
F A π ← F A π + 1 { f t ( x (cid:48) i ) (cid:54) =
y i } k {// Submit x (cid:48) i } 6: return: F A π {// A always submits k attacks} As illustrated for k = 1 all algorithms achieve the optimal (1 /e ) -deterministic competitive ratio in the stochastic setting.
Note that the noise level, σ 2 , appears to have a small impact on the performance of the algorithms (§E.1).
This substantiates our result in Thm.
2 indicating that C n -competitive algorithms only degrade by a small factor in the stochastic setting.
For k < 5 , V IRTUAL + achieves the best competitive ratio—empirically validating Thm 1—after which S INGLE -R EF is superior.
E XPERIMENTS
We investigate the feasibility of online adversarial attacks by considering an online version of the challenging NoBox setting (Bose et al., 2020) in which the adversary must generate attacks without any access, including queries, to the target model f t .
Instead, the adversary only has access to a surrogate f s which is similar to f t .
In particular, we pick at random a f t and f s from an ensemble of pre-trained models from various canonical architectures.
We perform experiments on the MNIST
LeCun & Cortes (2010) and CIFAR-10 Krizhevsky (2009) datasets where we simulate a D by generating 1000 permutations of the test set and feeding each instantiation to Alg. 2.
In practice, online adversaries compute the value V i
=
(cid:96) ( f s ( x (cid:48) i ) , y i ) of each data point in D by attacking f s using their ﬁxed attack strategy (where (cid:96) is the cross-entropy), but the decision to submit the attack to f t is done using an online algorithm A (see Alg. 2).
As representative attack strategies, we use the well-known FGSM attack (Goodfellow et al., 2015) and a universal whitebox attack in PGD (Madry et al., 2017).
We are most interested in evaluating the online fool rate, which is simply the ratio of successfully executed attacks against f t out of a possible of k attacks selected by A .
The architectures used for f s , f t , and additional metrics (e.g. competitive ratios) can be found in §E.
Baselines .
We rely on two main baselines, ﬁrst we use a NAIVE baseline–a lower bound–where the data points are picked uniformly at random, and an upper bound with the OPT baseline where attacks, while crafted using f s , are submitted by using the true value v i and thus utilizing f t . [Figure]
Q1:
Utility of using an online algorithm .
We ﬁrst investigate the utility of using an online algorithm,
A , in selecting data points to attack in comparison to the N AIVE baseline.
For a given permutation π and an attack method (FGSM or PGD), we compute the online fool rate of the N AIVE baseline and an A as F N AIVE π , F A π respectively.
In Fig.
4, we uniformly sample 20 permutations
π i ∼ S n , i ∈ [ n ] , of D and plot a scatter graph of points with coordinates ( F N AIVE
π
i , F A π i ) , for different A ’s, attacks with k = 1000 , and datasets.
The line y = x corresponds to the N AIVE baseline performance —i.e. coordinates ( F N AIVE π , F N AIVE π ) —and each point above that line corresponds to an A that outperforms the baseline on a given π i .
As observed, all A ’s signiﬁcantly outperform the N AIVE baseline with an average aggregate improvement of 7.5% and 34.1% on MNIST and CIFAR-10.
[Table]
Q2: Online Attacks on Non-Robust Classiﬁers
We now conduct experiments on non-robust
MNIST and CIFAR-10 classiﬁers.
We report the average performance of all online algorithms, and the optimal ofﬂine algorithm O PT in Tab. 1.
For MNIST, we ﬁnd that the two best online algorithms are S INGLE -R EF and our proposed V IRTUAL + which approach the upper bound provided by O PT .
For experiments with k < 5 please see §E.4.
For k = 10 and k = 100 , S INGLE -R EF is slightly superior while k = 1000 V IRTUAL + is the best method with an average relative improvement of 15 . 3% .
This is unsurprising as V IRTUAL
+ does not have any additional hyperparameters unlike
S INGLE -R EF which appears more sensitive to the choice of optimal thresholds and reference ranks, both of which are unknown beyond k = 100 and non-trivial to ﬁnd in closed form (see §E.2 for details).
On CIFAR-10, we observe that V IRTUAL + is the best approach regardless of attack strategy and the online attack budget k .
A notable observation is that even conventional whitebox adversaries like FGSM and PGD become strong blackbox transfer attack strategies when using an appropriate
[Table]
Q3:
Online Attacks on Robust Classiﬁers .
We now test the feasibility of online attacks against classiﬁers robustiﬁed using adversarial training by adapting the public Madry Challenge (Madry et al., 2017) to the online setting.
We report the average performance of each A in Table 2.
We observe that V IRTUAL + is the best online algorithms, outperforming V IRTUAL and O PTIMISTIC , in all settings except for k = 10 on MNIST where S INGLE -R EF is slightly better.
Q4:
Differences between the online and ofﬂine setting .
The online threat model presents several interesting phenomenon that we now highlight.
First we observe that a stronger attack (e.g. PGD)—in comparison to FGSM—in the ofﬂine setting doesn’t necessarily translate to an equivalently stronger attack in the online setting.
Such an observation was ﬁrst made in the conventional ofﬂine transfer setting by Madry et al.
, but we argue the online setting further exacerbates this phenomenon.
We explain this phenomenon in Fig.
5a & 5b by plotting the distribution of the loss values and the ratio of unsuccessful attacks to total attacks for PGD and FGSM.
We see that for the PGD attack high-density regions of unsuccessful attacks can be found even for high surrogate loss values and as a result can lead A further astray by picking unsuccessful data points—which may be top- k in surrogate loss values—to conduct a transfer attack.
A similar initially counter-intuitive observation can be made when comparing the online fool rate on robust and non-robust classiﬁers.
While it is natural to expect the online fool rate to be lower on robust model we empirically observe the opposite in Tab. 1 and 2.
To understand this phenomena we plot the distribution of the ratio of unsuccessful attacks to total attacks as a function f s ’s loss in Fig.
5c and observe non-robust models have a heavier tail making it harder for A to pick successful attacks purely based on loss (see §F).
[Figure]
[Figure]
R ELATED W ORK
Adversarial attacks .
The idea of attacking deep networks was ﬁrst introduced in (Szegedy et al., 2014; Goodfellow et al., 2015), and recent years have witnessed the introduction of challenging threat models, such as blackbox Chen et al.
; Ilyas et al.
; Jiang et al.
; Bose et al.
Chakraborty et al.
(2018) as well as defense strategies Madry et al.
; Tramèr et al.
Ding et al.
Closest to our setting are adversarial attacks against real-time systems Gong et al.
(2019a;b) and deep reinforcement learning agents
Lin et al.
(2017); Sun et al.
However, unlike our work, these approaches are not online algorithms and do not impose online constraints.
k -secretary .
The classical secretary problem was originally proposed by Gardner (1960) and later solved in Dynkin (1963) with an (1 /e )
-optimal algorithm.
Kleinberg (2005) introduced the k -secretary problem and an asymptotically optimal algorithm achieving a competitive ratio of 1 − Θ( (cid:112) 1 /k ) .
As outlined in §3.2 for general k an optimal algorithms exist Chan et al.
but requires the analysis of involved LPs that grow with the size of n .
A parallel line of work dubbed the prophet secretary problem, considers online problems where—unlike §4.1—some information on the distribution of values is known a priori Azar et al.
(2014; 2018); Esfandiari et al.
Secretary problems have also been applied to machine learning by informing the online algorithm about the inputs before execution Antoniadis et al.
; Dütting et al.
Finally, other interesting secretary settings include playing with adversaries Bradac et al.
; Kaplan et al.
C ONCLUSION
In this paper, we formulate the online adversarial attack problem, a novel threat model to study adversarial attacks on streaming data.
We propose V IRTUAL +, a simple yet practical online algorithm that enables attackers to select easy to fool data points while being theoretically the best single threshold algorithm for k < 5 .
We further introduce the stochastic k -secretary problem and prove fundamental results on the competitive ratio of any online algorithm operating in this new setting.
Our work sheds light on the tight coupling between optimally selecting data points using an online algorithm and the ﬁnal attack success rate, enabling weak adversaries to perform on par with stronger ones at no additional cost.
Investigating, the optimal threshold values for larger values of k along with competitive analysis for the general setting is a natural direction for future work.
E THICS S TATEMENT
We introduce the online threat model which aims to capture a new domain for adversarial attack research against streaming data.
Such a threat model exposes several new security and privacy risks.
For example, using online algorithms, adversaries may now tailor their attack strategy to attacking a small subset of streamed data but still cause signiﬁcant damage to downstream models e.g. the control system of an autonomous car.
On the other hand our research also highlights the need and importance of stateful defence strategies that are capable of mitigating such online attacks.
On the theoretical side the development and analysis of V IRTUAL + has many potential applications outside of adversarial attacks broadly categorized as resource allocation problems.
As a concrete example one can consider advertising auctions which provide the main source of monetization for a variety of internet services including search engines, blogs, and social networking sites.
Such a scenario is amenable to being modelled as a secretary problem as an advertiser may be able to estimate accurately the bid required to win a particular auction, but may not be privy to the trade off for future auctions.
R EPRODUCIBILITY STATEMENT
Throughout the paper we tried to provide as many details as possible in order for the results of the paper to be reproducible.
In particular, we provide a detailed description of V IRTUAL +in Alg.and we explain how to combine any attacker (e.g. PGD) with an online algorithm to form an online adversarial attack in Alg.
We provide a general description of the experimental setup in §5, further details with the speciﬁc architecture of the models and hyper-parameters used are provided in §E.2.
We also provided conﬁdence intervals with our experiments every time it was possible to do so.
Finally the code used to produce the experimental results is provided with the supplementary materials and will be made public after the review process.
A P ROOF OF C OMPETITIVE R ATIO FOR V IRTUAL +A LGORITHM
As an illustrative example that aids in understanding the full general-
k proof for for the competitive ratio V IRTUAL
+ we now prove Theorem 1 for k = 2 from the main paper.
Theorem 3.
For k = 2 , the competitive ratio achieved by V IRTUAL + algorithm is equal to,
[Equation]
Particularly for t = α · n , α ∈ (0 , 1) we get
[Equation] [Equation]
Thus, asymptotically we have
[Equation]
t n p j +
[Figure]
Pick first
[Figure]
at
[Figure]
Pick
[Figure]
Proof.
First note that by Albers & Ladewig (2020, Lemma 3.3) we can show that the competitive ratio for the k -secretary problem for a monotone algorithm is equal to
[Equation] [Equation]
where i a is the index of the a th secretary picked by the ofﬂine solution —i.e. i a is a top- k secretary of D .
By Lemma 2 V IRTUAL + is a monotone algorithm
and we may use Eq.
Now, let us focus on the case k = 2 .
When calculating the probability of either of the top two items in D being picked by the V IRTUAL + we must ﬁrst compute the probability of one of the top-2 items being picked during the selection phase (time step t + 1 . . .
n ).
Now notice that V IRTUAL + picks an item at time step j + 1 if and only if this is a top-2 item with respect to all of D and | S A | ≤ 2 at time-step j + 1 .
Let top- 2 j denote the two largest elements observed by A up to and inclusive of time step j .
Thus, for a ∈ { 1 , 2 } , we have
[Equation]
Now, we compute P ( | S A | ≤ 2 at time-step j + 1) by decomposing this probability into the following two events: A.)
| S A | = 0 where the selection set is empty and B.)
the event | S A | = 1 where exactly one item has been picked.
We now analyze each event in turn.
Event A .
In order for the event | S A | = 0 to occur it implies that the algorithm does not select any items in the ﬁrst j rounds.
This means both two top- 2 j elements must have appeared in the sampling phase.
Thus the probability for this event is exactly t ( t − 1)
j ( j − 1) .
Event B .
The second event is when | S A | = 1 —i.e. the algorithm picks exactly one element in the ﬁrst j rounds.
The computation of this event is illustrated in Figure 6.
Let’s say that an element is picked at time step p .
Now to compute the probability of Event B occurring we ﬁrst make the following two observations:
Observation 1:
In order for exactly one element to be picked at the time step p ≤ j , this element must be one of the top- 2 j elements.
Furthermore, this implies the other of the top- 2 j element —i.e. the one not picked at p must have appeared in the sampling phase.
Note that if both top- 2 j elements appear after the sampling phase, the condition would be satisﬁed twice and two elements would be selected instead of exactly one, and if they both appeared during the sampling phase we return to Event A. As a result, the probability for this condition is given by
[Equation]
Observation 2: By observation 1.
we know that the online algorithm A picks one of the top- 2 j at time step p and the fact that the event under consideration is | S A | = 1 the reference list R from time step p to j + 1 must contain both top- 2 j elements.
However, for
A to pick only at p we also need to ensure that no elements are picked prior t to p .
Therefore, before time step p the reference list must contain top- 2 p .
Again by observation 1, we know that R already contains one of the top- 2 j elements therefore we know it contains one of the top- 2 p elements.
Thus the probability of ensuring that the second top- 2 p elements is also within R by time step p is ( t − 1) ( p − 2) .
Finally, since there are two top elements and they may appear in any order we must count the probability of Event B occurring twice.
Overall we get:
[Equation]
Total probability:
[Equation]
Now we will use the following lemma
Lemma 1.
For any differentiable function f and any a < b , we have,
[Equation]
Proof.
[Equation]
Thus,
[Equation]
and by summing for n = a . . .
b we get the desired lemma.
Applying this lemma to f ( x ) = 1+2ln( x/t ) x, a = t and b = n − 1 , we get
[Equation]
Now for t = αn where α ∈ (0 , 1) and as n −→ ∞ , that lower-bound becomes
[Equation]
The constant term of the RHS is a concave function of α that is maximized for α ∗ ≈ 0 .
Thus, our algorithms achieves competitive ratio larger than 0 . 42737 .
B C OMPETITIVE R ATIO G ENERAL k
We now prove our main result for the competitive ratio of V IRTUAL + for k ≥ 2 .
The theorem statement is reproduced here for convenience.
Theorem 1.
The competitive ratio of V IRTUAL + for k ≥ 2 with threshold t k
= αn can asymptotically be lower bounded by the following concave optimization problem,
C k
[Equation]
(cid:33)
[Equation]
where a m =   k k ( k − 1) k − m − k m m ! 
 ( − 1) m +1 .
Particularly, we get C 2 ≥ 0 .
427 , C 3 ≥ .
457 , C 4 ≥ .
4769 outperforming Albers & Ladewig (2020).
t n p
[Equation]
Pick first
[Equation]
Pick any top elements at k j +
Length of sampling phase
[Figure]
Proof.
First note that by Albers & Ladewig (2020, Lemma 3.3) we can show that the competitive ratio for the k -secretary problem for a monotone algorithm is equal to
[Equation]
(18) where i a is the index of the a th secretary picked by the optimal ofﬂine solution —i.e. i a is a top- k secretary of D .
By Lemma 2 V IRTUAL + is a monotone algorithm
and we may use Eq.
[Equation] [Equation]
Now, we compute P ( | S A | < k at time-step j + 1) by decomposing this probability into smaller events P ( | S A | = ν at time-step j + 1) where ν ∈ [0 , . . .
, k − 1] .
We may compute the probability of P ( | S A | =
ν at time-step j + 1) in the following manner.
First, let us consider the scenario where ν elements are selected by V IRTUAL + at time steps p 1 , p 2 , . . .
, p ν .
Now, in order for an element to be selected at position p ν that element must be one of the top k elements up to time-step j + 1 .
Therefore we have a factor k/j in our equation.
Now, in order to guarantee that no elements are picked after the position p
ν
we additionally need to ensure that the remaining top- k up to j + 1 elements appear before p ν which results in a factor of (cid:0) p ν − 1 k − 1 (cid:1) / (cid:0) j − 1 k − 1 (cid:1) .
Similarly, we may recursively calculate the corresponding factor for each position p ν − 1 . . .
p 1 .
However, we also need to guarantee that no elements are picked within the time interval [ t +1 . . .
p 1 − 1] —i.e. before p 1 .
The probability for this occurring is then (cid:0) tk (cid:1) / (cid:0) p 1 − 1 k (cid:1) as this corresponds an ordering where the top- k elements up to p 1 − 1 all appear in the sampling phase.
Thus, the probability p k,νt,j := P ( | S A | = ν at time-step j + 1) is :
[Equation]
Therefore, the probability of not exceeding k -selections, p k t,j = (cid:80) k − 1 ν =
p k,ν t,j , to get before time step j + 1 is:
[Equation]
where we deﬁne Λ p i :
= 1 p i − k .
The total competitive ratio is then:
[Equation]
Now using Lemma 3 we can bound it:
[Equation]
[Equation] [Equation]
Now notice that:
[Equation]
Using the identity in Eq.
25 we compute the competitive ratio as:
[Equation]
For threshold t = αn where α ∈ (0 , 1) and as n −→ ∞ our competitive rate becomes:
[Equation]
Finally let us show that f
[Equation]
where a m =  
[Equation]
( k
[Equation]
− k m
[Equation] [Equation]
is concave.
To do so we just compute it second derivative and show that f (cid:48)(cid:48) ( α ) ≤ 0 .
We have, f
[Equation]
=
[Equation]
By using the deﬁnition of a m , we can verify that
[Equation]
and k
[Equation]
k − 1) a k
[Equation]
Thus we ﬁnally get,
[Equation]
where we use the fact that since α ∈ [0 , 1] , we have α log(1 /α ) ≥ 0 .
Deﬁnition B.1.
An algorithm is called monotone if the probabilities of selecting items i and j satisfy p i ≥ p j whenever the item values v i > v j holds for any two items.
Lemma 2.
V IRTUAL
+ is a monotone algorithm.
Proof.
In order to prove that V IRTUAL + is monotone as deﬁned in Deﬁnition B.1 we must prove that p i ≥ p j (where p i is the probability of picking the item i ) for any two items where v i > v j .
Without loss of generality let us consider a decreasing ordering of n -elements based on their values —i.e. v 1 > v 2 > · · · > v n .
We prove that p i ≥ p i +1 for all i ∈ [1 , . . . , n − 1] by showing that for each input sequence where v i + is accepted, there exists a unique input sequence where v i is accepted.
Let us consider a permutation π where v i +1 appeared and was accepted at time step a while v i appeared at time step b .
By swapping v i and v i +1 we obtain a new permutation π (cid:48) where v i now appears at a and v i +1 at b .
We now study the two following cases.
Case
a
< b .
If a < b notice that the reference set, R , and the selected set S A , are exactly the same at time step a for both permutations π and π (cid:48) .
Therefore, if v i +1 was accepted at time step a in permutation π then v i will also be accepted at time step a in permutation π (cid:48) since v i >
v i +1 .
Case 2: a > b .
If a > b notice that R —by deﬁnition of V IRTUAL +—at time step a contains top- k elements observed in the ﬁrst a − 1 time steps.
Now the k -th element in R at time-step a must satisfy,
[Equation]
where R a [ · ] [ k ] corresponds to the k -element in the reference set for a speciﬁc permutation at time step a .
Hence, we know that v i >
v i +1 ≥ R aπ [ k ] ≥ R a π (cid:48)
[ k ] as v i +1 was assumed to be picked.
Furthermore, the S A and R is the same for permutations π and π (cid:48) at time-step b .
Now by our primary assumption that v i +1 is picked at time-step a > b in π this means that v i must be v i ≥ R bπ
[ k ] since v i > v i +1 .
However, observe that v i and R bπ [ k ] cannot be consecutive in value as v i +1 appears at time-step a > b in permutation π .
This implies that v i +1 must also be selected at time step b in permutation π (cid:48) since v i and v i +1 are consecutive in value.
By a similar argument based on consecutive order of values between time steps a and b precisely the same elements will be selected in both π and π (cid:48) .
The argument that v i > v i +1 ≥ R aπ [ k ] ≥ R aπ (cid:48)
[ k ] implies that if v i +1 is selected in permutation π , v i will also be selected in permutation π (cid:48) .
The claim then follows by applying the inequality p i ≥ p
i +1 in an iterative fashion.
Lemma 3.
Let f i , i = 1 . . .
k be decreasing positive functions then we have
[Equation]
Proof.
The main proof step involves in ﬁrst noticing that since the functions f i , i = 1 . . .
k are decreasing and are positive we have,
[Equation]
[Equation]
Thus, by summing this inequality for p k = a k . . .
p k − 1 , we get
[Equation]
Now, because the functions f i , i = 1 . . .
k are decreasing and positive we have,
[Equation]
where for the last inequality we used the fact that x k − 1 ∈
[ p k − 1 , p k − 1 + 1] .
Finally, by summing for p k − 1 = a k − 1 . . .
p k − 2 , we get,
[Equation]
Using a recursive argument we ﬁnally get,
[Equation]
[Table]
C P ROOF OF T HEOREM 2 AND EMPIRICAL VERIFICATION
We now prove Theorem 2 in detail, reproduced here for convenience
[Table]
Theorem 2.
Let us assume that V IRTUAL + observes independent random variables
V i following
Assumption 1.
Its stochastic competitive ratio C s can be bounded as follows,
[Equation]
Proof.
In the Stochastic case we use the same beginning proof as in the non-stochastic case until Eq. 19.
Let us consider i a such that V i a ∈ top k
{V i } ,
[Equation]
Now, in the stochastic case P ( i a ∈ S A at time-step j + 1) not only depends on the set S A not being full but also that i a corresponds to a top- k elements in { v i } .
Because of the expectation over permutations, the event of the knapsack being fulled at time step j +
is independent from what happens at timestep j + 1 .
Thus we can write that
[Equation]
Finally, we just need to notice that P [ V i ∈ top k {V i } | v i ∈ top k { v i } ] does not depend on the value observed and thus leave to the same computation as in the non-stochastic case.
Similarly, we have
[Equation]
In conclusion it leads to
[Equation]
C. E MPIRICAL Q UANTIFICATION OF A SSUMPTION
Assumption 1 requires that a percentage of top- k true values v i remain top- k under noise.
We now empirically quantify the strength of this assumption in both of our datasets MNIST and CIFAR-10.
Note that unlike in theorem we cannot enforce any structure on the random variables that act as surrogate losses as provided by f s .
Despite this, we ﬁnd that in all cases the overlap between the top- k sets is non-zero which enables the effective use of online algorithms for picking candidate attack points as shown in table 4.
[Table]
D C LASSICAL O NLINE A LGORITHMS FOR S ECRETARY P ROBLEMS
All single threshold online algorithm described in this paper include: V IRTUAL , O PTIMISTIC and
S INGLE -R EF .
Each online algorithm consists of two phases — sampling phase followed by selection phase — and an optimal stopping point t which is used by the algorithm to transition between the phases.
We now brieﬂy summarize these two phases for the aforementioned online algorithms.
Sampling Phase - V IRTUAL , O PTIMISTIC and S INGLE -R EF .
In the sampling phase, the algorithms passively observe all data points up to a pre-speciﬁed time index t , but also maintains a sorted reference list R consisting of the k elements with the largest values V ( i ) seen.
Thus the R contains a list of elements sorted by decreasing value.
That is R [ k ] is the index of the k -th largest element in R and V ( R [ k ]) is its corresponding value.
The elements in R are kept for comparison but are crucially not selected in the sampling phase.
D. V IRTUAL A LGORITHM
Selection Phase - V IRTUAL algorithm .
Subsequently, in the selection phase, i > t , when an item with value V ( i ) is observed an irrevocable decision is made of whether the algorithm should select i into S .
To do so, the Virtual algorithm simply checks if the value of the k -th smallest element in R ,
V ( R [ k ]) , is smaller than V ( i ) in addition to possibly updating the set R .
The full Virtual algorithm is presented in Algorithm 1.
Algorithm
Inputs: t ∈ [ k . . .
n − k ] , R = ∅ , S A = ∅ Sampling phase: Observe the ﬁrst t data points and construct a list R with the indices of the top k data points seen.
sort ensures: V ( R [1]) ≥ V ( R [2]) · · · ≥ V ( R [ k ]) .
Selection phase (at time i > t ): 1: if V ( i ) ≥ V ( R [ k ]) and R [ k ] > t then 2:
3: else if V ( i ) ≥ V ( R [ k ]) and R [ k ] ≤ t then element
Selection Phase - O PTIMISTIC algorithm .
In the optimistic algorithm, i is selected if and only if
V ( i ) ≥ V ( R [ last ]) .
Whenever i is selected, R [ last ] is removed from the list R , but no new elements are ever added to R .
Thus, intuitively, elements are selected when they beat one of the remaining reference points from R .
We call this algorithm “optimistic” because it removes the reference point
R [ last ] even if V ( i ) exceeds, say, V ( R [1]) .
Thus, it implicitly assumes that it will see additional very valuable elements in the future, which will be added when their values exceed those of the remaining, more valuable, R [ a ] , a ∈ [ k ] .
D.
S
Selection Phase - S INGLE -R EF algorithm .
In the S INGLE -R EF algorithm, i is selected if and only if V ( i ) ≥ V ( R [ r ]) and we haven’t already selected k elements.
We call this algorithm single
Inputs: t ∈ [ k . . .
n − k ] , R = ∅ , S A = ∅ .
Sampling phase (up to time t ): Observe the ﬁrst t data points and construct a list R with the indices of the top k data points seen.
sort ensures: V ( R [1]) ≥ V ( R [2]) · · · ≥ V ( R [ k ]) .
Set last =
k , to be the index of the last element in R .
Selection phase (at time i > t ): 1: if V ( i ) ≥ V ( R [ last ]) then 2:
R = { R \ { R [ last ] }} {// Update R by taking out R [ k ] }
[Table] [Table]
=
[Table]
{ i }} {// Select element i }
[Table]
[Table]
i ← i +reference algorithm because we always compare incoming elements to one single reference element, that was determined in the sampling phase.
Inputs: t ∈ [ k . . .
n − k ] , R = ∅ , S A = ∅ , r ∈
[ k ] (reference rank)
Sampling phase (up to time t ): Observe the ﬁrst t data points and construct a list R with the indices of the top k data points seen.
Let s r =
R
[ r ] be the r -th best item from the sampling phase.
Selection phase (at time i > t ): 1:
{// Choose the ﬁrst k items better than s
} i ←
[Table]
E A DDITIONAL E XPERIMENTAL R ESULTS E. A DDITIONAL R ESULTS ON S YNTHETIC D ATA
We now provide additional results on Synthetic Data with varying levels of noise added to each item in D .
In particular, we investigate in ﬁgure 8 online algorithms in the face of no noise —i.e. σ 2 = 0 , σ 2 = 1 , and σ 2 = 5 in addition to σ 2 = 10 reported in ﬁgure 3.
The deterministic setting corresponds to σ 2 = 0 while σ 2 = 1 and σ 2 = 1 correspond to the stochastic setting as introduced in section 4.
E. E XPERIMENTAL D ETAILS
We provide more details about the experiments presented in section 5.
For further details we also invite the reader to look at the code provided with the supplementary materials.
The complete code to reproduce results can be found https
://anonymous.4open.science/r/ OnlineAttacks-
Attack strategies We use two different attack strategies the Fast Gradient Sign Method (FGSM)
(Goodfellow et al., 2015) and 40 iterations of the PGD attack (Madry et al., 2017) with l ∞ .
Hyper-parameters of online algorithms
All the online algorithms except S INGLE -R EF have a single hyper-parameters to choose which is the length of the sampling phase t .
For V IRTUAL and
O PTIMISTIC we use t = (cid:98) te (cid:99) which is the value suggested by theory in Babaioff et al.
For V IRTUAL + we use t = αn as found by solving the maximization problem for a speciﬁc k in
Theorem 1.
S INGLE -R EF has two hyper-parameters to choose the threshold t ( c in the original paper) and reference rank r .
For k = 1 ... 100 the values are given in Albers & Ladewig (2020) and are numerical solutions to combinatorial optimization problems.
However, for k = 1000 no values are speciﬁed and we choose c = 0 .
13 and r = 40 through grid search.
Indeed, these values may not be optimal ones but we leave the choice of better values as future work.
MNIST model architectures For table 1, f s and f t are chosen randomly from an ensemble of trained classiﬁers.
The ensemble is composed of ﬁve different architectures described in table 5, with 5 trained models per architecture.
CIFAR model architectures For table 1, f s and f t are chosen randomly from an ensemble of trained classiﬁers.
The ensemble is composed of ﬁve different architectures: VGG-16 (Simonyan & Zisserman, 2015), ResNet-18 (RN-18)
(He et al., 2016), Wide ResNet (WR) (Zagoruyko &
Komodakis, 2016), DenseNet-121 (DN-121) (Huang et al., 2017) and Inception-V3 architectures (Inc-V3)
(Szegedy et al., 2016), with 5 trained models per architecture.
E. A DDITIONAL METRICS
In addition to the results provided in table 1, we also provide two other metrics here: the stochastic competitive ratio in table 6 and the knapsack ratio table 7.
Where the knapsack ratio is deﬁned as the sum value of S A —i.e. the sum of total loss, as selected by the online algorithm divided by the
[Table]
Conv
[Table] [Table]
[Table] [Table]
Relu
Dropout
[Table] [Table]
FC
[Table]
Dropout
[Table]
Conv
[Table] [Table]
Conv
[Table]
[Table] [Table]
MaxPool
[Table]
Dropout
[Table]
Conv
[Table] [Table]
Conv
[Table]
Conv
[Table]
value of S ∗ selected by the optimal ofﬂine algorithm.
We observe that the competitive ratio is not always a good metric to compare the actual performance of the different algorithms, since sometimes the online algorithm with the best competitive ratio is not the algorithm with the best fool rate.
The knapsack ratio on the other hand seems to be a much better proxy for the actual performance of the algorithms, this is due to the fact that we’re interested in picking elements that have have a good chance to fool the target classiﬁer but are not necessarily the best possible attack.
[Table] [Table]
[Table] [Table]
E. A DDITIONAL RESULTS
Same architecture In addition to table 1 we also provide some results on MNIST where f s and f t always have the same architecture but have different weights.
This is a slightly less challenging setting as shown in Bose et al.
, we also observe that in this setting the adversaries are very effective against the target model.
[Table]
[Table] [Table]
[Table]
F D ISTRIBUTION OF V ALUES O BSERVED B Y O NLINE A LGORITHMS
In this section we further investigate performance disparity of online algorithms against robust and non-robust models for CIFAR-10 as observed in Tables 1 and 2.
We hypothesize that one possible explanation can be found through analyzing the ratio distribution of values
V i ’s for unsuccessful and successful attacks as observed by the online algorithm when attacking each model type.
However, note that eventhough an online adversary may employ a ﬁxed attack strategy to craft an attack x (cid:48)
=
ATT ( x ) the scale of values in each setting are not strictly comparable as the attack is performed on different model types.
In other words, given an ATT it is signiﬁcantly more difﬁcult to attack a robust model and thus we can expect a lower V i when compared to attacking a non-robust model.
Thus to investigate the difference in efﬁcacy of online attacks we pursue a distributional argument.
Indeed, distributions of V
i ’s observed, for a speciﬁc permutation of D , may drastically affect the performance of the online algorithms.
Consider for instance, if the V i ’s that correspond to successful attacks cannot be distinguished from the ones that are unsuccessful.
In such a case one cannot hope to use an online algorithm—that only observes V i ’s—to always correctly pick successful attacks.
In Figure 9 we visualize the distribution of V
i ’s of unsuccessful and successful attacks as a density ratio for CIFAR-10 robust and non-robust models.
We plot the distribution of V
i ’s (x-axis) with respect to the density ratio of unsuccessful versus successful attack vectors (y-axis) as provided by a kernel density estimator.
As observed, the tail of the distribution in the non-robust case is heavier than in the robust case which indicates that there are many data points with high values that lead to unsuccessful attacks.
Furthermore, this also suggests one explanation for the higher efﬁcacy of online algorithms against robust models: fewer attacks are successful but they are easier to differentiate from unsuccessful ones.
Importantly, this implies that given an online attack budget k (cid:28) n higher online fool rates can be achieved against robust models as the selected data points turn adversarial with higher probability when compared to non-robust models. [Figure] i
[Figure]
G F URTHER C OMPARISON T O P RIOR W ORK
While we consider —to the best of our knowledge—that our work is the only truly online threat model.
Our setting is the only one considering that data points are only ever observed once, and a decision to attack must be made at the moment and cannot be reversed retroactively.
For example, (Gong et al., 2019b) consider replay attacks on Voice-Controlled Systems whereby streamed audio input is captured with a recording device, and then the entire sequence is spoofed and replayed back.
Unlike online attacks that we consider, they can manipulate the whole sequence retroactively and do not have to make an irreversible decision to attack at a given timestep.
Similarly, both (Lin et al., 2017; Sun et al., 2020) consider adversarial attacks against deep reinforcement learning agents.
Like us, they consider an adversarial budget that limits the number of points to attack to avoid detection.
However, unlike us, they require whitebox access to the target model in order to train another predictive model by interacting with the environment, which can later be used to inform “when to attack”.
Thus the datapoints appearing at test time may already be seen and scored during the training period.
The dichotomy between collecting data for potentially an inﬁnite time horizon before attacking is at odds with our online threat model as a data point can only be observed once.
As a result, none of these works can be used within our online threat model and are not appropriate baselines.