Original: M ODEL -T ARGETED P OISONING A TTACKS WITH P ROVABLE C ONVERGENCE
Revised: M ODEL -T ARGETED P OISONING A TTACKS WITH P ROVABLE C ONVERGENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In a poisoning attack, an adversary with control over a small fraction of the training data attempts to select that data in a way that induces a model that misbehaves in a particular way desired by the adversary, such as misclassifying certain inputs.
Revised: In a poisoning attack, an adversary with control over a small fraction of the training data attempts to select that data in a way that induces a model that misbehaves in a particular way desired by the adversary, such as misclassifying certain inputs.

Original: We propose an efﬁcient poisoning attack that can target a desired model based on online convex optimization.
Revised: We propose an efﬁcient poisoning attack that can target a desired model based on online convex optimization.

Original: Unlike previous model-targeted poisoning attacks, our attack comes with provable convergence to any achievable target classiﬁer.
Revised: Unlike previous model-targeted poisoning attacks, our attack comes with provable convergence to any achievable target classiﬁer.

Original: The distance from the induced classiﬁer to the target classiﬁer is inversely proportional to the square root of the number of poisoning points.
Revised: The distance from the induced classiﬁer to the target classiﬁer is inversely proportional to the square root of the number of poisoning points.

Original: We also provide a lower bound on the minimum number of poisoning points needed to achieve a given target classiﬁer.
Revised: We also provide a lower bound on the minimum number of poisoning points needed to achieve a given target classiﬁer.

Original: Our attack is the ﬁrst model-targeted poisoning attack that provides provable convergence, and in our experiments it either exceeds or matches the best state-of-the-art attacks in terms of attack success rate and distance to the target model.
Revised: Our attack is the ﬁrst model-targeted poisoning attack that provides provable convergence, and in our experiments it either exceeds or matches the best state-of-the-art attacks in terms of attack success rate and distance to the target model.

Original: In addition, as an online attack our attack can incrementally determine nearly optimal poisoning points.
Revised: In addition, as an online attack our attack can incrementally determine nearly optimal poisoning points.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: State-of-the-art machine learning models require a large amount of labeled training data, which often depends on collecting data and labels from untrusted sources.
Revised: State-of-the-art machine learning models require a large amount of labeled training data, which often depends on collecting data and labels from untrusted sources.

Original: A typical application is email spam ﬁltering, where a spam detector ﬁlters out spam messages based on features (e.g., presence of certain words) and periodically updates the model based on newly received emails labeled by users.
Revised: A typical application is email spam ﬁltering, where a spam detector ﬁlters out spam messages based on features (e.g., presence of certain words) and periodically updates the model based on newly received emails labeled by users.

Original: In such a setting, spammers can generate “non-spam” messages by injecting non-related words or benign words, and when models are trained on these “non-spam” messages, the ﬁltering accuracy will drop signiﬁcantly (Lowd & Meek, 2005).
Revised: In such a setting, spammers can generate “non-spam” messages by injecting non-related words or benign words, and when models are trained on these “non-spam” messages, the ﬁltering accuracy will drop signiﬁcantly (Lowd & Meek, 2005).

Original: Such attacks are known as poisoning attacks , and a training process that collects labels or data from untrusted sources is potentially vulnerable to them.
Revised: Such attacks are known as poisoning attacks , and a training process that collects labels or data from untrusted sources is potentially vulnerable to them.

Original: Poisoning attacks can be categorized into objective-driven attacks and model-targeted attacks depending on whether a target model is considered in the attack process.
Revised: Poisoning attacks can be categorized into objective-driven attacks and model-targeted attacks depending on whether a target model is considered in the attack process.

Original: Objective-driven attacks have a speciﬁc attacker objective and aim to achieve the attack objective by generating the poisoning points; model-targeted attacks have a speciﬁc target classiﬁer in mind and aim to induce that target classiﬁer by generating a minimal number of poisoning points.
Revised: Objective-driven attacks have a speciﬁc attacker objective and aim to achieve the attack objective by generating the poisoning points; model-targeted attacks have a speciﬁc target classiﬁer in mind and aim to induce that target classiﬁer by generating a minimal number of poisoning points.

Original: Objective-driven attacks are most commonly studied in the existing literature.
Revised: Objective-driven attacks are most commonly studied in the existing literature.

Original: The attacker objective is typically one of two extremes: indiscriminate attacks, where the adversary’s goal is simply to decrease the overall accuracy of the model (Biggio et al., 2012; Xiao et al., 2012; Mei & Zhu, 2015b;
Revised: The attacker objective is typically one of two extremes: indiscriminate attacks, where the adversary’s goal is simply to decrease the overall accuracy of the model (Biggio et al., 2012; Xiao et al., 2012; Mei & Zhu, 2015b;

Original: Steinhardt et al., 2017; Koh et al., 2018); and instance-targeted attacks, where the goal is to produce a classiﬁer that misclassiﬁes a particular known input (Shafahi et al., 2018; Zhu et al., 2019; Koh & Liang, 2017).
Revised: Steinhardt et al., 2017; Koh et al., 2018); and instance-targeted attacks, where the goal is to produce a classiﬁer that misclassiﬁes a particular known input (Shafahi et al., 2018; Zhu et al., 2019; Koh & Liang, 2017).

Original: Recently,
Revised: Recently,

Original: Jagielski et al.
Revised: Jagielski et al.

Original: (2019) introduced a more realistic attacker objective known as a subpopulation attack, where the goal is to increase the error rate or obtain a particular output for a deﬁned subpopulation of the data distribution.
Revised: (2019) introduced a more realistic attacker objective known as a subpopulation attack, where the goal is to increase the error rate or obtain a particular output for a deﬁned subpopulation of the data distribution.

Original: Attacker objectives for realistic attacks are diverse and designing a uniﬁed and effective attack strategy for different attacker objectives is hard.
Revised: Attacker objectives for realistic attacks are diverse and designing a uniﬁed and effective attack strategy for different attacker objectives is hard.

Original: Gradient-based local optimization is most commonly used to construct poisoning points for a particular attacker objective (Biggio et al., 2012; Xiao et al., 2012; Mei & Zhu, 2015b; Koh & Liang, 2017; Shafahi et al., 2018; Zhu et al., 2019).
Revised: Gradient-based local optimization is most commonly used to construct poisoning points for a particular attacker objective (Biggio et al., 2012; Xiao et al., 2012; Mei & Zhu, 2015b; Koh & Liang, 2017; Shafahi et al., 2018; Zhu et al., 2019).

Original: Although these attacks can be modiﬁed to ﬁt other attacker objectives, since they are based on local optimization techniques they can easily get stuck into bad local optima and fail to ﬁnd effective sets of poisoning points (Steinhardt et al., 2017; Koh et al., 2018).
Revised: Although these attacks can be modiﬁed to ﬁt other attacker objectives, since they are based on local optimization techniques they can easily get stuck into bad local optima and fail to ﬁnd effective sets of poisoning points (Steinhardt et al., 2017; Koh et al., 2018).

Original: To circumvent the issue of local optima,
Revised: To circumvent the issue of local optima,

Original: Steinhardt et al.
Revised: Steinhardt et al.

Original: (2017) formulate an indiscriminate attack as a min-max optimization problem and solve it efﬁciently using online convex optimization techniques.
Revised: (2017) formulate an indiscriminate attack as a min-max optimization problem and solve it efﬁciently using online convex optimization techniques.

Original: However, the strong min-max attack only applies to the indiscriminate setting.
Revised: However, the strong min-max attack only applies to the indiscriminate setting.

Original: In contrast, model-targeted attacks incorporate the attacker objective into a target model and hence, the target model can reﬂect any attacker objective.
Revised: In contrast, model-targeted attacks incorporate the attacker objective into a target model and hence, the target model can reﬂect any attacker objective.

Original: Thus, the same model-targeted attack methods can be directly applied to a range of indiscriminate and subpopulation attacks just by ﬁnding a suitable target model.
Revised: Thus, the same model-targeted attack methods can be directly applied to a range of indiscriminate and subpopulation attacks just by ﬁnding a suitable target model.

Original: Mei & Zhu (2015b)
Revised: Mei & Zhu (2015b)

Original: ﬁrst introduced a target model into a poisoning attack, but their attack is still based on gradient-based local optimization techniques and suffers from bad local optima (Steinhardt et al., 2017; Koh et al., 2018).
Revised: ﬁrst introduced a target model into a poisoning attack, but their attack is still based on gradient-based local optimization techniques and suffers from bad local optima (Steinhardt et al., 2017; Koh et al., 2018).

Original: Koh et al.
Revised: Koh et al.

Original: (2018) proposed the KKT attack, which converts the complicated bi-level optimization into a simple convex optimization problem utilizing the KKT condition, avoiding the local optima issues.
Revised: (2018) proposed the KKT attack, which converts the complicated bi-level optimization into a simple convex optimization problem utilizing the KKT condition, avoiding the local optima issues.

Original: However, their attack only works for margin based losses and does not provide any guarantee on the number of poisoning points required to converge to the target classiﬁer.
Revised: However, their attack only works for margin based losses and does not provide any guarantee on the number of poisoning points required to converge to the target classiﬁer.

Original: In this work, we focus on model-targeted attacks and aim to understand the feasibility of a poisoning adversary to induce any target model.
Revised: In this work, we focus on model-targeted attacks and aim to understand the feasibility of a poisoning adversary to induce any target model.

Original: In particular, we ﬁnd both theoretical and empirical bounds on the sufﬁcient (and necessary) number of poisoning points to get close to a speciﬁc target classier.
Revised: In particular, we ﬁnd both theoretical and empirical bounds on the sufﬁcient (and necessary) number of poisoning points to get close to a speciﬁc target classier.

Original: Contributions.
Revised: Contributions.

Original: Our main contributions involve developing a principled and general model-targeted poisoning attack strategy, along with a proof that the model it induces converges to the target model.
Revised: Our main contributions involve developing a principled and general model-targeted poisoning attack strategy, along with a proof that the model it induces converges to the target model.

Original: Our poisoning method takes as input a target model, and produces a set of poisoning points.
Revised: Our poisoning method takes as input a target model, and produces a set of poisoning points.

Original: We prove that the model induced by training on the original training data with these points added, converges to the target classiﬁer as the number of poison points increases, given that the loss function is convex and proper regularization is adopted in training (Theorem 4.1).
Revised: We prove that the model induced by training on the original training data with these points added, converges to the target classiﬁer as the number of poison points increases, given that the loss function is convex and proper regularization is adopted in training (Theorem 4.1).

Original: Previous model-targeted attacks lack of such convergence guarantees.
Revised: Previous model-targeted attacks lack of such convergence guarantees.

Original: We then prove a lower bound on the minimum number of poisoning points needed to reach the target model (Theorem 4.2), given that the loss function for empirical risk minimization is convex.
Revised: We then prove a lower bound on the minimum number of poisoning points needed to reach the target model (Theorem 4.2), given that the loss function for empirical risk minimization is convex.

Original: Such a lower bound can be used to estimate the optimality of model-targeted poisoning attacks and also indicate the intrinsic hardness of attacking different targets.
Revised: Such a lower bound can be used to estimate the optimality of model-targeted poisoning attacks and also indicate the intrinsic hardness of attacking different targets.

Original: Our attack is also efﬁcient in incremental poisoning scenario as it works in an online fashion and can incrementally ﬁnd poisoning points that are nearly optimal.
Revised: Our attack is also efﬁcient in incremental poisoning scenario as it works in an online fashion and can incrementally ﬁnd poisoning points that are nearly optimal.

Original: Previous model-targeted attacks work with ﬁxed number of poisoning points and need to know the poisoning budget in advance.
Revised: Previous model-targeted attacks work with ﬁxed number of poisoning points and need to know the poisoning budget in advance.

Original: We run experiments to compare our attack to the state-of-the-art model-targeted attack (Koh et al., 2018).
Revised: We run experiments to compare our attack to the state-of-the-art model-targeted attack (Koh et al., 2018).

Original: We ﬁrst evaluate the convergence our attack to the target model and ﬁnd that, under same number of poisoning points, classiﬁers induced by our attack are closer to the target models than the best known attack, for all the target classiﬁers we tried.
Revised: We ﬁrst evaluate the convergence our attack to the target model and ﬁnd that, under same number of poisoning points, classiﬁers induced by our attack are closer to the target models than the best known attack, for all the target classiﬁers we tried.

Original: Then, we evaluate the success rate of our attack, and ﬁnd that it has superior performance than the state-of-the-art in the more realistic subpopulation attack scenario, and comparable performance in the conventional indiscriminate attack scenario (Section 5).
Revised: Then, we evaluate the success rate of our attack, and ﬁnd that it has superior performance than the state-of-the-art in the more realistic subpopulation attack scenario, and comparable performance in the conventional indiscriminate attack scenario (Section 5).

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: The poisoning attack proposed in this paper applies to multi-class prediction tasks or regression problems (by treating the response variable as an additional data feature), but for simplicity of presentation we consider a binary prediction task, h : X → Y , where X ⊆ R d and Y = { +1 , − 1 } .
Revised: The poisoning attack proposed in this paper applies to multi-class prediction tasks or regression problems (by treating the response variable as an additional data feature), but for simplicity of presentation we consider a binary prediction task, h : X → Y , where X ⊆ R d and Y = { +1 , − 1 } .

Original: The prediction model h is characterized by parameters θ ∈ Θ ⊆ R d .
Revised: The prediction model h is characterized by parameters θ ∈ Θ ⊆ R d .

Original: We deﬁne the non-negative convex loss on an individual point, ( x, y ) , as l ( θ ; x, y )
Revised: We deﬁne the non-negative convex loss on an individual point, ( x, y ) , as l ( θ ; x, y )

Original: (e.g., hinge loss for SVM model).
Revised: (e.g., hinge loss for SVM model).

Original: We also deﬁne the empirical loss over a set of points A as L ( θ ; A )
Revised: We also deﬁne the empirical loss over a set of points A as L ( θ ; A )

Original: = (cid:80) ( x,y
Revised: = (cid:80) ( x,y

Original: ) ∈
Revised: ) ∈

Original: A l ( θ ; x, y ) .
Revised: A l ( θ ; x, y ) .

Original: We adopt the game-theoretic formalization of the poisoning attack process from Steinhardt et al.
Revised: We adopt the game-theoretic formalization of the poisoning attack process from Steinhardt et al.

Original: (2017) to describe our model-targeted attack scenario:
Revised: (2017) to describe our model-targeted attack scenario:

Original: N data points are drawn uniformly at random from the true data distribution over X × Y and form the clean training set, D c . 2.
Revised: N data points are drawn uniformly at random from the true data distribution over X × Y and form the clean training set, D c . 2.

Original: The adversary, with knowledge of D c , the model training process and the model space Θ , generates a target classiﬁer θ p ∈ Θ that satisﬁes the attack goal.
Revised: The adversary, with knowledge of D c , the model training process and the model space Θ , generates a target classiﬁer θ p ∈ Θ that satisﬁes the attack goal.

Original: The adversary produces a set of poisoning points, D p , with the knowledge of D c , model training process, Θ and θ p . 4.
Revised: The adversary produces a set of poisoning points, D p , with the knowledge of D c , model training process, Θ and θ p . 4.

Original: Model builder trains the model on D c ∪ D p and produces a classiﬁer, θ atk .
Revised: Model builder trains the model on D c ∪ D p and produces a classiﬁer, θ atk .

Original: The adversary’s goal is that the induced classiﬁer, θ atk , is close to the desired target classiﬁer, θ p (Section 4.2 discusses how this distance is measured).
Revised: The adversary’s goal is that the induced classiﬁer, θ atk , is close to the desired target classiﬁer, θ p (Section 4.2 discusses how this distance is measured).

Original: Step 2 corresponds to the target classiﬁer generation process.
Revised: Step 2 corresponds to the target classiﬁer generation process.

Original: Our attack works for any target classiﬁer, and in the paper we do not focus on the question of how to ﬁnd the best target classiﬁer to achieve a particular adversarial goal but simply adopt the heuristic target classiﬁer generation process from Koh et al.
Revised: Our attack works for any target classiﬁer, and in the paper we do not focus on the question of how to ﬁnd the best target classiﬁer to achieve a particular adversarial goal but simply adopt the heuristic target classiﬁer generation process from Koh et al.

Original: Step 3 corresponds to our model-targeted poisoning attack, and is also the main contribution of the paper.
Revised: Step 3 corresponds to our model-targeted poisoning attack, and is also the main contribution of the paper.

Original: We assume the model builder trains a model through empirical risk minimization (ERM) and the training process details are known to the attacker:
Revised: We assume the model builder trains a model through empirical risk minimization (ERM) and the training process details are known to the attacker:

Original: [Equation]
Revised: [Equation]

Original: where R ( θ ) is the regularization function (e.g., 12 (cid:107) θ
Revised: where R ( θ ) is the regularization function (e.g., 12 (cid:107) θ

Original: [Equation]
Revised: [Equation]

Original: 22 for SVM model).
Revised: 22 for SVM model).

Original: Threat Model.
Revised: Threat Model.

Original: We assume an adversary with full knowledge of training data, model space and training process.
Revised: We assume an adversary with full knowledge of training data, model space and training process.

Original: Although this may be unrealistic for many scenarios, this setting allows us to focus on a particular aspect of poisoning attacks, and is the setting use in many prior works (Biggio et al., 2011;
Revised: Although this may be unrealistic for many scenarios, this setting allows us to focus on a particular aspect of poisoning attacks, and is the setting use in many prior works (Biggio et al., 2011;

Original: Mei & Zhu, 2015b;
Revised: Mei & Zhu, 2015b;

Original: Steinhardt et al., 2017; Koh et al., 2018; Shafahi et al., 2018).
Revised: Steinhardt et al., 2017; Koh et al., 2018; Shafahi et al., 2018).

Original: We assume an addition-only attack where the attacker only adds poisoning points into the clean training set.
Revised: We assume an addition-only attack where the attacker only adds poisoning points into the clean training set.

Original: A stronger attacker may be able to modify or remove existing points, but this typically requires administrative access to the system.
Revised: A stronger attacker may be able to modify or remove existing points, but this typically requires administrative access to the system.

Original: The added points are unconstrained, other than being value elements of the input space.
Revised: The added points are unconstrained, other than being value elements of the input space.

Original: They can have arbitrary features and labels, which enables us to perform the worst case analysis on the robustness of models against addition-only poisoning attacks.
Revised: They can have arbitrary features and labels, which enables us to perform the worst case analysis on the robustness of models against addition-only poisoning attacks.

Original: Although some previous works also allow arbitrary selectiong of the poisoning points (Biggio et al., 2011;
Revised: Although some previous works also allow arbitrary selectiong of the poisoning points (Biggio et al., 2011;

Original: Mei & Zhu, 2015b;
Revised: Mei & Zhu, 2015b;

Original: Steinhardt et al., 2017; Koh et al., 2018), others put different restrictions on the poisoning appoints.
Revised: Steinhardt et al., 2017; Koh et al., 2018), others put different restrictions on the poisoning appoints.

Original: A clean-label attack assumes adversaries can only perturb the features of the data, but the label is given by an oracle labeler (Koh & Liang, 2017;
Revised: A clean-label attack assumes adversaries can only perturb the features of the data, but the label is given by an oracle labeler (Koh & Liang, 2017;

Original: Shafahi et al., 2018; Zhu et al., 2019; Huang et al., 2020).
Revised: Shafahi et al., 2018; Zhu et al., 2019; Huang et al., 2020).

Original: In label-ﬂipping attacks, adversaries are only allowed to change the labels (Biggio et al., 2011; Xiao et al., 2012; 2015; Jagielski et al., 2019).
Revised: In label-ﬂipping attacks, adversaries are only allowed to change the labels (Biggio et al., 2011; Xiao et al., 2012; 2015; Jagielski et al., 2019).

Original: These restricted attacks are weaker than the poisoning attacks without restrictions (Koh et al., 2018;
Revised: These restricted attacks are weaker than the poisoning attacks without restrictions (Koh et al., 2018;

Original: Hong et al., 2020).
Revised: Hong et al., 2020).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The most commonly used poisoning strategy is gradient-based attack.
Revised: The most commonly used poisoning strategy is gradient-based attack.

Original: Gradient-based attacks iteratively modify a candidate poisoning point (ˆ x, ˆ y ) in the set D p based on the test loss deﬁned on ˆ x (keeping ˆ y ﬁxed).
Revised: Gradient-based attacks iteratively modify a candidate poisoning point (ˆ x, ˆ y ) in the set D p based on the test loss deﬁned on ˆ x (keeping ˆ y ﬁxed).

Original: This kind of attack was ﬁrst studied on SVM models (Biggio et al., 2012), and later extended to linear and logistic regression (Mei & Zhu, 2015b), and recently to larger neural network models (Koh & Liang, 2017; Yang et al., 2017; Muñoz-González et al., 2017; Shafahi et al., 2018; 
Revised: This kind of attack was ﬁrst studied on SVM models (Biggio et al., 2012), and later extended to linear and logistic regression (Mei & Zhu, 2015b), and recently to larger neural network models (Koh & Liang, 2017; Yang et al., 2017; Muñoz-González et al., 2017; Shafahi et al., 2018; Zhu et al., 2019; Huang et al., 2020). Jagielski et al.

Original: Zhu et al., 2019; Huang et al., 2020).
Revised: (2018) also studied gradient attacks and principled defenses on linear regression tasks. Their work studies linear regression while in this paper, we mainly focus on binary classiﬁcation, although our attack can also be extended to regression tasks. More importantly, our attack aims to induce a target model by generating poisoning points while Jagielski et al. (2018)’s attack tries to increase the Mean Squared Error of the linear regression task with a ﬁxed poisoning budget.

Original: In addition to classiﬁcation  tasks, gradient-based poisoning attacks are also applied to topic modeling (Mei & Zhu, 2015a), collaborative ﬁltering (Li et al., 2016) and algorithmic fairness (Solans et al., 2020).
Revised: In addition to classiﬁcation and regression tasks, gradient-based poisoning attacks are also applied to topic modeling (Mei & Zhu, 2015a), collaborative ﬁltering (Li et al., 2016) and algorithmic fairness (Solans et al., 2020).

Original: Besides the gradient-based attacks, researchers also utilize generative adversarial networks to craft poisoning points efﬁciently for larger neural networks, however, the effectiveness of the attack is limited (Yang et al., 2017; Muñoz-González et al., 2019).
Revised: Besides the gradient-based attacks, researchers also utilize generative adversarial networks to craft poisoning points efﬁciently for larger neural networks, however, the effectiveness of the attack is limited (Yang et al., 2017; Muñoz-González et al., 2019).

Original: The strongest attacks so far are the
Revised: The strongest attacks so far are the

Original: KKT attack (Koh et al., 2018) and the min-max attack (Steinhardt et al., 2017; Koh et al., 2018).
Revised: KKT attack (Koh et al., 2018) and the min-max attack (Steinhardt et al., 2017; Koh et al., 2018).

Original: However, the KKT attack cannot scale well for multi-class classiﬁcation and is limited to margin based losses (Koh et al., 2018).
Revised: However, the KKT attack cannot scale well for multi-class classiﬁcation and is limited to margin based losses (Koh et al., 2018).

Original: The min-max attack only works for indiscriminate attack setting, but additionally provides a certiﬁcate on worst case test loss for a ﬁxed number of poisoning points.
Revised: The min-max attack only works for indiscriminate attack setting, but additionally provides a certiﬁcate on worst case test loss for a ﬁxed number of poisoning points.

Original: We are also inspired by Steinhardt et al.
Revised: We are also inspired by Steinhardt et al.

Original: (2017) to adopt online convex optimization to instantiate our model-targeted attack, but now deals more general attack scenario.
Revised: (2017) to adopt online convex optimization to instantiate our model-targeted attack, but now deals more general attack scenario.

Original: We also distinguish ourselves from the poisoning attack against online learning (Wang & Chaudhuri, 2018).
Revised: We also distinguish ourselves from the poisoning attack against online learning (Wang & Chaudhuri, 2018).

Original: The attack against online learning considers a setting where training data arrives in a streaming manner while we consider the ofﬂine setting with training data being ﬁxed.
Revised: The attack against online learning considers a setting where training data arrives in a streaming manner while we consider the ofﬂine setting with training data being ﬁxed.

Original: Another line of work studies “targeted” poisoning attacks where an adversary gaurantees to increase the probability of an arbitrary “bad” property (Mahloujifar et al., 2019a; 2017; 2019b), as long as that property has some non-negligible chance of naturally happening.
Revised: Another line of work studies “targeted” poisoning attacks where an adversary gaurantees to increase the probability of an arbitrary “bad” property (Mahloujifar et al., 2019a; 2017; 2019b), as long as that property has some non-negligible chance of naturally happening.

Original: These attacks cannot be applied in the model-targeted setting as the probability of naturally producing a speciﬁc target model is often negligible.
Revised: These attacks cannot be applied in the model-targeted setting as the probability of naturally producing a speciﬁc target model is often negligible.

Original: Related to our
Revised: Related to our

Original: Theorem 4.2, Ma et al.
Revised: Theorem 4.2, Ma et al.

Original: (2019) also derived a lower bound on number of poisoning points (to induce a target model), but their lower bound only applies when differential privacy is deployed during the model training process (and hence hurts model utility), which is different from our problem setting.
Revised: (2019) also derived a lower bound on number of poisoning points (to induce a target model), but their lower bound only applies when differential privacy is deployed during the model training process (and hence hurts model utility), which is different from our problem setting.

Original:  P OISONING A TTACK WITH A T ARGET M ODEL
Revised:  P OISONING A TTACK WITH A T ARGET M ODEL

Original: Our new poisoning attack determines a target model and selects poisoning points to achieve that target model.
Revised: Our new poisoning attack determines a target model and selects poisoning points to achieve that target model.

Original: The target model generation is not our focus and we adopt the heuristic approach proposed by Koh et al.
Revised: The target model generation is not our focus and we adopt the heuristic approach proposed by Koh et al.

Original: For the new poisoning attack, ﬁrst, we show the algorithm that generates the poisoning points in Section 4.1 and then prove that the generated poisoning points, once added to the clean data, can produce a classiﬁer that asymptotically converges to the target classiﬁer in Section 4.2.
Revised: For the new poisoning attack, ﬁrst, we show the algorithm that generates the poisoning points in Section 4.1 and then prove that the generated poisoning points, once added to the clean data, can produce a classiﬁer that asymptotically converges to the target classiﬁer in Section 4.2.

Original: 4. M ODEL -T ARGETED P OISONING WITH O NLINE L EARNING
Revised: M ODEL -T ARGETED P OISONING WITH O NLINE L EARNING

Original: The main idea of our model-targeted poisoning attack, as outlined in Algorithm 1, is to sequentially add a point into the training set that have maximum loss difference between the intermediate model obtained so far and the target model, and by training models on the updated training set, we actually minimize the gap in the loss of the intermediate classiﬁer and the target classiﬁer.
Revised: The main idea of our model-targeted poisoning attack, as outlined in Algorithm 1, is to sequentially add a point into the training set that have maximum loss difference between the intermediate model obtained so far and the target model, and by training models on the updated training set, we actually minimize the gap in the loss of the intermediate classiﬁer and the target classiﬁer.

Original: Repeating the process then eventually generates classiﬁers that have similar loss distribution as the target classiﬁer.
Revised: Repeating the process then eventually generates classiﬁers that have similar loss distribution as the target classiﬁer.

Original: We show in Section 4.2 why similar loss distribution implies convergence.
Revised: We show in Section 4.2 why similar loss distribution implies convergence.

Original: Input: D c , the loss functions ( L and l ), θ p
Revised: Input: D c , the loss functions ( L and l ), θ p

Original: Output: D p 1: D p = ∅2: while stop criteria not met do 3: θ t =
Revised: Output: D p 1: D p = ∅ do 3: θ t =

Original: arg min L ( θ ; D c ∪ D p ) 4: ( x ∗ , y ∗ ) =
Revised: arg min L ( θ ; D c ∪ D p ) 4: ( x ∗ , y ∗ ) =

Original: arg max
Revised: arg max

Original: X×Y l ( θ t ; x, y ) − l
Revised: X×Y l ( θ t ; x, y ) − l

Original: ( θ p ; x, y )5:
Revised: ( θ p ; x, y )

Original: D p =D p ∪ { ( x ∗ , y ∗ ) }6: end while 7: return D p
Revised: D p = p ∪ { ( x ∗ , y ∗ ) }

Original: Algorithm 1 requires the input of clean training set D c , the Loss function ( L for set of points and l for individual point) and the target model θ p .
Revised: Algorithm 1 requires the input of clean training set D c , the Loss function ( L for set of points and l for individual point) and the target model θ p .

Original: The output from Algorithm 1 will be the set of poisoning points D p .
Revised: The output from Algorithm 1 will be the set of poisoning points D p .

Original: The algorithm is simple: ﬁrst, adversaries train the intermediate model θ t on the mixture of clean and poisoning points D c ∪ D p with D p an empty set in ﬁrst iteration (Line 3).
Revised: The algorithm is simple: ﬁrst, adversaries train the intermediate model θ t on the mixture of clean and poisoning points D c ∪ D p with D p an empty set in ﬁrst iteration (Line 3).

Original: The adversary then searches for the point that maximizes the loss difference between θ t and θ p (Line 4).
Revised: The adversary then searches for the point that maximizes the loss difference between θ t and θ p (Line 4).

Original: After the point of maximum loss difference is found, it is added to the poisoning set D p
Revised: After the point of maximum loss difference is found, it is added to the poisoning set D p

Original: (Line 5).
Revised: (Line 5).

Original: The whole process repeats until the stop condition is satisﬁed in Line 2.
Revised: The whole process repeats until the stop condition is satisﬁed in Line 2.

Original: The stop condition is ﬂexible and it can take various forms: 1) adversary has a budget T on number of poisoning points, and the algorithm halts when the algorithm runs for T iterations; 2) the intermediate classiﬁer θ t is closer to the target classiﬁer (than a preset threshold (cid:15) ) in terms of the maximum loss difference, and more details regarding this distance metric will be introduced in Section 4.2; 3) adversary has some requirement on the accuracy and the algorithm terminates when θ t satisﬁes the accuracy requirement.
Revised: The stop condition is ﬂexible and it can take various forms: 1) adversary has a budget T on number of poisoning points, and the algorithm halts when the algorithm runs for T iterations; 2) the intermediate classiﬁer θ t is closer to the target classiﬁer (than a preset threshold (cid:15) ) in terms of the maximum loss difference, and more details regarding this distance metric will be introduced in Section 4.2; 3) adversary has some requirement on the accuracy and the algorithm terminates when θ t satisﬁes the accuracy requirement.

Original: Since we focus on producing a classiﬁer close to the target model, we adopt the second stop criterion that measures the distance with respect to the maximum loss difference, and report results based on this criterion in Section 5.
Revised: Since we focus on producing a classiﬁer close to the target model, we adopt the second stop criterion that measures the distance with respect to the maximum loss difference, and report results based on this criterion in Section 5.

Original: A nice property of Algorithm 1 is that the classiﬁer θ atk trained on D c ∪D p is close to the target model θ p and asymptotically converges to θ p .
Revised: A nice property of Algorithm 1 is that the classiﬁer θ atk trained on D c ∪D p is close to the target model θ p and asymptotically converges to θ p .

Original: Details of the convergence will be shown in the next section.
Revised: Details of the convergence will be shown in the next section.

Original: The algorithm may appear to be slow, particularly for larger models due to requirement of repeatedly training a model in line 3.
Revised: The algorithm may appear to be slow, particularly for larger models due to requirement of repeatedly training a model in line 3.

Original: However, this is not an issue.
Revised: However, this is not an issue.

Original: First, as will be shown in next section, the algorithm is an online optimization process and line 3 corresponds to solving the online optimization problem exactly.
Revised: First, as will be shown in next section, the algorithm is an online optimization process and line 3 corresponds to solving the online optimization problem exactly.

Original: However, people often use the very efﬁcient online gradient descent method to approximately solve the problem and its asymptotic performance is the same (Shalev-Shwartz, 2012).
Revised: However, people often use the very efﬁcient online gradient descent method to approximately solve the problem and its asymptotic performance is the same (Shalev-Shwartz, 2012).

Original: Second, if we solve the optimization problem exactly, we can add multiple copies of ( x ∗ , y ∗ ) into
Revised: Second, if we solve the optimization problem exactly, we can add multiple copies of ( x ∗ , y ∗ ) into

Original: D p each time.
Revised: D p each time.

Original: This reduces the overall iteration number, and hence reduces the number of times retraining models.
Revised: This reduces the overall iteration number, and hence reduces the number of times retraining models.

Original: The proof of convergence will be similar.
Revised: The proof of convergence will be similar.

Original: For simplicity in interpreting the results, we do not use this in our experiments and add only one copy of ( x ∗ , y ∗ ) each iteration.
Revised: For simplicity in interpreting the results, we do not use this in our experiments and add only one copy of ( x ∗ , y ∗ ) each iteration.

Original: However, we also tested the performance by adding two copies of ( x ∗ , y ∗ ) and ﬁnd that the attack results are nearly the same while the efﬁciency is improved signiﬁcantly.
Revised: However, we also tested the performance by adding two copies of ( x ∗ , y ∗ ) and ﬁnd that the attack results are nearly the same while the efﬁciency is improved signiﬁcantly.

Original: For example, for the experiments we tried on
Revised: For example, for the experiments we tried on

Original: MNIST-17 dataset, by adding 2 copies of points, with same number of poisoning points, the attack success rate decreases at most by 0.7% while the execution time is reduced approximately by half.
Revised: MNIST-17 dataset, by adding 2 copies of points, with same number of poisoning points, the attack success rate decreases at most by 0.7% while the execution time is reduced approximately by half.

Original: 4. C ONVERGENCE OF O UR P OISONING A TTACK
Revised: 4. C ONVERGENCE OF O UR P OISONING A TTACK

Original: Before proving the convergence of Algorithm 1, we need to measure the distance of the model θ atk trained on D c ∪ D p to the target model θ p .
Revised: Before proving the convergence of Algorithm 1, we need to measure the distance of the model θ atk trained on D c ∪ D p to the target model θ p .

Original: First, we deﬁne a general closeness measure based on their prediction performance which we will use to state our convergence theorem:
Revised: First, we deﬁne a general closeness measure based on their prediction performance which we will use to state our convergence theorem:

Original: Deﬁnition 1 (Loss-based distance) .
Revised: Deﬁnition 1 (Loss-based distance and (cid:15) -close) .

Original: For two models θ 1 and θ 2 , a space X × Y and a loss function l ( θ ; x, y ) , we deﬁne loss-based distance D l, X , Y :
Revised: For two models θ 1 and θ 2 , a space X × Y and a loss function l ( θ ; x, y ) , we deﬁne loss-based distance D l, X , Y :

Original: Θ × Θ → R as
Revised: Θ × Θ → R as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: and we say model θ 1 is (cid:15) -close to model θ 2 when the loss-based distance from θ 1 to θ 2 is upper bounded by (cid:15) .

Original: Why is loss-based distance a meaningful notion of closeness?
Revised: Why is loss-based distance a meaningful notion of closeness?

Original: We argue that this notion captures the “behavorial” distance between two models.
Revised: We argue that this notion captures the “behavorial” distance between two models.

Original: Namely, if θ 1 is (cid:15) -close (as measured by loss-based distance) to θ 2 and vice versa, then θ 1 and θ 2 would have almost equal loss on all the points, meaning that they have almost the same behavior across all the space.
Revised: Namely, if θ 1 is (cid:15) -close (as measured by loss-based distance) to θ 2 and vice versa, then θ 1 and θ 2 would have almost equal loss on all the points, meaning that they have almost the same behavior across all the space.

Original: Note that our deﬁnition of distance does not have the symmetry property of metrics and hence is not a metric.
Revised: Note that our deﬁnition of distance does not have the symmetry property of metrics and hence is not a metric.

Original: However, it has some other properties of metrics in the space of attainable models.
Revised: However, it has some other properties of metrics in the space of attainable models.

Original: For example, if some model θ is attainable using ERM, no model could have negative distance to it.
Revised: For example, if some model θ is attainable using ERM, no model could have negative distance to it.

Original: To further show the value of this distance notion, in Appendix B we demonstrate an O ( (cid:15) ) upper bound on the (cid:96) 1 norm of difference between two models that are (cid:15) -close with respect to loss-based distance for the special case of Hinge loss.
Revised: To further show the value of this distance notion, in Appendix B we demonstrate an O ( (cid:15) ) upper bound on the (cid:96) 1 norm of difference between two models that are (cid:15) -close with respect to loss-based distance for the special case of Hinge loss.

Revised: In the rest of the paper, we will use terms (cid:15) -close or (cid:15) -closeness to denote that a model is (cid:15) away from another model based on the loss-based distance.

Original: Our convergence theorem uses the loss-based distance to establish that the attack of Algorithmconverges to the target classiﬁer:
Revised: Our convergence theorem uses the loss-based distance to establish that the attack of Algorithmconverges to the target classiﬁer:

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: After at most T steps, Algorithm 1 will produce the poisoning set D p and the classiﬁer trained on D c ∪ D p is (cid:15) -close to θ p , with respect to loss-based distance, D l, X , Y , for
Revised: After at most T steps, Algorithm 1 will produce the poisoning set D p and the classiﬁer trained on D c ∪ D p is (cid:15) -close to θ p , with respect to loss-based distance, D l, X , Y , for

Original: [Equation]
Revised: [Equation]

Original: where, γ is a constant for a given θ p and classiﬁcation task, and α ( T ) is the regret of the online algorithm when the loss function used for training is convex.
Revised: where, γ is a constant for a given θ p and classiﬁcation task, and α ( T ) is the regret of the online algorithm when the loss function used for training is convex.

Original: Remark 1.
Revised: Remark 1.

Original: Online learning algorithms with sublinear regret bound can be applied to show the convergence.
Revised: Online learning algorithms with sublinear regret bound can be applied to show the convergence.

Original: Here, we adopt results from McMahan (2017).
Revised: Here, we adopt results from McMahan (2017).

Original: Speciﬁcally, α ( T ) is in the order of
Revised: Speciﬁcally, α ( T ) is in the order of

Original: O (log T ) ) and we have (cid:15) ≤
Revised: O (log T ) ) and we have (cid:15) ≤

Original: O ( log T T ) when the loss function is additionally Lipschitz continuous and the regularizer R ( θ ) is strongly convex, and (cid:15) → 0 when T → + ∞ .
Revised: O ( log TT ) when the loss function is additionally Lipschitz continuous and the regularizer R ( θ ) is strongly convex, and (cid:15) → 0 when T → + ∞ .

Original: α ( T ) is also in the order of
Revised: α ( T ) is also in the order of

Original: O (log T ) when the loss function used for training is strongly convex and the regularizer is convex.
Revised: O (log T ) when the loss function used for training is strongly convex and the regularizer is convex.

Original: Proof idea.
Revised: Proof idea.

Original: The full proof of Theorem 4.1 is in Appendix A.
Revised: The full proof of Theorem 4.1 is in Appendix A.

Original: Here, we only summarize the high level proof idea.
Revised: Here, we only summarize the high level proof idea.

Original: The key idea is to frame the poisoning problem as an online learning problem.
Revised: The key idea is to frame the poisoning problem as an online learning problem.

Original: In this formulation, each step of the online learning problem corresponds to the i th poison point ( x i , y i ) .
Revised: In this formulation, each step of the online learning problem corresponds to the i th poison point ( x i , y i ) .

Original: In particular, the loss function at iteration i of the online learning problem is set to l ( · ; x i , y i ) .
Revised: In particular, the loss function at iteration i of the online learning problem is set to l ( · ; x i , y i ) .

Original: Then, we show that by deﬁning the parameters of the online learning problem in a careful way, the output of the follow-the-leader (FTL) algorithm (Shalev-Shwartz, 2012) at iteration i is a model that is identical to training a model on a dataset consisting of the clean points and the ﬁrst i − 1 poisoning points.
Revised: Then, we show that by deﬁning the parameters of the online learning problem in a careful way, the output of the follow-the-leader (FTL) algorithm (Shalev-Shwartz, 2012) at iteration i is a model that is identical to training a model on a dataset consisting of the clean points and the ﬁrst i − 1 poisoning points.

Original: On the other hand, the way the poisoning points are selected, we can show that at the i th iteration the maximum loss difference between the target model and the best induced model so far would be smaller than the regret of the FTL algorithm divided by the number of poisoning points.
Revised: On the other hand, the way the poisoning points are selected, we can show that at the i th iteration the maximum loss difference between the target model and the best induced model so far would be smaller than the regret of the FTL algorithm divided by the number of poisoning points.

Original: The convergence bound of Theorem 4.1 boils down to regret analysis of the algorithm based on the loss function.
Revised: The convergence bound of Theorem 4.1 boils down to regret analysis of the algorithm based on the loss function.

Original: Since we are assuming the loss function is convex with a strongly convex regularizer (or a strongly convex loss function with a convex regularizer), we can show that the regret is bounded by
Revised: Since we are assuming the loss function is convex with a strongly convex regularizer (or a strongly convex loss function with a convex regularizer), we can show that the regret is bounded by

Original: O (log T ) and hence the loss distance between the induced model and the target model converges to 0.
Revised: O (log T ) and hence the loss distance between the induced model and the target model converges to 0.

Original: Implications of Theorem 4.1 The theorem says that the loss-based distance of the model trained on D c ∪ D p to the target model correlates to the loss difference between the target model and the clean model θ c (trained on D c ) on D c , and correlates inversely with the number of poisoning points.
Revised: Implications of Theorem 4.1 The theorem says that the loss-based distance of the model trained on D c ∪ D p to the target model correlates to the loss difference between the target model and the clean model θ c (trained on D c ) on D c , and correlates inversely with the number of poisoning points.

Original: Therefore, it implies 1) if the target classiﬁer θ p has lower loss on D c , then it is easier to achieve the target model, and 2) with more poisoning points, we get closer to the target classiﬁer and our attack will be more effective.
Revised: Therefore, it implies 1) if the target classiﬁer θ p has lower loss on D c , then it is easier to achieve the target model, and 2) with more poisoning points, we get closer to the target classiﬁer and our attack will be more effective.

Original: The theorem also justiﬁes the motivation behind the heuristic method in Koh et al.
Revised: The theorem also justiﬁes the motivation behind the heuristic method in Koh et al.

Original: (2018) to select a target classiﬁer with lower loss on clean data.
Revised: (2018) to select a target classiﬁer with lower loss on clean data.

Original: For the indiscriminate attack scenario, we also improve the heuristic approach by adaptively updating the model and producing target classiﬁers with much lower loss on the clean set.
Revised: For the indiscriminate attack scenario, we also improve the heuristic approach by adaptively updating the model and producing target classiﬁers with much lower loss on the clean set.

Original: This helps to empirically validate our theorem.
Revised: This helps to empirically validate our theorem.

Original: Details of the original and improved heuristic approach, and relevant experiments are in
Revised: Details of the original and improved heuristic approach, and relevant experiments are in

Original: Appendix D.1.
Revised: Appendix D.1.

Original: 4. L OWER B OUND ON THE N UMBER OF POISONING P OINTS
Revised: 4. L OWER B OUND ON THE N UMBER OF POISONING P OINTS

Original: We ﬁrst provide the lower bound on number of poisoning points required for producing the target classiﬁer in addition only setting (Theorem 4.2), and then explain how the lower bound estimation can be incorporated into Algorithm 1.
Revised: We ﬁrst provide the lower bound on number of poisoning points required for producing the target classiﬁer in addition only setting (Theorem 4.2), and then explain how the lower bound estimation can be incorporated into Algorithm 1.

Original: The intuition behind the theorem below is, when the number of poisoning points added to the clean training set is smaller than the lower bound, there always exists a classiﬁer θ with lower loss compared to θ p and hence the target classiﬁer cannot be attained.
Revised: The intuition behind the theorem below is, when the number of poisoning points added to the clean training set is smaller than the lower bound, there always exists a classiﬁer θ with lower loss compared to θ p and hence the target classiﬁer cannot be attained.

Original: Theorem 4.2 (Lower Bound) .
Revised: Theorem 4.2 (Lower Bound) .

Original: Given a target classiﬁer θ p , to reproduce θ p by adding the poisoning set D p into D c , the number of poisoning points |D p | cannot be lower than
Revised: Given a target classiﬁer θ p , to reproduce θ p by adding the poisoning set D p into D c , the number of poisoning points |D p | cannot be lower than

Original: [Equation]
Revised: [Equation]

Original: The formula for the lower bound can be easily incorporated into Algorithm 1 to obtain tighter theoretical lower bound.
Revised: The formula for the lower bound can be easily incorporated into Algorithm 1 to obtain tighter theoretical lower bound.

Original: We simply need to check all of the intermediate classiﬁer θ t produced during the attack process and replace θ with θ t , and the lower bound can be computed for the pair of θ t and θ p .
Revised: We simply need to check all of the intermediate classiﬁer θ t produced during the attack process and replace θ with θ t , and the lower bound can be computed for the pair of θ t and θ p .

Original: Algorithm 1 then additionally returns the lower bound, which is the highest lower bound computed from our poisoning procedure.
Revised: Algorithm 1 then additionally returns the lower bound, which is the highest lower bound computed from our poisoning procedure.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We ﬁrst describe our experimental setup regarding the datasets, models, attacks and target classiﬁers.
Revised: We ﬁrst describe our experimental setup regarding the datasets, models, attacks and target classiﬁers.

Original: Next, we present the experimental results by showing the convergence of Algorithm 1, the comparison of attack success rates to state-of-the-art poisoning attack, and the theoretical lower bound for inducing a given target classiﬁer and its gap to the number of poisoning points used by our attack.
Revised: Next, we present the experimental results by showing the convergence of Algorithm 1, the comparison of attack success rates to state-of-the-art poisoning attack, and the theoretical lower bound for inducing a given target classiﬁer and its gap to the number of poisoning points used by our attack.

Original: We are most interested in subpopulation attacks, since they correspond to the more realistic attacker goal of impacting the classiﬁer outputs for a targeted subpopulation.
Revised: We are most interested in subpopulation attacks, since they correspond to the more realistic attacker goal of impacting the classiﬁer outputs for a targeted subpopulation.

Original: For completeness, we also evaluate our attack in the indiscriminate poisoning scenario but defer details on those experiments to
Revised: For completeness, we also evaluate our attack in the indiscriminate poisoning scenario but defer details on those experiments to

Original: Appendix D. Our ﬁndings for the indiscriminate attacks are that the attack gradually and consistently converges to the target model in terms of the maximum loss difference and the Euclidean distance to the target, with attack success rates that are comparable to the state-of-the-art attack (unlike the subpopulation attacks, where our attack produces superior results).
Revised: Appendix D. Our ﬁndings for the indiscriminate attacks are that the attack gradually and consistently converges to the target model in terms of the maximum loss difference and the Euclidean distance to the target, with attack success rates that are comparable to the state-of-the-art attack (unlike the subpopulation attacks, where our attack produces superior results).

Original: Dataset, Model and Attacks.
Revised: Dataset, Model and Attacks.

Original: For the subpopulation attack experiments, we use the Adult dataset (Dua & Graff, 2017).
Revised: For the subpopulation attack experiments, we use the Adult dataset (Dua & Graff, 2017).

Original: This dataset was used for evaluation by the ﬁrst subpopulation attack  paper (Jagielski et al., 2019).
Revised: This dataset was used for evaluation by the ﬁrst subpopulation attack [Figure] paper (Jagielski et al., 2019).

Revised: We downsampled the Adult dataset to ensure it is class-balanced and we ended up having 15,682 training and 7,692 test examples.

Original: We conduct experiments on linear SVM model and compare our model-targeted poisoning attack in Algorithm 1 to the state-of-the-art KKT attack (Koh et al., 2018).
Revised: We conduct experiments on linear SVM model and compare our model-targeted poisoning attack in Algorithm 1 to the state-of-the-art KKT attack (Koh et al., 2018).

Original: We do not include the model-targeted attack from Mei & Zhu (2015b) because it underperforms the KKT attack (Koh et al., 2018).
Revised: We do not include the model-targeted attack from Mei & Zhu (2015b) because it underperforms the KKT attack (Koh et al., 2018).

Original: We also do not include objective-driven attacks because their attack goals are different from ours.
Revised: We also do not include objective-driven attacks because our main goal here is to evaluate how well our attack approaches a given target model, across a range of target models.

Original: However, model-targeted attacks can be compared to objective-driven attacks  by choosing the target model in a careful way and we show some heuristics of choosing such“better” target model in Appendix E.
Revised: Model-targeted attacks can be compared to objective-driven attacks with regards to a given attacker objective by choosing the target model in a careful way. We show some heuristics of choosing such target models in Appendix E.

Original: Both our attack and the KKT attack take as input a target classiﬁer and the original training data, and output a set of poisoning points selected with the goal that the induced classiﬁer is as close as[Figure] possible to the target classiﬁer.
Revised: Both our attack and the KKT attack take as input a target classiﬁer and the original training data, and output a set of poisoning points selected with the goal that the induced classiﬁer is as close as possible to the target classiﬁer.

Original: We compare the effectiveness of the attacks in selecting poisoning points that converge to a given target classiﬁer by testing the attacks using the same target model.
Revised: We compare the effectiveness of the attacks in selecting poisoning points that converge to a given target classiﬁer by testing the attacks using the same target model.

Original: The KKT attack requires a target number of poisoning points as an input while our attack is more ﬂexible and can either take a target number of poisoning points or a threshold for (cid:15) -close distance to the target model.
Revised: The KKT attack requires a target number of poisoning points as an input while our attack is more ﬂexible and can either take a target number of poisoning points or a threshold for (cid:15) -close distance to the target model.

Original: Since we do not know the number of poisoning points needed to reach some attacker goal in advance for the KKT attack, we ﬁrst run our attack and produce a classiﬁer that satisﬁes the selected (cid:15) -close distance threshold.
Revised: Since we do not know the number of poisoning points needed to reach some attacker goal in advance for the KKT attack, we ﬁrst run our attack and produce a classiﬁer that satisﬁes the selected (cid:15) -close distance threshold.

Original: The loss function is set as the hinge loss since we target an SVM model in our experiments and we set (cid:15)
Revised: The loss function is set as the hinge loss since we target an SVM model in our experiments and we set (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 01 for all these experiments.
Revised: 01 for all these experiments.

Original: Then, we use the size of the poisoning set returned from our attack (denoted by n p ) as the input to the KKT attack for the target number of poisons needed.
Revised: Then, we use the size of the poisoning set returned from our attack (denoted by n p ) as the input to the KKT attack for the target number of poisons needed.

Original: We also compare the two attacks with varying numbers of poisoning points up to n p .
Revised: We also compare the two attacks with varying numbers of poisoning points up to n p .

Original: For the KKT attack, its entire optimization process must be rerun whenever the target number of poisoning points changes.
Revised: For the KKT attack, its entire optimization process must be rerun whenever the target number of poisoning points changes.

Original: Hence, it is infeasible to evaluate the KKT attack on many different poisoning set sizes.
Revised: Hence, it is infeasible to evaluate the KKT attack on many different poisoning set sizes.

Original: In our experiments, we run the KKT attack ﬁve poisoning set sizes: 0 . 2 · n p , 0 . 4 · n p , 0 . 6 · n p , 0 . 8 · n p , and n p .
Revised: In our experiments, we run the KKT attack ﬁve poisoning set sizes: 0 . 2 · n p , 0 . 4 · n p , 0 . 6 · n p , 0 . 8 · n p , and n p .

Original: In contrast, we simply run our attack for iterations up to the maximum number of poisoning points, collecting a data point for iteration up to n p .
Revised: In contrast, we simply run our attack for iterations up to the maximum number of poisoning points, collecting a data point for iteration up to n p .

Original: Subpopulations.
Revised: Subpopulations.

Original: We identify the subpopulations for the Adult dataset using k -means clustering techniques (ClusterMatch (Jagielski et al., 2019)) to obtain different clusters ( k = 20 in our case).
Revised: We identify the subpopulations for the Adult dataset using k -means clustering techniques (ClusterMatch (Jagielski et al., 2019)) to obtain different clusters ( k = 20 in our case).

Original: For each cluster, we select instances with label “<=50K” to form the subpopulation (indicating all instances in the subpopulation are in low income group).
Revised: For each cluster, we select instances with label “<=50K” to form the subpopulation (indicating all instances in the subpopulation are in low income group).

Original: This way of deﬁning subpopulation is rather arbitrary (in constrast to a more likely attack goal which would select subpopulations based on demographic characteristics), but enables us to simplify the analysis.
Revised: This way of deﬁning subpopulation is rather arbitrary (in constrast to a more likely attack goal which would select subpopulations based on demographic characteristics), but enables us to simplify the analysis.

Original: From the 20 subpopulations obtained, we select three subpopulations with the highest test accuracy on the clean model and they all have 100% test accuracy, indicating all instances in these subpopulations are correctly classiﬁed as low income.
Revised: From the 20 subpopulations obtained, we select three subpopulations with the highest test accuracy on the clean model and they all have 100% test accuracy, indicating all instances in these subpopulations are correctly classiﬁed as low income.

Original: This enables us to use “attack success rate” and “accuracy” without any ambiguity on the subpopulation—for each of our subpopulations, all instances are originally classiﬁed as low income, and the simulated attacker’s goal is to have them classiﬁed as high income.
Revised: This enables us to use “attack success rate” and “accuracy” without any ambiguity on the subpopulation—for each of our subpopulations, all instances are originally classiﬁed as low income, and the simulated attacker’s goal is to have them classiﬁed as high income.

Original: For each subpopulation, we use the heuristic approach from Koh et al.
Revised: For each subpopulation, we use the heuristic approach from Koh et al.

Original: (2018) to generate a target classiﬁer that that has 0% accuracy (100% attacker success) on the subpopulation, indicating that all subpopulation instances are now classiﬁed as high income.
Revised: (2018) to generate a target classiﬁer that that has 0% accuracy (100% attacker success) on the subpopulation, indicating that all subpopulation [Figure] [Table]

Original: Convergence.
Revised: Convergence.

Original: Figure 1 shows the convergence of Algorithm 1 using both maximum loss difference and Euclidean distance to the target.
Revised: Figure 1 shows the convergence of Algorithm 1 using both maximum loss difference and Euclidean distance to the target.

Original: The maximum number of poisons ( n p ) for the experiments is obtained when the classiﬁer from Algorithm 1 is 0 .
Revised: The maximum number of poisons ( n p ) for the experiments is obtained when the classiﬁer from Algorithm 1 is 0 .

Original: 01 -close to the target classiﬁer.
Revised: 01 -close to the target classiﬁer.

Original: Our attack steadily reduces the maximum loss difference and Euclidean distance to the target model, in contrast to the
Revised: Our attack steadily reduces the maximum loss difference and Euclidean distance to the target model, in contrast to the

Original: KKT attack which does not seem to converge towards the target model reliably.
Revised: KKT attack which does not seem to converge towards the target model reliably.

Original: Concretely, at the maximum number of poisons in Figure 1, both the maximum loss difference and Euclidean distance of our attack (to the target) is less than 2% of the corresponding distances of the KKT attack.
Revised: Concretely, at the maximum number of poisons in Figure 1, both the maximum loss difference and Euclidean distance of our attack (to the target) is less than 2% of the corresponding distances of the KKT attack.

Original: Attack Success.
Revised: Attack Success.

Original: Next, we compare the classiﬁers induced by the two attacks in terms of the attacker’s goal of reducing the test accuracy on the subpopulation.
Revised: Next, we compare the classiﬁers induced by the two attacks in terms of the attacker’s goal of reducing the test accuracy on the subpopulation.

Original: Figure 2 shows the accuracy results for the three subpopulations.
Revised: Figure 2 shows the accuracy results for the three subpopulations.

Original: For each test, the maximum number of poisoning points is obtained by running our attack with a target of 0 .
Revised: For each test, the maximum number of poisoning points is obtained by running our attack with a target of 0 .

Original: 01 -closeness (in loss based distance).
Revised: 01 -closeness (in loss based distance).

Original: For the three subpopulations, at[Figure] [Table] the maximum number of poisons, our attack is much more successful than the KKT attack—the induced classiﬁers have 0.5% accuracy compared to 15.4% accuracy for KKT on subpopulation 1, 0.0% compared to 6.9% on subpoulation 2, and 0.3% compared to 20.1% on subpoplation 3.
Revised: For the three subpopulations, at the maximum number of poisons, our attack is much more successful than the KKT attack—the induced classiﬁers have 0.5% accuracy compared to 15.4% accuracy for KKT on subpopulation 1, 0.0% compared to 6.9% on subpoulation 2, and 0.3% compared to 20.1% on subpoplation 3.

Original: Near Optimality of Our Attack.
Revised: Near Optimality of Our Attack.

Original: In order to show the optimality of our attack, we calculate a lower bound on the number of poisoning points needed to induce the model that is induced by the poisoning points that are found by our attack.
Revised: In order to show the optimality of our attack, we calculate a lower bound on the number of poisoning points needed to induce the model that is induced by the poisoning points that are found by our attack.

Original: We calculate this lower bound on the number of poisons using
Revised: We calculate this lower bound on the number of poisons using

Original: Theorem 4.2 (details in Section 4.3).
Revised: Theorem 4.2 (details in Section 4.3).

Original: Note that Theorem 4.2 provides a valid lower bound based on any intermediate model.
Revised: Note that Theorem 4.2 provides a valid lower bound based on any intermediate model.

Original: In order to get a lower bound on the number of poisoning points, we only need to use Theorem 4.2 on the encountered intermediate models and report the best one.
Revised: In order to get a lower bound on the number of poisoning points, we only need to use Theorem 4.2 on the encountered intermediate models and report the best one.

Original: We do this by running Algorithm 1 using the induced model (and not the previous target model) as the target model, terminating when the induced classiﬁer is 0 .
Revised: We do this by running Algorithm 1 using the induced model (and not the previous target model) as the target model, terminating when the induced classiﬁer is 0 .

Original: 01 -close to the given target model.
Revised: 01 -close to the given target model.

Original: We then consider all the intermediate classiﬁers that the algorithm induced across the iterations.
Revised: We then consider all the intermediate classiﬁers that the algorithm induced across the iterations.

Original: Our calculated lower bound shows that the gap between the lower bound and the number of used poison points is relatively small.
Revised: Our calculated lower bound shows that the gap between the lower bound and the number of used poison points is relatively small.

Original: This means our attack is nearly optimal in terms of minimizing the number of poisoning points needed to induce the target classiﬁer.
Revised: This means our attack is nearly optimal in terms of minimizing the number of poisoning points needed to induce the target classiﬁer.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a general poisoning framework with provable guarantees to reach any achievable target classiﬁer, along with a lower bound on the number of poisoning points needed.
Revised: We propose a general poisoning framework with provable guarantees to reach any achievable target classiﬁer, along with a lower bound on the number of poisoning points needed.

Original: Our attack is a generic tool that ﬁrst captures the goal of adversary as a target model, and then focuses on the power of attacks to induce that model.
Revised: Our attack is a generic tool that ﬁrst captures the goal of adversary as a target model, and then focuses on the power of attacks to induce that model.

Original: This separation enables future work to explore the effectiveness of poisoning attacks corresponding to different adversarial goals.
Revised: This separation enables future work to explore the effectiveness of poisoning attacks corresponding to different adversarial goals.

Original: Our framework also applies in scenarios where adversaries ﬁrst remove points and then add new points into the training set.
Revised: Our framework also applies in scenarios where adversaries ﬁrst remove points and then add new points into the training set.

Original: We have not considered defenses in this work, and it is important to study the effectiveness of our attack against data poisoning defenses.
Revised: We have not considered defenses in this work, and it is important to study the effectiveness of our attack against data poisoning defenses.

Original: Defenses may be designed the limit the search space of the points with maximum loss difference and increasing the number of poisoning points needed.
Revised: Defenses may be designed the limit the search space of the points with maximum loss difference and increasing the number of poisoning points needed.

Original: One limitation of our framework is the requirement in the convexity of the difference of loss functions to efﬁciently search for its maximum value, which excludes other commonly used loss functions such as logistic loss.
Revised: One limitation of our framework is the requirement in the convexity of the difference of loss functions to efﬁciently search for its maximum value, which excludes other commonly used loss functions such as logistic loss.

Original: However, our approach might still be effective in these cases by using local optimization techniques to search for poisoning points with maximum loss difference.
Revised: However, our approach might still be effective in these cases by using local optimization techniques to search for poisoning points with maximum loss difference.

Original: The convergence property of our attack holds (with strongly convex regularizer) for any Lipschitz and convex loss function, and does not require the loss difference to be convex.
Revised: The convergence property of our attack holds (with strongly convex regularizer) for any Lipschitz and convex loss function, and does not require the loss difference to be convex.

Original: A P ROOFS
Revised: A P ROOFS

Original: In this section, we provide the proofs of the main theorems shown in this paper.
Revised: In this section, we provide the proofs of the main theorems shown in this paper.

Original: For convenience, we restate all the theorems below while also referencing to the main paper.
Revised: For convenience, we restate all the theorems below while also referencing to the main paper.

Original: Before proving the main theorem, we introduce two new deﬁnitions and several lemmas to assist with the proof.
Revised: Before proving the main theorem, we introduce two new deﬁnitions and several lemmas to assist with the proof.

Original: Deﬁnition 2 (Attainable models) .
Revised: Deﬁnition 2 (Attainable models) .

Original: We say θ is C R -attainable with respect to loss function l and regularization function R if there exists a training set D such that
Revised: We say θ is C R -attainable with respect to loss function l and regularization function R if there exists a training set D such that

Original: [Equation]
Revised: [Equation]

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Let θ 

Revised: Let θ

Original: and θ 2 be two C R -attainable parameters for some C R > 

Revised: and θ 2 be two C R -attainable parameters for some C R >

Original: such that R ( θ 1 ) >
Revised: such that R ( θ 1 ) >

Original: R ( θ 2 ) .
Revised: R ( θ 2 ) .

Original: Then, sup
Revised: Then, sup

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider any attainable pairs of ( θ 1 , θ 2 ) such that R ( θ 1 ) >
Revised: Consider any attainable pairs of ( θ 1 , θ 2 ) such that R ( θ 1 ) >

Original: R ( θ 2 ) and let D 1 to be training set that the training algorithm produces the unique minimizer θ 1 .
Revised: R ( θ 2 ) and let D 1 to be training set that the training algorithm produces the unique minimizer θ 1 .

Original: Namely,
Revised: Namely,

Original: [Equation]
Revised: [Equation]

Original: Since θ 1 minimizes the total loss on D 1 uniquely, we have
Revised: Since θ 1 minimizes the total loss on D 1 uniquely, we have

Original: [Equation]
Revised: [Equation]

Original: By rearranging the above inequality and by an averaging argument, we have sup x,y
Revised: By rearranging the above inequality and by an averaging argument, we have sup x,y

Original: [Equation]
Revised: [Equation]

Original: (cid:1) .
Revised: (cid:1) .

Original: Now since R ( θ 1 ) > R ( θ 2 ) we have
Revised: Now since R ( θ 1 ) > R ( θ 2 ) we have

Original: [Equation]
Revised: [Equation]

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Let F be the family of all C R -attainable models.
Revised: Let F be the family of all C R -attainable models.

Original: For any θ 1 ∈ F , there is a constant γ where for all θ 2 ∈ F we have
Revised: For any θ 1 ∈ F , there is a constant γ where for all θ 2 ∈ F we have

Original: [Equation]
Revised: [Equation]

Original: where γ is a positive constant related to, θ 1 , C R and other model parameters (ﬁxed for a given classiﬁcation task).
Revised: where γ is a positive constant related to, θ 1 , C R and other model parameters (ﬁxed for a given classiﬁcation task).

Original: Proof.
Revised: Proof.

Original: We prove the lemma for γ = 1 − C R /C for
Revised: We prove the lemma for γ = 1 − C R /C for

Original: [Equation]
Revised: [Equation]

Original: First, note that by Lemma A.1 we have
Revised: First, note that by Lemma A.1 we have

Original: [Equation]
Revised: [Equation]

Original: which implies γ is positive.
Revised: which implies γ is positive.

Original: Now we consider two subcases based on the sign of R ( θ 2 ) − R ( θ 1 ) :
Revised: Now we consider two subcases based on the sign of R ( θ 2 ) − R ( θ 1 ) :

Original: Case 1: R ( θ 2 ) − R ( θ 1 ) ≥ 0 .
Revised: Case 1: R ( θ 2 ) − R ( θ 1 ) ≥ 0 .

Original: In this case the inequality is straightforward: sup
Revised: In this case the inequality is straightforward: sup

Original: [Equation]
Revised: [Equation]

Original: ,y
Revised: ,y

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is based on equation 2.
Revised: where the last inequality is based on equation 2.

Original: Case 2: R ( θ 2 ) − R ( θ 1 ) < 0 .
Revised: Case 2: R ( θ 2 ) − R ( θ 1 ) < 0 .

Original: From the deﬁnition of C we have
Revised: From the deﬁnition of C we have

Original: [Equation]
Revised: [Equation]

Original: Equivalently, we can say
Revised: Equivalently, we can say

Original: [Equation]
Revised: [Equation]

Original: Replacing R ( θ 2 ) − R ( θ 1 ) with the lower bound above completes the proof, namely sup x,y
Revised: Replacing R ( θ 2 ) − R ( θ 1 ) with the lower bound above completes the proof, namely sup x,y

Original: [Equation]
Revised: [Equation]

Original: With Deﬁnition 2 and the lemmas, we are ready to prove Theorem 4.1 (restating Theorem 4.1, from
Revised: With Deﬁnition 2 and the lemmas, we are ready to prove Theorem 4.1 (restating Theorem 4.1, from

Original: Section 4.2):
Revised: Section 4.2):

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: After at most T steps, Algorithm 1 will produce the poisoning set D p and the classiﬁer trained on D c ∪ D p is (cid:15) -close to θ p , with respect to loss-based distance, D l, X , Y , for
Revised: After at most T steps, Algorithm 1 will produce the poisoning set D p and the classiﬁer trained on D c ∪ D p is (cid:15) -close to θ p , with respect to loss-based distance, D l, X , Y , for

Original: [Equation]
Revised: [Equation]

Original: where, γ is a constant for a given θ p and classiﬁcation task, and α ( T ) is the regret of the online algorithm when the loss function used for training is convex.
Revised: where, γ is a constant for a given θ p and classiﬁcation task, and α ( T ) is the regret of the online algorithm when the loss function used for training is convex.

Original: The goal of the adversary is to get (cid:15) -close to θ p (in terms of the loss based distance) by injecting (potentially few) number of poisoned training data.
Revised: The goal of the adversary is to get (cid:15) -close to θ p (in terms of the loss based distance) by injecting (potentially few) number of poisoned training data.

Original: The algorithm is in essence an online learning problem and we transform Algorithm 1 into the form of standard online learning problem.
Revised: The algorithm is in essence an online learning problem and we transform Algorithm 1 into the form of standard online learning problem.

Original: Speciﬁcally, we adopt the follow the leader (FTL) framework to describe Algorithm 1 in the language of standard online learning problem.
Revised: Speciﬁcally, we adopt the follow the leader (FTL) framework to describe Algorithm 1 in the language of standard online learning problem.

Original: We ﬁrst describe the online learning setting considered in this paper and the notion of the regret.
Revised: We ﬁrst describe the online learning setting considered in this paper and the notion of the regret.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Let L be a class of loss functions, Θ set of possible models, A : (Θ × L ) ∗ → Θ an online learner and S : (Θ × L ) ∗ ×
Revised: Let L be a class of loss functions, Θ set of possible models, A : (Θ × L ) ∗ → Θ an online learner and S : (Θ × L ) ∗ ×

Original: Θ
Revised: Θ

Original: → L a strategy for picking loss functions in different rounds of online learning (adversarial environment in the context of online convex optimization).
Revised: → L a strategy for picking loss functions in different rounds of online learning (adversarial environment in the context of online convex optimization).

Original: We use
Revised: We use

Original: Regret ( A, S, T ) to denote the regret of A against S , in T rounds.
Revised: Regret ( A, S, T ) to denote the regret of A against S , in T rounds.

Original: Namely,
Revised: Namely,

Original: Regret ( A, S, T ) =
Revised: Regret ( A, S, T ) =

Original: [Equation]
Revised: [Equation]

Original: where θ i
Revised: where θ i

Original: =
Revised: =

Original: A (cid:0) ( θ 0 , l 0 ) , . . .
Revised: A (cid:0) ( θ 0 , l 0 ) , . . .

Original: , ( θ i − 1 , l i − 1 ) (cid:1) and l i = S (cid:0) ( θ 0 , l 0 ) , . . .
Revised: , ( θ i − 1 , l i − 1 ) (cid:1) and l i = S (cid:0) ( θ 0 , l 0 ) , . . .

Original: , ( θ i − 1 , l i − 1 ) , θ i (cid:1) .
Revised: , ( θ i − 1 , l i − 1 ) , θ i (cid:1) .

Original: With the online learning problem set up, we proceed to the main proof which ﬁrst describes Algorithm 1 in the FTL framework.
Revised: With the online learning problem set up, we proceed to the main proof which ﬁrst describes Algorithm 1 in the FTL framework.

Original: Proof of Theorem 4.1.
Revised: Proof of Theorem 4.1.

Original: The FTL framework proceeds by solving all the functions incurred during the previous online optimization steps, namely, A FTL (( θ 0 , l 0 ) , . .
Revised: The FTL framework proceeds by solving all the functions incurred during the previous online optimization steps, namely, A FTL (( θ 0 , l 0 ) , . .

Original: , ( θ i , l i )) =
Revised: , ( θ i , l i )) =

Original: arg min θ ∈ Θ (cid:80) ij =

Revised: arg min θ ∈ Θ (cid:80) ij =

Original: l i ( θ ) .
Revised: l i ( θ ) .

Original: Next, we describe how we design the i th loss function l i in each round of the online optimization.
Revised: Next, we describe how we design the i th loss function l i in each round of the online optimization.

Original: For the ﬁrst choice, A FTL chooses a random model θ 0 ∈ Θ .
Revised: For the ﬁrst choice, A FTL chooses a random model θ 0 ∈ Θ .

Original: In the ﬁrst round (round 0), S θ p uses the clean training set D c and the loss is set as
Revised: In the ﬁrst round (round 0), S θ p uses the clean training set D c and the loss is set as

Original: [Equation]
Revised: [Equation]

Original: According to the FTL framework, A FTL returns model that minimizes the loss on the clean training set D c using the structural empirical risk minimization.
Revised: According to the FTL framework, A FTL returns model that minimizes the loss on the clean training set D c using the structural empirical risk minimization.

Original: For the subsequent iterations ( i ≥ 1 ), the loss functions is deﬁned as, given the latest model θ i , S θ p ﬁrst ﬁnds ( x ∗ i , y ∗ i ) that maximizes the loss difference between θ i and a target model θ p .
Revised: For the subsequent iterations ( i ≥ 1 ), the loss functions is deﬁned as, given the latest model θ i , S θ p ﬁrst ﬁnds ( x ∗ i , y ∗ i ) that maximizes the loss difference between θ i and a target model θ p .

Original: Namely,
Revised: Namely,

Original: [Equation]
Revised: [Equation]

Original: and then chooses the i th loss function as follows:
Revised: and then chooses the i th loss function as follows:

Original: [Equation]
Revised: [Equation]

Original: Now we will see how FTL framework behaves when working on these loss functions at different iterations.
Revised: Now we will see how FTL framework behaves when working on these loss functions at different iterations.

Original: We use D ip to denote the set { ( x ∗ 1 , y ∗ 1 ) , . . .
Revised: We use D ip to denote the set { ( x ∗ 1 , y ∗ 1 ) , . . .

Original: , ( x ∗ i , y ∗ i ) } .
Revised: , ( x ∗ i , y ∗ i ) } .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: This means that A FTL algorithm, at each step, trains a new model over the combination of clean data and poison data so far ( i − 1 number of poisons).
Revised: This means that A FTL algorithm, at each step, trains a new model over the combination of clean data and poison data so far ( i − 1 number of poisons).

Original: Now we want to see what is the translation of the Regret ( A FTL , S θ p , T ) .
Revised: Now we want to see what is the translation of the Regret ( A FTL , S θ p , T ) .

Original: If we can prove an upper bound on regret, namely if we show
Revised: If we can prove an upper bound on regret, namely if we show

Original: Regret ( A FTL , S θ p , T ) ≤ α ( T ) for some function α , then we have
Revised: Regret ( A FTL , S θ p , T ) ≤ α ( T ) for some function α , then we have

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: T (cid:88)
Revised: T (cid:88)

Original: j = l j ( θ j ) −
Revised: j = l j ( θ j ) −

Original: T (cid:88)
Revised: T (cid:88)

Original: j = l
Revised: j = l

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: T (cid:88)
Revised: T (cid:88)

Original: j = l j (
Revised: j = l j (

Original: [Equation]
Revised: [Equation]

Original: j ) −
Revised: j ) −

Original: T (cid:88)
Revised: T (cid:88)

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: l j ( θ p
Revised: l j ( θ p

Original: [Equation]
Revised: [Equation]

Original: = L (
Revised: = L (

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: D c
Revised: D c

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: ≤ α ( T )
Revised: ≤ α ( T )

Original: Therefore we have
Revised: Therefore we have

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Based on Lemma A.2, we further have
Revised: Based on Lemma A.2, we further have

Original: [Equation]
Revised: [Equation]

Original: Above inequality states that average of the maximum loss difference in all previous rounds is bounded from above.
Revised: Above inequality states that average of the maximum loss difference in all previous rounds is bounded from above.

Original: Therefore, we know that among the T iterations, there exist an iteration j ∗ ∈
Revised: Therefore, we know that among the T iterations, there exist an iteration j ∗ ∈

Original: [ T ] (with lowest maximum loss difference) such that the maximum loss difference of θ j ∗ is (cid:15) -close to θ p with respect to the loss based distance where
Revised: [ T ] (with lowest maximum loss difference) such that the maximum loss difference of θ j ∗ is (cid:15) -close to θ p with respect to the loss based distance where

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.1 characterizes the dependencies of (cid:15) on α ( T ) and the constant term L ( θ p ; D c ) −
Revised: Theorem 4.1 characterizes the dependencies of (cid:15) on α ( T ) and the constant term L ( θ p ; D c ) −

Original: L ( θ c ; D c ) + N · C R · ( R ( θ p ) − R ( θ c )) .
Revised: L ( θ c ; D c ) + N · C R · ( R ( θ p ) − R ( θ c )) .

Original: To show the convergence of Algorithm 1, we need to ensure (cid:15) → 0 when T → + ∞ , which implies we need to show α ( T ) ≤ O ( √ T ) .
Revised: To show the convergence of Algorithm 1, we need to ensure (cid:15) → 0 when T → + ∞ , which implies we need to show α ( T ) ≤ O ( √ T ) .

Original: Following remark (restating Remark 1 in Section 4.2) and its proof shows the desired convergence.
Revised: Following remark (restating Remark 1 in Section 4.2) and its proof shows the desired convergence.

Original: Remark 1.
Revised: Remark 1.

Original: Online learning algorithms with sublinear regret bound can be applied to show the convergence.
Revised: Online learning algorithms with sublinear regret bound can be applied to show the convergence.

Original: Here, we adopt the regret analysis from McMahan (2017).
Revised: Here, we adopt the regret analysis from McMahan (2017).

Original: Speciﬁcally, α ( T ) is in the order of O (log T ) ) and we have (cid:15) ≤
Revised: Speciﬁcally, α ( T ) is in the order of O (log T ) ) and we have (cid:15) ≤

Original: O ( log TT ) when the loss function is Lipschitz continuous and the regularizer R ( θ ) is strongly convex, and (cid:15) → 0 when T → + ∞ .
Revised: O ( log TT ) when the loss function is Lipschitz continuous and the regularizer R ( θ ) is strongly convex, and (cid:15) → 0 when T → + ∞ .

Original: α ( T ) is also in the order of
Revised: α ( T ) is also in the order of

Original: O (log T ) when the loss function used for training is strongly convex and the regularizer is convex.
Revised: O (log T ) when the loss function used for training is strongly convex and the regularizer is convex.

Original: Our FTL framework formulation can utilize the existing logarithmic regret bound of adaptive FTL algorithm when the objective functions are strongly convex with respect to some norm (cid:107) · (cid:107) , as illustrated in Section 3.6 in McMahan (2017).
Revised: Our FTL framework formulation can utilize the existing logarithmic regret bound of adaptive FTL algorithm when the objective functions are strongly convex with respect to some norm (cid:107) · (cid:107) , as illustrated in Section 3.6 in McMahan (2017).

Original: For clarity in presentation, we ﬁrst restate their related results below.
Revised: For clarity in presentation, we ﬁrst restate their related results below.

Original: Setting 1 (Setting 1 in McMahan (2017)) .
Revised: Setting 1 (Setting 1 in McMahan (2017)) .

Original: Given a sequence of objective loss functions f 1 , f 2 , ..., f i and a sequence of incremental regularization functions r 0 , r 1 , ..., r i we consider an algorithm that selects the response point based on
Revised: Given a sequence of objective loss functions f 1 , f 2 , ..., f i and a sequence of incremental regularization functions r 0 , r 1 , ..., r i we consider an algorithm that selects the response point based on

Original: [Equation]
Revised: [Equation]

Original: We simplify the summation notation with f 1: i ( θ ) =
Revised: We simplify the summation notation with f 1: i ( θ ) =

Original: (cid:80) ij =1 f j ( θ ) .
Revised: (cid:80) ij =1 f j ( θ ) .

Original: Assume that r i is a convex function and satisfy r i ( θ ) ≥ 0 for i ∈ { 0 , 1 , 2 , ... } , against a sequence of convex loss functions
Revised: Assume that r i is a convex function and satisfy r i ( θ ) ≥ 0 for i ∈ { 0 , 1 , 2 , ... } , against a sequence of convex loss functions

Original: f i : R d →
Revised: f i : R d →

Original: R ∪ {∞} .
Revised: R ∪ {∞} .

Original: Further, letting h 0: i = r 0: i + f 1: i we assume dom h 0: i is non-empty.
Revised: Further, letting h 0: i = r 0: i + f 1: i we assume dom h 0: i is non-empty.

Original: Recalling θ i =
Revised: Recalling θ i =

Original: arg min θ h 0: i − 1 ( θ ) , we further assume ∂f i ( θ i ) is non-empty.
Revised: arg min θ h 0: i − 1 ( θ ) , we further assume ∂f i ( θ i ) is non-empty.

Original: We denote the dual norm of a norm (cid:107) · (cid:107) as (cid:107) · (cid:107) ∗ .
Revised: We denote the dual norm of a norm (cid:107) · (cid:107) as (cid:107) · (cid:107) ∗ .

Original: Theorem A.3 (Restatement of Theorem 1 in McMahan (2017)) .
Revised: Theorem A.3 (Restatement of Theorem 1 in McMahan (2017)) .

Original: Consider Setting 1, and suppose the r i are chosen such that r 0: i + f 1: i +1 is 1-strongly-convex w.r.t.
Revised: Consider Setting 1, and suppose the r i are chosen such that r 0: i + f 1: i +1 is 1-strongly-convex w.r.t.

Original: some norm (cid:107) · (cid:107) ( i ) · .
Revised: some norm (cid:107) · (cid:107) ( i ) · .

Original: If we deﬁne the regret of the algorithm with respect to a selected point θ ∗ as
Revised: If we deﬁne the regret of the algorithm with respect to a selected point θ ∗ as

Original: Regret T ( θ ∗ , f i ) ≡
Revised: Regret T ( θ ∗ , f i ) ≡

Original: T (cid:88) i =
Revised: T (cid:88) i =

Original: f i ( θ i ) −
Revised: f i ( θ i ) −

Original: T (cid:88) i =
Revised: T (cid:88) i =

Original: f i ( θ ∗ ) .
Revised: f i ( θ ∗ ) .

Original: Then, for any θ ∗ ∈ R d and for any T > 0 , with g i ∈
Revised: Then, for any θ ∗ ∈ R d and for any T > 0 , with g i ∈

Original: ∂f i ( θ i ) , we have
Revised: ∂f i ( θ i ) , we have

Original: [Equation]
Revised: [Equation]

Original: Corollary A.3.1 (Formalization of FTL result in Section 3.6 in McMahan (2017)) .
Revised: Corollary A.3.1 (Formalization of FTL result in Section 3.6 in McMahan (2017)) .

Original: In the FTL framework (no individual regularizer is used in the optimization procedure), suppose each loss function f
Revised: In the FTL framework (no individual regularizer is used in the optimization procedure), suppose each loss function f

Original: i is 1-strongly convex w.r.t.
Revised: i is 1-strongly convex w.r.t.

Original: a norm (cid:107) · (cid:107) , then we have
Revised: a norm (cid:107) · (cid:107) , then we have

Original: [Equation]
Revised: [Equation]

Original: with (cid:107) g i (cid:107) ∗ ≤ G .
Revised: with (cid:107) g i (cid:107) ∗ ≤ G .

Original: Proof.
Revised: Proof.

Original: The following proof is a restatement of the proof in Section 3.6 in McMahan (2017).
Revised: The following proof is a restatement of the proof in Section 3.6 in McMahan (2017).

Original: The proof follows from Theorem A.3.
Revised: The proof follows from Theorem A.3.

Original: Since we are considering the FTL framework, let r i ( θ ) = 

Revised: Since we are considering the FTL framework, let r i ( θ ) =

Original: for all i and deﬁne (cid:107) θ (cid:107) ( i ) = √ i (cid:107) θ (cid:107) .
Revised: for all i and deﬁne (cid:107) θ (cid:107) ( i ) = √ i (cid:107) θ (cid:107) .

Original: Observe that h 0: i (i.e., f 1: i ) is 1-strongly convex with respect to (cid:107) θ (cid:107) ( i ) (Lemma 3 in McMahan (2017)), and we have (cid:107) θ (cid:107) ( i ) , ∗ = 1 √ i (cid:107) θ (cid:107) ∗ .
Revised: Observe that h 0: i (i.e., f 1: i ) is 1-strongly convex with respect to (cid:107) θ (cid:107) ( i ) (Lemma 3 in McMahan (2017)), and we have (cid:107) θ (cid:107) ( i ) , ∗ = 1 √ i (cid:107) θ (cid:107) ∗ .

Original: Then by applying Theorem A.3, we have
Revised: Then by applying Theorem A.3, we have

Original: [Equation]
Revised: [Equation]

Original: Based on the inequality of (cid:80) Ti =1 1 /i ≤ 1 + log T and if we further assume (cid:107) g i (cid:107) ∗ ≤ G , then we can have
Revised: Based on the inequality of (cid:80) Ti =1 1 /i ≤ 1 + log T and if we further assume (cid:107) g i (cid:107) ∗ ≤ G , then we can have

Original: [Equation]
Revised: [Equation]

Original: Proof of Remark 1.
Revised: Proof of Remark 1.

Original: We will prove the logarithmic regret bound in Remark 1 utilizing Corollary A.3.1.
Revised: We will prove the logarithmic regret bound in Remark 1 utilizing Corollary A.3.1.

Original: First of all, our online learning process ﬁts into Setting 1.
Revised: First of all, our online learning process ﬁts into Setting 1.

Original: Speciﬁcally, we set r i ( θ ) = 0 for all i .
Revised: Speciﬁcally, we set r i ( θ ) = 0 for all i .

Original: For f i ( θ ) , when 1 ≤ i ≤ N , we set f i ( θ ) = 1 N L ( θ ; D c ) + C R · R ( θ ) (evenly distributing the term
Revised: For f i ( θ ) , when 1 ≤ i ≤ N , we set f i ( θ ) = 1 N L ( θ ; D c ) + C R · R ( θ ) (evenly distributing the term

Original: L ( θ ; D c ) + N · C R · R ( θ ) across N iterations) and when i ≥ N + 1 , we set f i ( θ ) =
Revised: L ( θ ; D c ) + N · C R · R ( θ ) across N iterations) and when i ≥ N + 1 , we set f i ( θ ) =

Original: l i − N ( θ ) .
Revised: l i − N ( θ ) .

Original: Details of l i can be referred from the proof of Theorem 4.1.
Revised: Details of l i can be referred from the proof of Theorem 4.1.

Original: Therefore, f i is 1-strongly convex with respect to a norm (cid:107) · (cid:107) (the norm is determined by the regularizer R ( θ ) and C R ).
Revised: Therefore, f i is 1-strongly convex with respect to a norm (cid:107) · (cid:107) (the norm is determined by the regularizer R ( θ ) and C R ).

Original: Further, l 0: i ( θ ) = f 1: N + i ( θ ) .
Revised: Further, l 0: i ( θ ) = f 1: N + i ( θ ) .

Original: In addition, the assumption that dom h 0: i is non-empty in Setting 1 means when if we train a classiﬁer on the poisoned data set, we can always return a model and hence the assumption is satisﬁed.
Revised: In addition, the assumption that dom h 0: i is non-empty in Setting 1 means when if we train a classiﬁer on the poisoned data set, we can always return a model and hence the assumption is satisﬁed.

Original: The assumption of the existence of subgradient ∂f i ( θ i ) in Setting 1 is also satisﬁed by the poisoning attack scenario.
Revised: The assumption of the existence of subgradient ∂f i ( θ i ) in Setting 1 is also satisﬁed by the poisoning attack scenario.

Original: The logarithmic regret of Regret ( A FTL , S θ p , T ) of our algorithm then follows from the result of
Revised: The logarithmic regret of Regret ( A FTL , S θ p , T ) of our algorithm then follows from the result of

Original: Regret T ( θ ∗ , f i ) in Corollary A.3.1.
Revised: Regret T ( θ ∗ , f i ) in Corollary A.3.1.

Original: Speciﬁcally, l 0: i ( θ ) = f 1: N + i ( θ ) is 1-strongly convex to norm (cid:107) · (cid:107) i = √ N + i (cid:107) · (cid:107) and since we assume the loss function is G -Lipschitz, we have (cid:107) g
Revised: Speciﬁcally, l 0: i ( θ ) = f 1: N + i ( θ ) is 1-strongly convex to norm (cid:107) · (cid:107) i = √ N + i (cid:107) · (cid:107) and since we assume the loss function is G -Lipschitz, we have (cid:107) g

Original: i (cid:107) ∗ ≤ G .
Revised: i (cid:107) ∗ ≤ G .

Original: Therefore, we have the logarithmic regret bound as:
Revised: Therefore, we have the logarithmic regret bound as:

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: We next provide the proof of the certiﬁed lower bound (restating Theorem 4.2 from Section 4.3):
Revised: We next provide the proof of the certiﬁed lower bound (restating Theorem 4.2 from Section 4.3):

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Given a target classiﬁer θ p , to reproduce θ p by adding the poisoning set D p into D c , the number of poisoning points |D p | cannot be lower than
Revised: Given a target classiﬁer θ p , to reproduce θ p by adding the poisoning set D p into D c , the number of poisoning points |D p | cannot be lower than

Original: [Equation]
Revised: [Equation]

Original: The main intuition behind the theorem is, when the the number of poisoning points added to the clean training set is lower than the certiﬁed lower bound, for structural empirical risk minimization problem (shown in equation 1 in the main paper), then target classiﬁer will always have higher loss than another classiﬁer and hence cannot be achieved.
Revised: The main intuition behind the theorem is, when the the number of poisoning points added to the clean training set is lower than the certiﬁed lower bound, for structural empirical risk minimization problem (shown in equation 1 in the main paper), then target classiﬁer will always have higher loss than another classiﬁer and hence cannot be achieved.

Original: Proof.
Revised: Proof.

Original: We ﬁrst show that for all models θ , we can derive a lower bound on the number of poison points required to get θ p .
Revised: We ﬁrst show that for all models θ , we can derive a lower bound on the number of poison points required to get θ p .

Original: Then since these lower bounds all hold, we can take the maximum over all of them and get a valid lower bound.
Revised: Then since these lower bounds all hold, we can take the maximum over all of them and get a valid lower bound.

Original: We ﬁrst show that for any model θ , the minimum number of poisoning points cannot be lower than
Revised: We ﬁrst show that for any model θ , the minimum number of poisoning points cannot be lower than

Original: [Equation]
Revised: [Equation]

Original: Let us denote the point corresponding to the supremium of the loss difference between θ and θ p as ( x ∗ , y ∗ ) 2 .
Revised: Let us denote the point corresponding to the supremium of the loss difference between θ and θ p as ( x ∗ , y ∗ ) 2 .

Original: Namely, l ( θ ; x ∗ , y ∗ ) − l ( θ p ; x ∗ , y ∗ ) =
Revised: Namely, l ( θ ; x ∗ , y ∗ ) − l ( θ p ; x ∗ , y ∗ ) =

Original: sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) .
Revised: sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) .

Original: Now suppose we can obtain θ p with lower number of poisoning points c < c ( θ ) .
Revised: Now suppose we can obtain θ p with lower number of poisoning points z < z ( θ ) .

Original: Assume there is a poisoning set D p with size c such that when added to D c would result in θ p .
Revised: Assume there is a poisoning set D p with size z such that when added to D c would result in θ p .

Original: We have sup x,y
Revised: We have sup x,y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: implying sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) + C R · ( R ( θ ) − R ( θ p )) > 0 .
Revised: implying sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) + C R · ( R ( θ ) − R ( θ p )) > 0 .

Original: Based on the assumption that c < c ( θ ) , and the fact that sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) + C R · ( R ( θ ) − R ( θ p )) > 0 , we have c · [Equation] l ( θ ; x ∗ , y ∗ ) − l ( θ p ; x ∗ , y ∗ )
Revised: Based on the assumption that z < z ( θ ) , and the fact that sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) + C R · ( R ( θ ) − R ( θ p )) > 0 , we have z

Original: [Equation]
Revised: [Equation]

Original: R
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: x

Original: [Equation]
Revised: [Equation]

Original: θ p
Revised: y

Original: [Equation]
Revised: [Equation]

Original: where the equality is based on the deﬁnition of c ( θ ) .
Revised: where the equality is based on the deﬁnition of z ( θ ) .

Original: On the other hand, by deﬁnition of ( x ∗ , y ∗ ) for any D p of size c , we have
Revised: On the other hand, by deﬁnition of ( x ∗ , y ∗ ) for any D p of size z , we have

Original: [Equation]
Revised: [Equation]

Original: − R ( θ p )) (cid:1) .
Revised: − R ( θ p )) (cid:1) .

Original: The above two inequality imply that for any set D p with size c we have
Revised: The above two inequalities imply that for any set D p with size z we have

Original: [Equation]
Revised: [Equation]

Original: which indicates that adding D p poisoning points into the training set D c , the model θ has lower loss compared to θ p , which is a contradiction to the assumption that θ p has lower loss on D c ∪ D p and can be achieved.
Revised: which indicates that adding D p poisoning points into the training set D c , the model θ has lower loss compared to θ p , which is a contradiction to the assumption that θ p has lowest loss on D c ∪ D p and can be achieved.

Original: Now, since θ p needs to have lower loss on D c ∪ D p compared to any classiﬁer θ ∈ Θ , the best lower bound is the supremium over all models in the model space Θ .
Revised: Now, since θ p needs to have lower loss on D c ∪ D p compared to any classiﬁer θ ∈ Θ , the best lower bound is the supremium over all models in the model space Θ .

Original: B R ELATING (cid:15) - CLOSENESS TO CLOSENESS OF PARAMETERS
Revised: B R ELATING (cid:15) - CLOSENESS TO CLOSENESS OF PARAMETERS

Original: In theorem below, we show how one can relate the notion of (cid:15) -closeness in Deﬁnition 1 in the main paper to closeness of parameters in the speciﬁc setting of hinge loss.
Revised: In theorem below, we show how one can relate the notion of (cid:15) -closeness in Deﬁnition 1 in the main paper to closeness of parameters in the speciﬁc setting of hinge loss.

Original: We use this just as an example to show that our notion of (cid:15) -closeness can be tightly related to the closeness of the models.
Revised: We use this just as an example to show that our notion of (cid:15) -closeness can be tightly related to the closeness of the models.

Original: Theorem B.1.
Revised: Theorem B.1.

Original: Consider the hinge loss function l ( θ ; x, y )
Revised: Consider the hinge loss function l ( θ ; x, y )

Original: = max(1 − y · (cid:104) x, θ (cid:105) , 0) for θ ∈ R d and x ∈ R d and y ∈ {− 1 , +1 } .
Revised: = max(1 − y · (cid:104) x, θ (cid:105) , 0) for θ ∈ R d and x ∈ R d and y ∈ {− 1 , +1 } .

Original: For θ, θ
Revised: For θ, θ

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: R
Revised: R

Original: d such that (cid:107) θ (cid:107) 1 ≤ r and (cid:107) θ (cid:48) (cid:107) 1 ≤ r for some r ≥ 1 , if θ is (cid:15) -close to θ (cid:48) in the loss based distance, for some (cid:15) ≤ 1 then, (cid:107) θ − θ (cid:48)
Revised: d such that (cid:107) θ (cid:107) 1 ≤ r and (cid:107) θ (cid:48) (cid:107) 1 ≤ r for some r ≥ 1 , if θ is (cid:15) -close to θ (cid:48) in the loss based distance, for some (cid:15) ≤ 1 then, (cid:107) θ − θ (cid:48)

Original: (cid:107) 1 ≤ r · (cid:15) .
Revised: (cid:107) 1 ≤ r · (cid:15) .

Original: Remark 2.
Revised: Remark 2.

Original: In Theorem B.1 above, an upper bound on the norm of θ and θ (cid:48) is (cid:112) d/C R .
Revised: In Theorem B.1 above, an upper bound on the norm of θ and θ (cid:48) is (cid:112) d/C R .

Original: however, the models that we care about in practice usually have smaller norms.
Revised: however, the models that we care about in practice usually have smaller norms.

Original: Proof.
Revised: Proof.

Original: We construct a point x ∗ as follows:
Revised: We construct a point x ∗ as follows:

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) θ (cid:107) 1 ≤ r we have
Revised: Since (cid:107) θ (cid:107) 1 ≤ r we have

Original: [Equation]
Revised: [Equation]

Original: and similarly since (cid:107) θ (cid:48) (cid:107) 1 ≤ r we have
Revised: and similarly since (cid:107) θ (cid:48) (cid:107) 1 ≤ r we have

Original: [Equation]
Revised: [Equation]

Original: Therefore by Inequalities equation 4 and equation 5 we have
Revised: Therefore by Inequalities equation 4 and equation 5 we have

Original: [Equation]
Revised: [Equation]

Original: which by Equation equation 3 implies
Revised: which by Equation equation 3 implies

Original: [Equation]
Revised: [Equation]

Original: Now since we know that, ∀ x ∈ R d , the loss difference between θ and θ (cid:48) is bounded by (cid:15) , the bound should also hold for the point ( x ∗ , − 1) , meaning that
Revised: Now since we know that, ∀ x ∈ R d , the loss difference between θ and θ (cid:48) is bounded by (cid:15) , the bound should also hold for the point ( x ∗ , − 1) , meaning that

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: C I NSTANTIATING T HEOREM 4.1 FOR THE CASE OF SVM
Revised: C I NSTANTIATING T HEOREM 4.1 FOR THE CASE OF SVM

Original: Here we show how to instantiate Theorem 4.1 for SVM with exact constants instead of the asymptotic notations.
Revised: Here we show how to instantiate Theorem 4.1 for SVM with exact constants instead of the asymptotic notations.

Original: We need to calculate the constant γ to get the exact constant.
Revised: We need to calculate the constant γ to get the exact constant.

Original: Imagine the feature domain is R d .
Revised: Imagine the feature domain is R d .

Original: Now we calculate the constant C as follows.
Revised: Now we calculate the constant C as follows.

Original: Let i ∗ θ = arg min
Revised: Let i ∗ θ = arg min

Original: i ∈
Revised: i ∈

Original: [ d ] | θ
Revised: [ d ] | θ

Original: [ i ] /θ p
Revised: [ i ] /θ p

Original: [ i ] | and α
Revised: [ i ] | and α

Original: θ =
Revised: θ =

Original: | θ
Revised: | θ

Original: [ i ∗ θ ] /θ p
Revised: [ i ∗ θ ] /θ p

Original: [ i ∗ θ ] | .
Revised: [ i ∗ θ ] | .

Original: Let x ∗ θ ∈ R d be a point where is equal to 0 everywhere and is equal to 1 /θ p [ i ∗ θ ] on the i ∗ coordinate.
Revised: Let x ∗ θ ∈ R d be a point where is equal to 0 everywhere and is equal to 1 /θ p [ i ∗ θ ] on the i ∗ coordinate.

Original: We have,
Revised: We have,

Original: [Equation]
Revised: [Equation]

Original: Now we can calculate C as follows
Revised: Now we can calculate C as follows

Original: [Equation]
Revised: [Equation]

Original: (By Inequality 6)
Revised: (By Inequality 7)

Original: [Equation]
Revised: [Equation]

Original: (By deﬁnition of α θ )
Revised: (By deﬁnition of α θ )

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Therefore γ ≥ 1 − 2 · C R · R ( θ p ) .
Revised: Therefore γ ≥ 1 − 2 · C R · R ( θ p ) .

Original: On the other hand, we can also calculate α ( T ) based on the exact form given in the proof of Theorem 4.1.
Revised: On the other hand, we can also calculate α ( T ) based on the exact form given in the proof of Theorem 4.1.

Original: D  I NDISCRIMINATE P OISONING A TTACKS
Revised: D A DDITIONAL E XPERIMENTS IN I NDISCRIMINATE S ETTING

Original: In this section, we evaluate the attacks in the conventional indiscriminate attack setting, where the attacker’s goal is just to reduce the overall accuracy of the model.
Revised: In this section, we evaluate the attacks in the conventional indiscriminate attack setting, where the attacker’s goal is just to reduce the overall accuracy of the model

Original: Datasets and Models.
Revised: Datasets and Models.

Original: For the indiscriminate attack, we use the MNIST-17 dataset, which consists of the digits 1 and 7 and is commonly used for evaluating indiscriminate poisoning attacks against binary classiﬁcation (Steinhardt et al., 2017; Biggio et al., 2012; Xiao et al., 2012).
Revised: For the indiscriminate attack, we use the MNIST-17 dataset, which consists of the digits 1 and 7 and is commonly used for evaluating indiscriminate poisoning attacks against binary classiﬁcation (Steinhardt et al., 2017; Biggio et al., 2012; Xiao et al., 2012).

Original: The MNISTdataset contains 13007 training data and 2163 test data.
Revised: The MNISTdataset contains 13007 training data and 2163 test data.

Original: The dataset contains 784 features and all the features are normalized into range [0 , 1] .
Revised: The dataset contains 784 features and all the features are normalized into range [0 , 1] .

Original: For completeness, the Adult dataset used for subpopulation attack is downsampled to form a class-balanced dataset and contains 15682 training data andtest data.
Revised: For completeness, the Adult dataset used for subpopulation attack is downsampled to form a class-balanced dataset and contains 15682 training data andtest data.

Original: The dataset contains 57 features and the features are also normalized into range [0 , 1] (except for the binary features).
Revised: The dataset contains 57 features and the features are also normalized into range [0 , 1] (except for the binary features).

Original: All of the processed datasets are included in the supplementary material.
Revised: All of the processed datasets are included in the supplementary material.

Original: We still adopt linear SVM model in the indiscriminate attack scenario.
Revised: We still adopt linear SVM model in the indiscriminate attack scenario.

Original: All of the models for both datasets set the regularization parameter C R = 0 . 09 .
Revised: All of the models for both datasets set the regularization parameter C R = 0 . 09 .

Original: The clean accuracy of SVM model on
Revised: The clean accuracy of SVM model on

Original: MNIST-17 dataset is 98.9% and the accuracy on Adult dataset is 78.5%.
Revised: MNIST-17 dataset is 98.9% and the accuracy on Adult dataset is 78.5%.

Original: Target Classiﬁers.
Revised: Target Classiﬁers.

Original: Accuracy of the clean MNIST-17 model has around 1% error rate on the test set.
Revised: Accuracy of the clean MNIST-17 model has around 1% error rate on the test set.

Original: For our experiment, we aim to generate three target classiﬁers with overall test accuracies around 5%, 10% and 15%.
Revised: For our experiment, we aim to generate three target classiﬁers with overall test accuracies around 5%, 10% and 15%.

Original: To generate target classiﬁers with desired error rates, we follow the heuristic strategy proposed by Koh et al.
Revised: To generate target classiﬁers with desired error rates, we follow the heuristic strategy proposed by Koh et al.

Original: (2018) to generate multiple candidate target classiﬁers, and then among all the valid candidate models that satisfy the error rate requirement we choose the one with lowest loss on the clean training set.
Revised: (2018) to generate multiple candidate target classiﬁers, and then among all the valid candidate models that satisfy the error rate requirement we choose the one with lowest loss on the clean training set.

Original: Using this approach, the ﬁnal target classiﬁers induced have overall test accuracy of 94.0%, 88.8% and 83.3% respectively.
Revised: Using this approach, the ﬁnal target classiﬁers induced have overall test accuracy of 94.0%, 88.8% and 83.3% respectively.

Original: (We describe a better way of ﬁnding the target classiﬁers in Appendix D.1, but for comparison purposes do not use those in the results here.)
Revised: (We describe a better way of ﬁnding the target classiﬁers in Appendix D.1, but for comparison purposes do not use those in the results here.)

Original: Convergence.
Revised: Convergence.

Original: We show the convergence of Algorithm 1 by reporting the maximum loss difference and Euclidean distance between the classiﬁer induced by the attack and the target classiﬁer.
Revised: We show the convergence of Algorithm 1 by reporting the maximum loss difference and Euclidean distance between the classiﬁer induced by the attack and the target classiﬁer.

Original: Figures 3a and 3b summarize the results for the target classiﬁer with a 10% error rate.
Revised: Figures 3a and 3b summarize the results for the target classiﬁer with a 10% error rate.

Original: The maximum number of poisoning points in the ﬁgure is obtained when the classiﬁer from Algorithm 1 is 0 .
Revised: The maximum number of poisoning points in the ﬁgure is obtained when the classiﬁer from Algorithm 1 is 0 .

Original: 1 -close to the target classiﬁer in the loss-based distance.
Revised: 1 -close to the target classiﬁer in the loss-based distance.

Original: In Figure 3, the classiﬁer induced by our algorithm steadily converges to the target classiﬁer both in the maximum loss difference and Euclidean distance, while the classiﬁer induced by the KKT attack diverges initially and then starts to converge to the target model.
Revised: In Figure 3, the classiﬁer induced by our algorithm steadily converges to the target classiﬁer both in the maximum loss difference and Euclidean distance, while the classiﬁer induced by the KKT attack diverges initially and then starts to converge to the target model.

Original: At the maximum number of points, the maximum loss difference of KKT-induced classiﬁer to the target is 0.46, compared to 0.1 for the classiﬁer induced by our attack.
Revised: At the maximum number of points, the maximum loss difference of KKT-induced classiﬁer to the target is 0.46, compared to 0.1 for the classiﬁer induced by our attack.

Original: For the Euclidean distance, the KKT-induced classiﬁer is 0.16 away, compared to 0.07 for the classiﬁer induced by our attack.
Revised: For the Euclidean distance, the KKT-induced classiﬁer is 0.16 away, compared to 0.07 for the classiﬁer induced by our attack.

Original: Attack Success.
Revised: Attack Success.

Original: We next compare the classiﬁer induced from our attack to the classiﬁer induced by the KKT attack in terms of their overall test accuracy.
Revised: We next compare the classiﬁer induced from our attack to the classiﬁer induced by the KKT attack in terms of their overall test accuracy.

Original: Similarly, the maximum number of poisoning
Revised: Similarly, the maximum number of poisoning

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: points in Figure 2 is obtained by running our attack with 0 .
Revised: points in Figure 2 is obtained by running our attack with 0 .

Original: 1 -closeness (in loss based distance) to the target as the input.
Revised: 1 -closeness (in loss based distance) to the target as the input.

Original: In terms of the test accuracy, our attack has a comparable attack success rate compared to the KKT attack.
Revised: In terms of the test accuracy, our attack has a comparable attack success rate compared to the KKT attack.

Original: Speciﬁcally, for target models of 5% and 10% error rates, both methods have almost identical performance, as shown in Figures 4a and 4b.
Revised: Speciﬁcally, for target models of 5% and 10% error rates, both methods have almost identical performance, as shown in Figures 4a and 4b.

Original: For the target model of 15% error rate (test accuracy is 83.3%), the KKT attack is more successful than our attack, inducing models with 82.7% accuracy (17.3% attack success rate) compared to 85.9% accuracy (14.1% attack success rate) for our attack.
Revised: For the target model of 15% error rate (test accuracy is 83.3%), the KKT attack is more successful than our attack, inducing models with 82.7% accuracy (17.3% attack success rate) compared to 85.9% accuracy (14.1% attack success rate) for our attack.

Original: Interestingly, in this case, the performance of models induced by the two attacks with fewer poisoning points our attack results in models with lower test accuracy (higher attacker success) than the KKT attack.
Revised: Interestingly, in this case, the performance of models induced by the two attacks with fewer poisoning points our attack results in models with lower test accuracy (higher attacker success) than the KKT attack.

Original: To summarize, for the indiscriminate scenario, our attack produces classiﬁers that have much closer distance to the target models than the KKT attack, and has comparable attack success rates with the KKT attack.
Revised: To summarize, for the indiscriminate scenario, our attack produces classiﬁers that have much closer distance to the target models than the KKT attack, and has comparable attack success rates with the KKT attack.

Original: Lower Bound on Number of Poisons.
Revised: Lower Bound on Number of Poisons.

Original: We next check the optimality of our attack in the indiscriminate attack scenario.
Revised: We next check the optimality of our attack in the indiscriminate attack scenario.

Original: Similar to the subpopulation attack setting, we still use Theorem 4.2 to compute the lower bound of the induced classiﬁer from our attack by using it as the input to Algorithm 1, and terminating when the induced classiﬁer is 0 .
Revised: Similar to the subpopulation attack setting, we still use Theorem 4.2 to compute the lower bound of the induced classiﬁer from our attack by using it as the input to Algorithm 1, and terminating when the induced classiﬁer is 0 .

Original: 1 -close to the given target model.
Revised: 1 -close to the given target model.

Original: Our calculated lower bound shows that there exists a relatively large gap between the number of poisoning points, especially for the induced classiﬁer from our attack for the target model of 5% error rate, where the
Revised: Our calculated lower bound shows that there exists a relatively large gap between the number of poisoning points, especially for the induced classiﬁer from our attack for the target model of 5% error rate, where the

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: lower bound is only 50% of the actual number of poisoning points used.
Revised: lower bound is only 50% of the actual number of poisoning points used.

Original: For the induced classiﬁer for the target model of 15% error rate, the gap between the number of poisoning points and the lower bound is smallest, with the lower bound taking 79% of the number of poisoning points.
Revised: For the induced classiﬁer for the target model of 15% error rate, the gap between the number of poisoning points and the lower bound is smallest, with the lower bound taking 79% of the number of poisoning points.

Original: The relatively large gap indicates that either the estimated lower bound is not tight or the attack itself is not close to optimal.
Revised: The relatively large gap indicates that either the estimated lower bound is not tight or the attack itself is not close to optimal.

Original: To gain more insights into this problem, we further show the computed lower bound at each iteration when running Algorithm 1 and Figure 5 summarizes the results.
Revised: To gain more insights into this problem, we further show the computed lower bound at each iteration when running Algorithm 1 and Figure 5 summarizes the results.

Original: From the Figure, we see that, the peak value of the curve (i.e., highest lower bound) always appears before the termination of the algorithm, indicating that the computed lower bound is likely to be tight and we may need to further improve the attack algorithm.
Revised: From the Figure, we see that, the peak value of the curve (i.e., highest lower bound) always appears before the termination of the algorithm, indicating that the computed lower bound is likely to be tight and we may need to further improve the attack algorithm.

Original: For completeness, we also repeat the lower bound computation process for classiﬁers induced from the KKT attack.
Revised: For completeness, we also repeat the lower bound computation process for classiﬁers induced from the KKT attack.

Original: The KKT induced classiﬁers are obtained by running the KKT attack with target classiﬁers of different error rates as target input.
Revised: The KKT induced classiﬁers are obtained by running the KKT attack with target classiﬁers of different error rates as target input.

Original: The target number of poisoning points of KKT attack is given by the size of poisoning set returned from our attack when our algorithm terminates when the induced classiﬁer is 0 .
Revised: The target number of poisoning points of KKT attack is given by the size of poisoning set returned from our attack when our algorithm terminates when the induced classiﬁer is 0 .

Original: 1 -close to the target model of different error rates.
Revised: 1 -close to the target model of different error rates.

Original: Then the lower bound computation process is identical to the above – we simply send the KKT induced classiﬁer as target input to Algorithm 1 and terminate it when the induced classiﬁer from our algorithm is 0 .
Revised: Then the lower bound computation process is identical to the above – we simply send the KKT induced classiﬁer as target input to Algorithm 1 and terminate it when the induced classiﬁer from our algorithm is 0 .

Original: 1 -close to the given target model (i.e., KKT induced classiﬁer).
Revised: 1 -close to the given target model (i.e., KKT induced classiﬁer).

Original: The results are summarized in Table 3 and we observe that there also exists a relatively large gap between the lower bound and the number of poisoning points used by the attack.
Revised: The results are summarized in Table 3 and we observe that there also exists a relatively large gap between the lower bound and the number of poisoning points used by the attack.

Original: Similarly, we also plot the lower bound computed at different iterations of Algorithm 1 in Figure 6, and ﬁnd that the peak value also appears before the termination of the algorithm, indicating that the lower bound might be tight and we need a stronger attack strategy to close the gap.
Revised: Similarly, we also plot the lower bound computed at different iterations of Algorithm 1 in Figure 6, and ﬁnd that the peak value also appears before the termination of the algorithm, indicating that the lower bound might be tight and we need a stronger attack strategy to close the gap.

Original: D. I MPROVED T ARGET G ENERATION P ROCESS
Revised: D. I MPROVED T ARGET G ENERATION P ROCESS

Original: The original heuristic approach works by ﬁnding different quantiles of training points that have higher loss on the clean model, ﬂipping their labels, repeating those points for multiple copies, and adding them to the clean training set.
Revised: The original heuristic approach works by ﬁnding different quantiles of training points that have higher loss on the clean model, ﬂipping their labels, repeating those points for multiple copies, and adding them to the clean training set.

Original: We ﬁnd that, in the process of trying different quantiles and copies of high loss points, if we also adaptively update the model where the high loss points are found (instead of just always ﬁxing it to be the clean model), we can generate a valid target classiﬁer with even lower loss on the clean training.
Revised: We ﬁnd that, in the process of trying different quantiles and copies of high loss points, if we also adaptively update the model where the high loss points are found (instead of just always ﬁxing it to be the clean model), we can generate a valid target classiﬁer with even lower loss on the clean training.

Original: Such an improved generation process can signiﬁcantly reduce the number
Revised: Such an improved generation process can signiﬁcantly reduce the number

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: of poisoning points needed to reach the same (cid:15) -closeness (with respect to the loss based distance)
Revised: of poisoning points needed to reach the same (cid:15) -closeness (with respect to the loss based distance)

Original: to the target classiﬁer, consistent with the claims in Theorem 4.1 in the main paper.
Revised: to the target classiﬁer, consistent with the claims in Theorem 4.1 in the main paper.

Original: In addition, we ﬁnd that, if we compare our attack with improved generation process to the KKT attack with the original generation process (Koh et al., 2018), we can also reach the desired target error rate much faster using our attack.
Revised: In addition, we ﬁnd that, if we compare our attack with improved generation process to the KKT attack with the original generation process (Koh et al., 2018), we can also reach the desired target error rate much faster using our attack.

Original: Validate Implication of Theorem 4.1.
Revised: Validate Implication of Theorem 4.1.

Original: We ﬁrst empirically validate the implication of Theorem 4.
Revised: We ﬁrst empirically validate the implication of Theorem 4.

Original: in the main paper: to obtain the same (cid:15) -closeness in loss-based distance, a target classiﬁer with lower loss on the clean training set D c requires fewer poisoning points.
Revised: in the main paper: to obtain the same (cid:15) -closeness in loss-based distance, a target classiﬁer with lower loss on the clean training set D c requires fewer poisoning points.

Original: Therefore, when adversaries have multiple target classiﬁers that satisfy the attack goal, the one with lower loss on clean training set is preferred.
Revised: Therefore, when adversaries have multiple target classiﬁers that satisfy the attack goal, the one with lower loss on clean training set is preferred.

Original: For both the original and improved target generation methods, we generate three target classiﬁers with error rates of 5%, 10% and 15%.
Revised: For both the original and improved target generation methods, we generate three target classiﬁers with error rates of 5%, 10% and 15%.

Original: The original target classiﬁer generation method returns classiﬁers with test accuracy of 94.0%, 88.8% and 82.3% respectively (also used in the previous experiments of indiscriminate attack).
Revised: The original target classiﬁer generation method returns classiﬁers with test accuracy of 94.0%, 88.8% and 82.3% respectively (also used in the previous experiments of indiscriminate attack).

Original: The improved target generation process returns target classiﬁers with approximately the same test accuracy (94.9%, 88.9% and 84.5%).
Revised: The improved target generation process returns target classiﬁers with approximately the same test accuracy (94.9%, 88.9% and 84.5%).

Original: However, for classiﬁers returned from the two generation methods of same error rate, the improved generation method produces classiﬁers with signiﬁcantly lower loss compared to the original generation approach.
Revised: However, for classiﬁers returned from the two generation methods of same error rate, the improved generation method produces classiﬁers with signiﬁcantly lower loss compared to the original generation approach.

Original: Table 4 compares the two target generation approaches by showing the number of poisoning points needed to get 0 .
Revised: Table 4 compares the two target generation approaches by showing the number of poisoning points needed to get 0 .

Original: 1 -close to the corresponding target model of same error rate.
Revised: 1 -close to the corresponding target model of same error rate.

Original: For example, for target models of 15% error rate, the model from the original approach has a total clean loss of 5428.4 while our improved method reduces it to 4641.6.
Revised: For example, for target models of 15% error rate, the model from the original approach has a total clean loss of 5428.4 while our improved method reduces it to 4641.6.

Original: With the reduced clean loss, getting 0 .
Revised: With the reduced clean loss, getting 0 .

Original: 1 -close to the target model generated from our improved process only requires 3206 poisoning points, while reaching the same distance from the target model produced by the original method would require 6762 poisoning points, a more than 50% reduction.
Revised: 1 -close to the target model generated from our improved process only requires 3206 poisoning points, while reaching the same distance from the target model produced by the original method would require 6762 poisoning points, a more than 50% reduction.

Original: End-to-end Comparison.
Revised: End-to-end Comparison.

Original: Figure 7 compares the two attacks in an end-to-end manner in terms of their test accuracy.
Revised: Figure 7 compares the two attacks in an end-to-end manner in terms of their test accuracy.

Original: With the improved target generation process, our attack can achieve the desired error rate much faster than the KKT attack with the original process.
Revised: With the improved target generation process, our attack can achieve the desired error rate much faster than the KKT attack with the original process.

Original: For the KKT attack with target model generated from the original process, we determine the target number of poisoning points using the size of poisoning set returned from running Algorithm 1 with 0 .
Revised: For the KKT attack with target model generated from the original process, we determine the target number of poisoning points using the size of poisoning set returned from running Algorithm 1 with 0 .

Original: 1 -closeness and target model from original process as inputs.
Revised: 1 -closeness and target model from original process as inputs.

Original: To run our attack with improved generation process, we terminate the algorithm when the size of the poisoning points is same as the number of poisoning points used by the KKT attack with original process.
Revised: To run our attack with improved generation process, we terminate the algorithm when the size of the poisoning points is same as the number of poisoning points used by the KKT attack with original process.

Original: Such a termination criteria helps us to ensure that both attacks use same number of poisoning points and can be compared easily.
Revised: Such a termination criteria helps us to ensure that both attacks use same number of poisoning points and can be compared easily.

Original: We also evaluate the KKT attack on fractions of the maximum target number of poisoning points (0.2, 0.4, 0.6, and 0.8), as in the previous experiments.
Revised: We also evaluate the KKT attack on fractions of the maximum target number of poisoning points (0.2, 0.4, 0.6, and 0.8), as in the previous experiments.

Original: The accuracy plot shows that our attack (with improved target model) can achieve the desired error rate (e.g., 10%) much faster than the KKT attack (with original target model), especially for the target classiﬁers of error rates of 10% and 15%.
Revised: The accuracy plot shows that our attack (with improved target model) can achieve the desired error rate (e.g., 10%) much faster than the KKT attack (with original target model), especially for the target classiﬁers of error rates of 10% and 15%.

Original: E C OMPARISON OF M ODEL -T ARGETED AND O BJECTIVE -D RIVEN A TTACKS
Revised: E C OMPARISON OF M ODEL -T ARGETED AND O BJECTIVE -D RIVEN A TTACKS

Original: Model-targeted and objective-driven attacks have different goals: using few poisoning points, the former aims to induce a given target classiﬁer while the latter tries to achieve some attacker objective such as increasing overall test error rate (in indiscriminate setting). Therefore, the two types of attacks are not directly comparable. However, model-targeted attacks can also perform well in achieving some attacker objective given that a“better” target model for  that  attacker goal is provided.
Revised: Although model-targeted attacks work to induce the given target classiﬁers by generating poisoning points, the end goal is still to achieve the attacker objectives encoded in the target models. In terms of the comparison to the objective-driven attacks, we ﬁrst demonstrate that, objective-driven attacks can be used to generate a target model, which can then be used as the target for a model-targeted attack, resulting in an attack that achieves the desired attacker objective with fewer poisoning points.

Revised: Then, we show that in order to have competitive performance against state-of-the-art objective-driven attacks (e.g., the min-max attack (Steinhardt et al., 2017)), the target classiﬁers should be generated carefully, such that the attacker objectives of the target classiﬁers can be achieved efﬁciently with model-targeted attacks using fewer poisoning points.

Original: A principled search of  such “better” target classiﬁer is out of the scope of this paper and we leave it for the future work.
Revised: Although the investigation of a systematic approach to generate such “desired” classiﬁers is out of the scope of this paper, in the indiscriminate setting, we have some empirical evidence.

Original: Here, we provide a heuristic approach of searching a “better” target model when one is interested in comparing the model-targeted attacks (e.g., the attack proposed in this paper) to some objective-driven attacks (e.g., the min-max attack (Steinhardt et al., 2017)) in indiscriminate setting – with target models of larger overall error rates (e.g., 10% or 15% error rate), one often can achieve the attacker  objective of producing some smaller overall error rate (e.g., 5%) using fewer number of poisoning points.
Revised: Speciﬁcally, we ﬁnd that target classiﬁers with lower loss on clean training set and higher error rates (higher than what are desired in the attacker objectives) often require fewer poisoning points to achieve the attacker objectives. Target Models Generated from Objective-driven Attacks. In our experiments, the target classiﬁers are generated from the label ﬂipping based objective driven attacks that are effective but need too many poisoning points to achieve their objective.

Original: For example, in Figure 7, with our model-targeted poisoning attack, if adversaries aim to produce a classiﬁer of some small error rate such as 5% (instead of inducing the given target classiﬁer) using fewer  number of poisoning points, then choosing target classiﬁers of 10% or 15% error rates can achieve the attacker goal much faster than using a target classiﬁer of 5% error rate.
Revised: Then, our attacks are deployed to achieve the same objective with fewer poisoning points. Table 5 shows the number of poisoning points used by the label-ﬂipping attack and our model-targeted attack, to achieve desired attack objectives of increasing the test error to certain amount. We can see that, using our attack, the number of poisoning points used by label-ﬂipping attacks can be saved up to 73%.

Original: Next, we compare our attack to the  min-max attack (Steinhardt et al., 2017)  in indiscriminate setting.
Revised: Comparison to the Min-Max Attack. Still using target classiﬁers generated from label-ﬂipping attacks, we show that our attack can outperform the state-of-the-art min-max attack (Steinhardt et al., 2017) at reducing the overall test accuracy, under same amount of poisoning points. Since we aim to produce target classiﬁers with lower loss on clean training set and higher error rates, we adopt the improved target model generation process described in Section D.1 (helps to reduce the loss on clean training set) and generate a classiﬁer of 15% error rate. With the target model, we terminate our attack when the induced model is 0 .

Original: We compare theaccuracy drops of two attacks under same number of poisoning points, but here we choose target classiﬁer of larger error rate for our attack (as it gives higher accuracy drop).
Revised: 1 -close to the target model in terms of the loss-based distance. However, to we compare our attack to the min-max attack conveniently, we compare their accuracy reduction at different poisoning ratios, which is the common evaluation strategy of objective-driven attacks. Table 6 summarizes the results.

Original: Speciﬁcally, we use a target classiﬁer of 10% error rate and the number of poisoning points used for the two attacks is 650 (0.05 poisoning ratio).
Revised: From the table, we observe that, compared to the min-max Attacker Objectives 5% [Table] [Table]

Original: Empirically, with theheuristically selected “better” target model, our attack can reduce the overall test accuracy from the clean accuracy of 98.9% to 96.4%.
Revised: [Table] Our Attack 1, 5, 6, increase the test error to certain amounts (i.e., 5%, 10% and 15%) and the target classiﬁers to our attack are generated by running the label-ﬂipping attacks with given attacker objectives. [Table]

Original: The min-max attack reduces  the test accuracy to 97.4% andour attack is slightly more successful than the min-max attack.
Revised: attack, our attack reduces more on the test accuracy under the same poisoning budget and the gap becomes larger when the poisoning ratio increases.