Original: M ODEL -T ARGETED P OISONING A TTACKS WITH P ROVABLE C ONVERGENCE
Revised: M ODEL -T ARGETED P OISONING A TTACKS WITH P ROVABLE C ONVERGENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In a poisoning attack, an adversary with control over a small fraction of the training data attempts to select that data in a way that induces a model that misbehaves in a particular way desired by the adversary, such as misclassifying certain inputs.
Revised: In a poisoning attack, an adversary with control over a small fraction of the training data attempts to select that data in a way that induces a model that misbehaves in a particular way desired by the adversary, such as misclassifying certain inputs.

Original: We propose an efﬁcient poisoning attack that can target a desired model based on online convex optimization.
Revised: We propose an efﬁcient poisoning attack that can target a desired model based on online convex optimization.

Original: Unlike previous model-targeted poisoning attacks, our attack comes with provable convergence to any achievable target classiﬁer.
Revised: Unlike previous model-targeted poisoning attacks, our attack comes with provable convergence to any achievable target classiﬁer.

Original: The distance from the induced classiﬁer to the target classiﬁer is inversely proportional to the square root of the number of poisoning points.
Revised: The distance from the induced classiﬁer to the target classiﬁer is inversely proportional to the square root of the number of poisoning points.

Original: We also provide a lower bound on the minimum number of poisoning points needed to achieve a given target classiﬁer.
Revised: We also provide a lower bound on the minimum number of poisoning points needed to achieve a given target classiﬁer.

Original: Our attack is the ﬁrst model-targeted poisoning attack that provides provable convergence, and in our experiments it either exceeds or matches the best state-of-the-art attacks in terms of attack success rate and distance to the target model.
Revised: Our attack is the ﬁrst model-targeted poisoning attack that provides provable convergence, and in our experiments it either exceeds or matches the best state-of-the-art attacks in terms of attack success rate and distance to the target model.

Original: In addition, as an online attack our attack can incrementally determine nearly optimal poisoning points.
Revised: In addition, as an online attack our attack can incrementally determine nearly optimal poisoning points.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: State-of-the-art machine learning models require a large amount of labeled training data, which often depends on collecting data and labels from untrusted sources.
Revised: State-of-the-art machine learning models require a large amount of labeled training data, which often depends on collecting data and labels from untrusted sources.

Original: A typical application is email spam ﬁltering, where a spam detector ﬁlters out spam messages based on features (e.g., presence of certain words) and periodically updates the model based on newly received emails labeled by users.
Revised: A typical application is email spam ﬁltering, where a spam detector ﬁlters out spam messages based on features (e.g., presence of certain words) and periodically updates the model based on newly received emails labeled by users.

Original: In such a setting, spammers can generate “non-spam” messages by injecting non-related words or benign words, and when models are trained on these “non-spam” messages, the ﬁltering accuracy will drop signiﬁcantly (Lowd & Meek, 2005).
Revised: In such a setting, spammers can generate “non-spam” messages by injecting non-related words or benign words, and when models are trained on these “non-spam” messages, the ﬁltering accuracy will drop signiﬁcantly (Lowd & Meek, 2005).

Original: Such attacks are known as poisoning attacks , and a training process that collects labels or data from untrusted sources is potentially vulnerable to them.
Revised: Such attacks are known as poisoning attacks , and a training process that collects labels or data from untrusted sources is potentially vulnerable to them.

Original: Poisoning attacks can be categorized into objective-driven attacks and model-targeted attacks depending on whether a target model is considered in the attack process.
Revised: Poisoning attacks can be categorized into objective-driven attacks and model-targeted attacks depending on whether a target model is considered in the attack process.

Original: Objective-driven attacks have a speciﬁc attacker objective and aim to achieve the attack objective by generating the poisoning points; model-targeted attacks have a speciﬁc target classiﬁer in mind and aim to induce that target classiﬁer by generating a minimal number of poisoning points.
Revised: Objective-driven attacks have a speciﬁc attacker objective and aim to achieve the attack objective by generating the poisoning points; model-targeted attacks have a speciﬁc target classiﬁer in mind and aim to induce that target classiﬁer by generating a minimal number of poisoning points.

Original: Objective-driven attacks are most commonly studied in the existing literature.
Revised: Objective-driven attacks are most commonly studied in the existing literature.

Original: The attacker objective is typically one of two extremes: indiscriminate attacks, where the adversary’s goal is simply to decrease the overall accuracy of the model (Biggio et al., 2012; Xiao et al., 2012; Mei & Zhu, 2015b;
Revised: The attacker objective is typically one of two extremes: indiscriminate attacks, where the adversary’s goal is simply to decrease the overall accuracy of the model (Biggio et al., 2012; Xiao et al., 2012; Mei & Zhu, 2015b;

Original: Steinhardt et al., 2017; Koh et al., 2018); and instance-targeted attacks, where the goal is to produce a classiﬁer that misclassiﬁes a particular known input (Shafahi et al., 2018; Zhu et al., 2019; Koh & Liang, 2017).
Revised: Steinhardt et al., 2017; Koh et al., 2018); and instance-targeted attacks, where the goal is to produce a classiﬁer that misclassiﬁes a particular known input (Shafahi et al., 2018; Zhu et al., 2019; Koh & Liang, 2017).

Original: Recently,
Revised: Recently,

Original: Jagielski et al.
Revised: Jagielski et al.

Original: (2019) introduced a more realistic attacker objective known as a subpopulation attack, where the goal is to increase the error rate or obtain a particular output for a deﬁned subpopulation of the data distribution.
Revised: (2019) introduced a more realistic attacker objective known as a subpopulation attack, where the goal is to increase the error rate or obtain a particular output for a deﬁned subpopulation of the data distribution.

Original: Attacker objectives for realistic attacks are diverse and designing a uniﬁed and effective attack strategy for different attacker objectives is hard.
Revised: Attacker objectives for realistic attacks are diverse and designing a uniﬁed and effective attack strategy for different attacker objectives is hard.

Original: Gradient-based local optimization is most commonly used to construct poisoning points for a particular attacker objective (Biggio et al., 2012; Xiao et al., 2012; Mei & Zhu, 2015b; Koh & Liang, 2017; Shafahi et al., 2018; Zhu et al., 2019).
Revised: Gradient-based local optimization is most commonly used to construct poisoning points for a particular attacker objective (Biggio et al., 2012; Xiao et al., 2012; Mei & Zhu, 2015b; Koh & Liang, 2017; Shafahi et al., 2018; Zhu et al., 2019).

Original: Although these attacks can be modiﬁed to ﬁt other attacker objectives, since they are based on local optimization techniques they can easily get stuck into bad local optima and fail to ﬁnd effective sets of poisoning points (Steinhardt et al., 2017; Koh et al., 2018).
Revised: Although these attacks can be modiﬁed to ﬁt other attacker objectives, since they are based on local optimization techniques they can easily get stuck into bad local optima and fail to ﬁnd effective sets of poisoning points (Steinhardt et al., 2017; Koh et al., 2018).

Original: To circumvent the issue of local optima,
Revised: To circumvent the issue of local optima,

Original: Steinhardt et al.
Revised: Steinhardt et al.

Original: (2017) formulate an indiscriminate attack as a min-max optimization problem and solve it efﬁciently using online convex optimization techniques.
Revised: (2017) formulate an indiscriminate attack as a min-max optimization problem and solve it efﬁciently using online convex optimization techniques.

Original: However, the strong min-max attack only applies to the indiscriminate setting.
Revised: However, the strong min-max attack only applies to the indiscriminate setting.

Original: In contrast, model-targeted attacks incorporate the attacker objective into a target model and hence, the target model can reﬂect any attacker objective.
Revised: In contrast, model-targeted attacks incorporate the attacker objective into a target model and hence, the target model can reﬂect any attacker objective.

Original: Thus, the same model-targeted attack methods can be directly applied to a range of indiscriminate and subpopulation attacks just by ﬁnding a suitable target model.
Revised: Thus, the same model-targeted attack methods can be directly applied to a range of indiscriminate and subpopulation attacks just by ﬁnding a suitable target model.

Original: Mei & Zhu (2015b)
Revised: Mei & Zhu (2015b)

Original: ﬁrst introduced a target model into a poisoning attack, but their attack is still based on gradient-based local optimization techniques and suffers from bad local optima (Steinhardt et al., 2017; Koh et al., 2018).
Revised: ﬁrst introduced a target model into a poisoning attack, but their attack is still based on gradient-based local optimization techniques and suffers from bad local optima (Steinhardt et al., 2017; Koh et al., 2018).

Original: Koh et al.
Revised: Koh et al.

Original: (2018) proposed the KKT attack, which converts the complicated bi-level optimization into a simple convex optimization problem utilizing the KKT condition, avoiding the local optima issues.
Revised: (2018) proposed the KKT attack, which converts the complicated bi-level optimization into a simple convex optimization problem utilizing the KKT condition, avoiding the local optima issues.

Original: However, their attack only works for margin based losses and does not provide any guarantee on the number of poisoning points required to converge to the target classiﬁer.
Revised: However, their attack only works for margin based losses and does not provide any guarantee on the number of poisoning points required to converge to the target classiﬁer.

Original: In this work, we focus on model-targeted attacks and aim to understand the feasibility of a poisoning adversary to induce any target model.
Revised: In this work, we focus on model-targeted attacks and aim to understand the feasibility of a poisoning adversary to induce any target model.

Original: In particular, we ﬁnd both theoretical and empirical bounds on the sufﬁcient (and necessary) number of poisoning points to get close to a speciﬁc target classier.
Revised: In particular, we ﬁnd both theoretical and empirical bounds on the sufﬁcient (and necessary) number of poisoning points to get close to a speciﬁc target classier.

Original: Contributions.
Revised: Contributions.

Original: Our main contributions involve developing a principled and general model-targeted poisoning attack strategy, along with a proof that the model it induces converges to the target model.
Revised: Our main contributions involve developing a principled and general model-targeted poisoning attack strategy, along with a proof that the model it induces converges to the target model.

Original: Our poisoning method takes as input a target model, and produces a set of poisoning points.
Revised: Our poisoning method takes as input a target model, and produces a set of poisoning points.

Original: We prove that the model induced by training on the original training data with these points added, converges to the target classiﬁer as the number of poison points increases, given that the loss function is convex and proper regularization is adopted in training (Theorem 4.1).
Revised: We prove that the model induced by training on the original training data with these points added, converges to the target classiﬁer as the number of poison points increases, given that the loss function is convex and proper regularization is adopted in training (Theorem 4.1).

Original: Previous model-targeted attacks lack of such convergence guarantees.
Revised: Previous model-targeted attacks lack of such convergence guarantees.

Original: We then prove a lower bound on the minimum number of poisoning points needed to reach the target model (Theorem 4.2), given that the loss function for empirical risk minimization is convex.
Revised: We then prove a lower bound on the minimum number of poisoning points needed to reach the target model (Theorem 4.2), given that the loss function for empirical risk minimization is convex.

Original: Such a lower bound can be used to estimate the optimality of model-targeted poisoning attacks and also indicate the intrinsic hardness of attacking different targets.
Revised: Such a lower bound can be used to estimate the optimality of model-targeted poisoning attacks and also indicate the intrinsic hardness of attacking different targets.

Original: Our attack is also efﬁcient in incremental poisoning scenario as it works in an online fashion and can incrementally ﬁnd poisoning points that are nearly optimal.
Revised: Our attack is also efﬁcient in incremental poisoning scenario as it works in an online fashion and can incrementally ﬁnd poisoning points that are nearly optimal.

Original: Previous model-targeted attacks work with ﬁxed number of poisoning points and need to know the poisoning budget in advance.
Revised: Previous model-targeted attacks work with ﬁxed number of poisoning points and need to know the poisoning budget in advance.

Original: We run experiments to compare our attack to the state-of-the-art model-targeted attack (Koh et al., 2018).
Revised: We run experiments to compare our attack to the state-of-the-art model-targeted attack (Koh et al., 2018).

Original: We ﬁrst evaluate the convergence our attack to the target model and ﬁnd that, under same number of poisoning points, classiﬁers induced by our attack are closer to the target models than the best known attack, for all the target classiﬁers we tried.
Revised: We ﬁrst evaluate the convergence our attack to the target model and ﬁnd that, under same number of poisoning points, classiﬁers induced by our attack are closer to the target models than the best known attack, for all the target classiﬁers we tried.

Original: Then, we evaluate the success rate of our attack, and ﬁnd that it has superior performance than the state-of-the-art in the more realistic subpopulation attack scenario, and comparable performance in the conventional indiscriminate attack scenario (Section 5).
Revised: Then, we evaluate the success rate of our attack, and ﬁnd that it has superior performance than the state-of-the-art in the more realistic subpopulation attack scenario, and comparable performance in the conventional indiscriminate attack scenario (Section 5).

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: The poisoning attack proposed in this paper applies to multi-class prediction tasks or regression problems (by treating the response variable as an additional data feature), but for simplicity of presentation we consider a binary prediction task, h : X → Y , where X ⊆ R d and Y = { +1 , − 1 } .
Revised: The poisoning attack proposed in this paper applies to multi-class prediction tasks or regression problems (by treating the response variable as an additional data feature), but for simplicity of presentation we consider a binary prediction task, h : X → Y , where X ⊆ R d and Y = { +1 , − 1 } .

Original: The prediction model h is characterized by parameters θ ∈ Θ ⊆ R d .
Revised: The prediction model h is characterized by parameters θ ∈ Θ ⊆ R d .

Original: We deﬁne the non-negative convex loss on an individual point, ( x, y ) , as l ( θ ; x, y )
Revised: We deﬁne the non-negative convex loss on an individual point, ( x, y ) , as l ( θ ; x, y )

Original: (e.g., hinge loss for SVM model).
Revised: (e.g., hinge loss for SVM model).

Original: We also deﬁne the empirical loss over a set of points A as L ( θ ; A )
Revised: We also deﬁne the empirical loss over a set of points A as L ( θ ; A )

Original: = (cid:80) ( x,y
Revised: = (cid:80) ( x,y

Original: ) ∈
Revised: ) ∈

Original: A l ( θ ; x, y ) .
Revised: A l ( θ ; x, y ) .

Original: We adopt the game-theoretic formalization of the poisoning attack process from Steinhardt et al.
Revised: We adopt the game-theoretic formalization of the poisoning attack process from Steinhardt et al.

Original: (2017) to describe our model-targeted attack scenario:
Revised: (2017) to describe our model-targeted attack scenario:

Original: N data points are drawn uniformly at random from the true data distribution over X × Y and form the clean training set, D c . 2.
Revised: N data points are drawn uniformly at random from the true data distribution over X × Y and form the clean training set, D c . 2.

Original: The adversary, with knowledge of D c , the model training process and the model space Θ , generates a target classiﬁer θ p ∈ Θ that satisﬁes the attack goal.
Revised: The adversary, with knowledge of D c , the model training process and the model space Θ , generates a target classiﬁer θ p ∈ Θ that satisﬁes the attack goal.

Original: The adversary produces a set of poisoning points, D p , with the knowledge of D c , model training process, Θ and θ p . 4.
Revised: The adversary produces a set of poisoning points, D p , with the knowledge of D c , model training process, Θ and θ p . 4.

Original: Model builder trains the model on D c ∪ D p and produces a classiﬁer, θ atk .
Revised: Model builder trains the model on D c ∪ D p and produces a classiﬁer, θ atk .

Original: The adversary’s goal is that the induced classiﬁer, θ atk , is close to the desired target classiﬁer, θ p (Section 4.2 discusses how this distance is measured).
Revised: The adversary’s goal is that the induced classiﬁer, θ atk , is close to the desired target classiﬁer, θ p (Section 4.2 discusses how this distance is measured).

Original: Step 2 corresponds to the target classiﬁer generation process.
Revised: Step 2 corresponds to the target classiﬁer generation process.

Original: Our attack works for any target classiﬁer, and in the paper we do not focus on the question of how to ﬁnd the best target classiﬁer to achieve a particular adversarial goal but simply adopt the heuristic target classiﬁer generation process from Koh et al.
Revised: Our attack works for any target classiﬁer, and in the paper we do not focus on the question of how to ﬁnd the best target classiﬁer to achieve a particular adversarial goal but simply adopt the heuristic target classiﬁer generation process from Koh et al.

Original: Step 3 corresponds to our model-targeted poisoning attack, and is also the main contribution of the paper.
Revised: Step 3 corresponds to our model-targeted poisoning attack, and is also the main contribution of the paper.

Original: We assume the model builder trains a model through empirical risk minimization (ERM) and the training process details are known to the attacker:
Revised: We assume the model builder trains a model through empirical risk minimization (ERM) and the training process details are known to the attacker:

Original: [Equation]
Revised: [Equation]

Original: where R ( θ ) is the regularization function (e.g., 12 (cid:107) θ
Revised: where R ( θ ) is the regularization function (e.g., 12 (cid:107) θ

Original: [Equation]
Revised: [Equation]

Original: 22 for SVM model).
Revised: 22 for SVM model).

Original: Threat Model.
Revised: Threat Model.

Original: We assume an adversary with full knowledge of training data, model space and training process.
Revised: We assume an adversary with full knowledge of training data, model space and training process.

Original: Although this may be unrealistic for many scenarios, this setting allows us to focus on a particular aspect of poisoning attacks, and is the setting used in many prior works (Biggio et al., 2011; Mei & Zhu, 2015b;
Revised: Although this may be unrealistic for many scenarios, this setting allows us to focus on a particular aspect of poisoning attacks, and is the setting used in many prior works (Biggio et al., 2011; Mei & Zhu, 2015b;

Original: Steinhardt et al., 2017; Koh et al., 2018; Shafahi et al., 2018).
Revised: Steinhardt et al., 2017; Koh et al., 2018; Shafahi et al., 2018).

Original: We assume an addition-only attack where the attacker only adds poisoning points into the clean training set.
Revised: We assume an addition-only attack where the attacker only adds poisoning points into the clean training set.

Original: A stronger attacker may be able to modify or remove existing points, but this typically requires administrative access to the system.
Revised: A stronger attacker may be able to modify or remove existing points, but this typically requires administrative access to the system.

Original: The added points are unconstrained, other than being value elements of the input space.
Revised: The added points are unconstrained, other than being value elements of the input space.

Original: They can have arbitrary features and labels, which enables us to perform the worst case analysis on the robustness of models against addition-only poisoning attacks.
Revised: They can have arbitrary features and labels, which enables us to perform the worst case analysis on the robustness of models against addition-only poisoning attacks.

Original: Although some previous works also allow arbitrary selection of the poisoning points (Biggio et al., 2011; Mei &
Revised: Although some previous works also allow arbitrary selection of the poisoning points (Biggio et al., 2011; Mei &

Original: Zhu, 2015b; Steinhardt et al., 2017; Koh et al., 2018), others put different restrictions on the poisoning appoints.
Revised: Zhu, 2015b; Steinhardt et al., 2017; Koh et al., 2018), others put different restrictions on the poisoning appoints.

Original: A clean-label attack assumes adversaries can only perturb the features of the data, but the label is given by an oracle labeler (Koh & Liang, 2017;
Revised: A clean-label attack assumes adversaries can only perturb the features of the data, but the label is given by an oracle labeler (Koh & Liang, 2017;

Original: Shafahi et al., 2018; Zhu et al., 2019; Huang et al., 2020).
Revised: Shafahi et al., 2018; Zhu et al., 2019; Huang et al., 2020).

Original: In label-ﬂipping attacks, adversaries are only allowed to change the labels (Biggio et al., 2011; Xiao et al., 2012; 2015; Jagielski et al., 2019).
Revised: In label-ﬂipping attacks, adversaries are only allowed to change the labels (Biggio et al., 2011; Xiao et al., 2012; 2015; Jagielski et al., 2019).

Original: These restricted attacks are weaker than the poisoning attacks without restrictions (Koh et al., 2018;
Revised: These restricted attacks are weaker than the poisoning attacks without restrictions (Koh et al., 2018;

Original: Hong et al., 2020).
Revised: Hong et al., 2020).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The most commonly used poisoning strategy is gradient-based attack.
Revised: The most commonly used poisoning strategy is gradient-based attack.

Original: Gradient-based attacks iteratively modify a candidate poisoning point (ˆ x, ˆ y ) in the set D p based on the test loss deﬁned on ˆ x (keeping ˆ y ﬁxed).
Revised: Gradient-based attacks iteratively modify a candidate poisoning point (ˆ x, ˆ y ) in the set D p based on the test loss deﬁned on ˆ x (keeping ˆ y ﬁxed).

Original: This kind of attack was ﬁrst studied on SVM models (Biggio et al., 2012), and later extended to linear and logistic regression (Mei & Zhu, 2015b), and recently to larger neural network models (Koh & Liang, 2017; Yang et al., 2017; Muñoz-González et al., 2017; Shafahi et al., 2018; Zhu et al., 2019; Huang et al., 2020).
Revised: This kind of attack was ﬁrst studied on SVM models (Biggio et al., 2012), and later extended to linear and logistic regression (Mei & Zhu, 2015b), and recently to larger neural network models (Koh & Liang, 2017; Yang et al., 2017; Muñoz-González et al., 2017; Shafahi et al., 2018; Zhu et al., 2019; Huang et al., 2020).

Original: Jagielski et al.
Revised: Jagielski et al.

Original: (2018) also studied gradient attacks and principled defenses on linear regression tasks.
Revised: (2018) also studied gradient attacks and principled defenses on linear regression tasks.

Original: Their work studies linear regression while in this paper, we mainly focus on binary classiﬁcation, although our attack can also be extended to regression tasks.
Revised: Their work studies linear regression while in this paper, we mainly focus on binary classiﬁcation, although our attack can also be extended to regression tasks.

Original: More importantly, our attack aims to induce a target model by generating poisoning points while Jagielski et al.
Revised: More importantly, our attack aims to induce a target model by generating poisoning points while Jagielski et al.

Original: (2018)’s attack tries to increase the Mean Squared Error of the linear regression task with a ﬁxed poisoning budget.
Revised: (2018)’s attack tries to increase the Mean Squared Error of the linear regression task with a ﬁxed poisoning budget.

Original: In addition to classiﬁcation and regression tasks, gradient-based poisoning attacks are also applied to topic modeling (Mei & Zhu, 2015a), collaborative ﬁltering (Li et al., 2016) and algorithmic fairness (Solans et al., 2020).
Revised: In addition to classiﬁcation and regression tasks, gradient-based poisoning attacks are also applied to topic modeling (Mei & Zhu, 2015a), collaborative ﬁltering (Li et al., 2016) and algorithmic fairness (Solans et al., 2020).

Original: Besides the gradient-based attacks, researchers also utilize generative adversarial networks to craft poisoning points efﬁciently for larger neural networks, however, the effectiveness of the attack is limited (Yang et al., 2017; Muñoz-González et al., 2019).
Revised: Besides the gradient-based attacks, researchers also utilize generative adversarial networks to craft poisoning points efﬁciently for larger neural networks, however, the effectiveness of the attack is limited (Yang et al., 2017; Muñoz-González et al., 2019).

Original: The strongest attacks so far are the
Revised: The strongest attacks so far are the

Original: KKT attack (Koh et al., 2018) and the min-max attack (Steinhardt et al., 2017; Koh et al., 2018).
Revised: KKT attack (Koh et al., 2018) and the min-max attack (Steinhardt et al., 2017; Koh et al., 2018).

Original: However, the KKT attack cannot scale well for multi-class classiﬁcation and is limited to margin based losses (Koh et al., 2018).
Revised: However, the KKT attack cannot scale well for multi-class classiﬁcation and is limited to margin based losses (Koh et al., 2018).

Original: The min-max attack only works for indiscriminate attack setting, but additionally provides a certiﬁcate on worst case test loss for a ﬁxed number of poisoning points.
Revised: The min-max attack only works for indiscriminate attack setting, but additionally provides a certiﬁcate on worst case test loss for a ﬁxed number of poisoning points.

Original: We are also inspired by Steinhardt et al.
Revised: We are also inspired by Steinhardt et al.

Original: (2017) to adopt online convex optimization to instantiate our model-targeted attack, but now deals more general attack scenario.
Revised: (2017) to adopt online convex optimization to instantiate our model-targeted attack, but now deals more general attack scenario.

Original: We also distinguish ourselves from the poisoning attack against online learning (Wang & Chaudhuri, 2018).
Revised: We also distinguish ourselves from the poisoning attack against online learning (Wang & Chaudhuri, 2018).

Original: The attack against online learning considers a setting where training data arrives in a streaming manner while we consider the ofﬂine setting with training data being ﬁxed.
Revised: The attack against online learning considers a setting where training data arrives in a streaming manner while we consider the ofﬂine setting with training data being ﬁxed.

Original: Another line of work studies “targeted” poisoning attacks where an adversary gaurantees to increase the probability of an arbitrary “bad” property (Mahloujifar et al., 2019a; 2017; 2019b), as long as that property has some non-negligible chance of naturally happening.
Revised: Another line of work studies “targeted” poisoning attacks where an adversary gaurantees to increase the probability of an arbitrary “bad” property (Mahloujifar et al., 2019a; 2017; 2019b), as long as that property has some non-negligible chance of naturally happening.

Original: These attacks cannot be applied in the model-targeted setting as the probability of naturally producing a speciﬁc target model is often negligible.
Revised: These attacks cannot be applied in the model-targeted setting as the probability of naturally producing a speciﬁc target model is often negligible.

Original: Related to our
Revised: Related to our

Original: Theorem 4.2, Ma et al.
Revised: Theorem 4.2, Ma et al.

Original: (2019) also derived a lower bound on number of poisoning points (to induce a target model), but their lower bound only applies when differential privacy is deployed during the model training process (and hence hurts model utility), which is different from our problem setting.
Revised: (2019) also derived a lower bound on number of poisoning points (to induce a target model), but their lower bound only applies when differential privacy is deployed during the model training process (and hence hurts model utility), which is different from our problem setting.

Original:  P OISONING A TTACK WITH A T ARGET M ODEL
Revised:  P OISONING A TTACK WITH A T ARGET M ODEL

Original: Our new poisoning attack determines a target model and selects poisoning points to achieve that target model.
Revised: Our new poisoning attack determines a target model and selects poisoning points to achieve that target model.

Original: The target model generation is not our focus and we adopt the heuristic approach proposed by Koh et al.
Revised: The target model generation is not our focus and we adopt the heuristic approach proposed by Koh et al.

Original: For the new poisoning attack, ﬁrst, we show the algorithm that generates the poisoning points in Section 4.1 and then prove that the generated poisoning points, once added to the clean data, can produce a classiﬁer that asymptotically converges to the target classiﬁer in Section 4.2.
Revised: For the new poisoning attack, ﬁrst, we show the algorithm that generates the poisoning points in Section 4.1 and then prove that the generated poisoning points, once added to the clean data, can produce a classiﬁer that asymptotically converges to the target classiﬁer in Section 4.2.

Original: M ODEL -T ARGETED P OISONING WITH O NLINE L EARNING
Revised: M ODEL -T ARGETED P OISONING WITH O NLINE L EARNING

Original: The main idea of our model-targeted poisoning attack, as outlined in Algorithm 1, is to sequentially add a point into the training set that have maximum loss difference between the intermediate model obtained so far and the target model, and by training models on the updated training set, we actually minimize the gap in the loss of the intermediate classiﬁer and the target classiﬁer.
Revised: The main idea of our model-targeted poisoning attack, as outlined in Algorithm 1, is to sequentially add a point into the training set that have maximum loss difference between the intermediate model obtained so far and the target model, and by training models on the updated training set, we actually minimize the gap in the loss of the intermediate classiﬁer and the target classiﬁer.

Original: Repeating the process then eventually generates classiﬁers that have similar loss distribution as the target classiﬁer.
Revised: Repeating the process then eventually generates classiﬁers that have similar loss distribution as the target classiﬁer.

Original: We show in Section 4.2 why similar loss distribution implies convergence.
Revised: We show in Section 4.2 why similar loss distribution implies convergence.

Original: Input: D c , the loss functions ( L and l ), θ p
Revised: Input: D c , the loss functions ( L and l ), θ p

Original: Output: D p 1: D p = ∅ do 3: θ t =
Revised: Output: D p 1: D p = ∅ do 3: θ t =

Original: arg min L ( θ ; D c ∪ D p ) 4: ( x ∗ , y ∗ ) =
Revised: arg min L ( θ ; D c ∪ D p ) 4: ( x ∗ , y ∗ ) =

Original: arg max
Revised: arg max

Original: X×Y l ( θ t ; x, y ) − l
Revised: X×Y l ( θ t ; x, y ) − l

Original: ( θ p ; x, y )
Revised: ( θ p ; x, y )

Original: D p = p ∪ { ( x ∗ , y ∗ ) }
Revised: D p = p ∪ { ( x ∗ , y ∗ ) }

Original: Algorithm 1 requires the input of clean training set D c , the Loss function ( L for set of points and l for individual point) and the target model θ p .
Revised: Algorithm 1 requires the input of clean training set D c , the Loss function ( L for set of points and l for individual point) and the target model θ p .

Original: The output from Algorithm 1 will be the set of poisoning points D p .
Revised: The output from Algorithm 1 will be the set of poisoning points D p .

Original: The algorithm is simple: ﬁrst, adversaries train the intermediate model θ t on the mixture of clean and poisoning points D c ∪ D p with D p an empty set in ﬁrst iteration (Line 3).
Revised: The algorithm is simple: ﬁrst, adversaries train the intermediate model θ t on the mixture of clean and poisoning points D c ∪ D p with D p an empty set in ﬁrst iteration (Line 3).

Original: The adversary then searches for the point that maximizes the loss difference between θ t and θ p (Line 4).
Revised: The adversary then searches for the point that maximizes the loss difference between θ t and θ p (Line 4).

Original: After the point of maximum loss difference is found, it is added to the poisoning set D p
Revised: After the point of maximum loss difference is found, it is added to the poisoning set D p

Original: (Line 5).
Revised: (Line 5).

Original: The whole process repeats until the stop condition is satisﬁed in Line 2.
Revised: The whole process repeats until the stop condition is satisﬁed in Line 2.

Original: The stop condition is ﬂexible and it can take various forms: 1) adversary has a budget T on number of poisoning points, and the algorithm halts when the algorithm runs for T iterations; 2) the intermediate classiﬁer θ t is closer to the target classiﬁer (than a preset threshold (cid:15) ) in terms of the maximum loss difference, and more details regarding this distance metric will be introduced in Section 4.2; 3) adversary has some requirement on the accuracy and the algorithm terminates when θ t satisﬁes the accuracy requirement.
Revised: The stop condition is ﬂexible and it can take various forms: 1) adversary has a budget T on number of poisoning points, and the algorithm halts when the algorithm runs for T iterations; 2) the intermediate classiﬁer θ t is closer to the target classiﬁer (than a preset threshold (cid:15) ) in terms of the maximum loss difference, and more details regarding this distance metric will be introduced in Section 4.2; 3) adversary has some requirement on the accuracy and the algorithm terminates when θ t satisﬁes the accuracy requirement.

Original: Since we focus on producing a classiﬁer close to the target model, we adopt the second stop criterion that measures the distance with respect to the maximum loss difference, and report results based on this criterion in Section 5.
Revised: Since we focus on producing a classiﬁer close to the target model, we adopt the second stop criterion that measures the distance with respect to the maximum loss difference, and report results based on this criterion in Section 5.

Original: A nice property of Algorithm 1 is that the classiﬁer θ atk trained on D c ∪D p is close to the target model θ p and asymptotically converges to θ p .
Revised: A nice property of Algorithm 1 is that the classiﬁer θ atk trained on D c ∪D p is close to the target model θ p and asymptotically converges to θ p .

Original: Details of the convergence will be shown in the next section.
Revised: Details of the convergence will be shown in the next section.

Original: The algorithm may appear to be slow, particularly for larger models due to requirement of repeatedly training a model in line 3.
Revised: The algorithm may appear to be slow, particularly for larger models due to requirement of repeatedly training a model in line 3.

Original: However, this is not an issue.
Revised: However, this is not an issue.

Original: First, as will be shown in next section, the algorithm is an online optimization process and line 3 corresponds to solving the online optimization problem exactly.
Revised: First, as will be shown in next section, the algorithm is an online optimization process and line 3 corresponds to solving the online optimization problem exactly.

Original: However, people often use the very efﬁcient online gradient descent method to approximately solve the problem and its asymptotic performance is the same (Shalev-Shwartz, 2012).
Revised: However, people often use the very efﬁcient online gradient descent method to approximately solve the problem and its asymptotic performance is the same (Shalev-Shwartz, 2012).

Original: Second, if we solve the optimization problem exactly, we can add multiple copies of ( x ∗ , y ∗ ) into
Revised: Second, if we solve the optimization problem exactly, we can add multiple copies of ( x ∗ , y ∗ ) into

Original: D p each time.
Revised: D p each time.

Original: This reduces the overall iteration number, and hence reduces the number of times retraining models.
Revised: This reduces the overall iteration number, and hence reduces the number of times retraining models.

Original: The proof of convergence will be similar.
Revised: The proof of convergence will be similar.

Original: For simplicity in interpreting the results, we do not use this in our experiments and add only one copy of ( x ∗ , y ∗ ) each iteration.
Revised: For simplicity in interpreting the results, we do not use this in our experiments and add only one copy of ( x ∗ , y ∗ ) each iteration.

Original: However, we also tested the performance by adding two copies of ( x ∗ , y ∗ ) and ﬁnd that the attack results are nearly the same while the efﬁciency is improved signiﬁcantly.
Revised: However, we also tested the performance by adding two copies of ( x ∗ , y ∗ ) and ﬁnd that the attack results are nearly the same while the efﬁciency is improved signiﬁcantly.

Original: For example, for the experiments we tried on
Revised: For example, for the experiments we tried on

Original: MNIST-17 dataset, by adding 2 copies of points, with same number of poisoning points, the attack success rate decreases at most by 0.7% while the execution time is reduced approximately by half.
Revised: MNIST-17 dataset, by adding 2 copies of points, with same number of poisoning points, the attack success rate decreases at most by 0.7% while the execution time is reduced approximately by half.

Original: 4. C ONVERGENCE OF O UR P OISONING A TTACK
Revised: 4. C ONVERGENCE OF O UR P OISONING A TTACK

Original: Before proving the convergence of Algorithm 1, we need to measure the distance of the model θ atk trained on D c ∪ D p to the target model θ p .
Revised: Before proving the convergence of Algorithm 1, we need to measure the distance of the model θ atk trained on D c ∪ D p to the target model θ p .

Original: First, we deﬁne a general closeness measure based on their prediction performance which we will use to state our convergence theorem:
Revised: First, we deﬁne a general closeness measure based on their prediction performance which we will use to state our convergence theorem:

Original: Deﬁnition 1 (Loss-based distance and (cid:15) -close) .
Revised: Deﬁnition 1 (Loss-based distance and (cid:15) -close) .

Original: For two models θ 1 and θ 2 , a space X × Y and a loss function l ( θ ; x, y ) , we deﬁne loss-based distance D l, X , Y :
Revised: For two models θ 1 and θ 2 , a space X × Y and a loss function l ( θ ; x, y ) , we deﬁne loss-based distance D l, X , Y :

Original: Θ × Θ → R as
Revised: Θ × Θ → R as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and we say model θ 1 is (cid:15) -close to model θ 2 when the loss-based distance from θ 1 to θ 2 is upper bounded by (cid:15) .
Revised: and we say model θ 1 is (cid:15) -close to model θ 2 when the loss-based distance from θ 1 to θ 2 is upper bounded by (cid:15) .

Original: Why is loss-based distance a meaningful notion of closeness?
Revised: Why is loss-based distance a meaningful notion of closeness?

Original: We argue that this notion captures the “behavorial” distance between two models.
Revised: We argue that this notion captures the “behavorial” distance between two models.

Original: Namely, if θ 1 is (cid:15) -close (as measured by loss-based distance) to θ 2 and vice versa, then θ 1 and θ 2 would have almost equal loss on all the points, meaning that they have almost the same behavior across all the space.
Revised: Namely, if θ 1 is (cid:15) -close (as measured by loss-based distance) to θ 2 and vice versa, then θ 1 and θ 2 would have almost equal loss on all the points, meaning that they have almost the same behavior across all the space.

Original: Note that our  deﬁnition of  distance does not have the symmetry property of metrics and hence is not a metric.
Revised: Note that our general deﬁnition of loss based distance does not have the symmetry property of metrics and hence is not a metric.

Original: However, it has some other properties of metrics in the space of attainable models.
Revised: However, it has some other properties of metrics in the space of attainable models.

Original: For example, if some model θ is attainable using ERM, no model could have negative distance to it.
Revised: For example, if some model θ is attainable using ERM, no model could have negative distance to it.

Original: To further show the value of this distance notion, in Appendix B we demonstrate an O ( (cid:15) ) upper bound on the (cid:96) 1 norm of difference between two models that are (cid:15) -close with respect to loss-based distance for the special case of Hinge loss. 
Revised: To further show the value of this distance notion, in Appendix B we demonstrate an O ( (cid:15) ) upper bound on the (cid:96) 1 norm of difference between two models that are (cid:15) -close with respect to loss-based distance for the special case of Hinge loss. For Hinge loss, it also satisﬁes the bi-directional closeness , that is if θ 1 is (cid:15) -close to θ 2 , then θis O ( (cid:15) ) -close to θ (details can be found in Corollary B.2.1), and the proof details can be found in

Original:  In the rest of the paper, we will use terms (cid:15) -close or (cid:15) -closeness to denote that a model is (cid:15) away from another model based on the loss-based distance.
Revised: Appendix B. In the rest of the paper, we will use terms (cid:15) -close or (cid:15) -closeness to denote that a model is (cid:15) away from another model based on the loss-based distance.

Original: Our convergence theorem uses the loss-based distance to establish that the attack of Algorithmconverges to the target classiﬁer:
Revised: Our convergence theorem uses the loss-based distance to establish that the attack of Algorithmconverges to the target classiﬁer:

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: After at most T steps, Algorithm 1 will produce the poisoning set D p and the classiﬁer trained on D c ∪ D p is (cid:15) -close to θ p , with respect to loss-based distance, D l, X , Y , for
Revised: After at most T steps, Algorithm 1 will produce the poisoning set D p and the classiﬁer trained on D c ∪ D p is (cid:15) -close to θ p , with respect to loss-based distance, D l, X , Y , for

Original: [Equation]
Revised: [Equation]

Original: where, γ is a constant for a given θ p and classiﬁcation task, and α ( T ) is the regret of the online algorithm when the loss function used for training is convex.
Revised: where, γ is a constant for a given θ p and classiﬁcation task, and α ( T ) is the regret of the online algorithm when the loss function used for training is convex.

Original: Remark 1.
Revised: Remark 1.

Original: Online learning algorithms with sublinear regret bound can be applied to show the convergence.
Revised: Online learning algorithms with sublinear regret bound can be applied to show the convergence.

Original: Here, we adopt results from McMahan (2017).
Revised: Here, we adopt results from McMahan (2017).

Original: Speciﬁcally, α ( T ) is in the order of
Revised: Speciﬁcally, α ( T ) is in the order of

Original: O (log T ) ) and we have (cid:15) ≤
Revised: O (log T ) ) and we have (cid:15) ≤

Original: O ( log TT ) when the loss function is additionally Lipschitz continuous and the regularizer R ( θ ) is strongly convex, and (cid:15) → 0 when T → + ∞ .
Revised: O ( log TT ) when the loss function is additionally Lipschitz continuous and the regularizer R ( θ ) is strongly convex, and (cid:15) → 0 when T → + ∞ .

Original: α ( T ) is also in the order of
Revised: α ( T ) is also in the order of

Original: O (log T ) when the loss function used for training is strongly convex and the regularizer is convex.
Revised: O (log T ) when the loss function used for training is strongly convex and the regularizer is convex.

Original: Proof idea.
Revised: Proof idea.

Original: The full proof of Theorem 4.1 is in Appendix A.
Revised: The full proof of Theorem 4.1 is in Appendix A.

Original: Here, we only summarize the high level proof idea.
Revised: Here, we only summarize the high level proof idea.

Original: The key idea is to frame the poisoning problem as an online learning problem.
Revised: The key idea is to frame the poisoning problem as an online learning problem.

Original: In this formulation, each step of the online learning problem corresponds to the i th poison point ( x i , y i ) .
Revised: In this formulation, each step of the online learning problem corresponds to the i th poison point ( x i , y i ) .

Original: In particular, the loss function at iteration i of the online learning problem is set to l ( · ; x i , y i ) .
Revised: In particular, the loss function at iteration i of the online learning problem is set to l ( · ; x i , y i ) .

Original: Then, we show that by deﬁning the parameters of the online learning problem in a careful way, the output of the follow-the-leader (FTL) algorithm (Shalev-Shwartz, 2012) at iteration i is a model that is identical to training a model on a dataset consisting of the clean points and the ﬁrst i − 1 poisoning points.
Revised: Then, we show that by deﬁning the parameters of the online learning problem in a careful way, the output of the follow-the-leader (FTL) algorithm (Shalev-Shwartz, 2012) at iteration i is a model that is identical to training a model on a dataset consisting of the clean points and the ﬁrst i − 1 poisoning points.

Original: On the other hand, the way the poisoning points are selected, we can show that at the i th iteration the maximum loss difference between the target model and the best induced model so far would be smaller than the regret of the FTL algorithm divided by the number of poisoning points.
Revised: On the other hand, the way the poisoning points are selected, we can show that at the i th iteration the maximum loss difference between the target model and the best induced model so far would be smaller than the regret of the FTL algorithm divided by the number of poisoning points.

Original: The convergence bound of Theorem 4.1 boils down to regret analysis of the algorithm based on the loss function.
Revised: The convergence bound of Theorem 4.1 boils down to regret analysis of the algorithm based on the loss function.

Original: Since we are assuming the loss function is convex with a strongly convex regularizer (or a strongly convex loss function with a convex regularizer), we can show that the regret is bounded by
Revised: Since we are assuming the loss function is convex with a strongly convex regularizer (or a strongly convex loss function with a convex regularizer), we can show that the regret is bounded by

Original: O (log T ) and hence the loss distance between the induced model and the target model converges to 0.
Revised: O (log T ) and hence the loss distance between the induced model and the target model converges to 0.

Original: Implications of Theorem 4.1 The theorem says that the loss-based distance of the model trained on D c ∪ D p to the target model correlates to the loss difference between the target model and the clean model θ c (trained on D c ) on D c , and correlates inversely with the number of poisoning points.
Revised: Implications of Theorem 4.1 The theorem says that the loss-based distance of the model trained on D c ∪ D p to the target model correlates to the loss difference between the target model and the clean model θ c (trained on D c ) on D c , and correlates inversely with the number of poisoning points.

Original: Therefore, it implies 1) if the target classiﬁer θ p has lower loss on D c , then it is easier to achieve the target model, and 2) with more poisoning points, we get closer to the target classiﬁer and our attack will be more effective.
Revised: Therefore, it implies 1) if the target classiﬁer θ p has lower loss on D c , then it is easier to achieve the target model, and 2) with more poisoning points, we get closer to the target classiﬁer and our attack will be more effective.

Original: The theorem also justiﬁes the motivation behind the heuristic method in Koh et al.
Revised: The theorem also justiﬁes the motivation behind the heuristic method in Koh et al.

Original: (2018) to select a target classiﬁer with lower loss on clean data.
Revised: (2018) to select a target classiﬁer with lower loss on clean data.

Original: For the indiscriminate attack scenario, we also improve the heuristic approach by adaptively updating the model and producing target classiﬁers with much lower loss on the clean set.
Revised: For the indiscriminate attack scenario, we also improve the heuristic approach by adaptively updating the model and producing target classiﬁers with much lower loss on the clean set.

Original: This helps to empirically validate our theorem.
Revised: This helps to empirically validate our theorem.

Original: Details of the original and improved heuristic approach, and relevant experiments are in
Revised: Details of the original and improved heuristic approach, and relevant experiments are in

Original: Appendix D.1.
Revised: Appendix D.1.

Original: 4. L OWER B OUND ON THE N UMBER OF POISONING P OINTS
Revised: 4. L OWER B OUND ON THE N UMBER OF POISONING P OINTS

Original: We ﬁrst provide the lower bound on number of poisoning points required for producing the target classiﬁer in addition only setting (Theorem 4.2), and then explain how the lower bound estimation can be incorporated into Algorithm 1.
Revised: We ﬁrst provide the lower bound on number of poisoning points required for producing the target classiﬁer in addition only setting (Theorem 4.2), and then explain how the lower bound estimation can be incorporated into Algorithm 1.

Original: The intuition behind the theorem below is, when the number of poisoning points added to the clean training set is smaller than the lower bound, there always exists a classiﬁer θ with lower loss compared to θ p and hence the target classiﬁer cannot be attained.
Revised: The intuition behind the theorem below is, when the number of poisoning points added to the clean training set is smaller than the lower bound, there always exists a classiﬁer θ with lower loss compared to θ p and hence the target classiﬁer cannot be attained.

Original: Theorem 4.2 (Lower Bound) .
Revised: Theorem 4.2 (Lower Bound) .

Original: Given a target classiﬁer θ p , to reproduce θ p by adding the poisoning set D p into D c , the number of poisoning points |D p | cannot be lower than 
Revised: Given a target classiﬁer θ p , to reproduce θ p by adding the poisoning set D p into D c , the number of poisoning points |D p | cannot be lower than [Equation]

Revised: Corollary 4.2.1.

Revised: If we further assume bi-directional closeness in the loss based distance, we can also derive the lower bound on number of poisoning points needed to induce models that are (cid:15) -close to the target model.

Revised: More precisely, if θ 1 being (cid:15) -close to θ 2 implies that θ 2 is also k · (cid:15) close to θ 1 , then we have,

Original: [Equation]
Revised: [Equation]

Revised: where R ∗ is an upper bound on the nonnegative regularizer R ( θ ) .

Original: The formula for the lower bound  can be easily incorporated into Algorithm 1 to obtain tighter theoretical lower bound.
Revised: The formula for the lower bound in Theorem 4.2 (and also the lower bound in Corollary 4.2.1) can be easily incorporated into Algorithm 1 to obtain tighter theoretical lower bound.

Original: We simply need to check all of the intermediate classiﬁer θ t produced during the attack process and replace θ with θ t , and the lower bound can be computed for the pair of θ t and θ p .
Revised: We simply need to check all of the intermediate classiﬁer θ t produced during the attack process and replace θ with θ t , and the lower bound can be computed for the pair of θ t and θ p .

Original: Algorithm 1 then additionally returns the lower bound, which is the highest lower bound computed from our poisoning procedure.
Revised: Algorithm 1 then additionally returns the lower bound, which is the highest lower bound computed from our poisoning procedure.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We ﬁrst describe our experimental setup regarding the datasets, models, attacks and target classiﬁers.
Revised: We ﬁrst describe our experimental setup regarding the datasets, models, attacks and target classiﬁers.

Original: Next, we present the experimental results by showing the convergence of Algorithm 1, the comparison of attack success rates to state-of-the-art poisoning attack, and the theoretical lower bound for inducing a given target classiﬁer and its gap to the number of poisoning points used by our attack.
Revised: Next, we present the experimental results by showing the convergence of Algorithm 1, the comparison of attack success rates to state-of-the-art poisoning attack, and the theoretical lower bound for inducing a given target classiﬁer and its gap to the number of poisoning points used by our attack.

Original: We are most interested in subpopulation attacks, since they correspond to the more realistic attacker goal of impacting the classiﬁer outputs for a targeted subpopulation.
Revised: We are most interested in subpopulation attacks, since they correspond to the more realistic attacker goal of impacting the classiﬁer outputs for a targeted subpopulation.

Revised: Therefore, in the main body, we introduce the results of SVM model on the Adult dataset (Dua & Graff, 2017) in the subpopulation poisoning scenario.

Original: For completeness, we also evaluate our attack  in the indiscriminate poisoning scenario but defer details on those experiments to 
Revised: For completeness, we also evaluate our attack on SVM model on MNIST 1dataset in the indiscriminate poisoning scenario but defer details on those experiments to Appendix D.

Original: Appendix D. Our ﬁndings for the indiscriminate attacks are that the attack gradually and consistently converges to the target model in terms of the maximum loss difference and the Euclidean distance to the target, with attack success rates that are comparable to the state-of-the-art attack (unlike the subpopulation attacks, where our attack produces superior results). 
Revised: Our ﬁndings for the indiscriminate attacks are that the attack gradually and consistently converges to the target model in terms of the maximum loss difference and the Euclidean distance to the target, with attack success rates that are comparable to the state-of-the-art attack (unlike the subpopulation attacks, where our attack produces superior results). To further verify the universal effectiveness of our attack, we also evaluate our attack on additional dataset (Dogﬁsh) and model (Logistic regression), and more details can be found in Appendix F.

Original: Dataset, Model and Attacks.
Revised: Dataset, Model and Attacks.

Original: For the subpopulation attack experiments, we use the Adult dataset (Dua & Graff, 2017).
Revised: For the subpopulation attack experiments, we use the Adult dataset (Dua & Graff, 2017).

Original: This dataset was used for evaluation by the ﬁrst subpopulation attack[Figure] paper (Jagielski et al., 2019).
Revised: This dataset was used for evaluation by the ﬁrst subpopulation attack paper (Jagielski et al., 2019).

Original: We downsampled the Adult dataset to ensure it is class-balanced and we ended up having 15,682 training and 7,692 test examples.
Revised: We downsampled the Adult dataset to ensure it is class-balanced and we ended up having 15,682 training and 7,692 test examples.

Original: We conduct experiments on linear
Revised: We conduct experiments on linear

Original: SVM model and compare our model-targeted poisoning attack in Algorithm 1 to the state-of-the-art
Revised: SVM model and compare our model-targeted poisoning attack in Algorithm 1 to the state-of-the-art

Original: KKT attack (Koh et al., 2018).
Revised: KKT attack (Koh et al., 2018).

Original: We do not include the model-targeted attack from Mei & Zhu (2015b) because it underperforms the KKT attack (Koh et al., 2018).
Revised: We do not include the model-targeted attack from Mei & Zhu (2015b) because it underperforms the KKT attack (Koh et al., 2018).

Original: We also do not include objective-driven attacks because our main goal here is to evaluate how well our attack approaches a given target model, across a range of target models.
Revised: We also do not include objective-driven attacks because our main goal here is to evaluate how well our attack approaches a given target model, across a range of target models.

Original: Model-targeted attacks can be compared to objective-driven attacks with regards to a given attacker objective by choosing the target model in a careful way.
Revised: Model-targeted attacks can be compared to objective-driven attacks with regards to a given attacker objective by choosing the target model in a careful way.

Original: We show some heuristics of choosing such target models  in Appendix E.
Revised: We show some heuristics of choosing such target models and comparison to some objective-driven attacks in Appendix E.

Original: Both our attack and the KKT attack take as input a target classiﬁer and the original training data, and output a set of poisoning points selected with the goal that the induced classiﬁer is as close as possible to the target classiﬁer.
Revised: Both our attack and the KKT attack take as input a target classiﬁer and the original training data, and output a set of poisoning points selected with the goal that the induced classiﬁer is as close as possible to the target classiﬁer.

Original: We compare the effectiveness of the attacks in selecting poisoning points that converge to a given target classiﬁer by testing the attacks using the same target model.
Revised: We compare the effectiveness of the attacks in selecting poisoning points that converge to a given target classiﬁer by testing the attacks using the same target model.

Original: The KKT attack requires a target number of poisoning points as an input while our attack is more ﬂexible and can either take a target number of poisoning points or a threshold for (cid:15) -close distance to the target model.
Revised: The KKT attack requires a target number of poisoning points as an input while our attack is more ﬂexible and can either take a target number of poisoning points or a threshold for (cid:15) -close distance to the target model.

Original: Since we do not know the number of poisoning points needed to reach some attacker goal in advance for the KKT attack, we ﬁrst run our attack and produce a classiﬁer that satisﬁes the selected (cid:15) -close distance threshold.
Revised: Since we do not know the number of poisoning points needed to reach some attacker goal in advance for the KKT attack, we ﬁrst run our attack and produce a classiﬁer that satisﬁes the selected (cid:15) -close distance threshold.

Original: The loss function is set as the hinge loss since we target an SVM model in our experiments and we set (cid:15)
Revised: The loss function is set as the hinge loss since we target an SVM model in our experiments and we set (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 01 for all these experiments.
Revised: 01 for all these experiments.

Original: Then, we use the size of the poisoning set returned from our attack (denoted by n p ) as the input to the KKT attack for the target number of poisons needed.
Revised: Then, we use the size of the poisoning set returned from our attack (denoted by n p ) as the input to the KKT attack for the target number of poisons needed.

Original: We also compare the two attacks with varying numbers of poisoning points up to n p .
Revised: We also compare the two attacks with varying numbers of poisoning points up to n p .

Original: For the KKT attack, its entire optimization process must be rerun whenever the target number of poisoning points changes.
Revised: For the KKT attack, its entire optimization process must be rerun whenever the target number of poisoning points changes.

Original: Hence, it is infeasible to evaluate the KKT attack on many different poisoning set sizes.
Revised: Hence, it is infeasible to evaluate the KKT attack on many different poisoning set sizes.

Original: In our experiments, we run the KKT attack ﬁve poisoning set sizes: 0 . 2 · n p , 0 . 4 · n p , 0 . 6 · n p , 0 . 8 · n p , and n p .
Revised: In our experiments, we run the KKT attack ﬁve poisoning set sizes: 0 . 2 · n p , 0 . 4 · n p , 0 . 6 · n p , 0 . 8 · n p , and n p .

Original: In contrast, we simply run our attack for iterations up to the maximum number of poisoning points, collecting a data point for iteration up to n p .
Revised: In contrast, we simply run our attack for iterations up to the maximum number of poisoning points, collecting a data point for iteration up to n p .

Original: Subpopulations.
Revised: Subpopulations.

Original: We identify the subpopulations for the Adult dataset using k -means clustering techniques (ClusterMatch (Jagielski et al., 2019)) to obtain different clusters ( k = 20 in our case).
Revised: We identify the subpopulations for the Adult dataset using k -means clustering techniques (ClusterMatch (Jagielski et al., 2019)) to obtain different clusters ( k = 20 in our case).

Original: For each cluster, we select instances with label “<=50K” to form the subpopulation (indicating all instances in the subpopulation are in low income group).
Revised: For each cluster, we select instances with label “<=50K” to form the subpopulation (indicating all instances in the subpopulation are in low income group).

Original: This way of deﬁning subpopulation is rather arbitrary (in constrast to a more likely attack goal which would select subpopulations based on demographic characteristics), but enables us to simplify the analysis.
Revised: This way of deﬁning subpopulation is rather arbitrary (in constrast to a more likely attack goal which would select subpopulations based on demographic characteristics), but enables us to simplify the analysis.

Original: From the 20 subpopulations obtained, we select three subpopulations with the highest test accuracy on the clean model and they all have 100% test accuracy, indicating all instances in these subpopulations are correctly classiﬁed as low income.
Revised: From the 20 subpopulations obtained, we select three subpopulations with the highest test accuracy on the clean model and they all have 100% test accuracy, indicating all instances in these subpopulations are correctly classiﬁed as low income.

Original: This enables us to use “attack success rate” and “accuracy” without any ambiguity on the subpopulation—for each of our subpopulations, all instances are originally classiﬁed as low income, and the simulated attacker’s goal is to have them classiﬁed as high income.
Revised: This enables us to use “attack success rate” and “accuracy” without any ambiguity on the subpopulation—for each of our subpopulations, all instances are originally classiﬁed as low income, and the simulated attacker’s goal is to have them classiﬁed as high income.

Original: For each subpopulation, we use the heuristic approach from Koh et al.
Revised: For each subpopulation, we use the heuristic approach from Koh et al.

Original: (2018) to generate a target classiﬁer that has 0% accuracy (100% attacker success) on the subpopulation, indicating that all subpopulation instances 
Revised: (2018) to generate a target classiﬁer that has 0% accuracy (100% attacker success) on the subpopulation, indicating that all subpopulation instances are now classiﬁed as high income.

Original: [Figure]  [Table]
Revised: [Figure] [Figure] [Table]

Original: Convergence.
Revised: Convergence.

Original: Figure 1 shows the convergence of Algorithm 1 using both maximum loss difference and Euclidean distance to the target.
Revised: Figure 1 shows the convergence of Algorithm 1 using both maximum loss difference and Euclidean distance to the target.

Original: The maximum number of poisons ( n p ) for the experiments is obtained when the classiﬁer from Algorithm 1 is 0 .
Revised: The maximum number of poisons ( n p ) for the experiments is obtained when the classiﬁer from Algorithm 1 is 0 .

Original: 01 -close to the target classiﬁer.
Revised: 01 -close to the target classiﬁer.

Original: Our attack steadily reduces the maximum loss difference and Euclidean distance to the target model, in contrast to the
Revised: Our attack steadily reduces the maximum loss difference and Euclidean distance to the target model, in contrast to the

Original: KKT attack which does not seem to converge towards the target model reliably.
Revised: KKT attack which does not seem to converge towards the target model reliably.

Original: Concretely, at the maximum number of poisons in Figure 1, both the maximum loss difference and Euclidean distance of our attack (to the target) is less than 2% of the corresponding distances of the KKT attack.
Revised: Concretely, at the maximum number of poisons in Figure 1, both the maximum loss difference and Euclidean distance of our attack (to the target) is less than 2% of the corresponding distances of the KKT attack.

Original: Attack Success.
Revised: Attack Success.

Original: Next, we compare the classiﬁers induced by the two attacks in terms of the attacker’s goal of reducing the test accuracy on the subpopulation.
Revised: Next, we compare the classiﬁers induced by the two attacks in terms of the attacker’s goal of reducing the test accuracy on the subpopulation.

Original: Figure 2 shows the accuracy results for the three subpopulations.
Revised: Figure 2 shows the accuracy results for the three subpopulations.

Original: For each test, the maximum number of poisoning points is obtained by running our attack with a target of 0 .
Revised: For each test, the maximum number of poisoning points is obtained by running our attack with a target of 0 .

Original: 01 -closeness (in loss based distance).
Revised: 01 -closeness (in loss based distance).

Original: For the three subpopulations, at the maximum number of poisons, our attack is much more successful than the KKT attack—the induced classiﬁers have 0.5% accuracy compared to 15.4% accuracy for KKT on subpopulation 1, 0.0% compared to 6.9% on subpoulation 2, and 0.3% compared to 20.1% on subpoplation 3.
Revised: For the three subpopulations, at the maximum number of poisons, our attack is much more successful than the KKT attack—the induced classiﬁers have 0.5% accuracy compared to 15.4% accuracy for KKT on subpopulation 1, 0.0% compared to 6.9% on subpoulation 2, and 0.3% compared to 20.1% on subpoplation 3.

Original: Near Optimality of Our Attack.
Revised: Near Optimality of Our Attack.

Original: In order to show the optimality of our attack, we calculate a lower bound on the number of poisoning points needed to induce the model that is induced by the poisoning points that are found by our attack.
Revised: In order to show the optimality of our attack, we calculate a lower bound on the number of poisoning points needed to induce the model that is induced by the poisoning points that are found by our attack.

Original: We calculate this lower bound on the number of poisons using
Revised: We calculate this lower bound on the number of poisons using

Original: Theorem 4.2 (details in Section 4.3).
Revised: Theorem 4.2 (details in Section 4.3).

Original: Note that Theorem 4.2 provides a valid lower bound based on any intermediate model.
Revised: Note that Theorem 4.2 provides a valid lower bound based on any intermediate model.

Original: In order to get a lower bound on the number of poisoning points, we only need to use Theorem 4.2 on the encountered intermediate models and report the best one.
Revised: In order to get a lower bound on the number of poisoning points, we only need to use Theorem 4.2 on the encountered intermediate models and report the best one.

Original: We do this by running Algorithm 1 using the induced model (and not the previous target model) as the target model, terminating when the induced classiﬁer is 0 .
Revised: We do this by running Algorithm 1 using the induced model (and not the previous target model) as the target model, terminating when the induced classiﬁer is 0 .

Original: 01 -close to the given target model.
Revised: 01 -close to the given target model.

Original: We then consider all the intermediate classiﬁers that the algorithm induced across the iterations.
Revised: We then consider all the intermediate classiﬁers that the algorithm induced across the iterations.

Original: Our calculated lower bound  shows that the gap between the lower bound and the number of used poison points is relatively small.
Revised: Our calculated lower bound in Table 1 shows that the gap between the lower bound and the number of used poison points is relatively small.

Original: This means our attack is nearly optimal in terms of minimizing the number of poisoning points needed to induce the target classiﬁer.
Revised: This means our attack is nearly optimal in terms of minimizing the number of poisoning points needed to induce the target classiﬁer.

Original:  C ONCLUSION 
Revised:  C ONCLUSION AND D ISCUSSION

Original: We propose a general poisoning framework with provable guarantees to reach any achievable target classiﬁer, along with a lower bound on the number of poisoning points needed.
Revised: We propose a general poisoning framework with provable guarantees to reach any achievable target classiﬁer, along with a lower bound on the number of poisoning points needed.

Original: Our attack is a generic tool that ﬁrst captures the goal of adversary as a target model, and then focuses on the power of attacks to induce that model.
Revised: Our attack is a generic tool that ﬁrst captures the goal of adversary as a target model, and then focuses on the power of attacks to induce that model.

Original: This separation enables future work to explore the effectiveness of poisoning attacks corresponding to different adversarial goals.
Revised: This separation enables future work to explore the effectiveness of poisoning attacks corresponding to different adversarial goals.

Original: Our framework also applies in scenarios where adversaries ﬁrst remove points and then add new points into the training set.
Revised: Our framework also applies in scenarios where adversaries ﬁrst remove points and then add new points into the training set.

Original: We have not considered defenses in this work, and it is important to study the effectiveness of our attack against data poisoning defenses.
Revised: We have not considered defenses in this work, and it is important to study the effectiveness of our attack against data poisoning defenses.

Original: Defenses may be designed the limit the search space of the points with maximum loss difference and increasing the number of poisoning points needed.
Revised: Defenses may be designed the limit the search space of the points with maximum loss difference and increasing the number of poisoning points needed.

Original: One limitation of our framework is the requirement in the convexity of the difference of loss functions to efﬁciently search for its maximum value, which excludes other commonly used loss functions such as logistic loss.
Revised: One limitation of our framework is the requirement in the concavity of the difference of loss functions to efﬁciently search for its maximum value.

Original: However, our approach might still be effective in these cases by using local optimization techniques to search for poisoning points with  maximum loss difference.
Revised: However, our approach might still be effective in these cases by using local optimization techniques to search for poisoning points with (approximate) maximum loss difference and we have demonstrated this in the case of Logistic loss.

Revised: More formally, if the approximate maximum loss difference ˆ l found from local optimization techniques is within a constant factor from the globally optimal value l ∗ (i.e., ˆ l ≥ αl ∗ , 0 < α < 1 ), then we still enjoy a logarithmic convergence rate.

Original: The convergence property of our attack holds (with strongly convex regularizer) for any Lipschitz and convex loss function, and does not require the loss difference to be convex.
Revised: It is important to note that the convergence property of our attack holds (with strongly convex regularizer) for any Lipschitz and convex loss function, and does not require the loss difference to be concave.

Revised: The theoretical guarantees in the paper do not apply to non-convex models, although it might still be possible to apply our poisoning strategy empirically.

Revised: Incorporating online learning for non-convex functions might be one possible path to extend our theoretical analysis into non-convex settings.

Original: A P ROOFS
Revised: A P ROOFS

Original: In this section, we provide the proofs of the main theorems shown in this paper.
Revised: In this section, we provide the proofs of the main theorems shown in this paper.

Original: For convenience, we restate all the theorems below while also referencing to the main paper.
Revised: For convenience, we restate all the theorems below while also referencing to the main paper.

Original: Before proving the main theorem, we introduce two new deﬁnitions and several lemmas to assist with the proof.
Revised: Before proving the main theorem, we introduce two new deﬁnitions and several lemmas to assist with the proof.

Original: Deﬁnition 2 (Attainable models) .
Revised: Deﬁnition 2 (Attainable models) .

Original: We say θ is C R -attainable with respect to loss function l and regularization function R if there exists a training set D such that
Revised: We say θ is C R -attainable with respect to loss function l and regularization function R if there exists a training set D such that

Original: [Equation]
Revised: [Equation]

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Let θ 

Revised: Let θ

Original: and θ 2 be two C R -attainable parameters for some C R > 

Revised: and θ 2 be two C R -attainable parameters for some C R >

Original: such that R ( θ 1 ) >
Revised: such that R ( θ 1 ) >

Original: R ( θ 2 ) .
Revised: R ( θ 2 ) .

Original: Then, sup
Revised: Then, sup

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider any attainable pairs of ( θ 1 , θ 2 ) such that R ( θ 1 ) >
Revised: Consider any attainable pairs of ( θ 1 , θ 2 ) such that R ( θ 1 ) >

Original: R ( θ 2 ) and let D 1 to be training set that the training algorithm produces the unique minimizer θ 1 .
Revised: R ( θ 2 ) and let D 1 to be training set that the training algorithm produces the unique minimizer θ 1 .

Original: Namely,
Revised: Namely,

Original: [Equation]
Revised: [Equation]

Original: Since θ 1 minimizes the total loss on D 1 uniquely, we have
Revised: Since θ 1 minimizes the total loss on D 1 uniquely, we have

Original: [Equation]
Revised: [Equation]

Original: By rearranging the above inequality and by an averaging argument, we have sup x,y
Revised: By rearranging the above inequality and by an averaging argument, we have sup x,y

Original: [Equation]
Revised: [Equation]

Original: (cid:1) .
Revised: (cid:1) .

Original: Now since R ( θ 1 ) > R ( θ 2 ) we have
Revised: Now since R ( θ 1 ) > R ( θ 2 ) we have

Original: [Equation]
Revised: [Equation]

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Let F be the family of all C R -attainable models.
Revised: Let F be the family of all C R -attainable models.

Original: For any θ 1 ∈ F , there is a constant γ where for all θ 2 ∈ F we have
Revised: For any θ 1 ∈ F , there is a constant γ where for all θ 2 ∈ F we have

Original: [Equation]
Revised: [Equation]

Original: where γ is a positive constant related to, θ 1 , C R and other model parameters (ﬁxed for a given classiﬁcation task).
Revised: where γ is a positive constant related to, θ 1 , C R and other model parameters (ﬁxed for a given classiﬁcation task).

Original: Proof.
Revised: Proof.

Original: We prove the lemma for γ = 1 − C R /C for
Revised: We prove the lemma for γ = 1 − C R /C for

Original: [Equation]
Revised: [Equation]

Original: First, note that by Lemma A.1 we have
Revised: First, note that by Lemma A.1 we have

Original: [Equation]
Revised: [Equation]

Original: which implies γ is positive.
Revised: which implies γ is positive.

Original: Now we consider two subcases based on the sign of R ( θ 2 ) − R ( θ 1 ) :
Revised: Now we consider two subcases based on the sign of R ( θ 2 ) − R ( θ 1 ) :

Original: Case 1: R ( θ 2 ) − R ( θ 1 ) ≥ 0 .
Revised: Case 1: R ( θ 2 ) − R ( θ 1 ) ≥ 0 .

Original: In this case the inequality is straightforward: sup
Revised: In this case the inequality is straightforward: sup

Original: [Equation]
Revised: [Equation]

Original: ,y
Revised: ,y

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is based on equation 2.
Revised: where the last inequality is based on equation 2.

Original: Case 2: R ( θ 2 ) − R ( θ 1 ) < 0 .
Revised: Case 2: R ( θ 2 ) − R ( θ 1 ) < 0 .

Original: From the deﬁnition of C we have
Revised: From the deﬁnition of C we have

Original: [Equation]
Revised: [Equation]

Original: Equivalently, we can say
Revised: Equivalently, we can say

Original: [Equation]
Revised: [Equation]

Original: Replacing R ( θ 2 ) − R ( θ 1 ) with the lower bound above completes the proof, namely sup x,y
Revised: Replacing R ( θ 2 ) − R ( θ 1 ) with the lower bound above completes the proof, namely sup x,y

Original: [Equation]
Revised: [Equation]

Original: With Deﬁnition 2 and the lemmas, we are ready to prove Theorem 4.1 (restating Theorem 4.1, from
Revised: With Deﬁnition 2 and the lemmas, we are ready to prove Theorem 4.1 (restating Theorem 4.1, from

Original: Section 4.2):
Revised: Section 4.2):

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: After at most T steps, Algorithm 1 will produce the poisoning set D p and the classiﬁer trained on D c ∪ D p is (cid:15) -close to θ p , with respect to loss-based distance, D l, X , Y , for
Revised: After at most T steps, Algorithm 1 will produce the poisoning set D p and the classiﬁer trained on D c ∪ D p is (cid:15) -close to θ p , with respect to loss-based distance, D l, X , Y , for

Original: [Equation]
Revised: [Equation]

Original: where, γ is a constant for a given θ p and classiﬁcation task, and α ( T ) is the regret of the online algorithm when the loss function used for training is convex.
Revised: where, γ is a constant for a given θ p and classiﬁcation task, and α ( T ) is the regret of the online algorithm when the loss function used for training is convex.

Original: The goal of the adversary is to get (cid:15) -close to θ p (in terms of the loss based distance) by injecting (potentially few) number of poisoned training data.
Revised: The goal of the adversary is to get (cid:15) -close to θ p (in terms of the loss based distance) by injecting (potentially few) number of poisoned training data.

Original: The algorithm is in essence an online learning problem and we transform Algorithm 1 into the form of standard online learning problem.
Revised: The algorithm is in essence an online learning problem and we transform Algorithm 1 into the form of standard online learning problem.

Original: Speciﬁcally, we adopt the follow the leader (FTL) framework to describe Algorithm 1 in the language of standard online learning problem.
Revised: Speciﬁcally, we adopt the follow the leader (FTL) framework to describe Algorithm 1 in the language of standard online learning problem.

Original: We ﬁrst describe the online learning setting considered in this paper and the notion of the regret.
Revised: We ﬁrst describe the online learning setting considered in this paper and the notion of the regret.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Let L be a class of loss functions, Θ set of possible models, A : (Θ × L ) ∗ → Θ an online learner and S : (Θ × L ) ∗ ×
Revised: Let L be a class of loss functions, Θ set of possible models, A : (Θ × L ) ∗ → Θ an online learner and S : (Θ × L ) ∗ ×

Original: Θ
Revised: Θ

Original: → L a strategy for picking loss functions in different rounds of online learning (adversarial environment in the context of online convex optimization).
Revised: → L a strategy for picking loss functions in different rounds of online learning (adversarial environment in the context of online convex optimization).

Original: We use
Revised: We use

Original: Regret ( A, S, T ) to denote the regret of A against S , in T rounds.
Revised: Regret ( A, S, T ) to denote the regret of A against S , in T rounds.

Original: Namely,
Revised: Namely,

Original: Regret ( A, S, T ) =
Revised: Regret ( A, S, T ) =

Original: [Equation]
Revised: [Equation]

Original: where θ i
Revised: where θ i

Original: =
Revised: =

Original: A (cid:0) ( θ 0 , l 0 ) , . . .
Revised: A (cid:0) ( θ 0 , l 0 ) , . . .

Original: , ( θ i − 1 , l i − 1 ) (cid:1) and l i = S (cid:0) ( θ 0 , l 0 ) , . . .
Revised: , ( θ i − 1 , l i − 1 ) (cid:1) and l i = S (cid:0) ( θ 0 , l 0 ) , . . .

Original: , ( θ i − 1 , l i − 1 ) , θ i (cid:1) .
Revised: , ( θ i − 1 , l i − 1 ) , θ i (cid:1) .

Original: With the online learning problem set up, we proceed to the main proof which ﬁrst describes Algorithm 1 in the FTL framework.
Revised: With the online learning problem set up, we proceed to the main proof which ﬁrst describes Algorithm 1 in the FTL framework.

Original: Proof of Theorem 4.1.
Revised: Proof of Theorem 4.1.

Original: The FTL framework proceeds by solving all the functions incurred during the previous online optimization steps, namely, A FTL (( θ 0 , l 0 ) , . .
Revised: The FTL framework proceeds by solving all the functions incurred during the previous online optimization steps, namely, A FTL (( θ 0 , l 0 ) , . .

Original: , ( θ i , l i )) =
Revised: , ( θ i , l i )) =

Original: arg min θ ∈ Θ (cid:80) ij =

Revised: arg min θ ∈ Θ (cid:80) ij =

Original: l i ( θ ) .
Revised: l i ( θ ) .

Original: Next, we describe how we design the i th loss function l i in each round of the online optimization.
Revised: Next, we describe how we design the i th loss function l i in each round of the online optimization.

Original: For the ﬁrst choice, A FTL chooses a random model θ 0 ∈ Θ .
Revised: For the ﬁrst choice, A FTL chooses a random model θ 0 ∈ Θ .

Original: In the ﬁrst round (round 0), S θ p uses the clean training set D c and the loss is set as
Revised: In the ﬁrst round (round 0), S θ p uses the clean training set D c and the loss is set as

Original: [Equation]
Revised: [Equation]

Original: According to the FTL framework, A FTL returns model that minimizes the loss on the clean training set D c using the structural empirical risk minimization.
Revised: According to the FTL framework, A FTL returns model that minimizes the loss on the clean training set D c using the structural empirical risk minimization.

Original: For the subsequent iterations ( i ≥ 1 ), the loss functions is deﬁned as, given the latest model θ i , S θ p ﬁrst ﬁnds ( x ∗ i , y ∗ i ) that maximizes the loss difference between θ i and a target model θ p .
Revised: For the subsequent iterations ( i ≥ 1 ), the loss functions is deﬁned as, given the latest model θ i , S θ p ﬁrst ﬁnds ( x ∗ i , y ∗ i ) that maximizes the loss difference between θ i and a target model θ p .

Original: Namely,
Revised: Namely,

Original: [Equation]
Revised: [Equation]

Original: and then chooses the i th loss function as follows:
Revised: and then chooses the i th loss function as follows:

Original: [Equation]
Revised: [Equation]

Original: Now we will see how FTL framework behaves when working on these loss functions at different iterations.
Revised: Now we will see how FTL framework behaves when working on these loss functions at different iterations.

Original: We use D ip to denote the set { ( x ∗ 1 , y ∗ 1 ) , . . .
Revised: We use D ip to denote the set { ( x ∗ 1 , y ∗ 1 ) , . . .

Original: , ( x ∗ i , y ∗ i ) } .
Revised: , ( x ∗ i , y ∗ i ) } .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: This means that A FTL algorithm, at each step, trains a new model over the combination of clean data and poison data so far ( i − 1 number of poisons).
Revised: This means that A FTL algorithm, at each step, trains a new model over the combination of clean data and poison data so far ( i − 1 number of poisons).

Original: Now we want to see what is the translation of the Regret ( A FTL , S θ p , T ) .
Revised: Now we want to see what is the translation of the Regret ( A FTL , S θ p , T ) .

Original: If we can prove an upper bound on regret, namely if we show
Revised: If we can prove an upper bound on regret, namely if we show

Original: Regret ( A FTL , S θ p , T ) ≤ α ( T ) for some function α , then we haveT
Revised: Regret ( A FTL , S θ p , T ) ≤ α ( T ) for some function α , then we have

Original: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original:  which implies
Revised: T ) which implies

Original: T (cid:88)
Revised: T (cid:88)

Original: j = l j ( θ j ) −
Revised: j = l j ( θ j ) −

Original: T (cid:88)
Revised: T (cid:88)

Original: j = l
Revised: j = l

Original: [Equation]
Revised: [Equation]

Original: θ 
Revised: θ p

Revised: [Equation]

Revised: c

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: T (cid:88)
Revised: T (cid:88)

Original: j = l j ( [Equation] j ) −
Revised: j = l j ( θ j ) −

Original: T (cid:88)
Revised: T (cid:88)

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: l j ( θ p
Revised: l j ( θ p

Original: [Equation]
Revised: [Equation]

Original: = L (
Revised: = L (

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: D c
Revised: D c

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: ≤ α ( T )
Revised: ≤ α ( T )

Original: Therefore we have
Revised: Therefore we have

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Based on Lemma A.2, we further have
Revised: Based on Lemma A.2, we further have

Original: [Equation]
Revised: [Equation]

Original: Above inequality states that average of the maximum loss difference in all previous rounds is bounded from above.
Revised: Above inequality states that average of the maximum loss difference in all previous rounds is bounded from above.

Original: Therefore, we know that among the T iterations, there exist an iteration j ∗ ∈
Revised: Therefore, we know that among the T iterations, there exist an iteration j ∗ ∈

Original: [ T ] (with lowest maximum loss difference) such that the maximum loss difference of θ j ∗ is (cid:15) -close to θ p with respect to the loss based distance where
Revised: [ T ] (with lowest maximum loss difference) such that the maximum loss difference of θ j ∗ is (cid:15) -close to θ p with respect to the loss based distance where

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.1 characterizes the dependencies of (cid:15) on α ( T ) and the constant term L ( θ p ; D c ) −
Revised: Theorem 4.1 characterizes the dependencies of (cid:15) on α ( T ) and the constant term L ( θ p ; D c ) −

Original: L ( θ c ; D c ) + N · C R · ( R ( θ p ) − R ( θ c )) .
Revised: L ( θ c ; D c ) + N · C R · ( R ( θ p ) − R ( θ c )) .

Original: To show the convergence of Algorithm 1, we need to ensure (cid:15) → 0 when T → + ∞ , which implies we need to show α ( T ) ≤ O ( √ T ) .
Revised: To show the convergence of Algorithm 1, we need to ensure (cid:15) → 0 when T → + ∞ , which implies we need to show α ( T ) ≤ O ( √ T ) .

Original: Following remark (restating Remark 1 in Section 4.2) and its proof shows the desired convergence.
Revised: Following remark (restating Remark 1 in Section 4.2) and its proof shows the desired convergence.

Original: Remark 1.
Revised: Remark 1.

Original: Online learning algorithms with sublinear regret bound can be applied to show the convergence.
Revised: Online learning algorithms with sublinear regret bound can be applied to show the convergence.

Original: Here, we adopt the regret analysis from McMahan (2017).
Revised: Here, we adopt the regret analysis from McMahan (2017).

Original: Speciﬁcally, α ( T ) is in the order of O (log T ) ) and we have (cid:15) ≤
Revised: Speciﬁcally, α ( T ) is in the order of O (log T ) ) and we have (cid:15) ≤

Original: O ( log TT ) when the loss function is Lipschitz continuous and the regularizer R ( θ ) is strongly convex, and (cid:15) → 0 when T → + ∞ .
Revised: O ( log TT ) when the loss function is Lipschitz continuous and the regularizer R ( θ ) is strongly convex, and (cid:15) → 0 when T → + ∞ .

Original: α ( T ) is also in the order of
Revised: α ( T ) is also in the order of

Original: O (log T ) when the loss function used for training is strongly convex and the regularizer is convex.
Revised: O (log T ) when the loss function used for training is strongly convex and the regularizer is convex.

Original: Our FTL framework formulation can utilize the existing logarithmic regret bound of adaptive FTL algorithm when the objective functions are strongly convex with respect to some norm (cid:107) · (cid:107) , as illustrated in Section 3.6 in McMahan (2017).
Revised: Our FTL framework formulation can utilize the existing logarithmic regret bound of adaptive FTL algorithm when the objective functions are strongly convex with respect to some norm (cid:107) · (cid:107) , as illustrated in Section 3.6 in McMahan (2017).

Original: For clarity in presentation, we ﬁrst restate their related results below.
Revised: For clarity in presentation, we ﬁrst restate their related results below.

Original: Setting 1 (Setting 1 in McMahan (2017)) .
Revised: Setting 1 (Setting 1 in McMahan (2017)) .

Original: Given a sequence of objective loss functions f 1 , f 2 , ..., f i and a sequence of incremental regularization functions r 0 , r 1 , ..., r i we consider an algorithm that selects the response point based on
Revised: Given a sequence of objective loss functions f 1 , f 2 , ..., f i and a sequence of incremental regularization functions r 0 , r 1 , ..., r i we consider an algorithm that selects the response point based on

Original: [Equation]
Revised: [Equation]

Original: We simplify the summation notation with f 1: i ( θ ) =
Revised: We simplify the summation notation with f 1: i ( θ ) =

Original: (cid:80) ij =1 f j ( θ ) .
Revised: (cid:80) ij =1 f j ( θ ) .

Original: Assume that r i is a convex function and satisfy r i ( θ ) ≥ 0 for i ∈ { 0 , 1 , 2 , ... } , against a sequence of convex loss functions
Revised: Assume that r i is a convex function and satisfy r i ( θ ) ≥ 0 for i ∈ { 0 , 1 , 2 , ... } , against a sequence of convex loss functions

Original: f i : R d →
Revised: f i : R d →

Original: R ∪ {∞} .
Revised: R ∪ {∞} .

Original: Further, letting h 0: i = r 0: i + f 1: i we assume dom h 0: i is non-empty.
Revised: Further, letting h 0: i = r 0: i + f 1: i we assume dom h 0: i is non-empty.

Original: Recalling θ i =
Revised: Recalling θ i =

Original: arg min θ h 0: i − 1 ( θ ) , we further assume ∂f i ( θ i ) is non-empty.
Revised: arg min θ h 0: i − 1 ( θ ) , we further assume ∂f i ( θ i ) is non-empty.

Original: We denote the dual norm of a norm (cid:107) · (cid:107) as (cid:107) · (cid:107) ∗ .
Revised: We denote the dual norm of a norm (cid:107) · (cid:107) as (cid:107) · (cid:107) ∗ .

Original: Theorem A.3 (Restatement of Theorem 1 in McMahan (2017)) .
Revised: Theorem A.3 (Restatement of Theorem 1 in McMahan (2017)) .

Original: Consider Setting 1, and suppose the r i are chosen such that r 0: i + f 1: i +1 is 1-strongly-convex w.r.t.
Revised: Consider Setting 1, and suppose the r i are chosen such that r 0: i + f 1: i +1 is 1-strongly-convex w.r.t.

Original: some norm (cid:107) · (cid:107) ( i ) · .
Revised: some norm (cid:107) · (cid:107) ( i ) · .

Original: If we deﬁne the regret of the algorithm with respect to a selected point θ ∗ as
Revised: If we deﬁne the regret of the algorithm with respect to a selected point θ ∗ as

Original: Regret T ( θ ∗ , f i ) ≡
Revised: Regret T ( θ ∗ , f i ) ≡

Original: T (cid:88) i =
Revised: T (cid:88) i =

Original: f i ( θ i ) −
Revised: f i ( θ i ) −

Original: T (cid:88) i =
Revised: T (cid:88) i =

Original: f i ( θ ∗ ) .
Revised: f i ( θ ∗ ) .

Original: Then, for any θ ∗ ∈ R d and for any T > 0 , with g i ∈
Revised: Then, for any θ ∗ ∈ R d and for any T > 0 , with g i ∈

Original: ∂f i ( θ i ) , we have
Revised: ∂f i ( θ i ) , we have

Original: [Equation]
Revised: [Equation]

Original: Corollary A.3.1 (Formalization of FTL result in Section 3.6 in McMahan (2017)) .
Revised: Corollary A.3.1 (Formalization of FTL result in Section 3.6 in McMahan (2017)) .

Original: In the FTL framework (no individual regularizer is used in the optimization procedure), suppose each loss function f
Revised: In the FTL framework (no individual regularizer is used in the optimization procedure), suppose each loss function f

Original: i is 1-strongly convex w.r.t.
Revised: i is 1-strongly convex w.r.t.

Original: a norm (cid:107) · (cid:107) , then we have
Revised: a norm (cid:107) · (cid:107) , then we have

Original: [Equation]
Revised: [Equation]

Original: with (cid:107) g i (cid:107) ∗ ≤ G .
Revised: with (cid:107) g i (cid:107) ∗ ≤ G .

Original: Proof.
Revised: Proof.

Original: The following proof is a restatement of the proof in Section 3.6 in McMahan (2017).
Revised: The following proof is a restatement of the proof in Section 3.6 in McMahan (2017).

Original: The proof follows from Theorem A.3.
Revised: The proof follows from Theorem A.3.

Original: Since we are considering the FTL framework, let r i ( θ ) = 

Revised: Since we are considering the FTL framework, let r i ( θ ) =

Original: for all i and deﬁne (cid:107) θ (cid:107) ( i ) = √ i (cid:107) θ (cid:107) .
Revised: for all i and deﬁne (cid:107) θ (cid:107) ( i ) = √ i (cid:107) θ (cid:107) .

Original: Observe that h 0: i (i.e., f 1: i ) is 1-strongly convex with respect to (cid:107) θ (cid:107) ( i ) (Lemma 3 in McMahan (2017)), and we have (cid:107) θ (cid:107) ( i ) , ∗ = 1 √ i (cid:107) θ (cid:107) ∗ .
Revised: Observe that h 0: i (i.e., f 1: i ) is 1-strongly convex with respect to (cid:107) θ (cid:107) ( i ) (Lemma 3 in McMahan (2017)), and we have (cid:107) θ (cid:107) ( i ) , ∗ = 1 √ i (cid:107) θ (cid:107) ∗ .

Original: Then by applying Theorem A.3, we have
Revised: Then by applying Theorem A.3, we have

Original: [Equation]
Revised: [Equation]

Original: Based on the inequality of (cid:80) Ti =1 1 /i ≤ 1 + log T and if we further assume (cid:107) g i (cid:107) ∗ ≤ G , then we can have
Revised: Based on the inequality of (cid:80) Ti =1 1 /i ≤ 1 + log T and if we further assume (cid:107) g i (cid:107) ∗ ≤ G , then we can have

Original: [Equation]
Revised: [Equation]

Original: Proof of Remark 1.
Revised: Proof of Remark 1.

Original: We will prove the logarithmic regret bound in Remark 1 utilizing Corollary A.3.1.
Revised: We will prove the logarithmic regret bound in Remark 1 utilizing Corollary A.3.1.

Original: First of all, our online learning process ﬁts into Setting 1.
Revised: First of all, our online learning process ﬁts into Setting 1.

Original: Speciﬁcally, we set r i ( θ ) = 0 for all i .
Revised: Speciﬁcally, we set r i ( θ ) = 0 for all i .

Original: For f i ( θ ) , when 1 ≤ i ≤ N , we set f i ( θ ) = 1 N L ( θ ; D c ) + C R · R ( θ ) (evenly distributing the term
Revised: For f i ( θ ) , when 1 ≤ i ≤ N , we set f i ( θ ) = 1 N L ( θ ; D c ) + C R · R ( θ ) (evenly distributing the term

Original: L ( θ ; D c ) + N · C R · R ( θ ) across N iterations) and when i ≥ N + 1 , we set f i ( θ ) =
Revised: L ( θ ; D c ) + N · C R · R ( θ ) across N iterations) and when i ≥ N + 1 , we set f i ( θ ) =

Original: l i − N ( θ ) .
Revised: l i − N ( θ ) .

Original: Details of l i can be referred from the proof of Theorem 4.1.
Revised: Details of l i can be referred from the proof of Theorem 4.1.

Original: Therefore, f i is 1-strongly convex with respect to a norm (cid:107) · (cid:107) (the norm is determined by the regularizer R ( θ ) and C R ).
Revised: Therefore, f i is 1-strongly convex with respect to a norm (cid:107) · (cid:107) (the norm is determined by the regularizer R ( θ ) and C R ).

Original: Further, l 0: i ( θ ) = f 1: N + i ( θ ) .
Revised: Further, l 0: i ( θ ) = f 1: N + i ( θ ) .

Original: In addition, the assumption that dom h 0: i is non-empty in Setting 1 means when if we train a classiﬁer on the poisoned data set, we can always return a model and hence the assumption is satisﬁed.
Revised: In addition, the assumption that dom h 0: i is non-empty in Setting 1 means when if we train a classiﬁer on the poisoned data set, we can always return a model and hence the assumption is satisﬁed.

Original: The assumption of the existence of subgradient ∂f i ( θ i ) in Setting 1 is also satisﬁed by the poisoning attack scenario.
Revised: The assumption of the existence of subgradient ∂f i ( θ i ) in Setting 1 is also satisﬁed by the poisoning attack scenario.

Original: The logarithmic regret of Regret ( A FTL , S θ p , T ) of our algorithm then follows from the result of
Revised: The logarithmic regret of Regret ( A FTL , S θ p , T ) of our algorithm then follows from the result of

Original: Regret T ( θ ∗ , f i ) in Corollary A.3.1.
Revised: Regret T ( θ ∗ , f i ) in Corollary A.3.1.

Original: Speciﬁcally, l 0: i ( θ ) = f 1: N + i ( θ ) is 1-strongly convex to norm (cid:107) · (cid:107) i = √ N + i (cid:107) · (cid:107) and since we assume the loss function is G -Lipschitz, we have (cid:107) g
Revised: Speciﬁcally, l 0: i ( θ ) = f 1: N + i ( θ ) is 1-strongly convex to norm (cid:107) · (cid:107) i = √ N + i (cid:107) · (cid:107) and since we assume the loss function is G -Lipschitz, we have (cid:107) g

Original: i (cid:107) ∗ ≤ G .
Revised: i (cid:107) ∗ ≤ G .

Original: Therefore, we have the logarithmic regret bound as:
Revised: Therefore, we have the logarithmic regret bound as:

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: We next provide the proof of the certiﬁed lower bound (restating Theorem 4.2 from Section 4.3):
Revised: We next provide the proof of the certiﬁed lower bound (restating Theorem 4.2 from Section 4.3):

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Given a target classiﬁer θ p , to reproduce θ p by adding the poisoning set D p into D c , the number of poisoning points |D p | cannot be lower than
Revised: Given a target classiﬁer θ p , to reproduce θ p by adding the poisoning set D p into D c , the number of poisoning points |D p | cannot be lower than

Original: [Equation]
Revised: [Equation]

Original: The main intuition behind the theorem is, when the the number of poisoning points added to the clean training set is lower than the certiﬁed lower bound, for structural empirical risk minimization problem (shown in equation 1 in the main paper), then target classiﬁer will always have higher loss than another classiﬁer and hence cannot be achieved.
Revised: The main intuition behind the theorem is, when the the number of poisoning points added to the clean training set is lower than the certiﬁed lower bound, for structural empirical risk minimization problem (shown in equation 1 in the main paper), then target classiﬁer will always have higher loss than another classiﬁer and hence cannot be achieved.

Original: Proof.
Revised: Proof.

Original: We ﬁrst show that for all models θ , we can derive a lower bound on the number of poison points required to get θ p .
Revised: We ﬁrst show that for all models θ , we can derive a lower bound on the number of poison points required to get θ p .

Original: Then since these lower bounds all hold, we can take the maximum over all of them and get a valid lower bound.
Revised: Then since these lower bounds all hold, we can take the maximum over all of them and get a valid lower bound.

Original: We ﬁrst show that for any model θ , the minimum number of poisoning points cannot be lower than
Revised: We ﬁrst show that for any model θ , the minimum number of poisoning points cannot be lower than

Original: [Equation]
Revised: [Equation]

Original: Let us denote the point corresponding to the supremium of the loss difference between θ and θ p as ( x ∗ , y ∗ ) 2 .
Revised: Let us denote the point corresponding to the supremium of the loss difference between θ and θ p as ( x ∗ , y ∗ ) 2 .

Original: Namely, l ( θ ; x ∗ , y ∗ ) − l ( θ p ; x ∗ , y ∗ ) =
Revised: Namely, l ( θ ; x ∗ , y ∗ ) − l ( θ p ; x ∗ , y ∗ ) =

Original: sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) .
Revised: sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) .

Original: Now suppose we can obtain θ p with lower number of poisoning points
Revised: Now suppose we can obtain θ p with lower number of poisoning points

Original: z < z ( θ ) .
Revised: z < z ( θ ) .

Original: Assume there is a poisoning set D p with size z such that when added to D c would result in θ p .
Revised: Assume there is a poisoning set D p with size z such that when added to D c would result in θ p .

Original: We have sup x,y
Revised: We have sup x,y

Original: [Equation]
Revised: [Equation]

Original: [Equation] implying sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) + C R · ( R ( θ ) − R ( θ p )) > 0 .
Revised: implying sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) + C R · ( R ( θ ) − R ( θ p )) > 0 .

Original: Based on the assumption that z < z ( θ ) , and the fact that sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) + C R · ( R ( θ ) − R ( θ p )) > 0 , we havez [Equation] θ [Equation]
Revised: Based on the assumption that z < z ( θ ) , and the fact that sup x,y (cid:0) l ( θ ; x, y ) − l ( θ p ; x, y ) (cid:1) + C R · ( R ( θ ) − R ( θ p )) > 0 , we have

Original: x

Original: [Equation]

Original: y

Original: [Equation]
Revised: [Equation]

Original: where the equality is based on the deﬁnition of z ( θ ) .
Revised: where the equality is based on the deﬁnition of z ( θ ) .

Original: On the other hand, by deﬁnition of ( x ∗ , y ∗ ) for any D p of size z , we have
Revised: On the other hand, by deﬁnition of ( x ∗ , y ∗ ) for any D p of size z , we have

Original: [Equation]
Revised: [Equation]

Original: − R ( θ p )) (cid:1) .

Original: The above two inequalities imply that for any set D p with size
Revised: The above two inequalities imply that for any set D p with size

Original: z we have
Revised: z we have

Original: [Equation]
Revised: [Equation]

Original: which indicates that adding D p poisoning points into the training set D c , the model θ has lower loss compared to θ p , which is a contradiction to the assumption that θ p has lowest loss on D c ∪ D p and can be achieved.
Revised: which indicates that adding D p poisoning points into the training set D c , the model θ has lower loss compared to θ p , which is a contradiction to the assumption that θ p has lowest loss on D c ∪ D p and can be achieved.

Original: Now, since θ p needs to have lower loss on D c ∪ D p compared to any classiﬁer θ ∈ Θ , the best lower bound is the supremium over all models in the model space Θ . 
Revised: Now, since θ p needs to have lower loss on D c ∪ D p compared to any classiﬁer θ ∈ Θ , the best lower bound is the supremium over all models in the model space Θ . Corollary 4.2.1.

Revised: If we further assume bi-directional closeness in the loss based distance, we can also derive the lower bound on number of poisoning points needed to induce models that are (cid:15) -close to the target model.

Revised: More precisely, if θ 1 being (cid:15) -close to θ 2 implies that θ 2 is also k · (cid:15) close to θ 1 , then we have,

Revised: [Equation]

Original: B R ELATING (cid:15) - CLOSENESS TO CLOSENESS OF PARAMETERS
Revised: where R ∗ is an upper bound on the nonnegative regularizer R ( θ ) . Proof of Corollary 4.2.1. The lower bound for all (cid:15) -close models to the target classiﬁer is given exactly as follows:

Revised: [Equation]

Original: In theorem below, we show how one can relate the notion of (cid:15) -closeness in Deﬁnition 1 in the main paper to closeness of parameters in the speciﬁc setting of hinge loss.
Revised: where inf (cid:107) θ (cid:48) − θ p (cid:107) D l, X , Y ≤ (cid:15) denotes θ (cid:48) is (cid:15) -close to θ p in the loss based distance. However, the formulation above is a min-max optimization problem and hard to analytically compute the lower bound (by plugging the lower bound formula into Algorithm 1. Therefore, we need to make several relaxations such that the lower bound is computable. For any model θ (cid:48) that is (cid:15) -close to θ p , based on the bi-directional assumption, then θ p is k(cid:15) -close to θ (cid:48) .

Revised: Therefore we have,

Revised: L

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: and the inequalities are all based on the deﬁnition of θ p being k(cid:15) -close to θ (cid:48) .

Revised: Plugging the above inequalities into the formula of sup θ,θ (cid:48) for model θ (cid:48)

Revised: [Equation]

Revised: and with the assumption that 0 ≤ R ( θ ) ≤ R ∗ , ∀ θ ∈ Θ , we immediately have

Revised: [Equation]

Revised: Since the inequality holds for any θ (cid:48)

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: and hence z (cid:48) ( θ ) is a valid lower bound.

Revised: Remark 2 (Improving Results in Corollary 4.2.1) .

Original: {+Assuming 
+}
Revised: {+Assuming 
+}

Revised: ≤

Revised: R

Revised: θ ) ≤ R ∗ is not a strong assumption and actually can be satisﬁed by many common convex models.

Revised: For example, for SVM model with (cid:96) 2 regularizer (in fact, applies to any regularizer R ( θ ) with R ( 0 ) = 0 ), we have

Revised: R ( θ ) ≤ 1 C R

Revised: and hence R ∗ ≤ 1 C R .

Revised: Moreover, we can further tighten the lower bound by better bounding the term R ( θ (cid:48) )

Revised: − R ( θ ) .

Revised: Speciﬁcally, R ( θ (cid:48) ) − R ( θ ) =

Revised: R ( θ (cid:48) )

Revised: − R ( θ p ) + R ( θ p ) − R ( θ ) and we only need to have a tighter upper and lower bounds on R ( θ (cid:48) )

Revised: − R ( θ p ) utilizing some special properties of the loss functions.

Revised: For the constant k in the bi-directional closeness, we can also compute its value for some speciﬁc loss functions.

Revised: For example, for Hinge loss, we can compute the value based on Corollary B.2.1 in Section B.

Revised: B R ELATING CLOSENESS OF LOSS BASED DISTANCE TO CLOSENESS OF PARAMETERS

Revised: In theorem below, we show how one can relate the notion of (cid:15) -closeness in Deﬁnition 1 in the main paper to closeness of parameters in the speciﬁc setting of hinge loss.

Original: We use this just as an example to show that our notion of (cid:15) -closeness can be tightly related to the closeness of the models.
Revised: We use this just as an example to show that our notion of (cid:15) -closeness can be tightly related to the closeness of the models.

Original: Theorem B.1.
Revised: Theorem B.1.

Original: Consider the hinge loss function l ( θ ; x, y )
Revised: Consider the hinge loss function l ( θ ; x, y )

Original: = max(1 − y · (cid:104) x, θ (cid:105) , 0) for θ ∈ R d and x ∈ R d and y ∈ {− 1 , +1 } .
Revised: = max(1 − y · (cid:104) x, θ (cid:105) , 0) for θ ∈ R d and x ∈ R d and y ∈ {− 1 , +1 } .

Original: For θ, θ
Revised: For θ, θ

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: R
Revised: R

Original: d such that (cid:107) θ (cid:107) 1 ≤ r and (cid:107) θ (cid:48) (cid:107) 1 ≤ rfor some r ≥ 1 , if θ is (cid:15) -close to θ (cid:48) in the loss based distance,for some (cid:15) ≤ 1 then, (cid:107) θ − θ (cid:48) (cid:107) 1 ≤ r · (cid:15) .
Revised: d such that (cid:107) θ (cid:107) 1 ≤ r and (cid:107) θ (cid:48) (cid:107) 1 ≤ r , if θ is (cid:15) -close to θ (cid:48) in the loss based distance, then, (cid:107) θ − θ (cid:48) (cid:107) 1 ≤ r · (cid:15) .

Original: Remark 2.
Revised: Remark 3.

Original: In Theorem B.1 above, an upper bound on the norm of θ and θ (cid:48) is (cid:112) d/C R .
Revised: In Theorem B.1 above with (cid:96) 2 -regularizer, an upper bound on the (cid:96) 1 -norm of θ and θ (cid:48) is (cid:112) d/C R .

Original: however, the models that we care about in practice usually have smaller norms.
Revised: however, the models that we care about in practice usually have smaller norms.

Revised: Remark 3 can be obtained by plugging 0 ∈ R d and compare the resulting (regularized) optimization loss to the model θ ∗ that minimizes the model loss.

Original: Proof.
Revised: Proof of Theorem B.1.

Original: We construct a point x ∗ as follows:
Revised: We construct a point x ∗ as follows:

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) θ (cid:107) 1 ≤ r we have 
Revised: Since (cid:107) θ (cid:107) 1 ≤ r we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: and similarly since (cid:107) θ (cid:48) (cid:107) 1 ≤ r we have
Revised: and similarly since (cid:107) θ (cid:48) (cid:107) 1 ≤ r we have

Original: [Equation]
Revised: [Equation]

Original: Therefore by Inequalities equation 4 and equation 5 we have
Revised: Therefore by Inequalities equation 4 and equation 5 we have

Original: [Equation]
Revised: [Equation]

Original: which byEquation equation 3 implies
Revised: which by equation 3 implies

Original: [Equation]
Revised: [Equation]

Original: Now since we know that, ∀ x ∈ R d , the loss difference between θ and θ (cid:48) is bounded by (cid:15) , the bound should also hold for the point ( x ∗ , − 1) , meaning that
Revised: Now since we know that, ∀ x ∈ R d , the loss difference between θ and θ (cid:48) is bounded by (cid:15) , the bound should also hold for the point ( x ∗ , − 1) , meaning that

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Revised: Theorem B.2.

Revised: Consider the hinge loss function l ( θ ; x, y )

Revised: = max(1 − y · (cid:104) x, θ (cid:105) , 0) for θ ∈ R d and x ∈ R d and y ∈ {− 1 , +1 } .

Revised: For X = { x ∈ R d : (cid:107) x (cid:107) 1 ≤ q } and Y = {− 1 , +1 } , For any two models θ, θ (cid:48) if (cid:107) θ − θ (cid:48) (cid:107) 1 ≤ (cid:15) , then θ is q(cid:15) -close to θ (cid:48) in the loss based distance.

Revised: Namely,

Original: C I NSTANTIATING T HEOREM 4.1 FOR THE CASE OF SVM
Revised: [Equation] Proof. For any given θ and θ (cid:48) , by triangle inequality for maximum, we have [Equation]

Revised: Therefore, we have

Revised: [Equation]

Revised: Our goal is then to obtain an upper bound of O ( (cid:15) ) for max ( x,y )

Revised: ∈X×Y (cid:104) yx, θ (cid:48) − θ (cid:105) when (cid:107) θ − θ (cid:48) (cid:107) 1 ≤ (cid:15).

Revised: To maximize (cid:104) yx, θ (cid:48) − θ (cid:105) by choosing x

Revised: and y , we only need to ensure that sign yx i

Revised: =

Revised: sign θ i , i ∈

Revised: [ d ] .

Revised: Therefore, based on the assumption that 1 q (cid:107) x (cid:107) ≤ 1 (i.e., 1 q | x i | ≤ 1 , i ∈ [ d ]) we have

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: which concludes the proof.

Revised: Corollary B.2.1.

Revised: For Hinge loss, with Theorem B.1 and Theorem B.2, if θ is (cid:15) -close to θ (cid:48) , then θ (cid:48) is rq(cid:15) -close to θ .

Original: Here we show how to instantiate Theorem 4.1 for SVM with exact constants instead of the asymptotic notations.
Revised: Here we show how to instantiate Theorem 4.1 for SVM with exact constants instead of the asymptotic notations.

Original: We need to calculate the constant γ to get the exact constant.
Revised: We need to calculate the constant γ to get the exact constant.

Original: Imagine the feature domain is R d .
Revised: Imagine the feature domain is R d .

Original: Now we calculate the constant C as follows.
Revised: Now we calculate the constant C as follows.

Original: Let i ∗ θ = arg min
Revised: Let i ∗ θ = arg min

Original: i ∈
Revised: i ∈

Original: [ d ] | θ
Revised: [ d ] | θ

Original: [ i ] /θ p
Revised: [ i ] /θ p

Original: [ i ] | and α
Revised: [ i ] | and α

Original: θ =
Revised: θ =

Original: | θ
Revised: | θ

Original: [ i ∗ θ ] /θ p
Revised: [ i ∗ θ ] /θ p

Original: [ i ∗ θ ] | .
Revised: [ i ∗ θ ] | .

Original: Let x ∗ θ ∈ R d be a point where is equal to 0 everywhere and is equal to 1 /θ p [ i ∗ θ ] on the i ∗ coordinate.
Revised: Let x ∗ θ ∈ R d be a point where is equal to 0 everywhere and is equal to 1 /θ p [ i ∗ θ ] on the i ∗ coordinate.

Original: We have,
Revised: We have,

Original: [Equation]
Revised: [Equation]

Original: Now we can calculate C as follows
Revised: Now we can calculate C as follows

Original: [Equation]
Revised: [Equation]

Original: (By Inequality 7)
Revised: (By Inequality 7)

Original: [Equation]
Revised: [Equation]

Original: (By deﬁnition of α θ )
Revised: (By deﬁnition of α θ )

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: Therefore γ ≥ 1 − 2 · C R · R ( θ p ) .
Revised: Therefore γ ≥ 1 − 2 · C R · R ( θ p ) .

Original: On the other hand, we can also calculate α ( T ) based on the exact form given in the proof of Theorem 4.1.
Revised: On the other hand, we can also calculate α ( T ) based on the exact form given in the proof of Theorem 4.1.

Original: DA DDITIONAL E XPERIMENTS IN I NDISCRIMINATE S ETTING 
Revised: D I NDISCRIMINATE S ETTING E XPERIMENTS

Original: In this section, we evaluate the attacks in the conventional indiscriminate attack setting, where the attacker’s goal is just to reduce the overall accuracy of the model
Revised: In this section, we evaluate the attacks in the conventional indiscriminate attack setting, where the attacker’s goal is just to reduce the overall accuracy of the model.

Original: Datasets and Models.
Revised: Datasets and Models.

Original: For the indiscriminate attack, we use the MNIST-17 dataset, which consists of the digits 1 and 7 and is commonly used for evaluating indiscriminate poisoning attacks against binary classiﬁcation (Steinhardt et al., 2017; Biggio et al., 2012; Xiao et al., 2012).
Revised: For the indiscriminate attack, we use the MNIST 1–7 dataset, which consists of the digits 1 and 7 and is commonly used for evaluating indiscriminate poisoning attacks against binary classiﬁcation (Steinhardt et al., 2017; Biggio et al., 2012; Xiao et al., 2012).

Original: The MNISTdataset contains 13007 trainingdata and 2163 test data.
Revised: MNIST 1– contains 13,007 training and 2,163 test samples.

Original: The dataset contains 784 features and all the features are normalized into range [0 , 1] .
Revised: The dataset contains 784 features and all the features are normalized into range [0 , 1] .

Original: For completeness, the Adult dataset used for subpopulation attack is downsampled to form a class-balanced dataset and contains 15682 training data andtest data.
Revised: For completeness, the Adult dataset used for subpopulation attack is downsampled to form a class-balanced dataset and contains 15682 training data and 7692 test data.

Original: The dataset contains 57 features and the features are also normalized into range [0 , 1] (except for the binary features).
Revised: The dataset contains 57 features and the features are also normalized into range [0 , 1] (except for the binary features).

Original: All of the processed datasets are included in the supplementary material.
Revised: All of the processed datasets are included in the supplementary material.

Original: We still adopt linear SVM model in the indiscriminate attack scenario.
Revised: We still adopt linear SVM model in the indiscriminate attack scenario.

Original: All of the models for both datasets set the regularization parameter C R = 0 . 09 .
Revised: All of the models for both datasets set the regularization parameter C R = 0 . 09 .

Original: The clean accuracy of SVM model on MNIST-17 dataset is 98.9% and the accuracy on Adult dataset is 78.5%.
Revised: The clean accuracy of SVM model on MNIST 1–7 is 98.9% and the accuracy on Adult dataset is 78.5%.

Original: Target Classiﬁers.
Revised: Target Classiﬁers.

Original: Accuracy of the clean MNIST-17 model has around 1% error rate on the test set.
Revised: Accuracy of the clean MNIST 1–7 model has around 1% error rate on the test set.

Original: For our experiment, we aim to generate three target classiﬁers with overall test accuracies around 5%, 10% and 15%.
Revised: For our experiment, we aim to generate three target classiﬁers with overall test errors around 5%, 10% and 15%.

Original: To generate target classiﬁers with desired error rates, we follow the heuristic strategy proposed by Koh et al.
Revised: To generate target classiﬁers with desired error rates, we follow the heuristic strategy proposed by Koh et al.

Original: (2018) to generate multiple candidate target classiﬁers, and then among all the valid candidate models that satisfy the error rate requirement we choose the one with lowest loss on the clean training set.
Revised: (2018) to generate multiple candidate target classiﬁers, and then among all the valid candidate models that satisfy the error rate requirement we choose the one with lowest loss on the clean training set.

Original: Using this approach, the ﬁnal target classiﬁers induced have overall test accuracy of 94.0%, 88.8% and 83.3% respectively.
Revised: Using this approach, the ﬁnal target classiﬁers induced have overall test accuracy of 94.0%, 88.8% and 83.3% respectively.

Original: (We describe a better way of ﬁnding the target classiﬁers in Appendix D.1, but for comparison purposes do not use those in the results here.) 
Revised: (We describe a better way of ﬁnding the target classiﬁers in Appendix D.1, but for comparison purposes do not use those in the results here.) [Figure] [Figure]

Original: Convergence.
Revised: Convergence.

Original: We show the convergence of Algorithm 1 by reporting the maximum loss difference and Euclidean distance between the classiﬁer induced by the attack and the target classiﬁer.
Revised: We show the convergence of Algorithm 1 by reporting the maximum loss difference and Euclidean distance between the classiﬁer induced by the attack and the target classiﬁer.

Original: Figures 3a and 3b summarize the results for the target classiﬁer with a 10% error rate.
Revised: Figures 3a and 3b summarize the results for the target classiﬁer with a 10% error rate.

Original: The maximum number of poisoning points in the ﬁgure is obtained when the classiﬁer from Algorithm 1 is 0 .
Revised: The maximum number of poisoning points in the ﬁgure is obtained when the classiﬁer from Algorithm 1 is 0 .

Original: 1 -close to the target classiﬁer in the loss-based distance.
Revised: 1 -close to the target classiﬁer in the loss-based distance.

Original: In Figure 3, the classiﬁer induced by our algorithm steadily converges to the target classiﬁer both in the maximum loss difference and Euclidean distance, while the classiﬁer induced by the KKT attack diverges initially and then starts to converge to the target model.
Revised: In Figure 3, the classiﬁer induced by our algorithm steadily converges to the target classiﬁer both in the maximum loss difference and Euclidean distance, while the classiﬁer induced by the KKT attack diverges initially and then starts to converge to the target model.

Original: At the maximum number of points, the maximum loss difference of KKT-induced classiﬁer to the target is 0.46, compared to 0.1 for the classiﬁer induced by our attack.
Revised: At the maximum number of points, the maximum loss difference of KKT-induced classiﬁer to the target is 0.46, compared to 0.1 for the classiﬁer induced by our attack.

Original: For the Euclidean distance, the KKT-induced classiﬁer is 0.16 away, compared to 0.07 for the classiﬁer induced by our attack.
Revised: For the Euclidean distance, the KKT-induced classiﬁer is 0.16 away, compared to 0.07 for the classiﬁer induced by our attack.

Original: Attack Success.
Revised: Attack Success.

Original: We next compare the classiﬁer induced from our attack to the classiﬁer induced by the KKT attack in terms of their overall test accuracy.Similarly, the maximum number of poisoning [Figure] [Figure]
Revised: We next compare the classiﬁer induced from our attack to the classiﬁer induced by the KKT attack in terms of their overall test accuracy.

Original: [Table]

Original: [Figure] [Table] [Figure] points in Figure 2 is obtained by running our attack with 0 .
Revised: Similarly, the maximum number of poisoning points in Figure 4 is obtained by running our attack with 0 .

Original: 1 -closeness (in loss based distance) to the target as the input.
Revised: 1 -closeness (in loss based distance) to the target as the input.

Original: In terms of the test accuracy, our attack has a comparable attack success rate compared to the KKT attack.
Revised: In terms of the test accuracy, our attack has a comparable attack success rate compared to the KKT attack.

Original: Speciﬁcally, for target models of 5% and 10% error rates, both methods have almost identical performance, as shown in Figures 4a and 4b.
Revised: Speciﬁcally, for target models of 5% and 10% error rates, both methods have almost identical performance, as shown in Figures 4a and 4b.

Original: For the target model of 15% error rate (test accuracy is 83.3%), the KKT attack is more successful than our attack, inducing models with 82.7% accuracy (17.3% attack success rate) compared to 85.9% accuracy (14.1% attack success rate) for our attack.
Revised: For the target model of 15% error rate (test accuracy is 83.3%), the KKT attack is more successful than our attack, inducing models with 82.7% accuracy (17.3% attack success rate) compared to 85.9% accuracy (14.1% attack success rate) for our attack.

Original: Interestingly, in this case, the performance of models induced by the two attacks with fewer poisoning points our attack results in models with lower test accuracy (higher attacker success) than the KKT attack.
Revised: Interestingly, in this case, the performance of models induced by the two attacks with fewer poisoning points our attack results in models with lower test accuracy (higher attacker success) than the KKT attack.

Original: To summarize, for the indiscriminate scenario, our attack produces classiﬁers that have much closer distance to the target models than the KKT attack, and has comparable attack success rates with the KKT attack.
Revised: To summarize, for the indiscriminate scenario, our attack produces classiﬁers that have much closer distance to the target models than the KKT attack, and has comparable attack success rates with the KKT attack.

Original: Lower Bound on Number of Poisons.
Revised: Lower Bound on Number of Poisons.

Original: We next check the optimality of our attack in the indiscriminate attack scenario.
Revised: We next check the optimality of our attack in the indiscriminate attack scenario.

Original: Similar to the subpopulation attack setting, we still use Theorem 4.2 to compute the lower bound of the induced classiﬁer from our attack by using it as the input to Algorithm 1, and terminating when the induced classiﬁer is 0 .
Revised: Similar to the subpopulation attack setting, we still use Theorem 4.2 to compute the lower bound of the induced classiﬁer from our attack by using it as the input to Algorithm 1, [Table] [Figure] [Table]

Revised: [Figure]

Revised: and terminating when the induced classiﬁer is 0 .

Original: 1 -close to the given target model.
Revised: 1 -close to the given target model.

Original: Our calculated lower bound shows that there exists a relatively large gap between the number of poisoning points, especially for the induced classiﬁer from our attack for the target model of 5% error rate, where the 
Revised: In Table 2, our calculated lower bound shows that there exists a relatively large gap between the number of poisoning points, especially for the induced classiﬁer from our attack for the target model of 5% error rate, where the lower bound is only 50% of the actual number of poisoning points used. For the induced

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: lower bound is only 50% of the actual number of poisoning points used. For the induced classiﬁer for the target model of 15% error rate, the gap between the number of poisoning points and the lower bound is smallest, with the lower boundtaking 79% of the number of poisoning points.
Revised: classiﬁer for the target model of 15% error rate, the gap between the number of poisoning points and the lower bound is smallest, with the lower bound 79% of the number of poisoning points.

Original: The relatively large gap indicates that either the estimated lower bound is not tight or the attack itself is not close to optimal.
Revised: The relatively large gap indicates that either the estimated lower bound is not tight or the attack itself is not close to optimal.

Original: To gain more insights into this problem, we further show the computed lower bound at each iteration when running Algorithm 1 and Figure 5 summarizes the results.
Revised: To gain more insights into this problem, we further show the computed lower bound at each iteration when running Algorithm 1 and Figure 5 summarizes the results.

Original: From the Figure, we see that, the peak value of the curve (i.e., highest lower bound) always appears before the termination of the algorithm, indicating that the computed lower bound is likely to be tight and we may need to further improve the attack algorithm.
Revised: From the Figure, we see that, the peak value of the curve (i.e., highest lower bound) always appears before the termination of the algorithm, indicating that the computed lower bound is likely to be tight and we may need to further improve the attack algorithm.

Original: For completeness, we also repeat the lower bound computation process for classiﬁers induced from the KKT attack.
Revised: For completeness, we also repeat the lower bound computation process for classiﬁers induced from the KKT attack.

Original: The KKT induced classiﬁers are obtained by running the KKT attack with target classiﬁers of different error rates as target input.
Revised: The KKT induced classiﬁers are obtained by running the KKT attack with target classiﬁers of different error rates as target input.

Original: The target number of poisoning points of KKT attack is given by the size of poisoning set returned from our attack when our algorithm terminates when the induced classiﬁer is 0 .
Revised: The target number of poisoning points of KKT attack is given by the size of poisoning set returned from our attack when our algorithm terminates when the induced classiﬁer is 0 .

Original: 1 -close to the target model of different error rates.
Revised: 1 -close to the target model of different error rates.

Original: Then the lower bound computation process is identical to the above – we simply send the KKT induced classiﬁer as target input to Algorithm 1 and terminate it when the induced classiﬁer from our algorithm is 0 .
Revised: Then the lower bound computation process is identical to the above – we simply send the KKT induced classiﬁer as target input to Algorithm 1 and terminate it when the induced classiﬁer from our algorithm is 0 .

Original: 1 -close to the given target model (i.e., KKT induced classiﬁer).
Revised: 1 -close to the given target model (i.e., KKT induced classiﬁer).

Original: The results are summarized in Table 3 and we observe that there also exists a relatively large gap between the lower bound and the number of poisoning points used by the attack.
Revised: The results are summarized in Table 3 and we observe that there also exists a relatively large gap between the lower bound and the number of poisoning points used by the attack.

Original: Similarly, we also plot the lower bound computed at different iterations of Algorithm 1 in Figure 6, and ﬁnd that the peak value also appears before the termination of the algorithm, indicating that the lower bound might be tight and we need a stronger attack strategy to close the gap.
Revised: Similarly, we also plot the lower bound computed at different iterations of Algorithm 1 in Figure 6, and ﬁnd that the peak value also appears before the termination of the algorithm, indicating that the lower bound might be tight and we need a stronger attack strategy to close the gap.

Original: D. I MPROVED T ARGET G ENERATION P ROCESS
Revised: D. I MPROVED T ARGET G ENERATION P ROCESS

Original: The original heuristic approach works by ﬁnding different quantiles of training points that have higher loss on the clean model, ﬂipping their labels, repeating those points for multiple copies, and adding them to the clean training set.
Revised: The original heuristic approach works by ﬁnding different quantiles of training points that have higher loss on the clean model, ﬂipping their labels, repeating those points for multiple copies, and adding them to the clean training set.

Original: We ﬁnd that, in the process of trying different quantiles and copies of high loss points, if we also adaptively update the model where the high loss points are found (instead of just always ﬁxing it to be the clean model), we can generate a valid target classiﬁer with even lower loss on the clean training.Such an improved generation process can signiﬁcantly reduce the number [Table]
Revised: We ﬁnd that, in the process of trying different quantiles and copies of high loss points, if we also adaptively update the model where the high loss points are found (instead of just always ﬁxing it to be the clean model), we can generate a valid target classiﬁer with even lower loss on the clean training.

Original: [Figure] of poisoning points needed to reach the same (cid:15) -closeness (with respect to the loss based distance) to the target classiﬁer, consistent with the claims in Theorem 4.1 in the main paper.
Revised: Such an improved generation process can signiﬁcantly reduce the number of poisoning points needed to reach the same (cid:15) -closeness (with respect to the loss based distance) to the target classiﬁer, consistent with the claims in Theorem 4.1 in the main paper.

Original: In addition, we ﬁnd that, if we compare our attack with improved generation process to the KKT attack with the original generation process (Koh et al., 2018), we can also reach the desired target error rate much faster using our attack.
Revised: In addition, we ﬁnd that, if we compare our attack with improved generation process to the KKT attack with the original generation process (Koh et al., 2018), we can also reach the desired target error rate much faster using our attack.

Original: Validate Implication of Theorem 4.1.
Revised: Validate Implication of Theorem 4.1.

Original: We ﬁrst empirically validate the implication of Theorem 4.
Revised: We ﬁrst empirically validate the implication of Theorem 4.

Original: in the main paper: to obtain the same (cid:15) -closeness in loss-based distance, a target classiﬁer with lower loss on the clean training set D c requires fewer poisoning points.
Revised: in the main paper: to obtain the same (cid:15) -closeness in loss-based distance, a target classiﬁer with lower loss on the clean training set D c requires fewer poisoning points.

Original: Therefore, when adversaries have multiple target classiﬁers that satisfy the attack goal, the one with lower loss on clean training set is preferred.
Revised: Therefore, when adversaries have multiple target classiﬁers that satisfy the attack goal, the one with lower loss on clean training set is preferred.

Original: For both the original and improved target generation methods, we generate three target classiﬁers with error rates of 5%, 10% and 15%.
Revised: For both the original and improved target generation methods, we generate three target classiﬁers with error rates of 5%, 10% and 15%.

Original: The original target classiﬁer generation method returns classiﬁers with test accuracy of 94.0%, 88.8% and 82.3% respectively (also used in the previous experiments of indiscriminate attack).
Revised: The original target classiﬁer generation method returns classiﬁers with test accuracy of 94.0%, 88.8% and 82.3% respectively (also used in the previous experiments of indiscriminate attack).

Original: The improved target generation process returns target classiﬁers with approximately the same test accuracy (94.9%, 88.9% and 84.5%).
Revised: The improved target generation process returns target classiﬁers with approximately the same test accuracy (94.9%, 88.9% and 84.5%).

Original: However, for classiﬁers returned from the two generation methods of same error rate, the improved generation method produces classiﬁers with signiﬁcantly lower loss compared to the original generation approach.
Revised: However, for classiﬁers returned from the two generation methods of same error rate, the improved generation method produces classiﬁers with signiﬁcantly lower loss compared to the original generation approach.

Original: Table 4 compares the two target generation approaches by showing the number of poisoning points needed to get 0 .
Revised: Table 4 compares the two target generation approaches by showing the number of poisoning points needed to get 0 .

Original: 1 -close to the corresponding target model of same error rate.
Revised: 1 -close to the corresponding target model of same error rate.

Original: For example, for target models of 15% error rate, the model from the original approach has a total clean loss of 5428.4 while our improved method reduces it to 4641.6.
Revised: For example, for target models of 15% error rate, the model from the original approach has a total clean loss of 5428.4 while our improved method reduces it to 4641.6.

Original: With the reduced clean loss, getting 0 .
Revised: With the reduced clean loss, getting 0 .

Original: 1 -close to the target model generated from our improved process only requires 3206 poisoning points, while reaching the same distance from the target model produced by the original method would require 6762 poisoning points, a more than 50% reduction.
Revised: 1 -close to the target model generated from our improved process only requires 3206 poisoning points, while reaching the same distance from the target model produced by the original method would require 6762 poisoning points, a more than 50% reduction.

Original: End-to-end Comparison.
Revised: End-to-end Comparison.

Original: Figure 7 compares the two attacks in an end-to-end manner in terms of their test accuracy.
Revised: Figure 7 compares the two attacks in an end-to-end manner in terms of their test accuracy.

Original: With the improved target generation process, our attack can achieve the desired error rate much faster than the KKT attack with the original process.
Revised: With the improved target generation process, our attack can achieve the desired error rate much faster than the KKT attack with the original process.

Original: For the KKT attack with target model generated from the original process, we determine the target number of poisoning points using the size of poisoning set returned from running Algorithm 1 with 0 .
Revised: For the KKT attack with target model generated from the original process, we determine the target number of poisoning points using the size of poisoning set returned from running Algorithm 1 with 0 .

Original: 1 -closeness and target model from original process as inputs.
Revised: 1 -closeness and target model from original process as inputs.

Original: To run our attack with improved generation process, we terminate the algorithm when the size of the poisoning points is same as the number of poisoning points used by the KKT attack with original process.
Revised: To run our attack with improved generation process, we terminate the algorithm when the size of the poisoning points is same as the number of poisoning points used by the KKT attack with original process.

Original: Such a termination criteria helps us to ensure that both attacks use same number of poisoning points and can be compared easily.
Revised: Such a termination criteria helps us to ensure that both attacks use same number of poisoning points and can be compared easily.

Original: We also evaluate the KKT attack on fractions of the maximum target number of poisoning points (0.2, 0.4, 0.6, and 0.8), as in the previous experiments.
Revised: We also evaluate the KKT attack on fractions of the maximum target number of poisoning points (0.2, 0.4, 0.6, and 0.8), as in the previous experiments.

Original: The accuracy plot shows that our attack (with improved target model) can achieve the desired error rate (e.g., 10%) much faster than the KKT attack (with original target model), especially for the target classiﬁers of error rates of 10% and 15%.
Revised: The accuracy plot shows that our attack (with improved target model) can achieve the desired error rate (e.g., 10%) much faster than the KKT attack (with original target model), especially for the target classiﬁers of error rates of 10% and 15%.

Original: E C OMPARISON OF M ODEL -T ARGETED AND O BJECTIVE -D RIVEN A TTACKS
Revised: E C OMPARISON OF M ODEL -T ARGETED AND O BJECTIVE -D RIVEN A TTACKS

Original: Although model-targeted attacks work to induce the given target classiﬁers by generating poisoning points, the end goal is still to achieve the attacker objectives encoded in the target models.
Revised: Although model-targeted attacks work to induce the given target classiﬁers by generating poisoning points, the end goal is still to achieve the attacker objectives encoded in the target models.

Original: In terms of the comparison to the objective-driven attacks, we ﬁrst demonstrate that, objective-driven attacks can be used to generate a target model, which can then be used as the target for a model-targeted attack, resulting in an attack that achieves the desired attacker objective with fewer poisoning points.
Revised: In terms of the comparison to the objective-driven attacks, we ﬁrst demonstrate that, objective-driven attacks can be used to generate a target model, which can then be used as the target for a model-targeted attack, resulting in an attack that achieves the desired attacker objective with fewer poisoning points.

Original: Then, we show that in order to have competitive performance against state-of-the-art objective-driven attacks (e.g., the min-max attack (Steinhardt et al., 2017)), the target classiﬁers should be generated carefully, such that the attacker objectives of the target classiﬁers can be achieved efﬁciently with model-targeted attacks using fewer poisoning points.
Revised: Then, we show that in order to have competitive performance against state-of-the-art objective-driven attacks (e.g., the min-max attack (Steinhardt et al., 2017)), the target classiﬁers should be generated carefully, such that the attacker objectives of the target classiﬁers can be achieved efﬁciently with model-targeted attacks using fewer poisoning points.

Original: Although the investigation of a systematic approach to generate such “desired” classiﬁers is out of the scope of this paper, in the indiscriminate setting, we have some empirical evidence.
Revised: Although the investigation of a systematic approach to generate such “desired” classiﬁers is out of the scope of this paper, in the indiscriminate setting, we have some empirical evidence.

Original: Speciﬁcally, we ﬁnd that target classiﬁers with lower loss on clean training set and higher error rates (higher than what are desired in the attacker objectives) often require fewer poisoning points to achieve the attacker objectives.
Revised: Speciﬁcally, we ﬁnd that target classiﬁers with lower loss on clean training set and higher error rates (higher than what are desired in the attacker objectives) often require fewer poisoning points to achieve the attacker objectives.

Original: Target Models Generated from Objective-driven  Attacks.
Revised: Target Models Generated from Objective-driven Label-Flipping Attacks.

Original: In our experiments, the target classiﬁers are generated from the label ﬂipping based objective driven attacks that are effective but need too many poisoning points to achieve their objective.
Revised: In our experiments, the target classiﬁers are generated from the label ﬂipping based objective driven attacks that are effective but need too many poisoning points to achieve their objective.

Original: Then, our attacks are deployed to achieve the same objective with fewer poisoning points.
Revised: Then, our attacks are deployed to achieve the same objective with fewer poisoning points.

Original: Table 5 shows the number of poisoning points used by the label-ﬂipping attack and our model-targeted attack, to achieve desired attack objectives of increasing the test error to certain amount.
Revised: Table 5 shows the number of poisoning points used by the label-ﬂipping attack and our model-targeted attack, to achieve desired attack objectives of increasing the test error to certain amount.

Original: We can see that, using our attack, the number of poisoning points used by label-ﬂipping attacks can be saved up to 73%.
Revised: We can see that, using our attack, the number of poisoning points used by label-ﬂipping attacks can be saved up to 73%.

Original: Comparison to the Min-Max Attack.
Revised: Comparison to the Min-Max Attack.

Original: Still using target classiﬁers generated from label-ﬂipping attacks, we show that our attack can outperform the state-of-the-art min-max attack (Steinhardt et al., 2017) at reducing the overall test accuracy, under same amount of poisoning points.
Revised: Still using target classiﬁers generated from label-ﬂipping attacks, we show that our attack can outperform the state-of-the-art min-max attack (Steinhardt et al., 2017) at reducing the overall test accuracy, under same amount of poisoning points.

Original: Since we aim to produce target classiﬁers with lower loss on clean training set and higher error rates, we adopt the improved target model generation process described in Section D.1 (helps to reduce the loss on clean training set) and generate a classiﬁer of 15% error rate.
Revised: Since we aim to produce target classiﬁers with lower loss on clean training set and higher error rates, we adopt the improved target model generation process described in Section D.1 (helps to reduce the loss on clean training set) and generate a classiﬁer of 15% error rate.

Original: With the target model, we terminate our attack when the induced model is 0 .
Revised: With the target model, we terminate our attack when the induced model is 0 .

Original: 1 -close to the target model in terms of the loss-based distance.
Revised: 1 -close to the target model in terms of the loss-based distance.

Original: However, to we compare our attack to the min-max attack conveniently, we compare their accuracy reduction at different poisoning ratios, which is the common evaluation strategy of objective-driven attacks.
Revised: However, to we compare our attack to the min-max attack conveniently, we compare their accuracy reduction at different poisoning ratios (i.e., 5%, 15% and 30%), which is the common evaluation strategy of objective-driven attacks in the indiscriminate setting (Biggio et al., 2011; Steinhardt et al., 2017; Koh et al., 2018).

Original: Table 6 summarizes the results.
Revised: Table 6 summarizes the results.

Original: From the table, we observe that, compared to the min-max 
Revised: From the table, we observe that, compared to the min-max attack, our attack reduces more on the test accuracy under the same poisoning budget and the gap becomes larger when the poisoning ratio increases. Comparison to the Label-Flipping Subpopulation Attack. We also compare our attack to the label-ﬂipping subpopulation attack from Jagielski et al. This attack works by randomly sampling ﬁxed number (constrained by the poisoning budget) of instances from the training data of the subpopulation, ﬂipping their labels and then injecting them into to the original training set.

Original: Attacker Objectives 5%
Revised: Although this attack is very simple, it shows relatively high attack success when the goal is to cause misclassiﬁcation on the selected subpopulation (Jagielski et al., 2019). To be consistent with our experiments in Section 5, we assume the attacker objectives are still to induce a model that has 0% accuracy on a selected subpopulation. For each of the SVM and logistic regression models, we selected the three subpopulations with highest test accuracy (all end up having have 100% accuracy). In indiscriminate setting, we already observed that models with lower loss on clean training set and larger overall error rates can achieve attacker objectives of smaller error rates faster.

Revised: However, to leverage this observation into our subpopulation experiments, one challenge is the attacker objective is to have 100% test error on the subpopulation, but no classiﬁers can have test errors larger than 100%.

Revised: To tackle this, we select models with larger loss on training samples from the subpopulation, with a hope that this process is “equivalent” to selecting target models with larger error rates (on subpopulation) than 100%.

Revised: To this end, we heuristically select targeted models that satisfy the attacker objective, have larger loss on the training data from the subpopulation, and have relatively low loss on the entire clean training set.

Revised: Empirically, this selection strategy works better than the original target generation process (as done in Section 5) in achieving the attacker objectives.

Revised: A more detailed and systematic investigation of the target model search process is left as the future work.

Revised: To check the effectiveness of achieving the attacker objectives, we ﬁrst run our attack and terminate when our attack achieves the attacker objective to have 0% accuracy on the selected subpopulation, and record the number of poisoning points used.

Revised: Then, we run the random label-ﬂipping attack with

Revised: Attacker Objectives 5% Error 10% Error 15% Error

Revised: Label-ﬂipping Attack 6, 8, 10,

Revised: Our Attack 1, 5, 6,

Revised: -ﬂipping attacks and achieve similar attacker objectives using our attack with fewer poisoning points.

Revised: The attacker objectives are to increase the test error to certain amounts (i.e., 5%, 10% and 15%) and the target classiﬁers to our attack are generated by running the label-ﬂipping attacks with given attacker objectives.

Original: [Table] 
Revised: [Table] [Table]

Revised: Our Attack

Original: [Table] 
Revised: [Table] Label [Table]

Revised: Fipping

Original: [Table]
Revised: [Table]

Revised: the same number of poisoning points.

Revised: For both attacks, we report the ﬁnal test accuracies of the resulting models on the subpoplations.

Revised: The attack comparisons on different subpopulation clusters and models are given in Table 7.

Revised: Results in the table compare our attack and the label-ﬂipping attack over the three distinct subpopulation clusters for the SVM and logistic regression models.

Revised: Across all settings, our attack is considerably more successful.

Revised: The number of poisoning points needed to reach the 0% accuracy goal is small compared to the entire training set size (e.g., the maximum poisoning ratio is only 10.5%).

Revised: The gap between our attack and the label-ﬂipping attack is fairly small.

Revised: For example, for Cluster 1 in the SVM experiment, the label-ﬂipping attack is also quite successful and reduces the test accuracy to 2.8% (our attack achieves 0% accuracy).

Revised: We believe the success of label-ﬂipping attack is due to the following two reasons.

Revised: First, label-ﬂipping in the subpopulation setting can be successful because smaller subpopulations show some degree of locality and hence, injecting points (from the subpopulation) with ﬂipped labels can have a strong impact on the selected subpopulation.

Revised: This is conﬁrmed by empirical evidence that increasing the subpopulation size (i.e., reducing its locality) gradually reduces the label-ﬂipping effectiveness and the attack becomes much less effective in the indiscriminate setting (i.e., subpopulation is the entire population).

Revised: Second, the Adult dataset only contains 57 features, where 53 of them are binary features with additional constraints.

Revised: Therefore, the beneﬁt from optimizing the feature values is less signiﬁcant as the optimization search space of our attack is fairly limited.

Revised: F A DDITIONAL E XPERIMENTS

Revised: In this section, we provide the results of evaluating our attack and the KKT attack on SVM models for an additional dataset (Section F.1) and on logistic regression models for three datasets (Section F.2).

Revised: The results here are consistent with our ﬁndings in on the datasets and models in Section 5, but provide further evidence of the general effectiveness of our attack.

Revised: F. A TTACKS ON SVM TRAINED ON D OGFISH

Revised: In this section, we introduce the results of SVM model evaluated on the Dogﬁsh dataset.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: Dataset.

Revised: The Dogﬁsh dataset contains dog and ﬁsh images of dimensions 298 × 298 × 3 .

Revised: This dataset has been used as a binary classiﬁcation task by by previous works in evaluating poisoning attacks in the indisrcriminate setting (Koh & Liang, 2017;

Revised: Steinhardt et al., 2017; Koh et al., 2018).

Revised: To make classiﬁcation easy for linear models, both Steinhardt et al.

Revised: (2017) and Koh et al.

Revised: (2018) use the extracted features (of the original images) from the ImageNet Inception model (Szegedy et al., 2016) and then apply linear models to complete the classiﬁcation task.

Revised: We also adopt the extracted features for classiﬁcation, so each instance has 2,048 features.

Revised: The Dogﬁsh dataset has 1,800 training samples and 600 test samples.

Revised: As in the previous work, we conduct a conventional indiscriminate attack on it, where the adversary’s goal is to reduce the overall test accuracy.

Revised: Target Classiﬁers.

Revised: The clean accuracy of the SVM model on the Dogﬁsh dataset is 98.5%.

Revised: We generate target classiﬁers of overall test error rates around 10%, 20% and 30% using a similar (heuristic) target generation process (Koh et al., 2018) as in the case of MNIST 1-7 dataset.

Revised: The ﬁnal test accuracies of the obtained target classiﬁers are 89.3%, 78.3% and 67.2% respectively.

Revised: Attack Results.

Revised: The convergence of our attack is demonstrated in Figure 8, both in the loss-based distance and the actual model distance in (cid:96) -2 norm.

Original: {+The maximum number of poisons for the experiments is obtained when the classiﬁer from Algorithm 1 is 2 . 
+}
Revised: {+The maximum number of poisons for the experiments is obtained when the classiﬁer from Algorithm 1 is 2 . 
+}

Revised: -close to the target classiﬁer.

Revised: Our attack steadily converges to the target model for both metrics, and has a faster convergence rate than the KKT attack.

Revised: Similar observations are found in other attack settings, as summarized in Figure 9.

Revised: Our attack is slightly more successful than the KKT attack in all three target classiﬁers we tested.

Revised: For the models induced from our attack, in Table 8, we also show the gap between the number of poisoning points of our attack and the theoretical lower bound.

Revised: Although our attack can induce the target models using very few poisoning points (recall that the entire training set size is 1,800), there is still some gap to the theoretical lower bound.

Revised: We also repeated the same process on the model induced from the KKT attack and again observe gaps between the number of poisoning points and the corresponding lower bound.

Revised: These suggest that there is still potentially room for improvement in ﬁnding more efﬁcient model-targeted poisoning attacks.

Revised: F. A TTACKS ON L OGISTIC R EGRESSION

Revised: In this section, we evaluate our attack on the logistic regression models for the Adult, MNIST 1and Dogﬁsh datasets.

Revised: The convergence guarantee in the paper also holds for logistic regression model (more generally, holds for any Lipschitz and convex function with strongly convex regularizer).

Revised: However, for logistic regression, we may not be able to efﬁciently search for the globally optimal point with maximum loss difference (Line 4 in Algorithm 1) because the difference of two logistic losses is not concave.

Revised: Therefore, we adopt gradient descent strategy, using the Adam optimizer (Kingma & Ba, 2014) to search for the point that (approximately) maximizes the loss difference.

Revised: This is in contrast to the SVM model, where the difference of Hinge loss is piece-wise linear and we can deploy general (convex) solvers to search for the globally optimal point in each linear segment (Diamond & Boyd, 2016; Inc, 2020).

Revised: However, as will be demonstrated next, poisoning points with approximate maximum loss difference can still be very effective.

Revised: More formally, if the approximate maximum loss difference ˆ l found from local optimization techniques is within a constant factor from the globally optimal value l ∗ (i.e., ˆ l ≥ αl ∗ , 0 < α < 1 ), then we can still enjoy the logarithmic convergence rate.

Revised: A similar issue of global optimality also applies to the KKT attack (Koh et al., 2018), where the attack objective function is no longer convex for logistic regression models, and therefore, we also utilize gradient based technique to (approximately) solve the optimization problem.

Revised: Since the maximum loss difference found for logistic regression models may not be the globally optimal value, in these experiments we did not compute the lower bound (Theorem 4.2) on number of poisoning points to induce the poisoned model from our attack, which requires obtaining the actual maximum loss difference 3 .

Revised: Target Classiﬁers.

Revised: The clean accuracies of logistic regression models on the three datasets are 79.9% on Adult, 98.1% on MNIST 1-7 and 98.5% on Dogﬁsh.

Revised: Target classiﬁers for logistic regression models are generated similarly to their SVM counterpart on each dataset.

Revised: For Adult dataset, the subpopulations are generated exactly the same as in the SVM case and form a total of 20 subpopulations, where instances in each subpopulation all belong to the “low-income” group.

Revised: Among all the subpoulations, we select three with the highest test accuracy on the subpopulations and they all have 100% accuracy.

Revised: The target classiﬁers are then generated to have 0% accuracy on the subpopulations using the target generation method described in Section 5.

Revised: On the MNIST 1– dataset, target models are generated to have around 5%, 10% and 15% overall test errors and ﬁnal test accuracies of the obtained target models are 94.7%, 89.0% and 84.5%.

Revised: For Dogﬁsh, target models are generated to have around 10%, 20% and 20% overall test errors and the ﬁnal test accuracies of the resulting models are 89.0%, 79.5% and 67.3%.

Revised: Results on Adult.

Original: Our Attack 1, 5, 6, increase the test error to certain amounts (i.e., 5%, 10% and 15%) and the target classiﬁers to our attack are generated by running thelabel-ﬂipping attacks with given attacker objectives.
Revised: Figure 10 shows the effectiveness of our attack on logistic regression models trained on the Adult dataset, using the loss-based distance and the actual model distance in (cid:96) -2 norm. The maximum number of poisons for the experiments is obtained when the classiﬁer from Algorithm 1 is 0 . 05 -close to the target classiﬁer. Our attack steadily converges to the target model while the

Revised: KKT attack fails to have a reliable convergence.

Revised: Similar observations are also found in other attack settings, as shown in Figure 11.

Revised: Our attack is much more successful than the KKT attack, especially for the attack on Cluster 1.

Original: [Table]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Results on MNIST 1–7.

Revised: Figure 12 shows the convergance of our attack on LR models trained on the

Revised: MNIST 1–7.

Original: attack, our attack reduces more on the test accuracy under the same poisoning budget and the gap becomes larger when the poisoning ratio increases.
Revised: The maximum number of poisons for the experiments is obtained when the classiﬁer from Algorithm 1 is 0 . 1 -close to the target classiﬁer. Our attack converges to the target model while the KKT attack diverges. Similar observations are also found for other attack settings, as shown in

Revised: Figure 13.

Revised: In these settings, our attack is much more successful than the KKT attack.

Revised: In fact, the

Revised: KKT attack seems to not ﬁnd a useful set of poisoning points as its induced models did not show a signiﬁcant drop from the clean accuracy of 98.1%.

Revised: We suspect this is due to the highly non-convex nature of the attacker objective 4 when attacking logistic regression models.

Revised: In contrast, our attack only deals with maximizing the difference of two logistic losses, which is simpler than the KKT attacker objective, and results in a successful attack.

Revised: Results on Dogﬁsh.

Revised: Figure 14 shows attack results of logistic regression models on the Dogﬁsh dataset.

Revised: The maximum number of poisons for the experiments is obtained when the classiﬁer from

Original: {+Algorithm 1 is 1 . 
+}
Revised: {+Algorithm 1 is 1 . 
+}

Revised: -close to the target classiﬁer.

Revised: Our attack converges to the target model while the KKT attack fails to converge.

Revised: Attack success comparisons are given in Figure 15.

Revised: As with

Revised: MNIST 1–7, our attack succeeds in settings where the KKT attack does not lead to signiﬁcant accuracy drops from the clean accuracy of 98.5%.

Revised: We believe this is also because of the highly non-convex nature of the KKT attack objective.